Done some research on this and found the answer over on the Sims 4 Forum.
It seems somehow the lot got changed from a residential lot to a generic lot.
The solution was simple.
Just go to manage worlds, then change lot type to residential. It'll prompt you to evict your sims, go ahead and do so, then it will prompt to go into build mode, do so, then go back to world, then move your sims back in.
I can now ask visitors to leave again or they will leave on their own.