@shatinn I've actually been doing just that in my legacy challenge.
Perhaps because I'm using MC Command Center for that challenge, whenever I download any fabulous houses from the Gallery, the townies immediately move into those homes first.
I made two small neighborhoods of matching homes in Newcrest which were immediately filled while Oasis Springs remained completely empty--it was ridiculous.
So, I evicted those sims then set those homes to "Generic," a lot setting that doesn't require me to add anything to the lot.
The lot is no longer a residential one so no sims will move in. Also, no sims will visit a Generic lot except the ones you are controlling.
Anyway, that is how I'm getting around enabling "Fill Empty Houses" while saving the best houses for my favorite sims.
In my dynasty challenge, I am required to have a museum.
I've set that museum to "Generic" as well, and haven't seen a single visitor for 8 generations.
Good luck!