I've also installed this on a retail lot in a non-challenges save file.
On the single door separating the shop and living quarter, I toggled "Enable Lock" and selected "Visitors" only.
However, the computer in the living quarters had to be moved up to the roof (inaccessible to the sims).
Instead of tending the shop/customers, household members kept trying to run off and play BlicBloc or troll teh forums. No, no, not at the shop, people!