Manage Households: - My Household tab - Click image to see larger view
Q: Does it list households you played with?
A: Yes and the ones you've taken into CAS to edit. Along with other households that are available to play - in the Other Household tab.
Q: Can you remove them without deleting them in the world?
A: You cannot remove Sims out of Manage Households without them being deleted from your game.
Q: Edited households show up in Manage Households, but I'm not playing them.
A: This is due to you editing the Sims. Any time you edit a Sim, it will get saved to Manage Households.
-- Other Information:
The My Households tab has player created, played or unplayed households. Sims in the Other Households tab may loose progress information such as skill levels, relationships, etc.
If you don't want this to happen to any of the households, click the heart to the right under the household's photo to move them to the My Households tab.
Click the white plumbob in the upper left of the Sim household to make them unplayed.
In the My Households tab (which is automatically selected) you should be able to find what families are in-game and what world they are in.
The Sims you edit don't get saved to the library unless you save them to the library yourself.
Some Additional Information:
New Cullling Mechanics and Favourited Households
https://www.carls-sims-4-guide.com/forum/index.php?topic=29133.msg472099#msg472099