@StamperCat Moving your garden: You would, of course, lose the layout of your garden, but you could stash each of your plants in your household inventory before moving.
After rebuilding, you would simply take them out, rebuy your pots/planters, then set up our garden again.
Replacing the house: Replacing the house is more of a hassle than moving.
You would need to put away all your stuff in household inventory (including everything outside, like your plants).
Then you could (1) bulldoze your entire lot or (2) take apart your house yourself, deleting room by room).
After placing the new house, take everything back out again.
Basically, whether you replace your house with one from the gallery or move, you'll still need to stash the things you want to keep (including garden) in your household inventory.
Good luck!