If I'm not mistaken, they have children too, don't they?
But either way, go and meet his wife at the market, chat her up for a bit so she thinks you're friendly and form a group with her. Then from the map vew, click on their house and select "Go Here With Group." Once you're there, you can ask her if you can go inside.