Just so you know, it's a max of 8 Sims and 6 Pets, to a max of 10 creatures (Sims + Pets) per household. Meaning, if you had 6 Pets, you could have a max of only 2 Sims.
And, I'm sorry, but I can't help. I've never adopted strays (yet).
Oh, that's good to know. Now that I think of it, it really would be a full circus with 8 sims and 6 pets all under one roof.
About the stray - I reloaded my game, but it didn't help, so I just asked the dog to leave. Maybe there's some restriction that it needs to visit set amount of times before it will be willing to move in or something (I doubt it but can't come up with any solution), this was his second visit on my lot.
Edit. Maybe I was right about the number of visits - when the dog came for the third time, I got the adopt option after inviting him in. Before that, no amount of interactions would bring the option up.