For me it used to be if I invited someone over and had my sim in house when they showed up and I invited them in before they rang the door bell they wouldn't group automatically. But if I was outside or let them ring the bell they would group.
But since the last patch, no matter what as soon as they set foot on the lot they are grouped with my sim.
And I agree, it is kind of annoying.