Typically, you get that message when you form a group of 3 or more sims and one of the sims starts engaging in romantic interactions with someone. It doesn't even necessarily have to be with a member of the group. It makes the other members of the group uncomfortable, and they leave the group. However, I've never had it happen myself unless I was actually initiating romantic interactions.
I've also noticed that groups will form automatically if you select "Travel to X Lot With..." Likewise, when you invite someone over, it will form a group and grant them automatic access to the house if you don't greet them right away. When inviting someone over, you can prevent this by pausing the game as soon as they come on the lot and selecting the "Greet" interaction so it stops them from ringing the doorbell and then inviting them inside yourself. When traveling with members of your household, the only way to avoid making a group is to make each person travel separately.
Groups kind of act weird, and I'm not totally sure I have them figured out yet. One thing I've noticed about inviting a sim over is that you should clean up any old papers on the lot before doing so. If the game forms a group automatically, the mess will cause the invited sim to talk about how boring the outing was and leave before even getting inside the house (giving your sim a negative moodlet in the process). I've likewise disbanded groups that were formed by traveling immediately upon arriving at the destination, and everyone will say what a great time they had...even though they haven't actually done anything. To be totally honest, I think the entire group system is bugged, and doesn't work the way it was originally intended to work. That might explain why it's so confusing lol.