As others have mentioned, the ex-wife will no longer be a designated caregiver once the toddlers become children.
Once both toddlers age up, the wife won't come in uninvited. I have no idea why her "other guy" is able to visit, too.
How long until the younger daughter's birthday? Can you hold on until then? You could also lock all the doors to the house to "Everybody except household members."
Then, the father could come and go, but ex-wife and her other guy can't enter the house.