I'm not sure if this will help. I read this on an article with people using the illegal version of sims 4. But it still might help us who bought the game legally.
Before travelling, pause your game. Press M. This will make the map appear. Select destination and whoever you want to travel with. I also use M when travelling as it is easier compared to selecting the phone icon for travelling.
If this doesn't work, playing offline will be your best bet as the game usually plays faster on offline mode.