The only thing I've found is time, as in, game time. Eventually the ghostly relationships decline and then, after even more time, the ghostly relationships just 'go' from your panel.
It's such a pain. My game randomly culls 'living' friends etc too. I think that if we could have a separate 'tab' for the ghostly relationships that would be helpful. I always seem to end up with more ghosts in my relationship tab then living.