They can't hurt your sim in terms of causing physical harm or really bad moodlets, and it's easily one of my favorite careers in the game.
I did find out recently (the hard way of course) that sometimes attempts to banish a ghost (as opposed to a spirit) with the banisher can fail, and the ghost will fight with you. It's rare, I've only had it happen once so far, but it can happen - it's not that bad, they'll just dislike you even more and you can go back at them with the banisher. You'll still be able to finish your job, the mood hit is very minor (assuming you lost) and most importantly, you won't die. You can avoid that altogether by just convincing the ghosts to move on.
