Photography is the most irritating to level up, because rather than leaving the sim doing one activity like you do in pretty much all other skills, you have to take the pictures and it takes a while of repeatedly doing "take photo, pause, position camera, take picture, repeat". It's a lot more hands on which can be more fun, but the monotony is rather annoying in my opinion.
Charisma is difficult for the friend requirements if you go for leveling it via books. However, if you just level it through making the friends you need (and 20 friends is useful for not getting decay any longer), then it's pretty easy.
Otherwise... pretty much all skills are easy to level up. Some take longer than others (sculpting takes longer than painting, from what I see. That's more the problem of very few townies having savvy sculptor, however).
Supermaxing... Photography or any of the instruments is hardest. Photography because of all the collections you're trying to find, and I also find finding enough strangers to get the stranger photos tricky. Instruments because of the bug with the performing at parties challenge.