Noodles, I did that. I uninstalled, then re-installed to Late Night. I don't have any stuff packs. I've had a few glitches since then, but in Random Wishmasters could send my sims on vacation 6 or 7 times before coming home to clones. They could even take photographs. I am thinking about updating to the latest patch when EA comes out with the next one, however.
To answer your question, you'll lose memories, chest hair, dates, and some children-themed wallpaper. Oh, and I can't download the free stuff I got for uploading memories. But I'm glad I did, because I have much less anxiety, especially about the challenges.