Yes, you can remove everything from the downloads folder. However, I recommend that you don't delete the contents, but rather move them elsewhere. It would be best to put them on an external hard drive or a flash drive, but if you have neither, just put them somewhere else on your hard drive. This way you keep a back up in case you have to re-install. Re-downloading from the store is a real pain, especially when you have tons of content.
My download's folder is always empty, but I keep copies of my sims3packs on external hard drives.