Ever since the beginning of my android life...I have never formatted my SD card when flashing or updating a rom. There is so much on mine that it would be nuts. If anything...copy and paste all pictures etc... onto your computer and format the SD card. Then transfer all data back.
No... only the stuff the ROM requires... like BAMF Forever required removing the BAMF_Extra folder on the SD card, but that's all I've ever deleted... besides your Recovery backup files on your SD card... so I'd say absolutely NOT!!!
I've deleted some of the folder's I'm fairly sure I don't care about, but it's always a good idea to make a backup of the data before you delete it just in case.
Wouldn't it be just like if you bought a new SD card. You would think the rom would reinstall ever thing it needed. You would be able to a back up rom or anything like though.