I'm the same way. Do you have a bunch of files in the root of your card instead of sitting in a folder? Any important system files will not be on your SD card. That's where your music, pics, ringtones, data various apps use, etc. is stored.
A lot of apps create folders on the SD Card to store data and in most cases, the app may no longer be in use and the folder remains. i would suggest backing-up everything onto the computer and formating the SD Card. then you can go one by one and transfer things back onto the SD Card. This is a good time to keep photos/videos/music on your computer and move back the ones you think you'll be using.
Mine is the same way and I plan on cleaning it soon.