I would be leery of deleting anything, sometimes there are hidden things in those "empty" folders... It will be hard to fix the phone off you don't have a clue what you deleted hey... Leave well alone would be my suggestion here...
If you're not concerned/worried about what Tim1954 has alluded to, then unmount your SD card and remount it again. Doing this will cause the media scanner to scan for media files on the phone. The folder may disappear from the gallery if there's nothing in it.