Yup, use a DIFFERENT tagging program. Album art files were stored in each folder mostly for Window's sake, and it went into the folder to look for the image file in the containing folder to show the album art; since these are image files, your phone shows them up in the Gallery automatically. However, MP3 has evolved since then, and now album art is usually embedded within the file itself, avoiding this issue since no actual image file is present.
So, you have two options:
1) Re-tag your songs and make sure the 'embed album art in file' option is selected (I think it's a V2 or V3 tag option).
2) Find the folders for your music and choose to hide them from the Gallery.
BONUS: Maybe your songs are already tagged with the modern format and the image files are just a legacy thing. Try erasing the image from one folder and then playing a song from that album (oh, and reboot before playing just to be safe). If the Album Art is displayed correctly, I'd say you're safe deleting all the album art pictures as these are already within your MP3 files. If not, go to option one.