I was having the exact same problem with my GS4, with 2 different 64GB SDcards (the 2nd one even a Samsung branded card). I also suspected that it was a physical issue with the slot itself and faulty contacts. I'm guessing even just a momentary lapse of connectivity to the card causes the phone OS to throw up the error and then go through a remount to re-verify the card integrity. I even think that my original 64GB card eventually became corrupted from this happening so many times in 2 and a half years since I was unable to re-format it on phone or even on my PC.
What I just did right now to try to resolve the issue is to unmount the card, then remove it (no phone shut down or re-start required). I then removed the back and *very gently* pressed down on the metal SDcard 'dock', focusing on the edges of it where it connects to the phone body. If this doesn't work, I'll try your suggestion of using a spacer underneath the card, between it and the phone body. I'm also wondering if it would be worth it to try to place a small piece of tape over the entire card dock or at least the exposed part of the card itself (being careful not to block any other areas of the phone body) to prevent moisture from entering the dock area.
Additionally, I have also received "No SIM card" errors that required a phone restart to fix, though not as frequently as the SDcard errors, and I suspect that it's a similar root cause of momentary loss of contact to the card. Has anyone else had this? I suppose this is what happens when you have a phone with a removable back and openings on the back that allow moisture and high humidity to access the phone body and these card ports. This might also explain Samsung's noticeable silence on the issue, since they would have to admit that it's an actual phone hardware design problem.
Finally, as far as SD cards themselves, cards of 32GB and less should be formatted as FAT32, and cards larger than 32GB need to be formatted as exFAT. As far as I'm aware, NTFS formatted cards will not work in the phone.