Yes, there's some insignificant power consumption when you leave things plugged in, and it's minor on a per-device level. But between everything in your house, it does add up. Normally, when people mention Vampire or Standby Power, it's usually not for things like your toaster or a charging cable (although they can be classified as such), but it's for devices that have standby features, like a clock on a microwave or VCR, or the orange standby light on my TV or the red one on my Playstation 3 or my computer monitor's and tower's and webcam's standby lights. Those things draw more power than just my USB charging cable being plugged in, but everything together adds up. (
this article claims the can make up to 5-10% of your energy cost)
Sometime, turn all the lights off in your house and walk around and see how many standby lights and clocks are glowing, even though everything is "off"....
All that being said, what's one more thing plugged in? I'll take 3 charging mats, please. Take my money!