Check the watt-age/amperage of the charger you want to use. I believe the travel charger that comes with the N4 is 5W 1200mA (= 1.2A). Most phone USB chargers are 5W 1A chargers, but some cheap ones are crap that only identify themselves as a 500mA USB data port (D- and D+ lines aren't shorted together), so will charge the N4 much slower. A good, tablet charger like a 10W 2A charger for the Nexus 7 or Kindle Fire will do fine, the Nexus 4 shouldn't take any more current (the (A)mps part of the equation) than it wants. In my initial tests with Battery Monitor Widget Pro, I've noticed the N4 isn't taking much more than 500mA, even on an ASUS Nexus 7 charger. Couldn't tell you why.
The LG Nexus wireless charger (WCP-400) comes with a big ol' 1800mA adapter, and ... darn ... I can't read what the output is supposed to be without watchmaker's glasses. I suspect the WCP400 uses a fair amount of current powering the inductive charging schtuff.
But generally if you've got a 5W 1A charger from another phone, it should be fine. Use a good microUSB cable, not come cheap crap.
(Yes, it's true. I've found charging performance differs with some cables.)
iPhone-specific chargers may be wonky. Apple uses a weird protocol where the iDevices look for 2.0V and 2.7V on D- and D+ (if I recall correctly), an Apple device finds that and sucks down up to 1A, but on other chargers (conventional, with D- and D+ shorted), stop at 500mA regardless of what the charger is capable of. Chalk it up to Apple "thinking different." Read: proprietary.