1. USB charging maxes out at 500 mA based on the standard, it can't go higher than that when plugged into a computer. I'm not sure what the wall charger for the N4 puts out, but it's probably twice that, most of today's Android phone chargers range anywhere from 800 mA to a full 1 A (or 1000 mA). It's actually possible your computer doesn't even output that much out to USB.
2. Now, that's just what the charger is *quoted* to deliver. You can use a 3rd party app, like battery monitor widget or GSam to see what the actual power consumption of your device is during a charge. Your device's mA won't be at a full amp when charging, but posting those numbers may help diagnose if your device isn't up to par with what others are getting. You may also want to observe how much power your device is drawing when not charging to compare it to. If, for example you're drawing more than 500 mA from your battery at any given time, than that USB charger won't get you back into the positive even if it's working 100% correctly. (This is why a lot of older car chargers, which also are often rated at 500 mA, can't charge a device when navigation is running). I've definitely had a few after-market wall chargers that fall way short of their advertised power output as well.
3. Try different chargers, cables, power outlets. It's impossible to rule out the charger or cable itself until you do so.
Ultimately, even if your charger is OK and the phone and battery hardware is fine, the kernel installed with your OS can also affect charging times as well...which is why it's important to get feedback from folks here (that are running the same OS or ROM) on what "normal" is.
For the record, even without knowing any other info, I agree that 3 hours to charge off a wall charger is a bit much. Like Pizzadude, my S3 takes about an hour, give or take.