The usual culprit is a bad USB cable not making a solid connection, so I'd replace that first. If that doesn't work, then it's something inside the phone. If it has a user replaceable battery, replacing that would be the second suggestion. It could also be a bad connection somewhere in the charging circuit between the USB port and the battery. That would require a shop to properly diagnose and repair.
There's also a slim chance that the charger or battery is defective. Chargers usually last almost forever, and the a battery so bad that it can't accept a charge even when powered off is pretty rare (and not something I'd consider unless you're having other battery related issues that you didn't tell us about).