The amount of current the phone can draw from the charger is fixed. If the phone is using more than that amount of current, the extra current comes from the battery, so the amount of charge decreases - just more slowly than if the charger weren't plugged in.
But if you regularly discharge the phone to 20%, you could be killing it - lithium batteries
are not deep discharge batteries, so they shouldn't be discharged to less than 40%. (See
The Care and Feeding of Lithium Polymer Batteries.)
You can run
Ampere to see how much the phone is drawing, then plug in the charger and see how much current it's drawing from the charger to charge the battery. (If it shows negative current, it's most likely a bad battery.)