I was having the exact same problem! I found that switching out the battery with a new one solved the problem. Before I got a new battery, taking the battery out and putting it back in and keeping the battery off and on charger until it was full was a temporary fix.
I'm having the exact same problem with my Note 4, it takes hours to charge to 100% and discharges fast as well.
Getting a new battery seemed to help a little bit for me, but I'm not sure if it would improve things for your situation.