Is it possible that fast charging is screwing up these phone batteries? I charge with an Anker wall socket charger and it's fine. I have the new fast charger 2A charger but I rather stick to the slow charge.
The faster you charge a battery, the more accumulated wear and tear you put on the battery per charge. It really has to do with heat. The technical answer is - If you want the battery to last the longest possible time before replacement, charge it as slowly as you have time for.
Having said that, the incremental increase in wear and tear from 2A fast charge is very small, and even using actual QuickCharge is pretty minimal, so if you are buying good-quality batteries just plug the darned thing in when it's convenient and use whatever charge method you want. Our phones have replaceable batteries, which is probably why many of us are still carrying them. We are free from the worry about all that nonsense, because for $20 we can buy a good-quality replacement battery and 5 minutes later be enjoying that "new phone" battery life.
Of course, if you buy the eBay $5 batteries, charge those things slowly. Always. Some of them are of equivalent quality to OEM. Some are unstable pieces of garbage that will go Thermite the first time you throw 12V at them. There's only one way to find out which are which, really, and after you have found a bad one good luck finding a replacement Note 4 on swappa, assuming you were able to put out the fire and not looking for a new home.