If you regularly let it get down to 18%, the battery is probably already in pretty bad shape. 50% is the best charging point for longest battery life. 40% won't change it much. By the time you let it get down to 25% it's shortening the life a lot. (It's not the manufacturers, it's the way a lithium battery works. The best battery for cellphones - electrically, anyway - is a lead acid gel cell. But one large enough to run a Note 4 would be about as large as a Note 4 and weigh about 5 times as much as the phone does now with the lithium battery in it, so it's not practical. (You can discharge it to absolutely flat once in a while without shortening the life too much.)
As for how long it takes to charge, that depends on how discharged it is when you start charging it and whether it's on or off. And, if it's on, how much current the phone itself is using. (Any current used by the phone isn't being used to charge the battery, and the amount of current coming into the phone is fixed, whether the phone is on or off. The Note 4 should be drawing about 1200mA with the original wall charger.)
Draining very fast can be due to a bad battery, running loads of apps at once, an electrical short somewhere in the phone - it's not something that can be diagnosed remotely. Check your battery stats and see if anything other than Android OS, Android System, Screen or Phone Standby is using a lot of battery. Those should be the ones using the most.
(BTW, what the manufacturers call 'fast charge' is what the industry calls normal charge - 1 times the battery capacity. So for a Note 4, that would be a 3.22 Amp charge. The recommended charge is 2.415 Amps [3/4 capacity]. Also, fast charge only comes in from 0% to 50% and, since the battery should never be allowed to drop below 40%, it's more of a marketing gimmick than anything else. It probably cuts the 40% to 100% charge time by a few minutes. [Letting the battery drop low enough to actually save time with fast charge is a good way to need a replacement battery every 6 months, or even less.])