Also, some say that fast charging means faster draining. Is there any truth to that? Ive used the note4 w the fast charge enabled, would there be any advantage to turning fast charge off? It charges during the day while i sleep so im never "in a hurry" for it to charge.
Fast charging really isn't related to the speed at which your battery drains.
However, by nature, using a battery does small amounts of damage to it until it degrades. Fast charging can, in theory, accelerate that damage. While your battery is obviously designed to be continuously fast-charged, in general the slower you charge a battery the less damage you are doing to it. So to keep your battery at optimal capacity for the longest time, it's probably best to disable fast charging.
Especially in a situation where you are charging it overnight - if there's no hurry to charge the battery you are better off charging it slowly.
Having said all that, don't use a slow charger when you are sitting there waiting on a recharge to "baby" the battery. The amount of damage done is fairly trivial, and the Note 4 has a user-replaceable battery. Charge it in a manner that is convenient for you.
I tend to use a regular charger for my Note 4 when I charge it overnight, and only put it on the fast charger when, for example, I get home from work and have 15 minutes to dump some wattage into the battery so I can go out for dinner when I'm on call without worrying about charge anxiety (though now that I have a spare battery, I tend to just pop the back off and change batteries instead). I am, however, thinking about a fast charger for the car so I can dump power into the phone quickly while commuting home or to work. That might even eliminate the need to charge overnight.