Alright I deleted my battery saver app an cleared the cache an now my score is 48000+. So now I know it's a combination of them both for a low score.
The Note 4 does come with built in battery saving features... using a 3rd Party battery saver application is often not necessary and, sometimes, does more harm than good. The only battery 'saver' I've really come across that does what it sets out to do is one like Greenify... which limits how services wake up the phone.... Oftentimes, the limited power that you save by fussing with sync, wifi, etc... the app that does the administration uses all the power you save.
Now.. there is NOTHING wrong with battery saving modes, especially when they are built into the OEM software... those are far more integrated and typically work much better. Now, they do come at a cost... most often the phone's performance. But back to those low Antutu scores... a 40K score is still REALLY high. It's like having a power saving mode in your 600HP sports car that limits the power to 500HP, but you get a pretty decent boost in your gas mileage. Even with the power saving mode on, your Note 4 is still fast enough to pretty much do most everything you ask of it without breaking a sweat.