Is there any way I can keep my pie and eat it? That's what you're asking. Location services use a bit of battery (and wake the phone). You can't have it both ways, either turn them off and save battery or turn them on and use battery. If someone could invent a way of running electronic devices without power, he'd be VERY wealthy (if the power companies didn't make him disappear first).
I disagree completely.
In earlier android devices and os versions, there was definitely a difference in battery performance if location services was left on.
I always toggled it on/off for navigation on my previous devices.
I have been using location services, set to high accuracy, and leaving it on all the time, without a noticeable drain in battery performance on my Note 4.
I have my applications set up for limited background running. Most applications I use I don't need them checking in every 15-20 minutes, so I don't allow them.
I'm doing things with the Note 4 that "conventional android wisdom" always said don't do if you want to save battery, and I am not suffering from battery life issues.
Some of the "tried and true" statements for Android battery life no longer apply or require serious caveats with the Note 4.