Normally when apps go to the background, they shouldn't be using any negligible battery life, with the exception of those meant to run for syncing and such. Unless a bug pops up, most apps should be more or less dormant in the background.
A lot of people mistakenly believe Android operates like Windows, and think unused apps should be force closed. It's not like Windows, and your phone's memory takes up the same amount of battery power whether it's used or not. Android is meant to keep it full to save power by not having to initialize common apps all the time. So if you start arbitrarily closing out apps in effort to save battery life, you can actually decrease its life because Android will simply initialize another app in the background to have at the ready.
Being that your phone is a relatively old model, have you ever replaced the battery? If not, my guess is that's your problem. Unfortunately, I have no idea if anyone makes the Note 4 batteries anymore. Chances are, even "new" batteries may have been sitting on a shelf for years. If you don't replace it with a fresh one, then you run the risk of reduced battery life right out of the box due to its age alone.