Processor/GPU intensive activities like watching vids, gaming, and navigation, will eat more battery than just texting and talking on the phone. It's to be expected. Social media and internet surfing can also cause battery drain if the websites you visit use acceleration for whatever's on the page.
For me, my Note 4 battery can last anywhere from 4 to 7 days, and it's only because I mostly use it for phone and texts only. Even when I play a few games, and do use it for watching a video here or there, or navigation, I can still get more than 3 days out of a single charge. Right now, my SOT is at 2.5 hours and the phone was last charged 5 days and 12 hours ago and it still has 40%. That should tell you how little I've used my phone for this charge cycle. The Note 4 screen, while quite large, is still too small for my eyes, so I do things like gaming, internet, videos/movies, and other stuff on my Note 10.1 tablet. My tablet sometimes gets charged nightly even though it has a 8220mah battery.
Battery usage varies with everyone, so I wouldn't worry too much about it. Just charge when you need to, and/or get a few spare batteries and you'll be golden! I have 3 spare batteries for my Note 4 and while you'd think I wouldn't need them, given my normal usage, they come in handy when I go overnight camping, or when I just plain forget to charge my phone. Luckily, with the Note 4 we have the option to just pop a spare battery in... and that's just awesome!