It would be good to know what email app are you using. Are you using the stock email app that ships with the Note 4 or are you using the Gmail app?
If you are using the stock app, one of my only suggestions would be to try to force stop and clear both the cache and the data for the app by going to the device's settings -> application manager -> Look for the email app -> Click on the clear cache and clear data buttons.
If you are using the Gmail app, try that same procedure (but go to the Gmail app instead in the application manager). If that doesn't work, then I would proceed to check if there are updates available for the app (by going to the Play Store and going to the My Apps section and check if an update is available for Gmail).