I signed up just to add this comment...
This is quite annoying (and embarrassing in a quiet room when you're not expecting your phone to start shouting at you) and does seem to be a bug with the Note 4's calculator app. From reading the posts here, it looks like there is an issue with Settings-Personalisation-Accessibility-Services section. For me, it's when this service is enabled:
AutoVoice Google Now Integration
I've gone into the settings for that service and changed a few things, but nothing fixes the issue. However, as I'm using this service for integration with Tasker, I've been able to use Tasker to fix the problem. In Tasker:
1. Create a new Profile for the Calculator App
2. For the entry task, first store the Volume Media variable value %VOLM to a new variable named %mediaVolume. Then set the media volume to 0.
3. For the exit task, restore the media volume to the %mediaVolume value you stored in the entry task.
In simple terms, the media volume on my phone is now stored when the calculator app is launched, the media volume is then muted whilst using the calculator app to mask this annoying bug, and finally the media volume is restored to its old value when the calculator app is closed.
Problem solved (well covered up at least)