The solution is to turn off using voice command to answer/end calls.
On my S5, that option is not available through the main "Settings" icon as it is on other phones. On the S5 you need to bring up the "Phone" app and then:
Menu button / settings / Answering/Ending calls
Then uncheck the Using...