Jumping to voice input when I pick up the phone

[Stoutwoman]

Whenever I pick up the phone or put it down, the screen jumps somewhere I didn't ask it to go ...usually to voice input, or the apps tray, or the clock/timer.
Is there a way to make it stop doing this?
Best phone I've ever had, love it to bits, but this habit is starting to annoy.