My headphones have a button that is extremely sensitive so it randomly will trigger voice or it will stop or start my music, I might even switch to a cheap pair of headphones without a button.
That's the only solution - a pair of headphones with no button, or with a button with a stiffer spring. You normally can't modify the sensitivity of those little switches.