Disabling Talk Back worked for me on lollipop and kitkat. Go into Settings/Apps/All/Talk Back -> disable.
It does not remove the "ok google" functionality, which I like, and you don't have to disable the google search app either.
It just stops the annoying, constant "to use google voice...