I'm seeing precisely the same issues you have described in this thread and have tried all of the same rectification actions, resulting in the same outcomes.
I'm not sure if I had the problem before 5.1, but am wondering if something happened there.
Tis a shame as, at times, I find the voice activation to be genuinely useful on a device with a 6" screen. Still, I hope to see it corrected in future updates.
For others suffering with this problem, make sure to send some feedback directly to Google via the Search in-app link.