I was having same problem for a long time, very frustrating when music is continually interrupted by Google App I did not wish to use at present keeps freezing everything and demanding I say something.
But as much as Google App for Android seemed my enemy, Google Search on regular computer is my friend. I searched found this unanswered thread, and now have found the solution at another forum, from which I paste:
Navigate to Settings
Tap the General tab
Under "Personal" find "Language and Input"
Find "Google voice typing" and tap the Settings button (cog icon)
Tap "Ok Google" Detection
Under the "From the Google app" option, move the slider to the left.
That’s it, you have successfully turned OK Google off.
In case you can’t find the settings, there is another way:
Open the Google app
In the top left corner of the page, touch the Menu icon
Tap Settings > Voice > "OK Google" Detection
From here, you can choose when you want your phone to listen when you say "OK Google".
(Note on my particular phone, an LG39C, there was no General tab in settings, but I was successful in doing it through the 2nd recommended way through the Google App settings). Yay! Finally, no stupid interruptions when I had no intention of doing anything but listening to my music files on my phone. I can easily turn it back on when I wish to talk to Google.