Google assistant is AARRGGHH @#$%!

The short answer is, you can't.

It doesn't use very much battery if you don't have voice wake enabled though, and it shouldn't use any at all if you're not actively using the app.

Where do you turn off the voice wake enabled option, cannot seem to find it