I've had the same experience. To make it easier and safer, I've turned on "wake on voice in lock screen" so all I have to do is hit the power button (which is pretty easy to do by feel while I'm driving) and then say my wakeup command. It's not perfect, but it gets the job done.