If the phone is in your pocket it could be sensing your leg tapping the screen which will cause it to skip ahead or back. Also, if you have it in a plastic arm case, the protective plastic screen of the arm case could be causing screen taps.
I have more skips if I'm working outside with my phone in my pocket than if its sitting static on a desk or in my console. Google Assistant requires the "OK Google" keyword to wake it up and then the command "play next track", etc.
Are you hearing the google assistant tone before a track skips?
David