I'm not using the Jabra app, I'd rather not use any Jabra apps, I checked and they don't have any settings for this.
The issue is that when I use the Google Assistant by saying "OK Google" and other of its commands, my voice is picked up from the phone mic, instead of the headset mic...