You can also see which app is using high mode and it should be one of those apps. You can adjust the settings if possible, or revoke location permissions for the app.
I would advise against turning off high mode all together unless you never need gps for navigation or something. Although you can turn it on at that time too