I was having the exact same issue with my LG G4 and it was driving me nuts! I think I've eventually found the setting that stops it. No doubt there'll be a difference on your phone, but it might give you an idea where to look.
Go to: General settings > Language & Input > Speech - Voice...