What phone is this? If it's Samsung, this seems to be a somewhat known issue for them. What fixed it on my wife's S7 Edge was to go into the application manager and there's a USB settings app in the list (may need to select to view system apps). I went in it to clear the app data, then restated the phone and it's been fine since.
If you do a search about this issue on the forums, you'll find a few different threads about it.