I signed up JUST for this because I've had the same problem for about 1-2 weeks now and it has also been driving me nuts! Although none of the solutions provided were the answer, smooth4lyfe's suggestion started me on the right path and I think I just found the answer.
Use the notification bar, however don't just tap the driving mode button, long press it. This will open a "Driving mode settings" panel. From here you can change some settings. Tap "My place for Driving mode", then remove the check mark from "Car" (which is what I had) or whatever option you have there that has a check mark. Maybe it will be called something different.
I have yet to test this but I am fairly confident that this will solve my problem, and hopefully yours as well.
@Jaguarr40 - that's of no help. I have the manual and wasn't able to find the answer - it may be in there, but when you don't know exactly what feature is causing the problem (s voice, driving mode, Bluetooth) it makes it hard to find the solution
@scarryhumor - very mature (said with much sarcasm) (but also slightly humorous)
@jadmet - although this will eliminate the problem, it doesn't solve it, since as soon as the phone is reconnected to Bluetooth the problem will come back. In Ontario we'll get ticketed if we're caught on a cell phone while driving so having Bluetooth for me is critical. Not to mention fiddling with a phone while driving is irresponsible.