There's a setting you can click to help improve the accuracy of the hands free recognition.
Press Menu > Settings > Voice input & output > voice recognizer settings and make sure the checkmark is green or on under "Personalized recognition."
This will allow the recognition software to improve itself in adapting to your specific speech and phrasing. The more you use it, the better it will work (in theory). If you only use this once or twice, it's useless. Please also know that if the device is far away from you and you're trying to give it voice commands, there will be problems with the translation as many variables can affect the sensitivity of the recognition.
I've also read of claims that if you turn on the HAC mode (hearing aid compatibility) it will help with voice recognition but it will also lower your media volume. Its found under Call Settings..... I have never tried or tested this before so I'm not sure how credible this is but if you like to, try it out and see if it helps.