I've never had mine answer to another person's voice, I just have reliability issues with it while I'm driving. Assist Driving mode seems to cut in and out at will. That's what prompted me to try utter! I have to say folks.... Utter works great and uses Google now in many situations. It just seems more accurate and more reliable to me not to mention it reading back to you what you say before sending any messages. It really is ideal and works how I wish Google now would.
Posted via my MotoX
Did you disable, or uncheck Drive Assist when using Utter?