Mostly everyone I know disables S-Voice from the minute they pick up their phones and choose to use Google Now instead. That being said, if you use Google's Always Listening feature (Or S-Voice's for that matter), yeah, it'll drain your battery faster because your phone is essentially always awake, listening to everything it picks up until in detects the hot word to turn voice recognition on (OK Google or Hey Galaxy, depending on what you use).
What I do, since I don't need the phone to just wake up with my voice, is disable S-Voice (well, not disable it, just prevent it from running when I double tap the home button) and just use Google Now by long-pressing on the home button. Once there, you can tap the microphone or just say OK Google.