Hi all,
so much time has elapsed with solutions for only a part of you, and the same topic in several other forums has stalled... Today I found out something that might both surprise and help many of you:
I have been having the OP's issue with my NotePro (SM-P905, stock). No suspicious apps, and no way to deinstall the TouchWiz toggles of course.

Today I just replaced the TouchWiz launcher ("TouchWiz Start") with the Google Now Launcher -- and voila, still playing!!! No root required -- just install Google Now Launcher and select it under Settings -> Application Defaults. There is no big difference otherwise, the toggles (when swiping down) and all other TouchWiz features are unaffected. You will gain better "Ok Google"-experience in addition. You will have to re-arrange your start screens, though.
In fact, this finding will probably be relevant for many other phones as well, and potentially also for the OP: As an example, I have seen two different samples of the Moto G 2nd (XT1068) bought within 4 weeks in Berlin, one of them using "Launcher 3" from stock and the other with the Google Now Launcher by default. (Yes, slightly different build numbers.) The OP might have missed this difference when comparing his phone to his wife's, because it seems that the Android Settings item "Home", for switching between launchers, is only displayed when you have more than one launcher installed. Clever hide-and-seek, isn't it? :-$
Hope that helps (and happy New Year),
Mk