Welcome to Android Central! If you do a factory reset, go immediately to the Play Store (after finishing the Setup Wizard) and stop any app from automatically installing (or, if the Setup Wizard asks you if you want to restore all of your apps, say no), so that you can start with a clean slate. Also, make sure your Google account is secured with 2-step authentication:
https://www.google.com/landing/2step/
Then start reinstalling apps. Only install from the Google Play Store, and avoid apps that have to do with gambling, "free" music/video downloads, or porn. Be careful with games that are clear knockoffs of other really popular games, and avoid "RAM booster/cleaner" apps, since they often load you up with adware and other junk.
Also, make sure your devices have good screen locks. Never leave your device screen unlocked, unless it's in your hand.
It's always possible for people to snoop via wi-fi, especially if your wi-fi network isn't password protected. But unless they're skilled hackers, I doubt they can install malware remotely on your device. If you're really worried about this, use your phone's mobile data connection, not the home wi-fi network.