I have most of the bloatware disabled and only a few apps on my phone. I only download apps that don't require crazy permissions or run a lot of processes in the background.
I definitely notice lag. None of this 5 minute lag people speak of, but it takes the phone about 2-3 seconds to switch between tasks. For example if i'm browsing the web on chrome, and I get a text message. It takes the phone about 3 seconds to open up the text message. Also the camera always seems to lag for me. If I try to open the camera quick to capture something that just happened, I usually miss my opportunity because of the delay. It's very minor lag, but my iPhone was instant with everything...so i'm readjusting.