Just picked up a Note 3 for the wife and I plan to grab a $99 Moto X for myself.
My wife carries her phone in her purse and has never put it in her pocket, so size is not an issue. She wanted a bigger phone (coming from iPhone 4s) and when she saw the oversized Note, it was game over for anything else. I always have my phone in my jeans/shorts pocket, so even the HTC One, S4, X2 feel too big for me. She also uses the phone most often in the car, so with Bluetooth, it stays in her purse. I thought it was going to look ridiculous using it as a phone against your head, but it doesn't and I have to admit to loving the big screen, even if it's not for me, and the camera is awesome.
Have not noticed any lag, will be interesting to compare when my Moto X gets here. Does have a lot of dumb features, but turn them off and they stay off, no big deal.
And I think I like Touchwiz too. I have a vanilla android (4.3) Nexus 7 tablet and don't get all the hoopla over wanting a vanilla android experience. For updates and bloat it's a win, but some of the UI changes of do feel like improvements.