Honestly most people when it comes to their phones really only care if it does web,email,texting,calls, games, and facebook well. Then they add in other factors like looks, screen size, and price.
I switched to a S4 after owning 3g-5 on the iPhone side of things because I wanted a bigger screen phone. The number 1 app I use outside of the basics is Evernote and the experience is the same on both phones.
Sure I like all the customizations I can do in Android but if I switched back to the iPhone my basic needs would be met either way. If Apple had a 4.7 inch or bigger screen phone I probably wouldn't have even bother to look at the S4 or HTC One. I was holding my friend's iPhone 5 the other day because they needed help. I realized how much I hated the way Apple designed that screen.
But in the end of the day the only thing I really miss is iMessage, it is so well done by Apple and 3rd party solutions are not the same. Now by next year if Apple has a iPhone out with a larger than screen I may look into going back. I do not like absolutes about never buying an iPhone or Android etc. I have to say though not sure if I ever want a phone with a screen smaller than 5 inches again.