I used to have an iPhone 5 also before I switched to an LG G3. It was my first Android. It took a bit to get used to it, but I gave myself time to fiddle with it and learn it and was really happy once I got the hang of it. That switch prompted me to trade in my iPad for a Note Pro 12.2 tablet as well.
I switched my G3 to a Note 4 this week and I absolutely love it. I'm one that wanted the larger sized screen and I don't regret it. It's just so much easier to view things on the bigger real estate. The Note 4 is really fast, my internet and video playback speeds are great, and the screen is so vibrant and sharp. The rest of my family stayed with the iPhones, because they are just very easy to use and they aren't into customization. Would I personally go back, no. By the way, I downloaded Nova Launcher (and purchased the paid version) as many people have suggested and set my screens up just the way I want them. If you pulled the trigger today on the 4, come back and let us know what you think.