I was a long-time user of iPhone 4s, but when Apple announced large-screen models I happily jumped a wagon and got biggest screen iPhone 7+. It felt very unusual to hold at the beginning after having a small screen iPhone, but eventually I got used to this kind of size. Recently I took iPhone SE and it felt so small and nice in my hand that I decided to try it for a week. It was nice to hold and easy to put into any pocket, but the screen felt too small - for example, when I tried to read CNN news - it was almost unreadable despite retina display. So I returned back to my iPhone 7+.
So my conclusion is what you are using your phone. If it is just basic tasks like phoning and may be chatting then small screen is fine. However, things like reading web-sites or any other complex visual tasks definitely need bigger screen. I assume that's why Apple still offers from smallest models iPhone SE to largest iPhone X.