My Droid Bionic is 4.3", and that's more than big enough for me. I was actually looking at a 4" LG dual-core (midrange phone) that was $79, but then the Bionic went on clearance for free with better specs - and has the lapdock, which can be very useful if traveling and needing to take care of work emails / spreadsheets etc. The dual-role capability of the Bionic and Atrix lines are really good for a business user, though Android in general needs work with Exchange integration.
I think a lot of this screen size thing has to do with the physique of the person using the phone as well as what they want to use it for.
i.e., if you don't have / don't want / can't afford a modern tablet and you try to use your phone for 'double duty' as a tablet, then I can very much understand wanting a bigger phone screen. Also if you have big hands, you probably won't be able to use an iPhone (3.7" and 4.0" now) or any Android under 4.3" effectively. Again, totally understandable and totally situational.
It's all preference and situational.
With the iPhone moving up in screen size and now covering 3.7" and 4" with an upper-midrange (iPhone 4S) and high end (iPhone 5), it will be interesting to see if they gain market share by attracting a new segment to their fold based on screen size. Anyone with fingers bigger than 'medium size' really has problems using the iPhone 3-4S in my experience.