I really think it's unfair to talk about this handoff time right now. The phone isn't even out, so how can we really know what it's going to be like? And also, even it it's true, just do like the Evo/Epic users are doing, only turn on 4G when you need that speed. 3G is plenty fast for most of what you're doing for when you're wandering around, and then when you're settled somewhere (work or home or whatever) turn on 4G and enjoy it.
It's easy to complain about a rumor from an "I know a guy..." scenario, and people jump on that way to quickly.