For me going from 4G to 3G is about 10 seconds, but going from 3G to 4G varies a lot, often only a few seconds but once in a while as long as 90 seconds? (usually by this time I either just throw it in my pocket, or toggle in and out of Airplane mode) And there is no noticeable difference in speed on 4G whether it has been in 4G all day, or it was off all day and I just toggle it back on momentarily. Never a hang/freeze issue.
I just tried it on two phone, one is on the Global RUU and the other on the offical OTA ICS update... both go from 3G to 4G in about 5 seconds, just long enough for the mobile data icon to disappear for a couple seconds, then reappear and 4G is right there.
If you are in a good signal area, why are toggling 3G and 4G? Using 4G with good signal will not drain your battery anymore than 3G, although in areas of poor signal this it can drain quicker on 4G... I think personal testing should be done, because honestly unless the phone is toggling itself back and forth between 3G and 4G because of signal strength I don't think leaving 4G on uses anymore battery than 3G.
Also, just one of those questions I gotta ask... What firmware are you running? The official OTA update to ICS? Did you do a factory reset since you got it?