I don't care about the processor as long as there is no lag and ok battery life.
Then you care about the processor. Because battery life is directly a function of A) How much juice the battery can store (which is pretty much a constant in a given sized phone right now) and B) How efficient the CPU is at using the available power.
A faster CPU based on the same chipset will burn up the battery quicker; a faster CPU based on a smaller die process may use substantially less power even though it's faster. So you definitely should care.
As for it being smooth - unfortunately right now it's not really an issue of how fast the processor or the GPU is, it's the fact that less of the Android UI has gotten the full benefit of GPU hardware acceleration (this becomes even more true with many UI skins that OEMs put over it) then on the iPhone. This could be partially overcome with enough ram in a phone and a good enough garbage collector, but even then you'd eventually hit lag when you multi-tasked enough.
So for battery life it's the underlying processor design and architecture that matters, and for smoothness it's more a function of proper OS programming to take advantage of the GPU better.