Conclusions
These results should hopefully give some context to the effect of ART on battery life. Even though the outcomes don't explicitly favor ART, they also don't really contradict any one story on the Internet. If anything, this just proves that there are situations where both runtimes can shine. Additionally, there are many other optimizations that can’t be accurately measured, none of which are represented here and could be responsible for more significant experiences.
I expect the majority of the people claiming wildly better battery life are experiencing a placebo effect. After all, every test I ran demonstrated no more than about 2%-4% difference after burning through nearly 50% of the battery.
Still, I'm fairly certain early adopters won’t be swayed to return to Dalvik, and I'm not really sure they should be. If you’re on the fence about switching, I would still advise sticking to Dalvik, at least until the next version of Android. Aside from smoother animations and scrolling, this version certainly doesn’t offer any profound increase in performance or battery life, and it’s still prone to occasional bugs. At this stage, it’s still something best left to developers and some enthusiasts.
It’s important to remember that these underwhelming results are coming from a preview version of software the Android Team did not intend for regular users to even start using yet. The code isn’t optimized to the extent that we know it can be, and there are surely several safety checks to guard against bugs, all of which are adding overhead that will eventually become unnecessary. Of course, nobody wants ART to land in the crosshairs of an upcoming Bug Watch (wink), so let’s not be too hasty about stripping out those safety measures. Nonetheless, the current incarnation is simply about getting everything working and introducing a new runtime to the world. We should be looking forward to future versions for more tangible improvements.
I’m pretty sure I’ll be hammering out an even more thorough set of benchmarks when Android 4.5 (or 5.0, or whatever) launches in a few months. In the meantime, Part 4 is coming up really soon, so keep your eyes peeled!