Android Lollipop is Google's biggest update to its mobile OS yet, and it's claimed to offer a host of new features and performance enhancements that we just couldn't wait to get our hands on. So obviously we installed the Android Lollipop developer preview on our Nexus 5. Also see: Android Lollipop release date and new features
But does it really offer those claimed performance enhancements? Project Volta and the new ART runtime built into Android Lollipop do indeed make for an impressive boost to battery life - for example, Ars Technica measured an increase from 345- to 471 minutes for the Nexus 5. That's amazing. How else does Android Lollipop differ from KitKat? Check out our Android L vs KitKat video preview
But what about other aspects of performance, and is it worth upgrading your phone or tablet to a potentially buggy beta OS to benefit from those enhancements now? From our own experience of Android L, we'd say probably not. Particularly given that some popular third-party apps, for instance Dropbox, just don't play nicely with the OS right now (understandably, of course). Also see: First Android Lollipop screenshots
Clearly it's not all about the benchmarks, and as we've seen over recent times it's quite possible for manufacturers to cheat these tests. We've also seen on a single device wildly different results from tests run concurrently, which is why we always publish an average figure. Also see: What's the fastest phone 2014
The Nexus 5 running KitKat already offers a very smooth user experience. It offers this with Android Lollipop too, however, and it's doubtful that the average user would notice a difference in performance. Plus, Android L is still a developer preview, and performance may well change when the final version is released in the autumn. Also see: 35 best phones of 2014
In other words, then, benchmarks should all be taken with a pinch of salt. Obviously we ran them anyway - on a Nexus 5 running KitKat and another Nexus 5 running Android Lollipop. We ran our tests at exactly the same time and under the same test conditions.
And we found nothing to get excited over.
Graphics performance is the same, for example. We use GFXBench 3.0 to measure a phone's graphic potential, and the Nexus 5 measured 9fps in Manhattan and 24.1fps in T-Rex with both Android Lollipop and KitKat.
Geekbench 3 performance was actually a little lower with the Nexus 5 running Android L than it was running KitKat. For the Android Lollipop device the best results we were able to achieve were 759 points in the single-core test and 2,101 in the multi-core test. The Nexus 5 running KitKat put in a better performance, with 927 points single-core, and 2,744 multi-core.
Only in SunSpider did the Android L device show an improvement to performance. Here we measured 1178.6ms for Android L, and 1499.8ms for Android KitKat. However, it's interesting to note that one day previous we saw 774ms in the same test for Android L, which just goes to prove how reliable are benchmarks.
What about Android Lollipop vs iOS 8? We compare the two forthcoming OSes.
Follow Marie Brewis on Twitter.