"Look, all of these phones have the same hardware. Given the same OS (for the most part) and clock speed, they will all go roughly the same speed."
- Of course this is true, but what I'm testing is the ROMs against each other. I was meticulous in consistency during testing. So if the only variable is the ROM, and there is a significant difference in ALL the benchmarks I did then why wouldn't it be a good assumption that something in the better ROMs actually makes the phone perform better?
I'm not knocking your testing or the reason for testing. I appreciate the work that went into this.
A big difference? look at the actual numbers, not the bar lengths.
Let's look at Antutu, Reborn Vs Bumblebee. Reborn being the slowest and BB being the faster of the 2.2 roms (I don't count stock because of you using overclocks), your chart makes it seem like a decent difference but look at the actual numbers. 955 vs 1071, that is a bit over 120 points or about 10%.
10% speed increase is noticeable, but only barely. In computing, they say it takes 10% for you to
begin to notice a difference on anything other than a benchmark. Okay, so you have your 10%, the problem is, that is both ends of the spectrum. Most roms are less than that in terms of difference.
Harmonia and Bumblebee are only 20 points apart. That is 2%. Do you REALLY think you can tell without a benchmark? Even going back to Aphrodite 2.0, that is only 5%, odds are, you couldn't really tell. Even with the 10%, as soon as you load up some apps, that diminishes (which is why in computing we say it takes at least that). Keep in mind these are at the extreme ends. The difference between Harmonia and Bumblebee is almost nill. There is no way you can tell a difference in actual use (no offense to Obijohn, to his credit, BB will show a difference in other ways, just not normal use. I really respect what he has done).
So let's look at Quadrant.
This time we will go with Bumblebee versus AOSP.
This time we show 1071 vs 1275, about a 20% difference. The two should feel as though they are not even in the same class of phones. 20% is a massive jump in speed and an amount you should clearly be able to see and feel. It should be the same as going from a stock rom, to an overclocked Bumblebee, trust me, you
can tell a difference. And yet, AOSP feels pretty much the same. I'm not saying it is the same speed, it is faster, but there is no way you can convince me that it's 20% faster.
The Linpack scores are actually closer to the differences you can expect between them all. Which happens to show almost no difference. The only problem is that Linpack scores tend to fluctuate a bit, but if you average it out, it works out to being pretty accurate in my experience.
"And yet, even though according to this chart your phone triples in speed, you can't really tell much difference between it and other roms in terms of feel."
- User experience is anecdotal...if someone likes a ROM I'm all for them using it and being happy. What makes me happy is getting the best bang for the buck. Why benchmark computers? Because it's not always about user experience and satisfaction. Sometimes you need to know which piece of hardware or software is the best.
Which is exactly what I said, use the rom you like the best and not worry about the performance.
The rom that works best for you, will be the faster, better rom because it works for you.
A guy who needs to haul tons of stuff is better off driving a pickup as opposed to a Prius, even if the Prius gets better gas mileage and may be faster (top speed). Benchmarks become irrelevant when compared to need. The problem is that in this case we aren't comparing a pickup to a Prius, we are comparing a Prius with10 cup holders, a spare tire, and a cd player to a Prius with 4 cup holders and racing stripes. Yes, the one with stripes is faster, but by how much.
What you did is great and backs up what a few of us have been saying. Even expected results are a result. Which is GREAT!!! I'm not bashing you for it, I was thanking you for it. I was just trying to explain some of the false readings and expectations. Remember a benchmark is synthetic, it's not a real world test and may or may not indicate actual real world performance, which in this case is exactly the problem.
I give you much credit...what people do in their spare time for others who they don't even know and owe nothing to is exceptional. I'm not criticizing anyone who puts time into it, I wish I could. I wish I could build a kernel/ROM, and it wouldn't be perfect.
You could build a rom if you wanted to.
I did my first with Winrar and Notepad, and that hasn't changed much. From what I have seen of roms for other devices, it seems some aren't even doing that much.
Facebook
Twitter
Instagram