Samsung cheats at benchmarks

[snookasnoo]

AnandTech | Looking at CPU/GPU Benchmark Optimizations in Galaxy S 4

Nothing Samsung does should surprise anyone but this is underhanded even for them
They specially optimized Touchwiz for certain benchmarks to make it look faster than it is. This is explains why the Touchwiz version is "faster" than the pure Google version which never made any sense to me.
What are the odds Android Central will run this story...

 

[smooth4lyfe]

This is only for the international/Exynos version
Either way, the phone is fast as hell...it's the one I have

 

[garublador]

So the point is that benchmarks are useless if they don't actually measure how well the device performs?

 

[OhAlfie]

I'm plenty happy with my S4.

I don't give a **** about silly articles/studies like these. It's just a phone folks, ya need to relax and not spend so much energy trying to bi*ch/complain about it.

 

[Haalcyon]

I'm plenty happy with my S4.

I don't give a **** about silly articles/studies like these. It's just a phone folks, ya need to relax and not spend so much energy trying to bi*ch/complain about it.

Ah now why'd ya have to go and be the voice of reason?

Sent from me LTE Note 8.0

 

[Kevin OQuinn]

This is only for the international/Exynos version
Either way, the phone is fast as hell...it's the one I have

No it's not. Read the article. It's the Snapdragon version also, but only on the CPU side.

I remember reading all the threads that said something like "well, it benchmarks faster than the other phones" and people acting like that was a good thing. Very few folks actually said "benchmarks don't matter at all, only user experience." It's funny how the tune changes when it's discovered that Samsung was essentially cheating on those benchmarks.

This is a big deal, just like it was when GPU vendors were doing it on PC's. If any other manufacturer were doing it (I'm sure others are, they just haven't been caught yet) it would be just as big a deal.

Samsung, and others, know that people pay attention to benchmarks, and being fastest in any of them is a great marketing bullet point. That's why they do things like this, and why benchmarks will continue to not be indicative of a great user experience. I applaud Anandtech for doing the work to dig into this issues, and hope that if other manufacturers are doing it that it's also discovered.

 

[Jerry Hildenbrand]

 

[Aaron Watson]

Wait im confused. Isnt the purpose of those benchmarks is to show the full potential of the phone?

Posted via Android Central App

 

[Kevin OQuinn]

Wait im confused. Isnt the purpose of those benchmarks is to show the full potential of the phone?

Posted via Android Central App

Yes, but what they're doing is only giving certain apps (the benchmarks) the full performance of the phone. The rest see the "normal" version, with the "regular" clocks and governors.

 

[Kevin OQuinn]

Just a general warning that trolling won't be tolerated here. We can have a genuine discussion without resorting to obvious troll remarks and comments.

I'm extremely interested in this and want to have a conversation about it. I like fast hardware. I always want the fastest devices. Knowing that benchmarks are being cheated by manufacturers is kind of a big deal to me.

 

[Alex Dobie]

And this is why we never do synthetic benchmarking on AC.

 

[garublador]

I'm extremely interested in this and want to have a conversation about it. I like fast hardware. I always want the fastest devices. Knowing that benchmarks are being cheated by manufacturers is kind of a big deal to me.

It seems to me like this is a big problem with benchmarks. If they can be "cheated" then they're not actually benchmarking anything useful. It's not terribly ethical of Samsung to abuse a loophole, but it also calls into question the validity of the benchmark itself. As far as I can tell there is some financial gain to be made in writing these benchmarks. If they don't actually tell you anything useful about how a device performs and they're apparently charging someone for that, is Samsung bad for abusing it or good for pointing out that companies are charging for useless services? These benchmarking companies are claiming that OEM's need them to prove they're the fastest for marketing reasons, so why shouldn't OEM's strive to get their benchmark numbers as high as possible? I think that the companies that come up with these benchmarks are partially to blame for manufacturing the desire to cheat on the benchmarks they write and then leaving loopholes that allow the cheating to happen.

All that makes me glad when I hear stuff like this:

And this is why we never do synthetic benchmarking on AC.

 

[smooth4lyfe]

Yes, but what they're doing is only giving certain apps (the benchmarks) the full performance of the phone. The rest see the "normal" version, with the "regular" clocks and governors.

I don't really see this as the phone being slower than it really is, because if Samsung could do this, it means the phone has the potential to run at that speed. Even if normal apps use the processor at a lower speed, it could potentially be used at that speed (if rooted) so in a way, its still fast....right?

 

[Kevin OQuinn]

It seems to me like this is a big problem with benchmarks. If they can be "cheated" then they're not actually benchmarking anything useful. It's not terribly ethical of Samsung to abuse a loophole, but it also calls into question the validity of the benchmark itself. As far as I can tell there is some financial gain to be made in writing these benchmarks. If they don't actually tell you anything useful about how a device performs and they're apparently charging someone for that, is Samsung bad for abusing it or good for pointing out that companies are charging for useless services? These benchmarking companies are claiming that OEM's need them to prove they're the fastest for marketing reasons, so why shouldn't OEM's strive to get their benchmark numbers as high as possible? I think that the companies that come up with these benchmarks are partially to blame for manufacturing the desire to cheat on the benchmarks they write and then leaving loopholes that allow the cheating to happen.

All that makes me glad when I hear stuff like this:

Samsung is bad for doing it because they kept it a secret. So is any other company that might be doing it (samsung just got caught first).

It's also not a loophole, it using code to open up a performance level that isn't available anywhere else in the OS.

I don't really see this as the phone being slower than it really is, because if Samsung could do this, it means the phone has the potential to run at that speed. Even if normal apps use the processor at a lower speed, it could potentially be used at that speed (if rooted) so in a way, its still fast....right?

So you want to cheat and use the cheat in other areas? Remember, if you root and use the right kernel you can already overclock. This isn't the same at all.

It's not the phone "being slower than it is", it's samsung artificially inflating how fast it is. It's a subtle, but important, difference.


Keep in mind that a benchmark only shows how well the software communicates with the hardware. It tells you nothing about how fast the actual silicon is.

Sent from my Nexus 7 using Tapatalk HD

 

[Jerry Hildenbrand]

I don't really see this as the phone being slower than it really is, because if Samsung could do this, it means the phone has the potential to run at that speed. Even if normal apps use the processor at a lower speed, it could potentially be used at that speed (if rooted) so in a way, its still fast....right?

Samsung is bad for doing it because they kept it a secret. So is any other company that might be doing it (samsung just got caught first).

It's also not a loophole, it using code to open up a performance level that isn't available anywhere else in the OS.

So you want to cheat and use the cheat in other areas? Remember, if you root and use the right kernel you can already overclock. This isn't the same at all.

It's not the phone "being slower than it is", it's samsung artificially inflating how fast it is. It's a subtle, but important, difference.


Keep in mind that a benchmark only shows how well the software communicates with the hardware. It tells you nothing about how fast the actual silicon is.

Sent from my Nexus 7 using Tapatalk HD

I think the most important thing to find out is why Samsung keeps the GPU and CPU scaled back during normal, as advertised use. They must have a reason not to let it run wide open for apps other than the ones they whitelist in their benchmark booster code.

I'm sure a big part of it is battery drain, but what might these clock speeds be doing to the hardware.

 

[Archienj7]

I think the most important thing to find out is why Samsung keeps the GPU and CPU scaled back during normal, as advertised use. They must have a reason not to let it run wide open for apps other than the ones they whitelist in their benchmark booster code.

I'm sure a big part of it is battery drain, but what might these clock speeds be doing to the hardware.

Heat. Notice the heat during those benchmarks and hd games? Imagine it being that warm constantly, you'd be lucky to get six months to a year out of it. There were reports of overheating before IIRC and this could be why. I'm just theorizing here but it's entirely possible this is why along with the benchmark boost.

 

[jimbo1mcm]

It is interesting that Samsung would do this. There might be another explanation, because I would hope Samsung is smart enough to know that they would be caught.

 

[slackerjack]

At a guess battery life I probably the single largest concern

Posted via Android Central App

 

[slackerjack]

No it's not. Read the article. It's the Snapdragon version also, but only on the CPU side.

I remember reading all the threads that said something like "well, it benchmarks faster than the other phones" and people acting like that was a good thing. Very few folks actually said "benchmarks don't matter at all, only user experience." It's funny how the tune changes when it's discovered that Samsung was essentially cheating on those benchmarks.

This is a big deal, just like it was when GPU vendors were doing it on PC's. If any other manufacturer were doing it (I'm sure others are, they just haven't been caught yet) it would be just as big a deal.

Samsung, and others, know that people pay attention to benchmarks, and being fastest in any of them is a great marketing bullet point. That's why they do things like this, and why benchmarks will continue to not be indicative of a great user experience. I applaud Anandtech for doing the work to dig into this issues, and hope that if other manufacturers are doing it that it's also discovered.

This might be a similar concept to what GPU vendors ere doing with synthetic bench mark s but the methodology and result is completely different. GPU vendors cut workload out of he grapichs/physics code...resulting in a similar image (or sometimes identical) but what the GPU is actually doing is significantly less than the competitor...where as Samsung is wrenching up the clock speed, but doing the same work... Just faster. The reason socs aren't run at max capacity is heat, which affects battery life and soc lifespan as there's no active cooling solution applied (and barely passive)...

Likely this was done to show the max theoretical performance of the device, understanding that not all socs are going to bin the same and thus, won't tolerate higher clock speed is ton the same way. I'm not saying it was right...but I think we're in witch hunt territory here...people don't buy phones based on synthetic benchmarks like they buy graphics cards based on synthetic benchmarks... To put this much effort onto forcing results in a handful of synthetics that would matter to 1% of us isn't a good use of time or money.

From a features and side of box comparison... The Galaxy s4 already had the upper hand on the One, which was it's only real competition out of the gate... As stupid as that is...

Posted via Android Central App

 

[slackerjack]

It is interesting that Samsung would do this. There might be another explanation, because I would hope Samsung is smart enough to know that they would be caught.

Caught by whom? It's like 1% of smartphone users that even know what a synthetic benchmark is... And probably 1% of the 1% who would care of they were cheating

Posted via Android Central App