Samsung cheats at benchmarks

garublador

Well-known member
May 20, 2013
1,135
0
0
Visit site
The problem with benchmarks is that becasue they're made in a way that it's possible to optimize your device to do well on them, it only makes sense for everyone to optimize their device. Otherwise you aren't comparing anything useful. If some are optimizing and some aren't, then the benchmark is useless. It's on the people developing the benchmarks to make them in such a way that they give a realistic difference between devices. If they don't do that, then they're just selling smoke and mirrors. Blaming Samsung for benchmarks being smoke and mirrors isn't going to improve anything at all. We'll still have benchmarks that don't measure anything useful.
 

Dapper37

Member
Dec 28, 2012
10
0
0
Visit site
Seriously though, lets not forget this is a company that copied its way to the top with its mobile device hardware. even long before mobile was what it is today. Now their popular because of someone elses OS. I put nothing past samsumg. Sad but true.
 
Last edited:

Kedar

Well-known member
May 3, 2010
1,029
14
0
Visit site
This is no different than manufacturers claiming to get X hours battery life on their laptop/tablet/phone.

Kevin O'Quinn said:
Samsung, and others, know that people pay attention to benchmarks, and being fastest in any of them is a great marketing bullet point. That's why they do things like this, and why benchmarks will continue to not be indicative of a great user experience. I applaud Anandtech for doing the work to dig into this issues, and hope that if other manufacturers are doing it that it's also discovered.
Isn't it the fault of those reviewers to be using such benchmarks? Samsung doesn't market the S4 with Quadrant numbers.

Anyway, move on. Not a big deal. All Samsung did was put it at max clock speed... they didn't even overclock the device.
 

LegalAmerican

Well-known member
Feb 15, 2012
2,330
156
0
Visit site
AnandTech | Looking at CPU/GPU Benchmark Optimizations in Galaxy S 4

Nothing Samsung does should surprise anyone but this is underhanded even for them
They specially optimized Touchwiz for certain benchmarks to make it look faster than it is. This is explains why the Touchwiz version is "faster" than the pure Google version which never made any sense to me.
What are the odds Android Central will run this story...

Personally, I don't even think Samsung makes phones. I stay up at night thinking about all the evil in the world and how Samsung has created all of it. And then I run to AnandTech to see what they can confirm. All tech sites have their small biases (like you hinted at when you wondered if AC would run this story) but not AnandTech. They would never do any of that. They are the opposite of everything Samsung stands for and I will not sleep until I can confirm that Samsung isn't even a real company.
 

Kevin OQuinn

AC Team Emeritus
May 17, 2010
9,267
496
0
Visit site
This is no different than manufacturers claiming to get X hours battery life on their laptop/tablet/phone.


Isn't it the fault of those reviewers to be using such benchmarks? Samsung doesn't market the S4 with Quadrant numbers.

Anyway, move on. Not a big deal. All Samsung did was put it at max clock speed... they didn't even overclock the device.

So you're passing the buck on to the reviewers? Seriously?

To use your battery life example, are you also going to say that reviewers shouldn't test battery life against manufacturer claims then? We've always said that benchmarks should be taken with a huge grain of salt, and this is the reason why. As has been stated, though, we don't reach everyone, and there are very justifiable reasons why cheating the benchmarks can be bad for consumers, with the ones that stick out to me being 1) the marketing spin that can be put on it 2) the sites that post the results can rank high in Google search.

We could argue about what constitutes an overclock, also. Sure, they might not have exceeded the max freq that the chip can run at, but if 99.9% of the time it runs at one freq, and then in a few specific circumstances it gets clocked higher, isn't that by definition an overclock?

I'm not saying that it's a huge deal and should steer people away from the S4 (I've never said that about benchmarks relative to any device), but this isn't something that needs to be overlooked. It's the beginning of a terrible trend in mobile, just as it was a terrible trend in GPU's on PC's.
 

blitz118

Well-known member
Apr 22, 2010
618
39
0
Visit site
This is no different than manufacturers claiming to get X hours battery life on their laptop/tablet/phone.


Isn't it the fault of those reviewers to be using such benchmarks? Samsung doesn't market the S4 with Quadrant numbers.

Anyway, move on. Not a big deal. All Samsung did was put it at max clock speed... they didn't even overclock the device.

He's not going to let this go. He is going to drill this to no end.
 

slackerjack

Well-known member
Sep 3, 2010
662
26
0
Visit site
It's the beginning of a terrible trend in mobile, just as it was a terrible trend in GPU's on PC's.

Its a reality, and on that was bound to make its way over to the mobile space as it gained in popularity, you can abhore it all you want, but its never going to go away as long as users substitute personal success and achievement with some status symbol (phones in this case), that mankind hasn't been able to get past whipping out their genetalia and comparing size or getting into who can pee furthest contests...the idea that this won't continue is optimistic at best.

In a lot of ways, its like racism: completely stupid, propagated by those who continue to benefit from it...supported by the ignorant...and its never going to go away.

Posted via Android Central App
 

OhAlfie

Well-known member
Sep 30, 2012
868
11
18
Visit site
So you're passing the buck on to the reviewers? Seriously?

To use your battery life example, are you also going to say that reviewers shouldn't test battery life against manufacturer claims then? We've always said that benchmarks should be taken with a huge grain of salt, and this is the reason why. As has been stated, though, we don't reach everyone, and there are very justifiable reasons why cheating the benchmarks can be bad for consumers, with the ones that stick out to me being 1) the marketing spin that can be put on it 2) the sites that post the results can rank high in Google search.

We could argue about what constitutes an overclock, also. Sure, they might not have exceeded the max freq that the chip can run at, but if 99.9% of the time it runs at one freq, and then in a few specific circumstances it gets clocked higher, isn't that by definition an overclock?

I'm not saying that it's a huge deal and should steer people away from the S4 (I've never said that about benchmarks relative to any device), but this isn't something that needs to be overlooked. It's the beginning of a terrible trend in mobile, just as it was a terrible trend in GPU's on PC's.


At the end of the day, 99.99% of the folks who own this or any smartphone would say "Who cares!!". Seriously, the general public knows pretty much nothing about "bennchmarks" nor do they care. They just care about having the new "it" phone so they can take their vertical videos of their kids soccer game and listen to their Jay-Z while they jog. The amount of folks who "care" about this silly crap is so miniscule it's worthless (IMO) to keep beating the topic to death like some seem to want to do.
 

garublador

Well-known member
May 20, 2013
1,135
0
0
Visit site
So you're passing the buck on to the reviewers? Seriously?
I think it's fair to blame anyone giving credibility to benchmarks that are easy to manipulate.

It's the beginning of a terrible trend in mobile, just as it was a terrible trend in GPU's on PC's.
I'm not sure I'd call it a trend. I'd call it, "How benchmarks have always worked and why they only be used as a weak indication of performance in a specific area."
 

Jerry Hildenbrand

Space Cowboy
Staff member
Oct 11, 2009
5,569
2,797
113
Visit site
I think it's fair to blame anyone giving credibility to benchmarks that are easy to manipulate.

Been saying this for 3 years. Nobody listens to me.

LOL@ all your benchmarks that run in Dalvik.
1 -- It's no different than running them on your PC through the Android emulator. Neither is fully native, or has direct access to the hardware.
2 -- The Optimus 3D. LG showed off how great its benchmarks were, and they were really freaking high. You ever hear anyone mention the Optimus 3D (or the AT&T equivalent that nobody remembers) when they talk about the most "powerful" Android phones? Only if that person were also really freaking high ...
 
Last edited:

Kedar

Well-known member
May 3, 2010
1,029
14
0
Visit site
So you're passing the buck on to the reviewers? Seriously?
Yeah... they're the ones using benchmarks that are supposed to show the true potential of the phone. Samsung just optimized their devices to run those apps well.
It's the beginning of a terrible trend in mobile, just as it was a terrible trend in GPU's on PC's.
I don't get on here too much, so I really don't want to start a full scale discussion/argument, but... Mountains out of molehills man. =/
 

smooth4lyfe

Trusted Member
Sep 16, 2012
5,534
0
0
Visit site
Samsung's Official Response to the situation

Under ordinary conditions, the Galaxy S4 has been designed to allow a maximum GPU frequency of 533MHz. However, the maximum GPU frequency is lowered to 480MHz for certain gaming apps that may cause an overload, when they are used for a prolonged period of time in full-screen mode. Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance.

The maximum GPU frequencies for the Galaxy S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results.

We remain committed to providing our customers with the best possible user experience.

Samsung speaks out against allegations of benchmark-rigging | SamMobile
 

Kevin OQuinn

AC Team Emeritus
May 17, 2010
9,267
496
0
Visit site
I think I'm done replying. Simply because I'm saying far more than is being quoted. So my words are being used completely out of context.

I haven't said it's a huge deal. I haven't said samsung is the only one doing it. I haven't said benchmarks should be the basis of the decision to buy a phone. All I've said is that this is disingenuous of Samsung. It does a disservice to consumers. I would say the same if it were any other company.

As for their statement, I read the Anandtech article again. They ran demanding games (safe to assume full screen, right?), and got the lower GPU frequency. Also, to deny that they did any specific tuning for benchmarks, but the code is called "BenchmarkBooster".... I'll just leave it at that.

Eh, you guys have fun.

Sent from my Nexus 7 using Tapatalk HD
 

OhAlfie

Well-known member
Sep 30, 2012
868
11
18
Visit site
I think I'm done replying. Simply because I'm saying far more than is being quoted. So my words are being used completely out of context.

I haven't said it's a huge deal. I haven't said samsung is the only one doing it. I haven't said benchmarks should be the basis of the decision to buy a phone. All I've said is that this is disingenuous of Samsung. It does a disservice to consumers. I would say the same if it were any other company.

As for their statement, I read the Anandtech article again. They ran demanding games (safe to assume full screen, right?), and got the lower GPU frequency. Also, to deny that they did any specific tuning for benchmarks, but the code is called "BenchmarkBooster".... I'll just leave it at that.

Eh, you guys have fun.

Sent from my Nexus 7 using Tapatalk HD

No it doesn't. Only does that to the .0000000000001% of techy folks who think this stuff is actually important enough to complain about. The rest of the consumers don't care, heck they don't even know what a "benchmark" test is most likely.
 

marrzie

Well-known member
Nov 17, 2012
64
0
0
Visit site
Im glad 99.99999999999999% of people finally agree specs and benchmarks dont matter in a smartphone. That was a tired argument.
 

LegalAmerican

Well-known member
Feb 15, 2012
2,330
156
0
Visit site
Now they're just lying. Anandtech tested all of those apps and it never went above 480. They also completely ignore the fact that its called BenchmarkBooster.

Sent from my SGH-M919 using Tapatalk 2

I am 100% certain you've never owned or liked a device that was deviously programmed or designed to slightly skew the numbers (when they otherwise might not have been) in it's favor. Samsung could give you a brand new car and you'd complain about the trim. When a person despises a company so completely no matter what they say or do, said person starts to lose their credibility.

The phone industry is designed and operated with the intent to make money. As much as it possibly can. And that includes Apple and HTC as much as it does Samsung. Why do people act surprised when they find out that Jesus and Mother Theresa are not the CEO's of these companies? There will ALWAYS be inner workings in EVERY company that you wont agree with. It doesn't make them the epitome of all evil, and it also doesn't make your company of choice the best there ever was. I get annoyed when these upper management figures do shady business. But I can't change it, and as long as the device works to my expectations, AnandTech isn't going to convince me that the companies want to sacrifice my firstborn.
 

LegalAmerican

Well-known member
Feb 15, 2012
2,330
156
0
Visit site
No it doesn't. Only does that to the .0000000000001% of techy folks who think this stuff is actually important enough to complain about. The rest of the consumers don't care, heck they don't even know what a "benchmark" test is most likely.

I fear you're misunderstanding why they are complaining. It's all the notorious Samsung haters that have jumped on this. And if HTC was found to be doing it, all the HTC haters would be all over it. It's more about demonizing a company to some of these people than it is about the importance of honesty or true benchmark scores. This same crap happens in every thread that points out a negative aspect of whatever company.
 

JHBThree

Well-known member
Feb 15, 2012
4,096
147
0
Visit site
No it doesn't. Only does that to the .0000000000001% of techy folks who think this stuff is actually important enough to complain about. The rest of the consumers don't care, heck they don't even know what a "benchmark" test is most likely.

Until they go to a site like, say, Engadget, which includes all of the benchmarks Samsung was cheating at in their reviews. They'll look at the numbers and say 'hey, these are higher, so the S4 must be a faster/better device'.

It's the lack of knowledge of benchmark tests that makes this so egregious for regular consumers. Everyone here already knows to take them with a grain of salt; because regular consumers don't know that, they won't.
 

Latest posts

Members online

Trending Posts

Forum statistics

Threads
942,406
Messages
6,913,939
Members
3,158,399
Latest member
pauloxcavalcante