Samsung cheats at benchmarks

[Kevin OQuinn]

This might be a similar concept to what GPU vendors ere doing with synthetic bench mark s but the methodology and result is completely different. GPU vendors cut workload out of he grapichs/physics code...resulting in a similar image (or sometimes identical) but what the GPU is actually doing is significantly less than the competitor...where as Samsung is wrenching up the clock speed, but doing the same work... Just faster. The reason socs aren't run at max capacity is heat, which affects battery life and soc lifespan as there's no active cooling solution applied (and barely passive)...

Likely this was done to show the max theoretical performance of the device, understanding that not all socs are going to bin the same and thus, won't tolerate higher clock speed is ton the same way. I'm not saying it was right...but I think we're in witch hunt territory here...people don't buy phones based on synthetic benchmarks like they buy graphics cards based on synthetic benchmarks... To put this much effort onto forcing results in a handful of synthetics that would matter to 1% of us isn't a good use of time or money.

From a features and side of box comparison... The Galaxy s4 already had the upper hand on the One, which was it's only real competition out of the gate... As stupid as that is...

Posted via Android Central App

Except the average consumer trusts those benchmarks that the 1% of us don't.

In the article it was directly compared to what's GPU manufacturers have been caught doing.

Sent from my Nexus 7 using Tapatalk HD

 

[slackerjack]

You genuinely think 1% of consumers are looking at benchmarks to help justify a buying decision??...ha. Androidcentral users probably represent .01% of smartphone users...and Id guess even a miniscule % of them would do so.

Again, the concept of "cheating" is similar to what GPU vendors have done in the past...but there's a difference, those cheats have minimized the workload or register calls (or if you prefer "dumbed down" the benchmarks requirements, which is the whole point of doing a benchmark in the first place) to increase performance. Samsung hasn't done this...they've simply allowed the CPU to run the full workload at a higher frequency than it would normally run. I agree that its likely not representative of day today performance...but it's not the same thing as running less to move faster and achieve higher scores (like with real GPU cheating) and it proves out the Samsung hardware is capable of that performance in at least a limited case.

Let me draw a comparison for you that might make more sense: if you had two identical cars running a track to prove out which was faster, the GPU cheats of old would have changed the length of the track, or lightened the weight of the car. In this case, Samsung's removed the governed from their cars engine, this increasing the maximum output and increasing the chances that you'd run down the engine faster by nature of running it to hard "cheating" is the way you describe either, but the method by which you're "cheating" is completely different and the end result as well.

I understand also that the analogy is flawed because Samsung was never running an identical car to begin with, the SOC is different, and the memory is faster than its prime competitor...so it was natural to assume that it'd be faster anyway. If you recall, I tried to start a thread to talk about this a month ago that fell largely into the pit of "wah, I like my phone better so Samsung sucks" ( ad these things often do) and th le conversation went nowhere fast. I followed Jerry's test setup recommendation and it proved what we all though...just never posted the results...the Galaxy S 4 was faster...by some margin, its clocked higher and has faster memory when put on a level playing field wit lh the one...shocking...I know.

At a guess...it's likely done to show max theoretical performance...and I'll agree that it's wrong if you're trying to prove out day to day performance...but NOT the same as changing the test conditions to prove a specific advantage

Posted via Android Central App

 

[Aaron Watson]

You genuinely think 1% of consumers is looking at benchmarks to help justify a buying decision??...ha. Androidcentral users probably represent .01% of smartphone users...and Id guess even a miniscule % of them would do so.

Again, the concept of "cheating" is similar to what GPU vendors have done in the past...but there's a difference, those cheats have minimized the workload or register calls (or if you prefer "dumbed down" the benchmarks requirements, which is the whole point of doing a benchmark in the first place) to increase performance. Samsung hasn't done this...they've simply allowed the CPU to run the full workload at a higher frequency than it would normally run. I agree that its likely not representative of day today performance...but it's not the same thing as running less to move faster and achieve higher scores (like with real GPU cheating) and it proves out the Samsung hardware is capable of that performance in at least a limited case.

Let me draw a comparison for you that might make more sense: if you had two identical cars running a track to prove out which was faster, the GPU cheats of old would have changed the length of the track, or lightened the weight of the car. In this case, Samsung's removed the governed from their cars engine, this increasing the maximum output and increasing the chances that you'd run down the engine faster by nature of running it to hard "cheating" is the way you describe either, but the method by which you're "cheating" is completely different and the end result as well.

I understand also that the analogy is flawed because Samsung was never running an identical car to begin with, the SOC is different, and the memory is faster than its prime competitor...so it was natural to assume that it'd be faster anyway. If you recall, I tried to start a thread to talk about this a month ago that fell largely into the pit of "wah, I like my phone better so Samsung sucks" ( ad these things often do) and th le conversation went nowhere fast. I followed Jerry's test setup recommendation and it proved what we all though...just never posted the results...the Galaxy S 4 was faster...by some margin, its clocked higher and has faster memory when put on a level playing field wit lh the one...shocking...I know.

At a guess...it's likely done to show max theoretical performance...and I'll agree that it's wrong if you're trying to prove out day to day performance...but NOT the same as changing the test conditions to prove a specific advantage

Posted via Android Central App

If he thinks it's just Samsung that does that, news flash. Intel, AMD, Nividia all do that. Still doesn't make it right though.

Posted via Android Central App

 

[slackerjack]

If he thinks it's just Samsung that does that, news flash. Intel, AMD, Nividia all do that. Still doesn't make it right though.

Posted via Android Central App

Maybe, I dunno if it's as rampant as it was back in the day when people actually bought high end PC hardware...it was always a nice of a niche market, and it's now easily a 10th the size it was in its heyday

Posted via Android Central App

 

[vferrari]

The average phone user probably has one of two responses to this "scandal", either: "Huh?" or "Yawn"

Sent from my GT-N8013 using Android Central Forums

 

[slackerjack]

I'm plenty happy with my S4.

I don't give a **** about silly articles/studies like these. It's just a phone folks, ya need to relax and not spend so much energy trying to bi*ch/complain about it.

From an academic perspective, it's very important. And the guys at Anadtech are hardly writing troll bait articles like most tech sites (present company excluded of course, I have at lot of respect for what Phil and the team do here) Brian Klung, An and and the rest of Anand's team are consummate professionals and have forgotten more about modern computer technology than people could even conceive of. Hell, I work with the GPUs every day, writing somewhat complex AA/AF algorithms, I've been doing it for 15 years, and I'll still go read an article from Anandtech and learn something new almost every time.

Posted via Android Central App

 

[Farish]

Smartphones have put a big dent into the mobile gaming industry(DS, PSP, etc) because of their gaming capabilities.

When you have a potential buyer of a smartphone whose one of their primary concerns is gaming, benchmarks start to matter because they are believe to give an unbiased result.

If you Google what is the fastest smartphone for gaming,

This article appears in the top 5: Galaxy S4 Benchmarks Reveal It Boasts The Fastest Gaming

This does matter. This affects search engine results, people's opinions and changes word of mouth.

Remember we are talking about people's perceptions and billions of dollars.

I know that this optimization was for the Octa processor but it still doesn't change the fact that this kind of gamesmanship is going to hurt not help Samsung in the long run.

 

[Kevin OQuinn]

If he thinks it's just Samsung that does that, news flash. Intel, AMD, Nividia all do that. Still doesn't make it right though.

Posted via Android Central App

I'm not sure you read all of my comments in this thread. I specifically said that Samsung was just caught first, but I'm sure others do it.

Maybe, I dunno if it's as rampant as it was back in the day when people actually bought high end PC hardware...it was always a nice of a niche market, and it's now easily a 10th the size it was in its heyday

Posted via Android Central App

I actually do have a bit of knowledge about the PC gaming market, and believe it or not, it's at an almost all-time high right now. The fact that companies like nVidia see a market for a $1k GPU speaks volumes about the popularity of PC gaming, even if they will only sell a relatively small number of Titan's.

As an aside, just because Samsung is gaming the system in a different way than other companies have historically done doesn't make the result any different. Skewed numbers in benchmarks. Does it really matter how you do it?

 

[JHBThree]

I don't really see this as the phone being slower than it really is, because if Samsung could do this, it means the phone has the potential to run at that speed. Even if normal apps use the processor at a lower speed, it could potentially be used at that speed (if rooted) so in a way, its still fast....right?

No.

The normal user will never see the performance that the benchmarks indicate. It is built into the code to prevent that from happening. The guys at anandtech specifically tested apps and games, and the software wouldn't let it reach that higher performance outside of the benchmarking apps.

There is obviously a reason Samsung does this, but its a cheap move on their part. They're embarassed that they were caught, so it may be a while before they explain why they're allowing benchmarking apps to have access to a level of performance that regular users can't.

Sent from my SGH-M919 using Tapatalk 2

 

[smooth4lyfe]

No.

The normal user will never see the performance that the benchmarks indicate. It is built into the code to prevent that from happening. The guys at anandtech specifically tested apps and games, and the software wouldn't let it reach that higher performance outside of the benchmarking apps.

There is obviously a reason Samsung does this, but its a cheap move on their part. They're embarassed that they were caught, so it may be a while before they explain why they're allowing benchmarking apps to have access to a level of performance that regular users can't.

Sent from my SGH-M919 using Tapatalk 2

Yeah I'm curious as to know why, and I think it's wrong that they did it...I think if they came out clean from the beginning to say these Benchmarks show the capabilities of the phone but lowered it due to.... Then it would have been less of a big issue

Sent from my Galaxy S4 Exynos via Tapatalk 4... I LOVE THIS PHONE

 

[slackerjack]

Eh, there have always been super high end video card solutions for that relative price point... the $600 price point being the golden era equivalent of the 1k card 10 years ago is less about inflation and more about there now being able to add multiple gpus and gobs of memory to single card (foot long, extra power supply requirement "single card" solutions).

Once the consoles became powerful enough to stand in for high end hardware (circa 2007/8ish) the pc only game scene all but vanished.... It's cheaper and easier to develop for consoles and it makes less sense to develop for a platform with fewer user. Look at PC only titles in the last 5 years... It's dominated by world of Warcraft (and a handful of other MMOs) and like Metro Last light... pretty bleak of you ask me, especially when the changing of the guard (from Id and epic over to crytek) see pic platform as assn afterthought.... It's a pretty good sign that times have changed. The next wave of consoles is only going to drive another nail in that coffin. Microsoft isn't making pc gaming any more attractive either with Windows 8' Mac Osx has always been a joke of a gaming os ... and since that just leave consoles.... Well pretty easy to see how it happened. Who knows maybe windows 8 on the xbox one will help matters, but time will have to tell... as it stands today, the only less appealing gaming platform than the PC is the Wii u... and even that will eventually get some 1st party support

Posted via Android Central App

 

[Kevin OQuinn]

Eh, there have always been super high end video card solutions for that relative price point... the $600 price point being the golden era equivalent of the 1k card 10 years ago is less about inflation and more about there now being able to add multiple gpus and gobs of memory to single card (foot long, extra power supply requirement "single card" solutions).

Once the consoles became powerful enough to stand in for high end hardware (circa 2007/8ish) the pc only game scene all but vanished.... It's cheaper and easier to develop for consoles and it makes less sense to develop for a platform with fewer user. Look at PC only titles in the last 5 years... It's dominated by world of Warcraft (and a handful of other MMOs) and like Metro Last light... pretty bleak of you ask me, especially when the changing of the guard (from Id and epic over to crytek) see pic platform as assn afterthought.... It's a pretty good sign that times have changed. The next wave of consoles is only going to drive another nail in that coffin. Microsoft isn't making pc gaming any more attractive either with Windows 8' Mac Osx has always been a joke of a gaming os ... and since that just leave consoles.... Well pretty easy to see how it happened. Who knows maybe windows 8 on the xbox one will help matters, but time will have to tell... as it stands today, the only less appealing gaming platform than the PC is the Wii u... and even that will eventually get some 1st party support

Posted via Android Central App

I can find numbers that say steam has more registered accounts and active users than xbox live.

MS and Valve both release numbers about active users, and there are more on Steam. But I digress.


I want to know why samsung only allows benchmark apps to see this extra level of performance that no other app can. I also would love to know what other companies are doing the same thing (htc? Sony? LG?)

Sent from my Nexus 7 using Tapatalk HD

 

[aapold]

If you use a hex editor to edit the names of the boosted apps could you get better performance in demanding apps you actually do run frequently? Just swap glbenchmark with riptide2 and so on... Maybe even make an app to swap then out....

Posted via Android Central App

 

[Almeuit]

I'm plenty happy with my S4.

I don't give a **** about silly articles/studies like these. It's just a phone folks, ya need to relax and not spend so much energy trying to bi*ch/complain about it.

This.

Sent from my T-Mobile HTC One using AC Forums.

 

[donm527#IM]

If this news was posted about the iphone or windows phone then people here would be caring more and there would be no guessing why... the obvious answer would be to deceive consumers... whether joe consumer where they may not really know about benchmarks but doesnt hurt for salesman to point it out like a car salesman would point out the horsepower on any car they are selling. and to the tech geeks its a number to pump out because it may not the the primary factor on a purchase but its sticks in your mind... you want the best performing phone you can buy that you're gonna own for a while.

funny where so many say, i don't care... i love my phone.

may be true... but i would still care that samsung resorts to deceiving people to sell a phone.

it's silly to only increase performance on performance tests.. like carrying nitrous in a car and only spraying it on races. if the phone doesnt run at that spec 100% of the time for all apps, it's useless. obviously you'd be running into heat and battery issues. they pumped for bragging rights... but this wasnt the way to to it.

I want to know why samsung only allows benchmark apps to see this extra level of performance that no other app can. I also would love to know what other companies are doing the same thing (htc? Sony? LG?)

Sent from my Nexus 7 using Tapatalk HD

 

[Aquila]

Here is Samsung's statement:

일부 벤치마크 사이트에서 제기한 갤럭시 S4 테스트에 대해 말씀드립니다. [이슈와 팩트] :: SAMSUNG TOMORROW

(Samsung Tomorrow Site)

It seems like they're indicating that using the full capabilities of the GPU is possible in any app that uses full screen (no status bar).

It is interesting that they do not address the code found entitled, "benchmark booster" that is cited in the initial allegations, other than to say, "we did not use a specific tool on purpose to achieve higher benchmark scores". Obviously one of these things are untrue, and the "benchmark booster" in the S4's firmware included had a specific white list of benchmarks that were granted access to the highest speeds by way of a whitelist. That's some pretty specific code to not consider it a "specific tool" that is used intentionally.

I get that you have a PR office and have to save face, but sometimes saving face is really about honesty and transparency. "Yes, we do that, here is why we don't run your device at those speeds all day long..."

 

[aapold]

Anandtech says Samsung should be focusing on the user experience not artificial benchmarks. But knowing you have the baddest phone IS part of the user experience for the S4. So in effect Samsung is enhancing the user experience. It's kinda like putting a spoiler on a car so people think it goes faster.

Posted via Android Central App

 

[monsieurms]

Two things---if Samsung is cheating, shame on them. They shouldn't be. It is not acceptable.

Second, I own the S4. It's the fastest phone I've ever seen. How that comports with "cheating on benchmarks" I don't know, but it's a useful bottom line. Many of the pro articles I read, btw, recommending this phone also did "real world" testing, not just relying on benchmarks. Everything they did and said NO MATTER HOW TESTED was basically along the same line and same conclusion--blazing fast, about the fastest phone in the marketplace.

So, if chicanery is going on, just on general principle, I'm a little pissed off. It doesn't change my overall impression of the phone or the real world performance. It is is lightning fast. There's no lie in that.

 

[donm527#IM]

From what I read it's a great phone and i think most can agree that it's one of the best phones and performing phones this year comparable to the One. Heck they even use the same CPU so it they should all perform pretty well and pretty much all should be close... but stupid to cheat just to bump up the numbers for bragging rights.

an S4 would still be on my short list of phones to upgrade... but this is just a black eye to the company itself more than the phone... the funny ads versus apple are cool imo... pumping benchmark numbers and lieing to us... uncool.

being king of the benchmark hill is so shortlived anyways... why risk it with tricks like this? the new phones with the 800 series processors are coming out anyways so the s4 already went down on my shortlist.

Two things---if Samsung is cheating, shame on them. They shouldn't be. It is not acceptable.

Second, I own the S4. It's the fastest phone I've ever seen. How that comports with "cheating on benchmarks" I don't know, but it's a useful bottom line. Many of the pro articles I read, btw, recommending this phone also did "real world" testing, not just relying on benchmarks. Everything they did and said NO MATTER HOW TESTED was basically along the same line and same conclusion--blazing fast, about the fastest phone in the marketplace.

So, if chicanery is going on, just on general principle, I'm a little pissed off. It doesn't change my overall impression of the phone or the real world performance. It is is lightning fast. There's no lie in that.

 

[Citizen Coyote]

You genuinely think 1% of consumers are looking at benchmarks to help justify a buying decision??...ha. Androidcentral users probably represent .01% of smartphone users...and Id guess even a miniscule % of them would do so.

Again, the concept of "cheating" is similar to what GPU vendors have done in the past...but there's a difference, those cheats have minimized the workload or register calls (or if you prefer "dumbed down" the benchmarks requirements, which is the whole point of doing a benchmark in the first place) to increase performance. Samsung hasn't done this...they've simply allowed the CPU to run the full workload at a higher frequency than it would normally run. I agree that its likely not representative of day today performance...but it's not the same thing as running less to move faster and achieve higher scores (like with real GPU cheating) and it proves out the Samsung hardware is capable of that performance in at least a limited case.

Let me draw a comparison for you that might make more sense: if you had two identical cars running a track to prove out which was faster, the GPU cheats of old would have changed the length of the track, or lightened the weight of the car. In this case, Samsung's removed the governed from their cars engine, this increasing the maximum output and increasing the chances that you'd run down the engine faster by nature of running it to hard "cheating" is the way you describe either, but the method by which you're "cheating" is completely different and the end result as well.

I understand also that the analogy is flawed because Samsung was never running an identical car to begin with, the SOC is different, and the memory is faster than its prime competitor...so it was natural to assume that it'd be faster anyway. If you recall, I tried to start a thread to talk about this a month ago that fell largely into the pit of "wah, I like my phone better so Samsung sucks" ( ad these things often do) and th le conversation went nowhere fast. I followed Jerry's test setup recommendation and it proved what we all though...just never posted the results...the Galaxy S 4 was faster...by some margin, its clocked higher and has faster memory when put on a level playing field wit lh the one...shocking...I know.

At a guess...it's likely done to show max theoretical performance...and I'll agree that it's wrong if you're trying to prove out day to day performance...but NOT the same as changing the test conditions to prove a specific advantage

Posted via Android Central App

I agree that the vast majority of smartphone buyers don't care about benchmark numbers. However, they do care (or at least notice) buzzwords and simple "2x as fast!" marketing bullet points. Don't believe me? Look at almost every recent product reveal by Apple and Amazon, to take two examples. Invariably, somewhere in the presentation is a slide or two comparing their device to the competition, with "2x as fast" or "4x as fast" labels on pretty bar charts. These numbers by themselves mean little, but they're likely loosely based off some sort of benchmark (only so they can "prove" the numbers to curious tech journalists). The numbers also look impressive to fans and curious onlookers. Even if they don't go spouting out on forums how their device is 4x faster at rendering triangles than the competition, that little nugget can get lodged in their head when it comes time to buy a device. If that's what pushes them to buy Amazing Device over Brand X, then the company succeeded.

I think this is what these manipulated benchmarks boil down to: they allow the company's PR and marketing divisions to make those simplistic "2x faster" bullet points and slides. It also leads to misleading articles like someone else mentioned, where the high benchmarks are translated into improved gaming performance. It's dishonest, and we've seen it time and again over the past 15 years in the PC industry. Unless companies know people are watching and willing to call them out on cooked test results, they'll keep doing it.

All of the above, including what Slackerjaw said, is why I personally consider benchmarks to be an academic info nugget at best. As long as the device I'm using performs, I don't care what numbers it gets, nor do I care if Brand X is faster or slower since that's not what I'm using.