Page 1 of 6 1234 ... LastLast
Results 1 to 25 of 139
Like Tree92Likes
  1. Thread Author  Thread Author    #1  

    Default Samsung cheats at benchmarks



    Nothing Samsung does should surprise anyone but this is underhanded even for them
    They specially optimized Touchwiz for certain benchmarks to make it look faster than it is. This is explains why the Touchwiz version is "faster" than the pure Google version which never made any sense to me.
    What are the odds Android Central will run this story...
    iPhone 5s-Verizon
    Moto X Verizon
    Nexus 7 2013
    Nexus 10
    iPad Air-Verizon
  2. #2  

    Default Re: Samsung cheats at benchmarks

    This is only for the international/Exynos version
    Either way, the phone is fast as hell...it's the one I have
    - Using the Galaxy S5! -
    Thanked by 2:
    monsieurms and neiljay6 like this.
  3. #3  

    Default Re: Samsung cheats at benchmarks

    So the point is that benchmarks are useless if they don't actually measure how well the device performs?
    bp3dots likes this.
  4. #4  

    Default Re: Samsung cheats at benchmarks

    I'm plenty happy with my S4.

    I don't give a **** about silly articles/studies like these. It's just a phone folks, ya need to relax and not spend so much energy trying to bi*ch/complain about it.
  5. #5  
    Haalcyon's Avatar

    Posts
    5,733 Posts
    Global Posts
    6,219 Global Posts
    ROM
    CM 10.2

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by OhAlfie View Post
    I'm plenty happy with my S4.

    I don't give a **** about silly articles/studies like these. It's just a phone folks, ya need to relax and not spend so much energy trying to bi*ch/complain about it.
    Ah now why'd ya have to go and be the voice of reason?

    Sent from me LTE Note 8.0
    Live long and prosper.


    Thanked by 2:
    neiljay6 and MagicMike Kelley like this.
  6. #6  
    Kevin O'Quinn's Avatar
    AC Team Emeritus

    Posts
    8,756 Posts
    Global Posts
    8,933 Global Posts
    ROM
    Stock

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by smooth4lyfe View Post
    This is only for the international/Exynos version
    Either way, the phone is fast as hell...it's the one I have
    No it's not. Read the article. It's the Snapdragon version also, but only on the CPU side.

    I remember reading all the threads that said something like "well, it benchmarks faster than the other phones" and people acting like that was a good thing. Very few folks actually said "benchmarks don't matter at all, only user experience." It's funny how the tune changes when it's discovered that Samsung was essentially cheating on those benchmarks.

    This is a big deal, just like it was when GPU vendors were doing it on PC's. If any other manufacturer were doing it (I'm sure others are, they just haven't been caught yet) it would be just as big a deal.

    Samsung, and others, know that people pay attention to benchmarks, and being fastest in any of them is a great marketing bullet point. That's why they do things like this, and why benchmarks will continue to not be indicative of a great user experience. I applaud Anandtech for doing the work to dig into this issues, and hope that if other manufacturers are doing it that it's also discovered.
    Kevin F.I.M.T.K. O'Quinn Esq.
    Thanked by 13:
    + Show/Hide list of the thanked
  7. #7  

    Default Re: Samsung cheats at benchmarks

    ಠ_ಠ
    제리
    Thanked by 4:
    MERCDROID likes this.
  8. #8  

    Default Re: Samsung cheats at benchmarks

    Wait im confused. Isnt the purpose of those benchmarks is to show the full potential of the phone?

    Posted via Android Central App
    Thanked by:
  9. #9  
    Kevin O'Quinn's Avatar
    AC Team Emeritus

    Posts
    8,756 Posts
    Global Posts
    8,933 Global Posts
    ROM
    Stock

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by Aaron Watson View Post
    Wait im confused. Isnt the purpose of those benchmarks is to show the full potential of the phone?

    Posted via Android Central App
    Yes, but what they're doing is only giving certain apps (the benchmarks) the full performance of the phone. The rest see the "normal" version, with the "regular" clocks and governors.
    Kevin F.I.M.T.K. O'Quinn Esq.
  10. #10  
    Kevin O'Quinn's Avatar
    AC Team Emeritus

    Posts
    8,756 Posts
    Global Posts
    8,933 Global Posts
    ROM
    Stock

    Default Re: Samsung cheats at benchmarks

    Just a general warning that trolling won't be tolerated here. We can have a genuine discussion without resorting to obvious troll remarks and comments.

    I'm extremely interested in this and want to have a conversation about it. I like fast hardware. I always want the fastest devices. Knowing that benchmarks are being cheated by manufacturers is kind of a big deal to me.
    Kevin F.I.M.T.K. O'Quinn Esq.
  11. #11  
    Alex Dobie's Avatar
    Android Central Europe Editor

    Posts
    197 Posts

    Default Re: Samsung cheats at benchmarks

    And this is why we never do synthetic benchmarking on AC.
  12. #12  

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by Kevin O'Quinn View Post
    I'm extremely interested in this and want to have a conversation about it. I like fast hardware. I always want the fastest devices. Knowing that benchmarks are being cheated by manufacturers is kind of a big deal to me.
    It seems to me like this is a big problem with benchmarks. If they can be "cheated" then they're not actually benchmarking anything useful. It's not terribly ethical of Samsung to abuse a loophole, but it also calls into question the validity of the benchmark itself. As far as I can tell there is some financial gain to be made in writing these benchmarks. If they don't actually tell you anything useful about how a device performs and they're apparently charging someone for that, is Samsung bad for abusing it or good for pointing out that companies are charging for useless services? These benchmarking companies are claiming that OEM's need them to prove they're the fastest for marketing reasons, so why shouldn't OEM's strive to get their benchmark numbers as high as possible? I think that the companies that come up with these benchmarks are partially to blame for manufacturing the desire to cheat on the benchmarks they write and then leaving loopholes that allow the cheating to happen.

    All that makes me glad when I hear stuff like this:

    Quote Originally Posted by Alex Dobie View Post
    And this is why we never do synthetic benchmarking on AC.
  13. #13  

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by Kevin O'Quinn View Post
    Yes, but what they're doing is only giving certain apps (the benchmarks) the full performance of the phone. The rest see the "normal" version, with the "regular" clocks and governors.
    I don't really see this as the phone being slower than it really is, because if Samsung could do this, it means the phone has the potential to run at that speed. Even if normal apps use the processor at a lower speed, it could potentially be used at that speed (if rooted) so in a way, its still fast....right?
    - Using the Galaxy S5! -
  14. #14  
    Kevin O'Quinn's Avatar
    AC Team Emeritus

    Posts
    8,756 Posts
    Global Posts
    8,933 Global Posts
    ROM
    Stock

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by garublador View Post
    It seems to me like this is a big problem with benchmarks. If they can be "cheated" then they're not actually benchmarking anything useful. It's not terribly ethical of Samsung to abuse a loophole, but it also calls into question the validity of the benchmark itself. As far as I can tell there is some financial gain to be made in writing these benchmarks. If they don't actually tell you anything useful about how a device performs and they're apparently charging someone for that, is Samsung bad for abusing it or good for pointing out that companies are charging for useless services? These benchmarking companies are claiming that OEM's need them to prove they're the fastest for marketing reasons, so why shouldn't OEM's strive to get their benchmark numbers as high as possible? I think that the companies that come up with these benchmarks are partially to blame for manufacturing the desire to cheat on the benchmarks they write and then leaving loopholes that allow the cheating to happen.

    All that makes me glad when I hear stuff like this:
    Samsung is bad for doing it because they kept it a secret. So is any other company that might be doing it (samsung just got caught first).

    It's also not a loophole, it using code to open up a performance level that isn't available anywhere else in the OS.


    Quote Originally Posted by smooth4lyfe View Post
    I don't really see this as the phone being slower than it really is, because if Samsung could do this, it means the phone has the potential to run at that speed. Even if normal apps use the processor at a lower speed, it could potentially be used at that speed (if rooted) so in a way, its still fast....right?
    So you want to cheat and use the cheat in other areas? Remember, if you root and use the right kernel you can already overclock. This isn't the same at all.

    It's not the phone "being slower than it is", it's samsung artificially inflating how fast it is. It's a subtle, but important, difference.


    Keep in mind that a benchmark only shows how well the software communicates with the hardware. It tells you nothing about how fast the actual silicon is.

    Sent from my Nexus 7 using Tapatalk HD
    Kevin F.I.M.T.K. O'Quinn Esq.
    Patrick Schroedl and Etios like this.
  15. #15  

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by smooth4lyfe View Post
    I don't really see this as the phone being slower than it really is, because if Samsung could do this, it means the phone has the potential to run at that speed. Even if normal apps use the processor at a lower speed, it could potentially be used at that speed (if rooted) so in a way, its still fast....right?
    Quote Originally Posted by Kevin O'Quinn View Post
    Samsung is bad for doing it because they kept it a secret. So is any other company that might be doing it (samsung just got caught first).

    It's also not a loophole, it using code to open up a performance level that isn't available anywhere else in the OS.

    So you want to cheat and use the cheat in other areas? Remember, if you root and use the right kernel you can already overclock. This isn't the same at all.

    It's not the phone "being slower than it is", it's samsung artificially inflating how fast it is. It's a subtle, but important, difference.


    Keep in mind that a benchmark only shows how well the software communicates with the hardware. It tells you nothing about how fast the actual silicon is.

    Sent from my Nexus 7 using Tapatalk HD
    I think the most important thing to find out is why Samsung keeps the GPU and CPU scaled back during normal, as advertised use. They must have a reason not to let it run wide open for apps other than the ones they whitelist in their benchmark booster code.

    I'm sure a big part of it is battery drain, but what might these clock speeds be doing to the hardware.
    ಠ_ಠ
    제리
    bp3dots and Seggzy like this.
  16. #16  

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by Jerry Hildenbrand View Post
    I think the most important thing to find out is why Samsung keeps the GPU and CPU scaled back during normal, as advertised use. They must have a reason not to let it run wide open for apps other than the ones they whitelist in their benchmark booster code.

    I'm sure a big part of it is battery drain, but what might these clock speeds be doing to the hardware.
    Heat. Notice the heat during those benchmarks and hd games? Imagine it being that warm constantly, you'd be lucky to get six months to a year out of it. There were reports of overheating before IIRC and this could be why. I'm just theorizing here but it's entirely possible this is why along with the benchmark boost.
  17. #17  

    Default Re: Samsung cheats at benchmarks

    It is interesting that Samsung would do this. There might be another explanation, because I would hope Samsung is smart enough to know that they would be caught.
  18. #18  

    Default Re: Samsung cheats at benchmarks

    At a guess battery life I probably the single largest concern

    Posted via Android Central App
  19. #19  

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by Kevin O'Quinn View Post
    No it's not. Read the article. It's the Snapdragon version also, but only on the CPU side.

    I remember reading all the threads that said something like "well, it benchmarks faster than the other phones" and people acting like that was a good thing. Very few folks actually said "benchmarks don't matter at all, only user experience." It's funny how the tune changes when it's discovered that Samsung was essentially cheating on those benchmarks.

    This is a big deal, just like it was when GPU vendors were doing it on PC's. If any other manufacturer were doing it (I'm sure others are, they just haven't been caught yet) it would be just as big a deal.

    Samsung, and others, know that people pay attention to benchmarks, and being fastest in any of them is a great marketing bullet point. That's why they do things like this, and why benchmarks will continue to not be indicative of a great user experience. I applaud Anandtech for doing the work to dig into this issues, and hope that if other manufacturers are doing it that it's also discovered.
    This might be a similar concept to what GPU vendors ere doing with synthetic bench mark s but the methodology and result is completely different. GPU vendors cut workload out of he grapichs/physics code...resulting in a similar image (or sometimes identical) but what the GPU is actually doing is significantly less than the competitor...where as Samsung is wrenching up the clock speed, but doing the same work... Just faster. The reason socs aren't run at max capacity is heat, which affects battery life and soc lifespan as there's no active cooling solution applied (and barely passive)...

    Likely this was done to show the max theoretical performance of the device, understanding that not all socs are going to bin the same and thus, won't tolerate higher clock speed is ton the same way. I'm not saying it was right...but I think we're in witch hunt territory here...people don't buy phones based on synthetic benchmarks like they buy graphics cards based on synthetic benchmarks... To put this much effort onto forcing results in a handful of synthetics that would matter to 1% of us isn't a good use of time or money.

    From a features and side of box comparison... The Galaxy s4 already had the upper hand on the One, which was it's only real competition out of the gate... As stupid as that is...

    Posted via Android Central App
    skidoo likes this.
  20. #20  

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by jimbo1mcm View Post
    It is interesting that Samsung would do this. There might be another explanation, because I would hope Samsung is smart enough to know that they would be caught.
    Caught by whom? It's like 1% of smartphone users that even know what a synthetic benchmark is... And probably 1% of the 1% who would care of they were cheating

    Posted via Android Central App
  21. #21  
    Kevin O'Quinn's Avatar
    AC Team Emeritus

    Posts
    8,756 Posts
    Global Posts
    8,933 Global Posts
    ROM
    Stock

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by slackerjack View Post
    This might be a similar concept to what GPU vendors ere doing with synthetic bench mark s but the methodology and result is completely different. GPU vendors cut workload out of he grapichs/physics code...resulting in a similar image (or sometimes identical) but what the GPU is actually doing is significantly less than the competitor...where as Samsung is wrenching up the clock speed, but doing the same work... Just faster. The reason socs aren't run at max capacity is heat, which affects battery life and soc lifespan as there's no active cooling solution applied (and barely passive)...

    Likely this was done to show the max theoretical performance of the device, understanding that not all socs are going to bin the same and thus, won't tolerate higher clock speed is ton the same way. I'm not saying it was right...but I think we're in witch hunt territory here...people don't buy phones based on synthetic benchmarks like they buy graphics cards based on synthetic benchmarks... To put this much effort onto forcing results in a handful of synthetics that would matter to 1% of us isn't a good use of time or money.

    From a features and side of box comparison... The Galaxy s4 already had the upper hand on the One, which was it's only real competition out of the gate... As stupid as that is...

    Posted via Android Central App
    Except the average consumer trusts those benchmarks that the 1% of us don't.

    In the article it was directly compared to what's GPU manufacturers have been caught doing.

    Sent from my Nexus 7 using Tapatalk HD
    Kevin F.I.M.T.K. O'Quinn Esq.
  22. #22  

    Default

    You genuinely think 1% of consumers are looking at benchmarks to help justify a buying decision??...ha. Androidcentral users probably represent .01% of smartphone users...and Id guess even a miniscule % of them would do so.

    Again, the concept of "cheating" is similar to what GPU vendors have done in the past...but there's a difference, those cheats have minimized the workload or register calls (or if you prefer "dumbed down" the benchmarks requirements, which is the whole point of doing a benchmark in the first place) to increase performance. Samsung hasn't done this...they've simply allowed the CPU to run the full workload at a higher frequency than it would normally run. I agree that its likely not representative of day today performance...but it's not the same thing as running less to move faster and achieve higher scores (like with real GPU cheating) and it proves out the Samsung hardware is capable of that performance in at least a limited case.

    Let me draw a comparison for you that might make more sense: if you had two identical cars running a track to prove out which was faster, the GPU cheats of old would have changed the length of the track, or lightened the weight of the car. In this case, Samsung's removed the governed from their cars engine, this increasing the maximum output and increasing the chances that you'd run down the engine faster by nature of running it to hard "cheating" is the way you describe either, but the method by which you're "cheating" is completely different and the end result as well.

    I understand also that the analogy is flawed because Samsung was never running an identical car to begin with, the SOC is different, and the memory is faster than its prime competitor...so it was natural to assume that it'd be faster anyway. If you recall, I tried to start a thread to talk about this a month ago that fell largely into the pit of "wah, I like my phone better so Samsung sucks" ( ad these things often do) and th le conversation went nowhere fast. I followed Jerry's test setup recommendation and it proved what we all though...just never posted the results...the Galaxy S 4 was faster...by some margin, its clocked higher and has faster memory when put on a level playing field wit lh the one...shocking...I know.

    At a guess...it's likely done to show max theoretical performance...and I'll agree that it's wrong if you're trying to prove out day to day performance...but NOT the same as changing the test conditions to prove a specific advantage

    Posted via Android Central App
    Last edited by slackerjack; 07-30-2013 at 08:55 PM.
  23. #23  

    Default

    Quote Originally Posted by slackerjack View Post
    You genuinely think 1% of consumers is looking at benchmarks to help justify a buying decision??...ha. Androidcentral users probably represent .01% of smartphone users...and Id guess even a miniscule % of them would do so.

    Again, the concept of "cheating" is similar to what GPU vendors have done in the past...but there's a difference, those cheats have minimized the workload or register calls (or if you prefer "dumbed down" the benchmarks requirements, which is the whole point of doing a benchmark in the first place) to increase performance. Samsung hasn't done this...they've simply allowed the CPU to run the full workload at a higher frequency than it would normally run. I agree that its likely not representative of day today performance...but it's not the same thing as running less to move faster and achieve higher scores (like with real GPU cheating) and it proves out the Samsung hardware is capable of that performance in at least a limited case.

    Let me draw a comparison for you that might make more sense: if you had two identical cars running a track to prove out which was faster, the GPU cheats of old would have changed the length of the track, or lightened the weight of the car. In this case, Samsung's removed the governed from their cars engine, this increasing the maximum output and increasing the chances that you'd run down the engine faster by nature of running it to hard "cheating" is the way you describe either, but the method by which you're "cheating" is completely different and the end result as well.

    I understand also that the analogy is flawed because Samsung was never running an identical car to begin with, the SOC is different, and the memory is faster than its prime competitor...so it was natural to assume that it'd be faster anyway. If you recall, I tried to start a thread to talk about this a month ago that fell largely into the pit of "wah, I like my phone better so Samsung sucks" ( ad these things often do) and th le conversation went nowhere fast. I followed Jerry's test setup recommendation and it proved what we all though...just never posted the results...the Galaxy S 4 was faster...by some margin, its clocked higher and has faster memory when put on a level playing field wit lh the one...shocking...I know.

    At a guess...it's likely done to show max theoretical performance...and I'll agree that it's wrong if you're trying to prove out day to day performance...but NOT the same as changing the test conditions to prove a specific advantage

    Posted via Android Central App
    If he thinks it's just Samsung that does that, news flash. Intel, AMD, Nividia all do that. Still doesn't make it right though.

    Posted via Android Central App
  24. #24  

    Default Re: Samsung cheats at benchmarks

    Quote Originally Posted by Aaron Watson View Post
    If he thinks it's just Samsung that does that, news flash. Intel, AMD, Nividia all do that. Still doesn't make it right though.

    Posted via Android Central App
    Maybe, I dunno if it's as rampant as it was back in the day when people actually bought high end PC hardware...it was always a nice of a niche market, and it's now easily a 10th the size it was in its heyday

    Posted via Android Central App
  25. #25  

    Default Re: Samsung cheats at benchmarks

    The average phone user probably has one of two responses to this "scandal", either: "Huh?" or "Yawn"

    Sent from my GT-N8013 using Android Central Forums
    skipatrol likes this.
Page 1 of 6 1234 ... LastLast

Similar Threads

  1. Samsung Galaxy S4 Battery Drain after Update
    By missa986 in forum Sprint Samsung Galaxy S4
    Replies: 11
    Last Post: 11-17-2013, 09:05 PM
  2. Samsung Galaxy S3 Mini not starting
    By Maskatul Gaming in forum Samsung Galaxy S3 Mini
    Replies: 2
    Last Post: 08-22-2013, 10:44 PM
  3. Having problems rooting Samsung s4 verizon
    By Derek11598 in forum Samsung Galaxy S4 Rooting, ROMs & Hacks
    Replies: 3
    Last Post: 08-06-2013, 11:34 AM
  4. Half off Samsung Qi wireless charger, works with new N7, $31 shipped!
    By Brandorak in forum Google Nexus 7 Tablet (2013)
    Replies: 6
    Last Post: 08-04-2013, 07:17 AM
  5. Have you ever seen the Nexus 7 Mobile available at a Canadian Retailer ?
    By alnexus in forum Google Nexus 7 Tablet (2012)
    Replies: 4
    Last Post: 07-31-2013, 12:35 PM

Posting Permissions

B