Note 3?s benchmarking ?adjustments? inflate scores by up to 20%

[Sooks]

I wasn't aware that I compared anything to Apple. I made a statement that companies "cheating" on benchmarks should be unacceptable.

It's a little bit ironic that nVidia isn't cheating on any of them, at least according to Anandtech. Maybe they actually learned something from going through it back in the PC days.

\

Sorry not you specifically but from AnandTech.

Also, how is this cheating?

 

[jrham]

I don't understand how ramping up the processors to 100% is cheating, sure its not indicative of everyday use but it showing the potential SoC. Its not like they are circumventing part of the benchmark just running their chip full potential. I guess its all in how you view the subject but a little transparency would surely help.

Sent from my SGH-T889 using AC Forums mobile app

 

[Kevin OQuinn]

\

Sorry not you specifically but from AnandTech.

Also, how is this cheating?

I don't understand how ramping up the processors to 100% is cheating, sure its not indicative of everyday use but it showing the potential SoC. Its not like they are circumventing part of the benchmark just running their chip full potential. I guess its all in how you view the subject but a little transparency would surely help.

Sent from my SGH-T889 using AC Forums mobile app

It's not overclocking, like was discovered on the S4, which is technically worse. I'll try to explain it. During normal operation the CPU is constantly changing speed based on demand. That's great for power management. It means that the CPU can idle most of the time, and ramp up speed when you're doing stuff. If you were to look at a graph of CPU speed doing anything (and I mean anything, since nothing will ever peg the CPU to max speed continuously) it'll have peaks and valleys. Kind of like a heart monitor. It's very dynamic and is almost constantly changing, especially with the current crop of quad-core CPU's. Each core will also be able to operate dynamically, which helps with power draw and thermal management. That last point is something I will touch on further down.

The goal of a CPU is to get work done as fast as possible and then get back down to the idle state, where it's drawing the least amount of power. "Race to idle" is a term you may have heard before, and that's what it's referencing. You open an app and the CPU speed spikes, then goes back down. It did what it needed to do and went back to an idle state (or the slowest it could go while still giving good performance for what you're doing). So we've established that CPU speed is determined by the demands that you're putting on your device at that specific time. There's one other factor, though, and that's thermal management.

As you know, CPU and GPU's generate a decent amount of heat doing there job, especially when being pushed very hard. Games and benchmarks are apps that would stress both components. Manufacturer's have targets for thermals, and a max temp and time that temp can be maintained. Once that point is reached it will "throttle" the CPU/GPU, which basically means it will slow them down. That's for longevity and is something that's very common across different sectors. If you did any reading on the S4 Pro in the Nexus 4 specifically, you'll know that it was thermally limited when running benchmarks. LG/Google let the chip do all the management and didn't smooth performance out. Other manufacturer's are more conservative, and will establish limits that will give more consistent performance. Odds are that they know that most users would rather have a consistent experience over spurts of faster performance.

Based on what we know is going on then, OEM's have introduced a level of thermal and power management not available to any other app. While during normal operation a CPU/GPU will race to idle, when running a benchmark the speed is set to max the entire time. This prevents the CPU from powering down, which helps performance because it then doesn't have to speed back up. Over time, those milliseconds add up to lower benchmark scores. But what about the thermals? You can't just keep the speed at max and also keep thermals in check, right? They thought of that too, and have also changed the thermal thresholds (most likely) for this particular level of performance.

So no, they aren't overclocking. What they're doing, and this isn't just Samsung, is introducing a performance tier that's only available to the whitelisted apps. That brand new game that you just picked up that has amazing graphics will never see the type of performance that the benchmarks are seeing, because they aren't on the whitelist. The CPU/GPU will operate dynamically just like they would for another other app, and would only see max speed across all four cores on rare occasion.


Hopefully that all makes sense.

 

[Cary Quinn]

So no, they aren't overclocking. What they're doing, and this isn't just Samsung, is introducing a performance tier that's only available to the whitelisted apps. That brand new game that you just picked up that has amazing graphics will never see the type of performance that the benchmarks are seeing, because they aren't on the whitelist. The CPU/GPU will operate dynamically just like they would for another other app, and would only see max speed across all four cores on rare occasion.


Hopefully that all makes sense.

The question that should be asked then, is are we sure those other apps are not on the "whitelist", and if not, is there a way to add them to the list? Or is there simply some way to tweak the OS to open up more than "normal" levels of performance?

 

[Sooks]

It's not overclocking, like was discovered on the S4, which is technically worse. I'll try to explain it. During normal operation the CPU is constantly changing speed based on demand. That's great for power management. It means that the CPU can idle most of the time, and ramp up speed when you're doing stuff. If you were to look at a graph of CPU speed doing anything (and I mean anything, since nothing will ever peg the CPU to max speed continuously) it'll have peaks and valleys. Kind of like a heart monitor. It's very dynamic and is almost constantly changing, especially with the current crop of quad-core CPU's. Each core will also be able to operate dynamically, which helps with power draw and thermal management. That last point is something I will touch on further down.

The goal of a CPU is to get work done as fast as possible and then get back down to the idle state, where it's drawing the least amount of power. "Race to idle" is a term you may have heard before, and that's what it's referencing. You open an app and the CPU speed spikes, then goes back down. It did what it needed to do and went back to an idle state (or the slowest it could go while still giving good performance for what you're doing). So we've established that CPU speed is determined by the demands that you're putting on your device at that specific time. There's one other factor, though, and that's thermal management.

As you know, CPU and GPU's generate a decent amount of heat doing there job, especially when being pushed very hard. Games and benchmarks are apps that would stress both components. Manufacturer's have targets for thermals, and a max temp and time that temp can be maintained. Once that point is reached it will "throttle" the CPU/GPU, which basically means it will slow them down. That's for longevity and is something that's very common across different sectors. If you did any reading on the S4 Pro in the Nexus 4 specifically, you'll know that it was thermally limited when running benchmarks. LG/Google let the chip do all the management and didn't smooth performance out. Other manufacturer's are more conservative, and will establish limits that will give more consistent performance. Odds are that they know that most users would rather have a consistent experience over spurts of faster performance.

Based on what we know is going on then, OEM's have introduced a level of thermal and power management not available to any other app. While during normal operation a CPU/GPU will race to idle, when running a benchmark the speed is set to max the entire time. This prevents the CPU from powering down, which helps performance because it then doesn't have to speed back up. Over time, those milliseconds add up to lower benchmark scores. But what about the thermals? You can't just keep the speed at max and also keep thermals in check, right? They thought of that too, and have also changed the thermal thresholds (most likely) for this particular level of performance.

So no, they aren't overclocking. What they're doing, and this isn't just Samsung, is introducing a performance tier that's only available to the whitelisted apps. That brand new game that you just picked up that has amazing graphics will never see the type of performance that the benchmarks are seeing, because they aren't on the whitelist. The CPU/GPU will operate dynamically just like they would for another other app, and would only see max speed across all four cores on rare occasion.


Hopefully that all makes sense.

I'm a Network Engineer / Systems Administrator and it makes perfect sense. However I still don't see the point. The entire purpose of a benchmark is to run everything at max to see what the highest potential is (never expecting to run constantly at that speed/temp therefore never expecting to run at that potential) My system has SLI 780s which you know very well, when I 3D Mark them, they run hotter and faster then I ever intend to run them while working / gaming. However it is still fun to see what the max is. In regards to computers even the latest i7s park half the cores during most events, you have to go into the registry and force then to never park. I feel like this is standard practice for wanting to see how things run at absolute max and in my book is a fair benchmark.

 

[jrham]

Thank you, I understand that and didn't really need the long explanation but I appreciate it. I understand why many people are up in arms but its still their own processor doing actual work, nothing is being side stepped. What would you think if Samsung changed their code to run full tilt 100% of the time? (horrible battery life I know) I guess the biggest question one should have is whether they look at benchmarks as potential of the system or everyday use of the system? I myself could careless about benchmark numbers, what matters to me is everyday usability regardless of a benchmark score of 1 or 1 million.
I'd love this whole topic to go away because it's just another reason for people to fight, flame and troll (not call out anyone, just sayin)

Sent from my SGH-T889 using AC Forums mobile app

 

[mavrrick]

You actually hit the nail on the head. No one should look at a benchmark and think it has anything to do with real world performance other then the very specific task it is benchmarking. There is no good single task to benchmark every aspect of a any machine becuase there are to many variables. This doesn't apply just to Phones but any device. It is all about what are you trying to measure. If you are measureing raw CPU performance every step should be taken to max it out. If it is battery then every step should be taken to max it out.

There is a reason these are called synthetic benchmarks. They are about potential not about reality, that has always been the case.

 

[Kevin OQuinn]

The question that should be asked then, is are we sure those other apps are not on the "whitelist", and if not, is there a way to add them to the list? Or is there simply some way to tweak the OS to open up more than "normal" levels of performance?

When the initial work was being done with the S4 to see what was going on a variety of apps and games were tried, some of them VERY popular. None of those apps opened up this extra performance tier. Whether or not things could be added to list I'm not sure, that's beyond me (I'm not a dev), but I'm sure someone is trying to figure out how to do it.

The problem you'd run into would be running the phone at a thermal limit for an extended period of time beyond that of the benchmark. That could negatively impact longevity of the device itself, as well as the obvious impacts to battery life.

I'm a Network Engineer / Systems Administrator and it makes perfect sense. However I still don't see the point. The entire purpose of a benchmark is to run everything at max to see what the highest potential is (never expecting to run constantly at that speed/temp therefore never expecting to run at that potential) My system has SLI 780s which you know very well, when I 3D Mark them, they run hotter and faster then I ever intend to run them while working / gaming. However it is still fun to see what the max is. In regards to computers even the latest i7s park half the cores during most events, you have to go into the registry and force then to never park. I feel like this is standard practice for wanting to see how things run at absolute max and in my book is a fair benchmark.

Absolutely true about the benchmark on desktop GPU's, but that's not a fair comparison. As far as I know (and it's still possible they're being cheated) the GPU is still working within it's thermal limit that's imposed during any other operation (i.e. running a game). So it will throttle down during a benchmark if it needs to. Same with the CPU.

Sure, you can go and force the cores to never park, but that's a manual operation that you'd need to do before playing a game, and it will come with the same downside of unnecessarily running them at peak performance for relatively little gain.

The difference being that when you force max speeds artificially you're seeing theoretical performance and not real-world performance. I don't want to know what my hardware can do when running an artificial test that may or may even be indicative of real-world performance. I want to know how well it's going to run the apps and games that I use. A synthetic benchmark might do that, but not if it gets it's own special performance tier that's not available to those games. Absolute worst case scenario would be GLBench showing that XXXX device has the best performance of any device when running a graphics heavy load, like a game, but in real life can't even give a smooth experience without stuttering or heavily dropped frames. No, that's not what's happening here, but would be a worst case scenario.

Thank you, I understand that and didn't really need the long explanation but I appreciate it. I understand why many people are up in arms but its still their own processor doing actual work, nothing is being side stepped. What would you think if Samsung changed their code to run full tilt 100% of the time? (horrible battery life I know) I guess the biggest question one should have is whether they look at benchmarks as potential of the system or everyday use of the system? I myself could careless about benchmark numbers, what matters to me is everyday usability regardless of a benchmark score of 1 or 1 million.
I'd love this whole topic to go away because it's just another reason for people to fight, flame and troll (not call out anyone, just sayin)

Sent from my SGH-T889 using AC Forums mobile app

I think I answered that somewhat in my reply above.

 

[jrham]

Well its a topic I'm comfortable agreeing to disagree.

Sent from my SGH-T889 using AC Forums mobile app

 

[Sooks]

When the initial work was being done with the S4 to see what was going on a variety of apps and games were tried, some of them VERY popular. None of those apps opened up this extra performance tier. Whether or not things could be added to list I'm not sure, that's beyond me (I'm not a dev), but I'm sure someone is trying to figure out how to do it.

The problem you'd run into would be running the phone at a thermal limit for an extended period of time beyond that of the benchmark. That could negatively impact longevity of the device itself, as well as the obvious impacts to battery life.



Absolutely true about the benchmark on desktop GPU's, but that's not a fair comparison. As far as I know (and it's still possible they're being cheated) the GPU is still working within it's thermal limit that's imposed during any other operation (i.e. running a game). So it will throttle down during a benchmark if it needs to. Same with the CPU.

Sure, you can go and force the cores to never park, but that's a manual operation that you'd need to do before playing a game, and it will come with the same downside of unnecessarily running them at peak performance for relatively little gain.

The difference being that when you force max speeds artificially you're seeing theoretical performance and not real-world performance. I don't want to know what my hardware can do when running an artificial test that may or may even be indicative of real-world performance. I want to know how well it's going to run the apps and games that I use. A synthetic benchmark might do that, but not if it gets it's own special performance tier that's not available to those games. Absolute worst case scenario would be GLBench showing that XXXX device has the best performance of any device when running a graphics heavy load, like a game, but in real life can't even give a smooth experience without stuttering or heavily dropped frames. No, that's not what's happening here, but would be a worst case scenario.



I think I answered that somewhat in my reply above.

I guess I see both sides of it, as I do understand where you are coming from. Maybe its time to change benchmarking. Would be nice to be able to set a profile ie "Real world" or "Every day" then a "Top Performance" or something idk, but I understand where you are coming from.

 

[iceman64]

LG, HTC they all do this,

http://www.phonearena.com/news/The-...s-all-found-to-boost-benchmark-scores_id47914

Posted via Android Central App

 

[Kevin OQuinn]

LG, HTC they all do this,

The numbers always lie: HTC, LG, and Asus all found to boost benchmark scores

Posted via Android Central App

HERE's a much better link. They break it down pretty well IMO. Seems that they all do it to some degree except for Moto and nVidia. And Nexus doesn't, of course.

 

[kidstechno]

This is a problem that shouldn't exist in the first place. Benchmarks are very specific and not a representation of a device as a whole in any, way, shape, or form. It's pretty dumb but hey, when millions of dollars and subscribers are on the line, I'm sure those engineers felt some pressure one way or another.