The S9+ DXO score is misleading

[Kevin OQuinn]

Wait, I'm not here to bash. I'm only here to point out that an aggregate score is not necessarily indicative of performance WHERE IT MATTERS.

Case in point, the individual category scores from the S9+ compared to the Pixel 2. Zoom and bokeh are two primary reasons why the aggregate score is better on the S9+ (noise being the third, but that wouldn't make up the difference if you remove zoom and bokeh), even though the Pixel 2 probably does better in every other situation and does score higher in all other categories (again, aside from noise).

What I wouldn't mind is having a discussion about why DXO inherently handicaps phones that only have one rear camera versus two. It seems a bit.....disingenuous? I think I would like to see them give scores that don't include those categories, and then scores that do. I'd even be willing to say that leaving bokeh makes some sense since at least one company with a single rear camera makes claims to be able to do bokeh just fine (but doesn't quite match phones with two rear cameras).

https://twitter.com/mkbhd/status/969194721703849984?s=21 (shared his tweet for the reply with the individual category scores next to each other)

 

[rushmore]

I wish the US had the Sammy chip instead of the 845, since the CPU will be better for emulators. Lots better. GPU is a tiny fraction of weight for emulators. 845 will be good on GPU side though, so perhaps better for most.

 

[digitalbreak]

Google came out and said that you don’t need dual camera to get Portrait and zoom pictures and that Pixel 2 can do it in a single camera - they want to be compared with other dual lens system that do the same or similar thing. So it’s fair to compare Pixel 2 with the S9+. And also, S9 does not have Portrait mode like the Pixel 2 with the single camera.

Also, though we might have to wait for normal users to give feedback, I think the new camera system with variable aperture is a key differentiator with S9.

 

[Kevin OQuinn]

Google came out and said that you don’t need dual camera to get Portrait and zoom pictures and that Pixel 2 can do it in a single camera - they want to be compared with other dual lens system that do the same or similar thing. So it’s fair to compare Pixel 2 with the S9+. And also, S9 does not have Portrait mode like the Pixel 2 with the single camera.

Also, though we might have to wait for normal users to give feedback, I think the new camera system with variable aperture is a key differentiator with S9.

Agree that Google invited the comparison (said as much in my OP). I'm curious how the S9 would score, but for some reason.....they didn't test it. Given the individual category scores it's not far fetched to think that they knew it wouldn't beat the Pixel 2.

I'm also curious to see if the variable aperture has a real world impact on image quality.

 

[edubb256]

Wait, I'm not here to bash. I'm only here to point out that an aggregate score is not necessarily indicative of performance WHERE IT MATTERS.

Case in point, the individual category scores from the S9+ compared to the Pixel 2. Zoom and bokeh are two primary reasons why the aggregate score is better on the S9+ (noise being the third, but that wouldn't make up the difference if you remove zoom and bokeh), even though the Pixel 2 probably does better in every other situation and does score higher in all other categories (again, aside from noise).

What I wouldn't mind is having a discussion about why DXO inherently handicaps phones that only have one rear camera versus two. It seems a bit.....disingenuous? I think I would like to see them give scores that don't include those categories, and then scores that do. I'd even be willing to say that leaving bokeh makes some sense since at least one company with a single rear camera makes claims to be able to do bokeh just fine (but doesn't quite match phones with two rear cameras).

I don't follow the logic here. I don't think it is disingenuous at all. If they removed the categories, they would not be comparing the full capabilities of the cameras. The iphone 8+ has a better camera than the iphone 8 and the scores reflect. If someone is considering both phones, it is helpful to know how they compare.

It doesn't make sense to say that categories should only be considered if the manufacturer makes a claim they are good at that category. So if a company said their camera is not good at focusing, are you saying DXO's review should not consider focusing ability for that camera, so they can get a more competitive score??

"WHERE IT MATTERS" depends on what kind of pictures you take. If you take pictures of people, then zoom and bokeh are very important. If you you take landscapes, those things are not important. Anyone can look at the individual category scores and decide what matters to them.

 

[D13H4RD2L1V3]

This is something that I've been suspecting for a while since DxO released their new scoring system.

It just seems to favor devices that use a very specific approach to dual cameras, specifically those used for enhanced zoom capabilities. It just seems to put phones with different dual camera approaches or ones with single cameras at a disadvantage.

I don't follow the logic here. I don't think it is disingenuous at all. If they removed the categories, they would not be comparing the full capabilities of the cameras. The iphone 8+ has a better camera than the iphone 8 and the scores reflect. If someone is considering both phones, it is helpful to know how they compare.

It doesn't make sense to say that categories should only be considered if the manufacturer makes a claim they are good at that category. So if a company said their camera is not good at focusing, are you saying DXO's review should not consider focusing ability for that camera, so they can get a more competitive score??

"WHERE IT MATTERS" depends on what kind of pictures you take. If you take pictures of people, then zoom and bokeh are very important. If you you take landscapes, those things are not important. Anyone can look at the individual category scores and decide what matters to them.

What he's saying is that the overall aggregate score is misleading. The overall score for the S9 is 99, with the photo score being 104. However, if you actually read the scores awarded in each individual metric, the main reason the S9 scored much higher was the secondary camera on the S9+, meaning that it does much better than the P2 in the zoom category. This is a major contributor to why the S9+ scored more.

In fact, when it comes to the core photography performance (exposure/contrast, autofocus, texture, color), the Pixel wins, with the exception of noise, of which the S9's multi-frame processing seems to do a good job of clearing.

This is a perfect example of why aggregate scores shouldn't be treated as a sole measure of overall camera performance. When you first see it, you would think the S9 has finally beaten the Pixel 2 but when you see it further, you only realize that it beat it mostly in one category that the scoring system favors and that in most other areas, it isn't better than the P2.

 

[L0n3N1nja]

I couldn't care less about DX0 score. I frequently use manual controls and shoot in RAW, 2 things the Pixel won't do with its camera app.

 

[D13H4RD2L1V3]

I couldn't care less about DX0 score. I frequently use manual controls and shoot in RAW, 2 things the Pixel won't do with its camera app.

Not by default, but it does support the full Camera2 API.

 

[Fred98TJ]

Simply use an3rd party app. Many allow raw and have manual controls.

 

[edubb256]

This is something that I've been suspecting for a while since DxO released their new scoring system.

It just seems to favor devices that use a very specific approach to dual cameras, specifically those used for enhanced zoom capabilities. It just seems to put phones with different dual camera approaches or ones with single cameras at a disadvantage.

One of the main objectives of the new scoring system was to take into consideration manufacturers' efforts to improve zoom and bokeh. So I would expect phones that use dual cameras to enhance zoom to show better scores, all else held equal. If they didn't do that the iphone 8 would have the same score as the iphone 8+, which would be an absurd result.

I don't see evidence they are favoring an specific implementation approach, just the result. Evidence to the contrary is the fact that before the Galaxy S9+, the Pixel 2 was the highest rated camera of all, beating all of the cameras with dual lens setups.

It is inevitable when you aggregate a number to summarize a review like this, it will work better for some users than others.

DXO is transparent and discusses their methodology in detail. They say in their methodology they "weigh the various results in a way that most closely matches their importance in real-world applications as judged by mainstream users" but they add "However, every user is different, and each has differing priorities. For that reason, we encourage anyone reading our reviews to dig in past the Overall score into the sub-scores and written analysis."

 

[D13H4RD2L1V3]

One of the main objectives of the new scoring system was to take into consideration manufacturers' efforts to improve zoom and bokeh. So I would expect phones that use dual cameras to enhance zoom to show better scores, all else held equal. If they didn't do that the iphone 8 would have the same score as the iphone 8+, which would be an absurd result.

I don't see evidence they are favoring an specific implementation approach, just the result. Evidence to the contrary is the fact that before the Galaxy S9+, the Pixel 2 was the highest rated camera of all, beating all of the cameras with dual lens setups.

It is inevitable when you aggregate a number to summarize a review like this, it will work better for some users than others.

DXO is transparent and discusses their methodology in detail. They say in their methodology they "weigh the various results in a way that most closely matches their importance in real-world applications as judged by mainstream users" but they add "However, every user is different, and each has differing priorities. For that reason, we encourage anyone reading our reviews to dig in past the Overall score into the sub-scores and written analysis."

Here's the thing.

People are dumb. They just see the aggregate score and think the S9+ is better than the Pixel 2

Even though the S9+ is better than it in a few areas while the Pixel 2 is better in other areas.

I'm sure DxO wants to be as fair and transparent as possible. But I still can't shake the feeling that this scoring methodology puts a lot of phones with different dual camera implementations or single camera devices at a small disadvantage unless they can make up for the deficiency with superb post processing like the Pixel 2. If a phone has a dual camera implementation that enhances zoom, it's probably going to score more in that area than those that use it differently or only has one sensor. Which is fine but when it pushes a phone to an overall score that edges a phone that does better in the core essentials, it's a bit icky.

I understand why they did it. I just think that people are going to take the whole aggregate scoring thing the wrong way.

It's also a reminder that people need to read more into why a score is given.

 

[illdini]

Funny...I don't remember Pixel owners complaining about DXOMark when they ranked Pixel 2 the highest.

Phone camera hardware & software improve from release to release. I'm sure when the next iPhone is released, these forums will be flooded with complaints about the camera rating. Same with the next Pixel, Note, etc.

 

[D13H4RD2L1V3]

Funny...I don't remember Pixel owners complaining about DXOMark when they ranked Pixel 2 the highest.

Phone camera hardware & software improve from release to release. I'm sure when the next iPhone is released, these forums will be flooded with complaints about the camera rating. Same with the next Pixel, Note, etc.

It's not that the S9+ scored more.

It's just that the score itself can be misleading.

It would lead you to believe that the S9+ is better than the Pixel 2 overall but in actuality, it's better in zoom, noise and bokeh but not in other areas.

It's a good example that one shouldn't take an aggregate score for what it is. It's important that you examine just how they came to that score and why

 

[digitalbreak]

It's not that the S9+ scored more.

It's just that the score itself can be misleading.

It would lead you to believe that the S9+ is better than the Pixel 2 overall but in actuality, it's better in zoom, noise and bokeh but not in other areas.

It's a good example that one shouldn't take an aggregate score for what it is. It's important that you examine just how they came to that score and why

Yea, every time DxOMark publishes a score, we hear this a lot - “score can be misleading” - well, they are becoming and I think have become the mainstream media for rating smartphone cameras.

I wish everyone understood to dig deeper into individual scores but everyone will focus on the overall aggregate score as that is what matter at the end of the day when you talk numbers.

Hopefully DxOMark can improve the way they score things, but this is what we have today.

 

[Kevin OQuinn]

I don't follow the logic here. I don't think it is disingenuous at all. If they removed the categories, they would not be comparing the full capabilities of the cameras. The iphone 8+ has a better camera than the iphone 8 and the scores reflect. If someone is considering both phones, it is helpful to know how they compare.

It doesn't make sense to say that categories should only be considered if the manufacturer makes a claim they are good at that category. So if a company said their camera is not good at focusing, are you saying DXO's review should not consider focusing ability for that camera, so they can get a more competitive score??

"WHERE IT MATTERS" depends on what kind of pictures you take. If you take pictures of people, then zoom and bokeh are very important. If you you take landscapes, those things are not important. Anyone can look at the individual category scores and decide what matters to them.

"WHERE IT MATTERS" - Zoom and bokeh only matter in specific situations. The things that the Pixel 2 score higher in matter all the time, with every single image.

Having scores on the S9 would be very useful, but they didn't do that. As you stated, they scored both iPhones, which makes it easy to see just how much they weight zoom and bokeh. I hope they rate the S9 too at some point in the near future.

 

[D13H4RD2L1V3]

Yea, every time DxOMark publishes a score, we hear this a lot - “score can be misleading” - well, they are becoming and I think have become the mainstream media for rating smartphone cameras.

I wish everyone understood to dig deeper into individual scores but everyone will focus on the overall aggregate score as that is what matter at the end of the day when you talk numbers.

Hopefully DxOMark can improve the way they score things, but this is what we have today.

I just personally feel that trying to describe the performance of an entire camera system with a single set of numbers is pretty difficult, because it's an aggregate of multiple factors, some of which matter more than others.

Does the S9 have a good camera? Yes, maybe even great. Is DxO's scoring for individual metrics flawed? No, because I think it's accurate to what the camera performs.

But is the aggregate score a bit misleading at first? IMO, yeah, especially when they are getting heaps of mainstream attention.

 

[LeoRex]

"WHERE IT MATTERS" - Zoom and bokeh only matter in specific situations. The things that the Pixel 2 score higher in matter all the time, with every single image.

This....

This is the heart of the matter... DXO has been weighting zoom and portrait modes far too heavy since they were first introduced. They are both additional features designed to augment pictures, and are only used in certain circumstances. Things like exposure, focus, color, contrast, noise, detail and clarity, those are CORE features of a camera and apply to every picture taken.

So where people have issues here is that when it comes to an aggregate score, those secondary features seem to be weighted more than core features that are significantly more important. Why? Well, its trendy. The two dominant phone marks have dumped a ton of effort into those two features and they are the 'in' thing. Samsung and Apple want to sell phones, and it's becoming harder to differentiate as the platform matures and there's less fruit on the vine to pick from. Zoom and portraits are sexy... they make for good marketing campaigns and draw attention.

But they don't necessarily lead to better pictures... something that appears to not be that big a priority (at least from DXO's perspective)

 

[Itsa_Me_Mario]

Here's a comparison by feature:

 

[Itsa_Me_Mario]

I don't follow the logic here. I don't think it is disingenuous at all. If they removed the categories, they would not be comparing the full capabilities of the cameras. The iphone 8+ has a better camera than the iphone 8 and the scores reflect. If someone is considering both phones, it is helpful to know how they compare.

It doesn't make sense to say that categories should only be considered if the manufacturer makes a claim they are good at that category. So if a company said their camera is not good at focusing, are you saying DXO's review should not consider focusing ability for that camera, so they can get a more competitive score??

"WHERE IT MATTERS" depends on what kind of pictures you take. If you take pictures of people, then zoom and bokeh are very important. If you you take landscapes, those things are not important. Anyone can look at the individual category scores and decide what matters to them.

Short version is that the Pixel 2 XL is a better camera than the S9+, but loses on the aggregate score. The logic is that the stronger camera should probably win on total score.

 

[Chris Williams8]

"WHERE IT MATTERS" - Zoom and bokeh only matter in specific situations. The things that the Pixel 2 score higher in matter all the time, with every single image.

Having scores on the S9 would be very useful, but they didn't do that. As you stated, they scored both iPhones, which makes it easy to see just how much they weight zoom and bokeh. I hope they rate the S9 too at some point in the near future.

I'm not disagreeing with you but noise does matters in most cases no matter how bright or dark the photo is.
Especially if the S9 scored 20 points higher than the Pixel in that category.


At the most the Pixel beat the S9 by 7 points in those scores.

Just saying.