How We Test Cameras

We raised our standards for resolution, and therefore overall image quality, in our Certified Test Results from the Pop Photo Lab.

By John Owens | Published Sep 9, 2009 12:20 AM EDT

Gear & Reviews

We may earn revenue from the products available on this page and participate in affiliate programs. Learn more ›

First it was 10 megapixels, then 12MP, then 14MP. And now, as DSLRs hurdle the 21MP barrier, the standards for performance have taken a big leap upward.

That’s why we’re raising our standards for resolution, and therefore overall image quality, in our Certified Test Results from the Pop Photo Lab.


What does this mean to you? Plenty. Whether you’re in the market for a new camera (and who, deep down, isn’t?), or you simply want to compare the picture-taking potential of various models, this is important stuff. How Pop Photo sees cameras affects the way millions of people, from photographers to imaging engineers, see cameras, too. So I asked Technical Editor Philip Ryan to explain the reasons for and ramifications of the changes we’re making in our rating system. Underscoring the importance of the work being done in the Pop Photo Lab, I turn the podium over to Phil: The steady increase in sensor resolution in the past two years means that nearly every DSLR we’ve tested lately has scored Excellent ratings in both resolution and image quality, especially at lower ISOs. The recent spate of super-highmegapixel cameras from Canon, Nikon, and Sony has yielded numerical resolution scores that are off the charts, but without a corresponding boost in our descriptive rating (Excellent, Extremely High, Very High, High, Acceptable, Below Normal). Not anymore. While most of our tests, and the way we rate the results, remain untouched, we’ve changed the way we consider resolution. We still measure it the same way we always have, through analysis of the camera’s images of a test target. But we’ve upgraded our target to accommodate the higher resolution of today’s sensors. More important, starting with the Pentax K2000, cameras must now top 2500 lines of resolution to earn an Excellent rating, up from 1700. The table below shows exactly how each rating has changed. Don’t get us wrong-we remain impressed with the capabilities of the current crop of cameras. But the old scale made it hard to distinguish among different models. For instance, Sony’s 24.6MP Alpha 900 got the same rating as the 12.3MP Olympus E-30, even though the Sony was able to resolve over 1000 lines more than the Olympus in our test. Our new scale will make it easier to differentiate between cameras based on their respective resolving powers.

What does this mean to you? Plenty. Whether you’re in the market for a new camera (and who, deep down, isn’t?), or you simply want to compare the picture-taking potential of various models, this is important stuff. How Pop Photo sees cameras affects the way millions of people, from photographers to imaging engineers, see cameras, too. So I asked Technical Editor Philip Ryan to explain the reasons for and ramifications of the changes we’re making in our rating system. Underscoring the importance of the work being done in the Pop Photo Lab, I turn the podium over to Phil: The steady increase in sensor resolution in the past two years means that nearly every DSLR we’ve tested lately has scored Excellent ratings in both resolution and image quality, especially at lower ISOs. The recent spate of super-highmegapixel cameras from Canon, Nikon, and Sony has yielded numerical resolution scores that are off the charts, but without a corresponding boost in our descriptive rating (Excellent, Extremely High, Very High, High, Acceptable, Below Normal). Not anymore. While most of our tests, and the way we rate the results, remain untouched, we’ve changed the way we consider resolution. We still measure it the same way we always have, through analysis of the camera’s images of a test target. But we’ve upgraded our target to accommodate the higher resolution of today’s sensors. More important, starting with the Pentax K2000, cameras must now top 2500 lines of resolution to earn an Excellent rating, up from 1700. The table below shows exactly how each rating has changed. Don’t get us wrong-we remain impressed with the capabilities of the current crop of cameras. But the old scale made it hard to distinguish among different models. For instance, Sony’s 24.6MP Alpha 900 got the same rating as the 12.3MP Olympus E-30, even though the Sony was able to resolve over 1000 lines more than the Olympus in our test. Our new scale will make it easier to differentiate between cameras based on their respective resolving powers.


And since resolution plays a major part in our overall imagequality rating, we’re raising the bar for that, too. Under our old system, a camera had to achieve an Excellent rating for both resolution and color accuracy, as well as a Moderately Low or better rating in noise. Now, while Excellent color accuracy and resolution are still required, we’ve tightened things up by demanding a Low or better noise rating in order for a camera to be rated Excellent in overall image quality.Our other rating scales remain the same. For instance, our color accuracy scale is based on the Delta E system, founded in the science of color perception. In this scale, which puts a numerical value on the difference between colors, smaller numbers are better, and 0 denotes no difference.A Delta E of 1 is the smallest divergence trained observers can see between two color patches placed side by side. Most people would probably see a slight difference starting at a Delta E of 3 or 4. In our test, anything below a Delta E of 8 (significantly less than the color shift of most color-film negatives) is considered Excellent. Since there has been no change to the way people see color, there’s no need to change this scale.Similarly, our noise ratings are untouched. Using DxO Analyzer software, we measure the standard deviation of grayscale patches across a full range from dark to light. Someday, if camera makers manage to keep noise extremely low across all ISOs, we might have to change this scale, too. But as long as sensor technology continues to lean toward capturing detail rather than reducing noise, our ratings remain the same. What’s next? In the next few months, we’ll roll out a new noisetest target. We’ve been using the 24-patch GretagMacbeth ColorChecker chart, which is printed on a paper surface and includes six grayscale patches. But the latest super-high-res DSLRs can capture the surface texture, and we don’t want our software to be tricked into interpreting that as noise. So we are moving to a new, backlit, glass target with a completely smooth surface for each gray patch. It also has an additional nine gray patches from light to dark, to yield an even more accurate measurement. Want a more detailed description of our testing methods? Check out www.PopPhoto.com. We’re always looking for ways to improve our testing in the milliondollar-plus Pop Photo Lab. So as DSLR performance continues to soar, you can expect our Certified Test Results to keep pace.

And since resolution plays a major part in our overall imagequality rating, we’re raising the bar for that, too. Under our old system, a camera had to achieve an Excellent rating for both resolution and color accuracy, as well as a Moderately Low or better rating in noise. Now, while Excellent color accuracy and resolution are still required, we’ve tightened things up by demanding a Low or better noise rating in order for a camera to be rated Excellent in overall image quality.Our other rating scales remain the same. For instance, our color accuracy scale is based on the Delta E system, founded in the science of color perception. In this scale, which puts a numerical value on the difference between colors, smaller numbers are better, and 0 denotes no difference.A Delta E of 1 is the smallest divergence trained observers can see between two color patches placed side by side. Most people would probably see a slight difference starting at a Delta E of 3 or 4. In our test, anything below a Delta E of 8 (significantly less than the color shift of most color-film negatives) is considered Excellent. Since there has been no change to the way people see color, there’s no need to change this scale.Similarly, our noise ratings are untouched. Using DxO Analyzer software, we measure the standard deviation of grayscale patches across a full range from dark to light. Someday, if camera makers manage to keep noise extremely low across all ISOs, we might have to change this scale, too. But as long as sensor technology continues to lean toward capturing detail rather than reducing noise, our ratings remain the same. What’s next? In the next few months, we’ll roll out a new noisetest target. We’ve been using the 24-patch GretagMacbeth ColorChecker chart, which is printed on a paper surface and includes six grayscale patches. But the latest super-high-res DSLRs can capture the surface texture, and we don’t want our software to be tricked into interpreting that as noise. So we are moving to a new, backlit, glass target with a completely smooth surface for each gray patch. It also has an additional nine gray patches from light to dark, to yield an even more accurate measurement. Want a more detailed description of our testing methods? Check out www.PopPhoto.com. We’re always looking for ways to improve our testing in the milliondollar-plus Pop Photo Lab. So as DSLR performance continues to soar, you can expect our Certified Test Results to keep pace.