[John Sadowsky Participant Posts: 169]

The Optimum Digital Expsoure

on: November 21, 2020 at 8:42 pm

David Sutton wrote:

there was a discussion pointing out that “the top 3% contains 50% of your captured data!” was a myth. This idea is still doing the rounds. A search has brought up this: https://photomorrobay.wordpress.com/ but there are a lot of other references around if you look.

Dave – I checked out this reference – although I didn’t read the whole thing.  I will credit that article for using the term “discrete tonal value,” which I refer to as quantization levels, rather than “data” which, sorry, is just wrong.  Other than that, you are correct, that that article is pretty much off the rails.

It is important to always keep in mind that the 50% in the top stop rule only applies to linearly quantized data, which means linearly proportional to the photon count at the sensor.  Camera raw data is nearly linear.  sRGB, Adobe RGB, or ProPhoto RGB data is not linear.  RGB space data has been transformed by the inverse of the tone reproduction curve (TRC) of the particular RGB space.  Long story short, the TRC transformation is why 8 bits in sRGB or ProPhoto can have as much dynamic range as 14-bit raw data.

The Photomorrobay article wrong because it seems to be confusing raw data with RGB data.  Moreover, the claim that the horizontal axis of a histogram is a stops (logarithmic) scale is wrong.  But it is not a linear scale either.  Photo histograms are histograms of RGB data, so the horizontal axis is warped by the TRC of the particular RGB space.  This is explained in my article A Better Histogram, which includes true stops histograms.  In that article, I’ve shown where the stops levels are on both linear and RGB space histograms.  You might find that interesting.

JSS

[David Sutton Participant Posts: 5]

Re: The Optimum Digital Expsoure

Reply #1 on: November 21, 2020 at 9:27 pm

Hi John.
Thanks for your comments.
I made a short and dirty search and picked that article at random.
I was looking for someone who could explain the terms and do the math. I felt that the “top 3% contains 50% of your captured data” was false because I see no loss of information or lack of smooth tonal gradations in the midtones or shadows in my images, which would be expected if there was a lack of “data” there. Furthermore, if I photograph a dark to light gradient and apply some edits, the shadows don’t posterise first, the whole lot goes at once.
Thanks for your link. I have read it before, but I’ll raid the chocolate supply and go back over it in detail.

To add to the discussion on exposure, ten years ago I’d use UniWB and also ETTR exposure within an inch of its life because the signal to noise ratio on Canon sensors was “Consistently Ruinous And Problematic”, and I was photographing in difficult light. Now with my current mirrorless cameras I just turn the exposure compensation dial until I see detail in the highlights, and press the shutter button letting the shadows fall where they may. What I see is what I get. I guess this is closer to how I’d work with slide film. The shadow detail is fine after processing and 24 inch wide prints (the limit of my printer) look good.
David

[John Sadowsky Participant Posts: 169]

Re: The Optimum Digital Expsoure

Reply #2 on: November 21, 2020 at 10:20 pm

David Sutton wrote:

What I see is what I get.

Yeah – gotta love mirrorless.

I shoot mostly landscapes.  I shoot, review the histograms and blowout indicators in-camera, and then shoot some more.  I watch what the metering system is saying, but I only take that as a suggestion.  I pretty much make exposure adjustments based just on how the image looks in the viewfinder, and what reviewing of test shots tells me.  However, …

The big problem is that our in-camera exposure tools really suck.  Shots that the camera says have highlight blowout are fine or even underexposed when I get them in post.  I’ll give Bob credit as his techniques attempt to address that problem.  In-camera histograms and blowout indicators are based on sRGB transformation of the raw data.  Bob bypasses all that.  What we need is in-camera raw data blowout indicators.  As far as I know, no camera does that.

JSS

[Brian Kimball]

Re: The Optimum Digital Expsoure

Reply #3 on: November 21, 2020 at 11:17 pm

Physicist Emil Martinec’s 2008 paper Noise, Dynamic Range, and Bit Depth in Digital SLRs remains the authoritative source on why ETTR actually works and why it’s not because there’s more bits available to store the brightest highlights vs the midtones or shadows.

Here’s an excerpt from page 3, but it’s definitely worth reading it from the beginning:

A common maxim in digital photography is that image quality is maximized by “exposing to the right” (ETTR) — that is, raising the exposure as much as possible without clipping highlights. It is often stated that in doing so, one makes the best use of the “number of available levels” in the raw data. This explication for instance can be found in a much-quoted  tutorial on Luminous-Landscape.com. The thinking is that, because raw is a linear capture medium, each higher stop in exposure accesses the next higher bit in the digital data, and twice as many raw levels are used in encoding the raw capture. For instance, in a 12-bit file, the highest stop of exposure has 2048 levels, the next highest stop 1024 levels, the one below that 512 levels, and so on. Naively it would seem obvious that the highest quality image data would arise from concentrating the image histogram in the higher exposure zones, where the abundance of levels allows finer tonal transitions.

However, the issue is not the number of raw levels in any given segment of the raw data (as measured e.g. in stops down from raw saturation point). Rather, the point is that by exposing to the right, one achieves a higher signal to noise ratio in the raw data. The number of available raw levels has little to do with the proper reason to expose right, since as we have seen the noise rises with signal and in fact the many raw levels available in higher exposure zones are largely wasted in digitizing photon shot noise […]”

Regarding this Optimum Digital Exposure article… it seems like a smoothed over and aggressively marketed version of countless blogs on practical ETTR that have come before.  Sorry, these concepts and techniques are not new.  For example:

https://www.fastrawviewer.com/blog/spot-meter-exposure

https://www.fastrawviewer.com/blog/how-to-use-the-full-dynamic-range-of-your-camera

https://www.fastrawviewer.com/blog/exposure-bracketing

https://www.fastrawviewer.com/raw-histogram-for-culling

 

• This reply was modified 2 years, 10 months ago by Brian Kimball.
• This reply was modified 2 years, 10 months ago by Brian Kimball. Reason: cleaned up formatting and visible html

[David Sutton Participant Posts: 5]

Re: The Optimum Digital Expsoure

Reply #4 on: November 22, 2020 at 12:40 am

As far as I know, which is not a lot, Magic Lantern for Canon has a raw histogram, and therefore its auto ETTR feature should be accurate. I installed Magic Lantern on my 5D2 and the camera never blew up. And it uninstalled fine when I sold the camera.

The other issue with histograms being based on the camera jpeg, is the choice of sRGB or Adobe rgb for the colour space makes a difference. For Canon, one was more accurate for highlight clipping and the other for shadow clipping, but I can’t remember which was which.

Mostly, the dynamic range of modern sensors is so good I don’t worry about under exposing a tad. On my cameras, for those shots that are critical and where I don’t want to bracket, choosing the appropriate film simulation and bringing up the live histogram (particularly for the blue channel) and then decreasing the exposure by 1/3 of a stop will nail everything except small specular highlights. In the past with a Canon sensor I’d increase the exposure by 1/3 of a stop after bring up the histogram. Times change.

[Peter Gallagher]

Re: The Optimum Digital Expsoure

Reply #5 on: November 22, 2020 at 4:18 am

“Since ACR/Lr added Process Version 2012, it’s actually a lot more complicated largely because of Local Laplacian Filters…wait, WHAT?”

I found Jeff’s contribution to this discussion valuable, as always. Although I do not understand the details of the Laplacian pyramid paper, I think I can intuit from the image examples included there what Proc. 2102 was/is doing to capture the fine details of the exposure. I think I can see why it may be valuable at the extremes of exposure when we attempt to fill our images with light so that we edge up to both the upper and lower extremes. Jeff’s recommendations for local highlight and black/shadow recovery to really extract these preserved details suggest something I will try more often.

I habitually set highlight details to med-grey (with spot AEL) & then over expose the whole image by (with my current sensor) a slightly-conservative 1.66 EV. Even so this seems sometimes to over-expose otherwise uninteresting highlights in a distracting way (if they’re large). For example, a slice of sky that has that dreaded flat-overbright-grey tone. I haven’t tried to increase recovery using local adjustments, but the mechanisms of Proc. 2012 suggest I should (as Jeff recommends).

It’ s a treat to learn stuff.

 

• This reply was modified 2 years, 10 months ago by Peter Gallagher.

[Kevin Raber]

Re: The Optimum Digital Expsoure

Reply #6 on: November 22, 2020 at 4:21 am

“Since ACR/Lr added Process Version 2012, it’s actually a lot more complicated largely because of Local Laplacian Filters…wait, WHAT?”

I found Jeff’s contribution to this discussion valuable, as always. Although I do not understand the details of the Laplacian pyramid paper, I think I can intuit from the image examples included there what Proc. 2102 was/is doing to capture the fine details of the exposure. I think I can see why it may be valuable at the extremes of exposure when we attempt to fill our images with light so that we edge up to both the upper and lower extremes. Jeff’s recommendations for local highlight and black/shadow recovery to really extract these preserved details suggest something I will try more often.

I habitually set highlight details to med-grey (with spot AEL) & then over expose the whole image by (with my current sensor) a slightly-conservative 1.66 EV. Even so this seems sometimes to over-expose otherwise uninteresting highlights in a distracting way (if they’re large). For example, a slice of sky that has that dreaded flat-overbright-grey tone. I haven’t tried to increase recovery using local adjustments, but the mechanisms of Proc. 2012 suggest I should (as Jeff recommends).

It’ s a treat to learn stuff.

• This reply was modified 2 years, 9 months ago by Kevin Raber.

[John Sadowsky Participant Posts: 169]

Re: The Optimum Digital Expsoure

Reply #7 on: November 22, 2020 at 9:51 am

David Sutton wrote:

The other issue with histograms being based on the camera jpeg

This is a nit-pick.  JPEG is an image compression standard.  The first step in JPEG production is the transformation of raw data (14 bits, or 16 if you have a PhaseOne) to an 8-bit RGB color space, usually sRGB or Adobe (but actually, JPEGs can be produced using any RGB data).  That step includes the TRC compression curve, which is why 8-bits is enough.  The nit-pick is that histograms and blowout indicators are calculated from 8-bit RGB transformed data.  That data may also be used to produce a compressed JPEG file, but the compressed JPEG data is not used for histogram calculation.  Strictly speaking, the RGB color space transformation is not part of the JPEG standard.  It is very common to confuse “8-bit sRGB or Adobe RGB” with “JPEG” – and it’s OK because everybody knows what is meant.    (But, hey, I’m just a crazy engineer who actually reads boring documents like the JPEG standard.)

I don’t know Cannon.  I did a little googling on Magic Lantern and didn’t see anything there suggesting produces a raw data histogram.  I would think that would be something Cannon would highlight in their marketing.  Nonetheless, if Cannon has an ETTR metering mode, it may be based on raw data – which would be seriously fantastic!  But who knows.  Camera manufacturers don’t generally disclose those kinds of algorithmic details.

My Sony A7Riii has a “highlights” metering mode.  Their literature doesn’t explain what it does very well.  Is it a true ETTR?  Who knows?  I haven’t gotten good results from it.  If I shoot with highlights metering at EV = 0, I often have highlight blowout in post software (C1), and I’ve not found an EV setting that reliably fixes that.  So I don’t use that mode.

Mirrorless (and live-view in general) adds another dimension here.  Mirrorless cameras have to be constantly transforming from the sensor’s native color space to the RGB space of the electronic viewfinder.  That’s not sRGB, not Adobe RGB, but rather the native RGB for the LCD display.  (The industry standard for display color spaces is now P3, not sRGB.)  I would think that the live histogram, zebras, etc., is based on that data.  I also expect that metering uses that data, rather than actual raw data – but that’s just my guess.  There is a LOT of processing going on constantly in these cameras!

But yes, you and I are on the same page.  Most photographers don’t use the full DR of their cameras.  Blue channel saturation does horrible things to sky tones, and I find sky color is very difficult to adjust in post.  Look at my El Malpais example in the Better Histogram article, which was easily a full stop underexposed – yet easily recovered in post and it didn’t even need noise reduction in the highly pushed shadows.

 

JSS

[Srdjan Mitrovic Silver Member Posts: 14]

Re: The Optimum Digital Expsoure

Reply #8 on: November 22, 2020 at 1:07 pm

John Sadowsky wrote:

I don’t know Cannon.  I did a little googling on Magic Lantern and didn’t see anything there suggesting produces a raw data histogram.  I would think that would be something Cannon would highlight in their marketing.  Nonetheless, if Cannon has an ETTR metering mode, it may be based on raw data – which would be seriously fantastic!  But who knows.  Camera manufacturers don’t generally disclose those kinds of algorithmic details.

Magic Lantern is a non-Canon firmware written by reverse-engineering the original Canon firmware (link). It apparently contains various tools to make it easier to do ETTR. Canon R5 would be much more interesting to me if it had a Magic Lantern port. Luckily it doesn’t :).

AFAIK, the only cameras that support RAW histograms with the manufacturer’s firmware are the original Leica Monochrom and some Phase One cameras.

The in-camera JPEG settings affect the zebras and the histogram, but I have not noticed that AdobeRBG vs. sRGB makes a difference. Thom Hogan writes that on Nikon Z cameras, the sRGB setting is closer to the color LCD, but to use AdobeRGB when using UniWB.

Currently, the most accurate way to view an image histogram is using RawDigger or FastRawViewer applications. It would be great if we could do it in-camera. In the meantime, we either bracket or give up some DR to be safe.

[John Sadowsky Participant Posts: 169]

Re: The Optimum Digital Expsoure

Reply #9 on: November 22, 2020 at 4:46 pm

Srdjan Mitrovic wrote:

It would be great if we could do it in-camera.

Thanks!  I was not aware there was any raw histogramming.  Now we need to get the manufacturers to provide us these tools in-camera.

I’m still pissed that Sony doesn’t have focus stacks.

JSS

[Andrew Rodney Participant Posts: 330]

Re: The Optimum Digital Expsoure

Reply #10 on: November 23, 2020 at 2:25 pm

ETTR needs to go away, the term is optimal exposure. ETTR is a kludge it’s about a JPEG that has little to do with optimal exposure of raw. We just need a raw Histogram to evaluate raw exposure then we don’t need it (because we can then understand how our sensors behave, as we did with film, without any need for a Histogram). The people that provide RawDigger and FastRawViewer (with raw Histograms) have good info below, along with some other’s I’ve collected.

Articles on exposing for raw:

Home

http://schewephoto.com/ETTR/

The Optimum Digital Exposure

http://digitaldog.net/files/ExposeForRaw.pdf

https://www.fastrawviewer.com/blog/mystic-exposure-triangle

https://www.fastrawviewer.com/blog/red_flowers_photography_to-see-the-real-picture

https://www.rawdigger.com/howtouse/exposure-for-raw-or-for-jpegs

https://www.rawdigger.com/howtouse/beware-histogram

https://www.rawdigger.com/howtouse/calibrate-exposure-meter-to-improve-dynamic-range

Also, it’s worth pointing out again, ISO isn’t exposure. It might be used to calculate exposure but exposure is a set of simple attributes: shutter and F-stop that provides the amount of light into a capture device. Of course, the scene plays a role. Photographers can and have exposed optimally without paying attention to the automatic exposure recommended along with ISO.

Author “Color Management for Photographers" & "Photoshop CC Color Management" (pluralsight.com)”

[John Sadowsky Participant Posts: 169]

Re: The Optimum Digital Expsoure

Reply #11 on: November 23, 2020 at 4:27 pm

Andrew Rodney wrote:

ETTR needs to go away, the term is optimal exposure. ETTR is a kludge it’s about a JPEG that has little to do with optimal exposure of raw.

This whole discussion actually has nothing to do with JPEG.  JPEG is an image data compression standard.  JPEG takes in an RGB image with 8-bit color channels.  I direct you to CCITT  T.81 standard.  The transformation from raw data to an RGB color space is not part of JPEG.

The issue is the transformation to an RGB space such as sRGB, Adobe RGB, PhoPhoto, P3, or the camera’s display’s native color space.  Surely, as a color management expert, you understand this.   The fact that histograms and blowout indicators are calculated from data transformed means they are not actually telling us what is happening with the raw data channels.

I know it is common for photographers to conflate the term JPEG with 8-bit RGB data.  I guess it is OK to go along with this kind of technical transgression as long as they cause no harm.  However, practicing photographers work with multiple RGB spaces:  sRGB or P3 for web, ProPhoto as a working space, perhaps Adobe for printing.  So there are good reasons not to conflate JPEG with color space.   It is factually incorrect (you do not calculate histograms from Huffman encoded DCT coefficients).  It results in confusion because a file type is not a color space.  The day will come that HEIC takes over from JPEG, and what are we going to call it?  I really wish educators such as yourself would be more careful about the terminology you use.

Moreover, if ETTR is a kluge, as you say, then most certainly so is DiNatale’s definition.  “99% brightness in your software” is referring to levels in your software’s working RGB color space.  ETTR (based on your in-camera RGB histogram) vs. your software’s RGB working space – you’re still doing the same thing.

You are absolutely correct to point out the ISO is not part of exposure.

As pointed out in Srdjan Mitrovic’s post, image quality is maximized by maximizing exposure (photon collection) without clipping.  We should agree that that is the fundamental principle of exposure.  I will not call this optimal exposure because different photographers will have different artistic constraints on shutter speed and aperture.  ETTR, and DiNatali’s methods, are techniques to achieve that goal.

The other point pointed out in this forum is that ETTR, and DiNatali’s definition, are tightly linked to the principle of maximizing exposure only when shooting at base ISO.  When exposure is limited by shutter speed and aperture constraints without clipping the sensor’s dynamic range, then we aren’t using the full quantization dynamic range so we have latitude with ISO setting.

In either case, (a) sensor-limited at base ISO or (b) exposure-limited at higher ISO, a bit of underexposure is easily corrected in post-processing and the cost to image quality is negligible in case (a), or none in case (b).  That cannot be said when clipping occurs.

IMHO, there is no one perfect exposure methodology.  It is largely a matter of style and taste.  As long as you adhere to the principle of maximizing photon collection, how you achieve that goal is a matter of style, and the tremendous dynamic range of our modern cameras gives us a lot of flexibility.

JSS

[Author]

Posts