[John Sadowsky Participant Posts: 169]

A Better Histogram, human vision?

on: November 16, 2019 at 11:47 pm

Ken – thanks.  The DR of the human eye is a contentious issue – there are some rather ugly threads on DPReview about this.  I hesitate to go to far down that rabbit hole.  I admit I have a narrow point of view about this.

I believe a lot of the controversy comes from confusion adaptation with sensor DR.  I do not include adaptation in DR.  If we do include adaptation, the eye can adapt from night vision to brightest daylight over a range of more than 40 stops.  I don’t think anybody claims the DR of human vision is 40 stops.  There is a wide range of adaptations, some fast (like the iris) and some a lot slower.  Where do we draw the line between sensor DR and adaptation?  I claim we should do it the say way we do it for our cameras.  Adaptation (exposure and ISO) is not considered to be part of DR.  Its just the DR of the sensor.

The high resolution fovea of the eye has only a 15 degree field of view.  Photoreceptors in the eye saturate at about 100 photons, which is where the 6.5 stops number comes from.  Our perception of vision is based on the fusion of many foveal image fragments that occurs in the occipital lobe of the brain.  Those image fragments can have different iris and perhaps other fast retinal adaptation.  The process of human vision perception is similar to HDR merging and tone mapping.  But like HDR merging and tone mapping, those different (iris or otherwise) adaptations on different foveal fragments have the effect of compressing a wide DR (that represented by the input foveal fragments) to a narrower DR that we actually perceive.  So much of this boils down to being precise about which DR are we referring to.  The total DR represented by all the foveal fragments before fusion, or the DR of the fused image that we perceive.

The problem is that we can’t download images from our occipital lobes.  So we can’t easily measure the DR of perceived vision.  Still, I claim that DR is only 6-7 stops.  The best way to see this is with a stops gradient displayed on a high DR media.  However, here I offer two other arguments.  First, consider that our best prints can have DRs that fall in the range of ~6 stops (matte paper) to as much as 9 stops.  Most high end monitors have a 9 stop DR.  If we have the ability to actually see wider DR, then we would find those media to be much less satisfying.  Second, as I’ve shown in the article, the gamma levels used in post processing software (histograms and curve adjustments) only cover ~6.5 stops.  If the DR of vision perception was larger we see that deficiency in our post processing.

Ok, that’s my defense.  Again I recognize that it is based on a narrow concept of DR of human vision, and there are other valid points of view.

JSS

[John Sadowsky Participant Posts: 169]

Re: Morning Balloon and Clouds

Reply #1 on: November 2, 2019 at 10:13 pm

Now that is a nice pic!

JSS

[John Sadowsky]

Re: Lr and C1

Reply #2 on: October 31, 2019 at 11:04 pm

Jeff – thank you for your reply.  I am particularly please with your support for open standards.  I have further comments and questions.

“There are two basic flavors of DNG files, there’s raw file DNGs … and converting to DNG and there’s also a Linear DNG that is no longer fully raw.”  True. Raw data is data as it comes off the sensor’s ADC.  DNG supports raw data storage along with all the tags that store all the sensor specific data needed to linearize it.  DNG can be a true RAW file.  It is also true that DNG conversion produces “linearized-DNG” files.  I would add that there is a third format, the floating point-DNG, which is the DNG file that Lightroom’s HDR merge produces.  This came up this thread because Mike Nelson Pedde indicated that  C1 doesn’t recognize Lr’s HDR.dng files.  I pointed out that that is most likely because of the floating point data format used by those files.

By the way – all of these files are TIFF files.  TIFF is Tagged Image File Format.  The tags are like a table of contents for the file that point to various segments of the file contain data.  There are tags for image data, tags for metadata, tags for data format, etc.  TIFF-EP adds additional tags – mostly metadata.  Proprietary RAW files are TIFF files with proprietary tags – they point to data, but you don’t know how to interpret that data unless you pay a royalty and sign a nondisclosure.  DNG is a TIFF file with additional tags defined in the open standard DNG specification.  JPEG files also use TIFF tags.  An exceptions are the ICC profiles that use a different format.  ICC profiles are embedded in TIFF and RGB files (to specify the color space) – but again as data blocks pointed to by a TIFF tag.  I know this stuff because I’m write Mac software that accesses these files.  I have to understands these various file format standards in great detail.  Hopefully, you’ll see some of my work soon in an article on stops histograms that I submitted to this site.

“The funny thing is most people don’t know that Camera Raw and Lightroom (in the Develop module) always convert the proprietary raw file to DNG on the fly.”  Of course they do!  Moreover, it is not just Lr Develop and ACR, anybody’s raw conversion does exactly the same thing as the first step.  C1 is no different.  (I’m not saying they do a full DNG file-to-file conversion, but the definitely do linearization first.)  The linearization steps are described in Chapter 5 of the DNG specification, which is just 1 page of the spec.  These are linearization of the nonlinear photodiode response at the top end near full well capacity, and black level subtraction (which is a pattern bias largely due to the photodiode dark current).  It is a single input – single output correction that depends only on the characteristics of the sensor’s photodiodes.  It donesn’t change with time and there won’t be better new algorithms in the future.

So, you save your pristine raw data, but every time you want to access it you’re going to do the same linearization.  I just don’t see the point.  Just what are you preserving with the true raw data?  Why are you willing to forgo the clear benefits of an open standard – a point of commonality justified by everything we know about image processing theory?  Do you expect there will be an image that couldn’t get from your DNG conversion, but could from your proprietary RAW file?  If so, please explain how that might occur.

Lastly, I don’t understand your statement “make a small parameter change in the file, the entire DNG file needs to be backed up.”  I’ve heard this before, but I don’t understand where it comes from.  My understanding is that when you make adjustments in Lr, the adjustments are stored in the Lr catalog, or sidecar file, – not in the DNG file.  I tested this.  I checked the modified date in the Mac finder of a DNG file – which was the same as the import date.  I made adjustments, closed Lr, and back in the finder there was no charge to “modified date.”  If the DNG file is changed, why doesn’t the Mac Finder recognize that?  If I’m mistaken about this, I’d really like to understand what I’m missing here.

Please forgive me for my brashness.  I know I am a nobody in the photography world, but I do understand data storage engineering.  I’ve asked several photography experts these questions, and to date I haven’t gotten satisfactory answers.

JSS

• This reply was modified 4 years, 6 months ago by John Sadowsky.

[John Sadowsky]

Re: Starting a Club

Reply #3 on: October 30, 2019 at 8:56 pm

I’ve done some of these things.  We do have a community college – I was going to check out their program.

I attended a club in PA a couple of times about 1.5 hour drive one way.  They did a cool thing – you submit 5 picture each month, the organizer displays then anonymously and the group critiques.  That was actually fun.

Maryland Photography Alliance – bingo!  I did Google for clubs, but didn’t find that resource?  In fact, from MD Photo Alliance, I found a club in Hartford County that is rather close.  They have over 700 members.  What’s more surprising is that Meetup got me to the club in PA, but didn’t tell me about the Hartford county club.

This has been really useful.  Thanks so much!

JSS

• This reply was modified 4 years, 6 months ago by John Sadowsky.

[John Sadowsky Participant Posts: 169]

Re: Photography Clubs

Reply #4 on: October 30, 2019 at 10:10 am

OK – I’m starting to see how this works.  I was looking for “Start New Topic” – I see now you created a new forum “The Art of Photography > Photography Clubs”.  I’ll create a kick off topic there later today.

Thanks again!  We’ll run it up the flag pole and see if anybody salutes!

JSS

[John Sadowsky Participant Posts: 169]

Re: Photography Clubs

Reply #5 on: October 30, 2019 at 9:54 am

Thanks Christopher!  I’m a bit confused on how these forums work.  I assume you the gate keeper who approves and creates the new forums?  If so, where is it?  I thought it might go under the “Speakers Corner – Ideas, Opinion & Debate.”  Do you need an initial post to kick it off?

JSS

[John Sadowsky]

Re: Lr and C1

Reply #6 on: October 29, 2019 at 10:59 pm

Kevin Raber wrote:

You are welcome to rant John.  …

Thank you.  Kevin, I really appreciate your experience – both technical and artistic.  That’s why I read your article and watch your videos.  You are passionate about photography.  I am too.  I am also passionate about engineering.  I beg you to consider that the (independent) engineer’s point of view is also important here.

I claim that there is nothing that proprietary RAW files can do that you can’t do with DNG.  We can debate that.  This is a testable hypotheses.  It an issue that should be addressed in PhotoPXL forums, and elsewhere.

I am definitely interested in hearing Jeff Schewe‘s opinions.  I read his book The Digital Negative some years ago – it was very influential.  I’m not saying he was promoting DNG back then (although I thought he did, but regardless, he may have changed his opinion).  He did say this in his book:  “I need to point out that the DNG file format is advancing (currently at DNG spec version 1.4).”  Actually, today the current version is still 1.4, which was published in 2012.  In fact, DNG has only had to two widely adopted versions: 1.2 in 2008 and 1.4 in 2012.  The fact that there have only really been two versions indicates that this format deals with features of data storage that are so fundamental they really don’t change with time – otherwise, Adobe would have addressed that.  Moreover, the fact that there are such fundamental principles of image storage (and this is my turf) indicates that there is no need for proprietary RAW formats.  I’d be happy to debate the technical aspects of those statements – but not here.  However, note also that the current TIFF standard is 6.0 published in 1992, and I’ve noted that TIFF is still a popular image file format.  The basic JPEG compression standard (T.81) was also published in 1992.  There has been no reason to revise those standards because they got these fundamentals right.  They have withstood the test of time, as has DNG.

My basic gripe, however, was not addressed in any of the responses.  Proprietary RAW formats is one tool that this industry uses to stifle independent software development.  The other issue I mentioned is that manufacturers don’t publish their WiFi, BT and even USB APIs (Application Programming Interface) is even worse.  (As I pointed out, Sony used to do that, but they stopped – apparently to protect their in-house software.)  I beg you to consider my comparison to the open developer networks supported by Apple and Android.  Photographers could have substantially better software if this industry would adopt open standards.

If I’m right, then the photographic community is missing out on a plethora of innovative software that an independent developer community could provide.  Moreover, that developer community already exists!  We only need to turn them loose.

 

JSS

• This reply was modified 4 years, 6 months ago by John Sadowsky.
• This reply was modified 4 years, 6 months ago by John Sadowsky.

[John Sadowsky]

Re: Lr and C1

Reply #7 on: October 29, 2019 at 10:01 pm

Javier Santa wrote:

I have forgotten about DNG a long time ago for one main reason: You can convert a RAW to DNG but not the opposite. Maybe things have changed since then.

In Lr Preferences under the File Handling tab, there is a check box “Embed Original RAW File”.  So, you actually can go back to your RAW format from DNG.  I don’t recommend that, because I don’t believe anything is lost by DNG conversion.  You would be right to point out that it is silly to double your storage in that way.  I’m not concerned about DNG going away because it is an open standard.  Even if Adobe goes under, someone will open that standard document and write software for it.  You can’t say that for proprietary file formats because they are not open standards.  My point is that we are always better off with an open standard.

 

 

JSS

• This reply was modified 4 years, 6 months ago by John Sadowsky.

[John Sadowsky Participant Posts: 169]

Re: Yosemmite Marching Band

Reply #8 on: October 29, 2019 at 8:59 am

Next up!  Rappers if the Everglades?

JSS

[John Sadowsky Participant Posts: 169]

Re: Lr and C1

Reply #9 on: October 28, 2019 at 10:10 am

Kevin Raber wrote:

Frankly, I wouldn’t convert to DNG unless I’m years into my workflow.  Many RAW files contain references to sensor information like dead pixels or columns etc.  That’s not always converted over in DNG. Most RAW formats will be supported for a long time.  DNG was originally designed to be an archival format in case a camera manufacturer went out of business or a RAW processor couldn’t read certain older files.  Converting to DNG is an extra step in a workflow that may not offer any benefit.

Kevin, with substantial due respect, I think you missed a key point here.  I apologize in advance for my rant here.

First, there is no real reason for camera manufacturers not to adopt DNG.  If there is nothing a proprietary RAW file that DNG can’t do.  DNG has two ways to support damaged pixels, for example.  The only processing of data that is done in conversion to DNG is linearization and black level subtraction.  (There is not demosaicing, etc.)  Those things are corrections for camera imperfection, as calibrated at manufacture time.  They are future proof, so there is no reason not to do that once for archival storage.  That’s the whole concept of the “digital negative.”  On the other hand, there is a huge advantage to having one common RAW format.

In my opinion, and one can certainly disagree with my rant, the main reason of proprietary RAW files is that it is one of the tools the industry uses to shut down 3rd party software development.  The other is that they don’t publish APIs (Application Programming Interfaces).

They want to keep you in their “ecosystem.”  Software manufacturers must pay a royalty and sign a non-disclosure agreement in order to get details of RAW format specification.  That makes it next to impossible for the rather large freelance and small company software developer community to develop new apps.   (Check out the audience size at Apple’s WWDC.)  Almost all camera companies have their own post processing software, but have you ever seen a positive review of, say, Sony’s Imaging Edge, or Nikon’s Capture software?   C1 is the exception.  It was originally developed for Phase One cameras – and then expanded to support other formats.  But seriously, how does Imaging Edge stack up to C1?  I don’t see too many sites like PhotoPxl doing videos on Imaging Edge.  If the industry would adopt DNG for raw capture, and again there is no reason not to, like some enlightened companies (Leica, Pentax) have, then anybody’s software would run on any camera’s RAW files.  No, we can’t have that!  Proprietary RAW files are nothing more than a log they throw in road of progress.

Compare that to the Apple and Android model – which is to make all the APIs open and well supported, and turn the developer community loose to develop apps.  It is next to impossible to do this in digital photography, and photographers suffer because of it.

The other part of this story is that camera manufacturers generally don’t support open APIs for WiFi, BT or even USB.  If we had access to those APIs, the software developer community could do some really cool stuff.  We just need to set them loose.  For example, why can’t we do focus stacks on the later Sony cameras (other than using tethered – yuk – Imaging Edge)?  It could be done easily with an iPhone app.  In fact, Sony used to publish their WiFi APIs, but they stopped doing that a few years ago.  Why? – to protect their in house software? – which … – well this is probably a good place to end my rant.

 

JSS

[John Sadowsky Participant Posts: 169]

Re: Lr and C1

Reply #10 on: October 25, 2019 at 7:08 pm

Jeff Schewe wrote:

C1 (Phase One) doesn’t really fully support DNG…

I’m very impressed with the videos on Capture One that Kevin has provided.  I’m starting to have concerns about Adobe – I think Lr CC is a disaster.  The videos are compelling enough to make me seriously consider dumping the Adobe CC Photographer $10/month plan.

However, I’ve converted most of my RAW files to DNG, so not fully supporting DNG is a problem.  DNG converted files contain linearized raw data in a 16 bit fixed point format.  From the comments, it appears that C1 works fine with those file.  However, …

The likely reason that C1 doesn’t recognize the HDR-DNG files is that those files store data in a 16 bit floating point format.  Those files can have a 30 stop dynamic range.  I personally believe that the HDR-DNG approach is one of the two “right” ways to do HDR merging of multiple exposures.  The other “right” way is just layers in photoshop.  HDR tone mapping is nothing but ordinary post processing – with lots of shadow pushing.  Give me Lr HDR merge, and let me do the tone mapping!  By the way, you can HDR merge images of the same exposure in order to reduce noise.

By the way, it doesn’t have to be a DNG.  The TIFF standard supports floating point data formats.  Almost all image files (proprietary RAW, DNG, JPEG, …) are based on the TIFF standard.  So, floating point format is not an Adobe thing.

So this is a problem for C1.  It should do an HDR merge, and it should be able to work with floating point data as in HDR-DNG or otherwise.

JSS

[John Sadowsky]

Re: Micro 4/3

Reply #11 on: October 22, 2019 at 10:19 am

Brad Smith wrote:

Here’s a solution for you…..rent a big, long, heavy prime for your Sony and simultaneously rent a micro 4/3 body and the Oly 300. Shoot with both and carefully look at the files at printing size.  There is a lot of money at stake in your decision, so this would, I think, be money well spent.

Great advice!  I will definitely rent before spending $12,000.  Still, for wildlife photography I am concerned about lens speed.  I had a Nikon 70-300 mm on my D700, but at 300 mm its was f5.6.  It was great for parties with a powerful shoe flash, but I found it to be too slow for outdoor shooting at 300 mm  except in the brightest daylight, and not really long enough for a lot of wildlife.  No doubt about it – wildlife is an expensive hobby.

JSS

• This reply was modified 4 years, 6 months ago by John Sadowsky.

[Author]

Posts