Does pixel-shift increase resolution?

Viewing 15 posts - 1 through 15 (of 15 total)
  • Author
    Topic: Does pixel-shift increase resolution? Read 733 Times
  • Mike Nelson Pedde
    Mike Nelson Pedde
    Participant
    Posts: 263
    Cameras, Lenses and Shooting Gear
    on: February 15, 2020 at 5:50 pm

    A well-presented discussion from Jim Kasson:

    https://blog.kasson.com/the-last-word/does-pixel-shift-increase-resolution/

    _____
    Mike Nelson Pedde
    Victoria, BC
    https://www.wolfnowl.com/

    John Sadowsky
    John Sadowsky
    Participant
    Posts: 69
    Re: Does pixel-shift increase resolution?
    Reply #1 on: February 21, 2020 at 5:51 pm

    I would expect the practical issues would overwhelm the 16-way pixel shift, starting with the fact that you have to have a rock-solid tripod and the high-resolution glass.  And what about shutter shake?  I’ve tried some 4-way pixel shift on my a7RM3 – I really couldn’t see much difference even under close pixel peeping.  Compare pixel-shift to HDR merging, or even just image stack averaging of the same exposure for noise reduction.  Layer-alignment in Photoshop (and other software) is quite accurate.  But you can’t use that with pixel-shift.  Pick your poison – sharpness or noise?  Pixel-shift is limited to indoor studio photography, for studios located far from highways, rail tracks, or anything else that might produce a micro-shake.

    I would gladly swap pixel-shift for focus stacks (as in the Nikons).  Why doesn’t Sony give us something we can actually useful?  (Not to mention that the Sony software used to merge the pixel-shift images is pathetic, IMHO.)

    Having said that, as a signal processing engineer, I was not at all satisfied with that article.

    The underlying assumption is that pixels have a 100% fill factor, meaning 100% of the pixel area is active in light detecting.  That just can’t be true.  The surface area of a pixel includes the light-sensing photodiode and several other components including at least 4 CMOS transistors.  Modern sensors do have micro-lenses on top of the sensor pixels, and BSI moves metal layers to the dark side of the sensor.  Those things do improve fill factor, but it is still a pretty big stretch of the imagination to think they achieved close to 100% fill.

    If it were true (that these pixels have 100% area efficiency), then it is almost immediately obvious 16-way pixel-shift would have no resolution advantage over 4-way pixel-shift.  The resolution would be determined by the “sampling aperture” of the square pixel area.  Extra 1/2 pixel offsets of those squares don’t add anything to resolution over 4-way pixel-shift.  4-way still makes sense as it eliminates the need for interpolation in demosaicing interpolation.

    I’d be happy to back up what I just said with more details from Fourier transform and Shannon-Nyquist sampling theory.  Here it is in a nutshell.  MTF = the Fourier transform of the PSF = point spread function.  He uses the MTF because he can combine the lens MTF and the sampling aperture MTF by multiplication (as opposed to the convolution of PSFs in the spatial domain).  But who thinks in terms of spatial frequency?  He should have at least press the button on his Matlab software (or whatever he’s using) to transform back to the spatial domain to show us PSF.  Then we could actually measure the parameter that photographers actually use to quantify resolution = the circle-of-confusion = the diameter of the PSF.  MTF – WTF?

     

     

    JSS

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 40
    Re: Does pixel-shift increase resolution?
    Reply #2 on: February 24, 2020 at 3:24 pm

    John, Brandon Dube, Jack Hogan, and I have all done modeling/real-world comparisons that indicate that the effective fill factors of modern BSI sensors with conventional (GFX 100 and a7RIV-ish) microlenses is near to 100%. The GFX 50S and GFX 50R is an outlier, having an effective fill factor of between 50% (Jack’s and my measurements) and 76% (Brandon’s measurements).

    You seem to object to my presenting the results in the frequency domain. There is a long tradition of measuring resolution in the frequency domain. MTF curves are a large part of lens specifications, and are used in camera and lens tests. SQF, another long-established resolution metric, is based on frequency-domain measurements.

    These images were created for another purpose, but If you want to take a look at what I call combined point spread functions, there are a few here:

     

    https://blog.kasson.com/the-last-word/optical-low-pass-filters-and-high-resolution-cameras/

    Jim

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 40
    Re: Does pixel-shift increase resolution?
    Reply #3 on: February 24, 2020 at 4:12 pm

    If it were true (that these pixels have 100% area efficiency), then it is almost immediately obvious 16-way pixel-shift would have no resolution advantage over 4-way pixel-shift.  The resolution would be determined by the “sampling aperture” of the square pixel area.  Extra 1/2 pixel offsets of those squares don’t add anything to resolution over 4-way pixel-shift.

    Yep. MTF = abs(fft(psf)), and, extending the concept to the what I call the combined spread function, MTF = abs(fft(csf)). And that means something to you and me, but not to a lot of people reading my blog. That’s why I presented the material the way I did.

     

    Jim

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 40
    Re: Does pixel-shift increase resolution?
    Reply #4 on: February 24, 2020 at 4:15 pm

    MTF = the Fourier transform of the PSF = point spread function

    It’s the magnitude of the Fourier transform of the PSF. The MTF is not complex.

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 40
    Re: Does pixel-shift increase resolution?
    Reply #5 on: February 25, 2020 at 11:17 am

    MTF = the Fourier transform of the PSF = point spread function

    It’s the magnitude of the Fourier transform of the PSF. The MTF is not complex.

    I see I was unclear. The MTF, being the magnitude of the Fourier transform of the PSF, is a two-dimensional matrix of real (not complex) numbers. But what is usually presented as MTF is a one-dimensional slice through that 2D matrix. The slice direction is usually horizontal or vertical for camera-sensor-based testing, and sagittal and tangential for lens testing on an optical bench.

    John Sadowsky
    John Sadowsky
    Participant
    Posts: 69
    Re: Does pixel-shift increase resolution?
    Reply #6 on: February 26, 2020 at 12:28 am

    Jim, I’m still not buying a 100% fill factor. I don’t think you can make such a claim without access to proprietary manufacturing process and design data. Nonetheless, it seems to defy physics. As with all features of semiconductor manufacturing, microlenses are masked and deposited on the sensor chip. It is highly probable that these lenses have circular symmetry, as sharp corners would be challenging to etch or deposit on that scale. (There are images online of microlenses – all the ones I found are circular.)  A 100% efficient circular microlens inside of a square pixel is only 78% of the pixel area.  We’re not talking about Zeiss optics here.  There are going to be optical compromises.

    OTF = FT of PSF, MTF = |OTF|^2 – my bad.

    Yes, I do believe your use of the MFT in your blog confuses the issue.  As I stated, photographers do have an operational definition of resolution, and it is not pixel count.  It is the spatial domain circle-of-confusion (CoC), which can be measured as the RMS radius of the PSF.

    I mostly agree with your conclusions. It is the combined PSF of the lens and the sampling-aperture (100% ideal square fill or otherwise) that limits the resolution, not the sampling density (megapixels).  Still, I found your description of pixel-shift as a halving of pixel pitch with a 400% fill to be unnecessary and confusing.  After all, the pixel doesn’t change size.  Pixel-shift simply increases the sampling density, which expands the area of alias-free spatial frequencies.  You only needed one MFT chart.  Then simply mark on that chart where then alias cutoff point is for the various pixel-shift sampling densities.  With that, I’ll concede that the frequency domain may the right place to explain the aliasing, but not the resolution.  Either way, you’re still assuming that your readers understand something about Nyquist-Shannon sampling theory.

    Notice that a smaller fill factor implies better resolution. (The reason we want a high fill factor is to collect more photons, which improves SNR.)  That’s likely going to be counter-intuitive for most photographers.  Since a more compact PSF implies a broader MFT, the value of pixel-shift is enhanced for lower (more realistic) fill factors.  That’s the point I was trying to make when I challenged the 100% fill factor.

    JSS

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 40
    Re: Does pixel-shift increase resolution?
    Reply #7 on: February 26, 2020 at 5:06 pm

    Yes, I do believe your use of the MFT in your blog confuses the issue.  As I stated, photographers do have an operational definition of resolution, and it is not pixel count.  It is the spatial domain circle-of-confusion (CoC), which can be measured as the RMS radius of the PSF.

    Tell that to Zeiss:

    https://www.zeiss.com/content/dam/camera-lenses/files/service/download-center/datasheets/otus-lenses/datasheet-zeiss-otus-1485.pdf

    Or Nikon:

    https://www.nikonusa.com/en/learn-and-explore/a/products-and-innovation/what-is-a-lens-mtf-chart-how-do-i-read-it.html

    Or LensRentals:

    https://www.lensrentals.com/blog/2019/04/just-mtf-charts-sony-fe-mount-prime-lenses/

     

    Or Rodenstock:

    https://www.rodenstock-photo.com/Archiv/e_Rodenstock_Analog_Lenses_27-42__8226.pdf

    or Schneider:

    https://captureintegration.com/schneider-ls-lens-mtf-charts-for-phase-one-mamiya/

    Or Canon:

    https://www.usa.canon.com/internet/portal/us/home/learn/education/topics/article/2019/april/Reading-and-Understanding-Lens-MTF-Charts/Reading-and-Understanding-Lens-MTF-Charts

    Or Imatest:

    https://www.imatest.com/products/imatest-master/

    https://www.imatest.com/docs/sharpness/

    CoC diameters are only useful if the PSF is roughly circular, and lens aberrations refuse to steadfastly toe that line. That’s why the sagittal and tangential MTFs differ. And there is much confusion on computing the diameter of PSFs that aren’t pillboxes; for example, the usual approximated  diameter of the Airy disk is twice the radius of the first zero, which is unduly pessimistic.

     

    Jim

     

    John Sadowsky
    John Sadowsky
    Participant
    Posts: 69
    Re: Does pixel-shift increase resolution?
    Reply #8 on: February 26, 2020 at 9:47 pm

    OK – whatever … I hope you have a nice life.

    JSS

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 40
    Re: Does pixel-shift increase resolution?
    Reply #9 on: February 29, 2020 at 1:10 pm

    Notice that a smaller fill factor implies better resolution. (The reason we want a high fill factor is to collect more photons, which improves SNR.)

    Of course, you’re right about the resolution. A canonically ideal sampler is a point sampler, which has (a Dirac delta-function-ish) 0% fill factor, and the flattest MTF curve. However, especially since having long experience with the GFX 50S and GFX 50R with their 50% fill factors, I disagree that the only reason for wanting a high fill factor is to improve SNR. A fill factor of nearly 100% is useful for reducing aliasing, as I’ve shown here:

    https://blog.kasson.com/the-last-word/balancing-real-and-fake-detail-part-2/

     

    Ron Gentry
    Ron Gentry
    Participant
    Posts: 4
    Re: Does pixel-shift increase resolution?
    Reply #10 on: March 2, 2020 at 7:54 pm

    As a scientist, I find all this theoretical debate very interesting.  However, on September 24, 2019 I posted a real-world test on the PhotoPXL website (Reply #8245).  It was admittedly done under ideal conditions, but it showed a definite improvement in resolution for the 16-frame pixel-shift image over the 4-frame version.  I will gladly be taking advantage of this capability whenever conditions seem appropriate.

    Ron Gentry

    John Sadowsky
    John Sadowsky
    Participant
    Posts: 69
    Re: Does pixel-shift increase resolution?
    Reply #11 on: March 3, 2020 at 12:32 pm

    As a scientist, I find all this theoretical debate very interesting.  … I will gladly be taking advantage of this capability whenever conditions seem appropriate.

    If it works for you – that’s all that really matters!

    What kind of photography are you using pixel-shift for?  Studio work?

    I tried 4-way with my A7Riii on a mountain landscape circa fall 2018.  It was early morning with very still air, and I weighted down my tripod for extra stability.  I could see a difference with extreme pixel-peeping.  I used Sony’s Imaging Edge software, which I was not impressed with.  At the time there was at least one video online demonstrating bugs in Imaging Edge pixel-shift merge.  Is it any better today?  Are there software alternatives?

    JSS

    Ron Gentry
    Ron Gentry
    Participant
    Posts: 4
    Re: Does pixel-shift increase resolution?
    Reply #12 on: March 3, 2020 at 1:39 pm

    So far I’ve only used it in the studio.  Imaging Edge worked well for me.  There is at least one other software choice for merging pixel-shift images:  https://www.fastrawviewer.com/PixelShift2DNG, but I haven’t yet tried it.

    Ron Gentry

    John Sadowsky
    John Sadowsky
    Participant
    Posts: 69
    Re: Does pixel-shift increase resolution?
    Reply #13 on: March 3, 2020 at 9:18 pm

    Thanks, Ron – I’ll have to give pixel-shift a second shot.  I do like to see every pine needle on every tree on that mountainside a half-mile away.  🙂

    JSS

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 40
    Re: Does pixel-shift increase resolution?
    Reply #14 on: March 5, 2020 at 10:34 am

    I tried 4-way with my A7Riii on a mountain landscape circa fall 2018.  It was early morning with very still air, and I weighted down my tripod for extra stability.  I could see a difference with extreme pixel-peeping.  I used Sony’s Imaging Edge software, which I was not impressed with.  At the time there was at least one video online demonstrating bugs in Imaging Edge pixel-shift merge.  Is it any better today?  Are there software alternatives?

    Yes. https://www.fastrawviewer.com/PixelShift2DNG

    IE is now a bit better than it was.

     

    Jim

Viewing 15 posts - 1 through 15 (of 15 total)
  • You must be logged in to reply to this topic.