Sony A7R3 shoot compressed or uncompressed?

Viewing 15 posts - 1 through 15 (of 18 total)
  • Author
    Topic: Sony A7R3 shoot compressed or uncompressed? Read 2554 Times
  • Benoit Malphettes
    Benoit Malphettes
    Participant
    Posts: 10
    Cameras, Lenses and Shooting Gear
    on: September 3, 2019 at 6:50 pm

    Does it make a difference? and if so, where would I see it?

    Thanks in advance for enlightened opinions!

    Mike Nelson Pedde
    Mike Nelson Pedde
    Participant
    Posts: 509
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #1 on: September 3, 2019 at 8:43 pm

    Jim Kasson (who’s also present here) could probably give you an excellent answer to that question. You can find his articles on the A7RIII here: https://blog.kasson.com/category/a7riii/

     

    _____
    Mike Nelson Pedde
    Victoria, BC
    https://www.wolfnowl.com/

    Benoit Malphettes
    Benoit Malphettes
    Participant
    Posts: 10
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #2 on: September 6, 2019 at 12:28 am

    Thanks Mike, forgot about the invaluable blogs of Jim Kasson

     

    Mike Nelson Pedde
    Mike Nelson Pedde
    Participant
    Posts: 509
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #3 on: September 6, 2019 at 1:36 pm

    You’re most welcome! Jim’s posts are often way over my head, but I have The Last Word in my RSS because I always get something out of them.

    _____
    Mike Nelson Pedde
    Victoria, BC
    https://www.wolfnowl.com/

    Brian Sweeney
    Brian Sweeney
    Participant
    Posts: 9
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #4 on: September 14, 2019 at 10:03 pm

    https://www.colbybrownphotography.com/getting-the-most-out-of-your-sony-a7r-iii/

     

    According to the above article, and “some quick web-surfing”, Sony offers uncompressed RAW and lossy-compressed RAW. The lossy-compressed RAW can cause problems in certain instances. I’m surprised that Sony does not have “Lossless Compressed”. The original series only offered lossy-compressed RAW, adding uncompressed RAW was done after many users complained. The problem with the lossy-compressed RAW format is that it is “Chaotic”, you don’t know what will be lost as it depends on the structure of the image. These algorithms tend to magnify the effects of noise on the image, noise is high-frequency and lossy algorithms do poorly with high frequency.

     

    Unless Sony offers a Lossless-compressed mode, shoot uncompressed for anything important.

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 70
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #5 on: September 14, 2019 at 11:42 pm

    https://www.colbybrownphotography.com/getting-the-most-out-of-your-sony-a7r-iii/

    According to the above article, and “some quick web-surfing”, Sony offers uncompressed RAW and lossy-compressed RAW. The lossy-compressed RAW can cause problems in certain instances. I’m surprised that Sony does not have “Lossless Compressed”. The original series only offered lossy-compressed RAW, adding uncompressed RAW was done after many users complained. The problem with the lossy-compressed RAW format is that it is “Chaotic”, you don’t know what will be lost as it depends on the structure of the image.

    Once you understand how the artifacts manifest, it’s pretty easy to look at a scene and decide if you’ll need lossless or not. 95% or the time I don’t use it.

    These algorithms tend to magnify the effects of noise on the image, noise is high-frequency and lossy algorithms do poorly with high frequency.

     

    In most circumstances, noise tends to cover up the artifacts of craw.

    Unless Sony offers a Lossless-compressed mode, shoot uncompressed for anything important.

    I think that’s a huge overreaction. I don’t hesitate to use craw in many circumstances where quality if paramount.

     

    Jim

    Brian Sweeney
    Brian Sweeney
    Participant
    Posts: 9
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #6 on: September 15, 2019 at 6:53 am

    https://www.colbybrownphotography.com/getting-the-most-out-of-your-sony-a7r-iii/

    According to the above article, and “some quick web-surfing”, Sony offers uncompressed RAW and lossy-compressed RAW. The lossy-compressed RAW can cause problems in certain instances. I’m surprised that Sony does not have “Lossless Compressed”. The original series only offered lossy-compressed RAW, adding uncompressed RAW was done after many users complained. The problem with the lossy-compressed RAW format is that it is “Chaotic”, you don’t know what will be lost as it depends on the structure of the image. With the processing power and buffer storage available, lossless compression is not hard to implement and will not produce the artifacts of the Sony algorithm. With the processing power and buffer memory available, lossless compression schemes are available that avoid the artifacts produced by the Sony algorithm. The running difference algorithm requires very little processing power and buffer space, does well with oversampled data. Given the pixel count of the Sony sensor and resolution of the lenses used on it, this algorithm would do well.

    Once you understand how the artifacts manifest, it’s pretty easy to look at a scene and decide if you’ll need lossless or not. 95% or the time I don’t use it.

    These algorithms tend to magnify the effects of noise on the image, noise is high-frequency and lossy algorithms do poorly with high frequency.

    In most circumstances, noise tends to cover up the artifacts of craw.

    Unless Sony offers a Lossless-compressed mode, shoot uncompressed for anything important.

    I think that’s a huge overreaction. I don’t hesitate to use craw in many circumstances where quality if paramount.

    Jim

    Most people do not understand how the artifacts manifest themselves, and don’t have the time to assess if a particular scene will cause the issue. Sony is only attaining a 2:1 compression with a chaotic process. They could use a simple running difference scheme with an escape value to store full-values if the 8-bit difference had been exceeded. They could use Huffman code to store the frame, similar savings. With the processing power and buffer storage available, there are algorithms that could be implemented that do not produce the artifacts of the Sony algorithm. The running difference algorithm would do well given that the sensor is oversampling the resolution of most of the lenses used with it.

     

    Brian Sweeney
    Brian Sweeney
    Participant
    Posts: 9
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #7 on: September 15, 2019 at 9:37 am

    Some web-surfing: One reference that the compressed mode raw of the Sony A7RIII uses 12-bit pixels when shooting burst mode and long exposures. There was a link to a Sony statement about the switch to 12-bit, but Sony seems to have moved or removed it- link was dead. The first step of the Sony compression algorithm on the older series cameras (at least what has been documented through users online) is to reduce the 14-bit raw value to 12-bits. If you require pulling out details from shadows, it is best to have the full 14-bits available. The other effect this has is to lose fine detail in the image: 4 intensity values are binned into one value. This is like making a contour map of the (intensity/ 4). Noise in the image can move pixels near the edge of a contour line to the next bin. To see this one would implement the compression algorithm and run it on an uncompressed image and store the difference frame.

     

    If you like the results with compressed mode- use it. If you cannot see a difference, it works for you. If you do a lot of post-processing, uncompressed raw provides more information to work with. Memory cards are huge these days. You can always shoot uncompressed, and “ZIP” the raw files on your computer to store them when not in use or for archiving. Algorithms such as ZIP will compress an uncompressed raw image without decimating the data.

    Brian Sweeney
    Brian Sweeney
    Participant
    Posts: 9
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #8 on: September 15, 2019 at 1:03 pm

    https://www.ingentaconnect.com/contentone/ist/ei/2016/00002016/00000002/art00020?crawler=true

     

    A paper on how ARW v2 (Alpha Raw) works, and a program available to correct artifacts. Worth reading just for the reference to Blondie. But anyone with the album would realize the parallel lines were B&W.

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 70
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #9 on: September 15, 2019 at 1:49 pm

    https://www.colbybrownphotography.com/getting-the-most-out-of-your-sony-a7r-iii/

    According to the above article, and “some quick web-surfing”, Sony offers uncompressed RAW and lossy-compressed RAW. The lossy-compressed RAW can cause problems in certain instances. I’m surprised that Sony does not have “Lossless Compressed”. The original series only offered lossy-compressed RAW, adding uncompressed RAW was done after many users complained. The problem with the lossy-compressed RAW format is that it is “Chaotic”, you don’t know what will be lost as it depends on the structure of the image. With the processing power and buffer storage available, lossless compression is not hard to implement and will not produce the artifacts of the Sony algorithm. With the processing power and buffer memory available, lossless compression schemes are available that avoid the artifacts produced by the Sony algorithm. The running difference algorithm requires very little processing power and buffer space, does well with oversampled data. Given the pixel count of the Sony sensor and resolution of the lenses used on it, this algorithm would do well.

    Once you understand how the artifacts manifest, it’s pretty easy to look at a scene and decide if you’ll need lossless or not. 95% or the time I don’t use it.

    These algorithms tend to magnify the effects of noise on the image, noise is high-frequency and lossy algorithms do poorly with high frequency.

    In most circumstances, noise tends to cover up the artifacts of craw.

    Unless Sony offers a Lossless-compressed mode, shoot uncompressed for anything important.

    I think that’s a huge overreaction. I don’t hesitate to use craw in many circumstances where quality if paramount.

    Jim

    Most people do not understand how the artifacts manifest themselves, and don’t have the time to assess if a particular scene will cause the issue. Sony is only attaining a 2:1 compression with a chaotic process. They could use a simple running difference scheme with an escape value to store full-values if the 8-bit difference had been exceeded. They could use Huffman code to store the frame, similar savings. With the processing power and buffer storage available, there are algorithms that could be implemented that do not produce the artifacts of the Sony algorithm. The running difference algorithm would do well given that the sensor is oversampling the resolution of most of the lenses used with it.

    I was not commenting on whether Sony should offer lossless compression. My views on that issue are well known, and not too dissimilar from your own. But Sony has had ample time to hear many people express those sentiments, and they have not listened. The cameras are what they are. So my comments were directed to people who want to get the best out of the cameras they currently have, not to those who wish to use some camera that is not available in the marketplace.

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 70
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #10 on: September 15, 2019 at 3:11 pm

    Some web-surfing: One reference that the compressed mode raw of the Sony A7RIII uses 12-bit pixels when shooting burst mode and long exposures. There was a link to a Sony statement about the switch to 12-bit, but Sony seems to have moved or removed it- link was dead. The first step of the Sony compression algorithm on the older series cameras (at least what has been documented through users online) is to reduce the 14-bit raw value to 12-bits.

     

    13 bits, at the low end of the tone curve.

    Here’s the rough outlines of the algorithm:

    The tone curve applied to the 13-bit linear data is a linear interpolation between a curve defined by these input points: 0 1000 1600 2854 4058 8602, and these output points: 0 1000 1300 1613 1763 2047.

     

    The delta modulation extracts successive groups of 32 pixels within a single row. Because of the RGGB Bayer color filter array, that gives 16 green pixels in all rows, 16 red pixels in the odd rows, and 16 blue pixels in the even rows. Odd and even are defined assuming that the first row is row 1.

     

    As an example, an odd row would start out RGRGRGRGRG… and an even row would begin GBGBGBGBGB…

     

    For each group of 16 pixels, the maximum and minimum 11-bit values are found, and the indices of those values recorded as two four-bit numbers. The minimum is subtracted from the maximum to give a number that I call the span.

     

    Then a value called the step is calculated. If the span is equal to or less than 128, the step is 1. If the span is more than 128 and equal to or less than 256, the step is 2. . If the span is more than 256 and equal to or less than 512, the step is 4. If the span is more than 512 and equal to or less than 1024, the step is 8. Otherwise the step is 16.

     

    For each of the remaining 14 pixels, the minimum value is subtracted, and the result divided by the step size, truncated (or rounded; that’s what I did) and encoded as a 7-bit number which is stored. Thus we have 16 pixels encoded in 16 bytes (11*2 + 4*2 + 7*14, or 128 bits). The algorithm proceeds across all the columns in a row, then moves to the next row down, does it again, until it runs out of rows. That’s what’s in the raw file.

     

    The process is reversed in the raw converter. The original 11-tone compressed values are recovered by inverting the delta modulation algorithm – I leave that as an exercise for the interested student – and the tone curve is removed by a linear interpolation between a curve defined by these input points: 0 : 0 1000 1300 1613 1763 2047, and these output points: 0 1000 1600 2854 4058 8602.

     

    Brian Sweeney
    Brian Sweeney
    Participant
    Posts: 9
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #11 on: September 15, 2019 at 4:44 pm

    Thankyou for the explaining the new algorithm. My background with digital imaging was more towards machine processing of data. I have a bias towards getting every last bit of information off the sensor, then processing it. The types of artifacts that are produced by lossy compression schemes would ruin  image data for those purposes. Lossless compression schemes can be implemented with about the same compression efficiency as that of the Sony algorithm. It strikes me that they came up with what they think is a good algorithm, and decided to stick with it.

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 70
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #12 on: September 15, 2019 at 5:36 pm

    Thankyou for the explaining the new algorithm.

     

    Just to be clear, this is the same algorithm the a7x cameras have used since day one.

    Jim Kasson
    Jim Kasson
    Participant
    Posts: 70
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #13 on: September 15, 2019 at 5:44 pm

    To see this one would implement the compression algorithm and run it on an uncompressed image and store the difference frame.

    The way you calculate the difference frame is important.

     

    https://blog.kasson.com/the-last-word/a-tough-test-of-sony-raw-compression/

    Brian Sweeney
    Brian Sweeney
    Participant
    Posts: 9
    Re: Sony A7R3 shoot compressed or uncompressed?
    Reply #14 on: September 15, 2019 at 6:31 pm

    To see this one would implement the compression algorithm and run it on an uncompressed image and store the difference frame.

    The way you calculate the difference frame is important.

    https://blog.kasson.com/the-last-word/a-tough-test-of-sony-raw-compression/

    What I mean by this: Implement the compression algorithm in software to post-process an uncompressed raw image. Compress the image, decompress it and then subtract from the original uncompressed image. That will show what is being lost by the compression algorithm.

     

    For doing a running sum difference frame on a Mosaic Sensor: do a running difference of elements of the same color on the same row. Store the differences as 8-bit signed values from -127 through +126, reserve 127 as an escape value to store the actual value for when the difference exceeds -127:126. If the encoded difference frame exceeds the original image size, switch to storing an uncompressed frame. Computationally expedient, best to store the original image if the running difference takes too much space. The point response of the sensor+lens is important, tends to work best on oversampled data. Worked 30 years ago.

    My Micro-Nikkor AFD 200/4 is one of the sharpest lenses that I own. Online reports state that it is well matched to a 16MPixel sensor. I wonder how many Sony A7RIII images are taken using lenses that outperform the sensor.

Viewing 15 posts - 1 through 15 (of 18 total)
  • You must be logged in to reply to this topic.