'extra' lossless compression for camera raw images

Discussion in 'Digital Photography' started by Sachin Garg, Mar 18, 2008.

  1. Sachin Garg

    Sachin Garg Guest

    I just finished writing a small tool (sgraw) which can losslessly compress
    camera raw images, usually between 20-to-60%. (Regulars here might recall I
    have mentioned this 'possibility' earlier, I finally put in some hours to
    actually build a working demo :)

    http://www.sachingarg.com/compression/sgraw/

    That would be savings between 1GB to 3GB for every 5GB of data.

    Even when camera has already compressed the raw image (which isn't always
    the case), its still possible to squeeze out extra compression as cameras
    have to use simple algorithms for speed.

    Compression is totally lossless, we get back the exact file we start with
    (just like with zip), so both pixel information and meta-data are perfectly
    restored.

    This is just the first quick hack version (and supports all nikon and fuji
    cameras), so it obviously wouldn't make sense to actually use it for backups
    etc, but if there is enough interest I can build it further to support other
    raw formats, along with improved compression and speed.

    Your thoughts on this?

    Sachin Garg [India]
    www.sachingarg.com | www.imagecompression.info
     
    Sachin Garg, Mar 18, 2008
    #1
    1. Advertising

  2. Sachin Garg

    Tony Polson Guest

    "Sachin Garg" <> wrote:

    >
    >I just finished writing a small tool (sgraw) which can losslessly compress
    >camera raw images, usually between 20-to-60%. (Regulars here might recall I
    >have mentioned this 'possibility' earlier, I finally put in some hours to
    >actually build a working demo :)
    >
    > http://www.sachingarg.com/compression/sgraw/
    >
    >That would be savings between 1GB to 3GB for every 5GB of data.
    >
    >Even when camera has already compressed the raw image (which isn't always
    >the case), its still possible to squeeze out extra compression as cameras
    >have to use simple algorithms for speed.
    >
    >Compression is totally lossless, we get back the exact file we start with
    >(just like with zip), so both pixel information and meta-data are perfectly
    >restored.
    >
    >This is just the first quick hack version (and supports all nikon and fuji
    >cameras), so it obviously wouldn't make sense to actually use it for backups
    >etc, but if there is enough interest I can build it further to support other
    >raw formats, along with improved compression and speed.
    >
    >Your thoughts on this?



    You should patent it.
     
    Tony Polson, Mar 18, 2008
    #2
    1. Advertising

  3. Sachin Garg

    Mike Mills Guest

    "Sachin Garg" <> wrote in
    news:frp6tm$8dv$:

    >
    > I just finished writing a small tool (sgraw) which can losslessly
    > compress camera raw images, usually between 20-to-60%. (Regulars
    > here might recall I have mentioned this 'possibility' earlier, I
    > finally put in some hours to actually build a working demo :)
    >
    > http://www.sachingarg.com/compression/sgraw/
    >
    > That would be savings between 1GB to 3GB for every 5GB of data.
    >

    Do you have any experience with freeware tools [preferably
    commandline] which do Lossy Compression of jpegs?
    The tool by JDO works well but it is proprietary and not freeware.
    It can regularly wring out 30% savings in byte size and still retain
    the "viewability" as a jpg file. Of course quality will be lost
    depending on the quality of the input..
     
    Mike Mills, Mar 18, 2008
    #3
  4. Sachin Garg

    Ali Guest

    The RAW file is already pretty low in file size compared to the PSD and TIFF
    file size. Harddrive space is also pretty low in cost too. So, would I use
    it? Unfortunately no.




    "Sachin Garg" <> wrote in message
    news:frp6tm$8dv$...
    >
    > I just finished writing a small tool (sgraw) which can losslessly compress
    > camera raw images, usually between 20-to-60%. (Regulars here might recall
    > I have mentioned this 'possibility' earlier, I finally put in some hours
    > to actually build a working demo :)
    >
    > http://www.sachingarg.com/compression/sgraw/
    >
    > That would be savings between 1GB to 3GB for every 5GB of data.
    >
    > Even when camera has already compressed the raw image (which isn't always
    > the case), its still possible to squeeze out extra compression as cameras
    > have to use simple algorithms for speed.
    >
    > Compression is totally lossless, we get back the exact file we start with
    > (just like with zip), so both pixel information and meta-data are
    > perfectly restored.
    >
    > This is just the first quick hack version (and supports all nikon and fuji
    > cameras), so it obviously wouldn't make sense to actually use it for
    > backups etc, but if there is enough interest I can build it further to
    > support other raw formats, along with improved compression and speed.
    >
    > Your thoughts on this?
    >
    > Sachin Garg [India]
    > www.sachingarg.com | www.imagecompression.info
    >
    >
     
    Ali, Mar 18, 2008
    #4
  5. Sachin Garg

    Steve Guest

    On Wed, 19 Mar 2008 01:28:47 +0530, "Sachin Garg"
    <> wrote:

    >
    >I just finished writing a small tool (sgraw) which can losslessly compress
    >camera raw images, usually between 20-to-60%. (Regulars here might recall I
    >have mentioned this 'possibility' earlier, I finally put in some hours to
    >actually build a working demo :)
    >
    > http://www.sachingarg.com/compression/sgraw/
    >
    >That would be savings between 1GB to 3GB for every 5GB of data.
    >
    >Even when camera has already compressed the raw image (which isn't always
    >the case), its still possible to squeeze out extra compression as cameras
    >have to use simple algorithms for speed.


    How much more can you squeeze out of a compressed raw image? When I
    shoot raw, I use Nikon compressed raw. I just tried bzip2 on a few
    files. I get about 9% further compression. I don't know exactly what
    compression algorithm Nikon uses but I doubt it's much more
    complicated than a requantization of the data to reduce the bits per
    pixel. Nothing near as complicated as what bzip2 is doing.

    9% savings isn't worth the effort for me. But 20-60% definitely would
    be if you get that with compressed raw.

    Steve
     
    Steve, Mar 19, 2008
    #5
  6. Sachin Garg

    Alan Browne Guest

    Sachin Garg wrote:
    > I just finished writing a small tool (sgraw) which can losslessly compress
    > camera raw images, usually between 20-to-60%. (Regulars here might recall I
    > have mentioned this 'possibility' earlier, I finally put in some hours to
    > actually build a working demo :)
    >
    > http://www.sachingarg.com/compression/sgraw/
    >
    > That would be savings between 1GB to 3GB for every 5GB of data.
    >
    > Even when camera has already compressed the raw image (which isn't always
    > the case), its still possible to squeeze out extra compression as cameras
    > have to use simple algorithms for speed.
    >
    > Compression is totally lossless, we get back the exact file we start with
    > (just like with zip), so both pixel information and meta-data are perfectly
    > restored.
    >
    > This is just the first quick hack version (and supports all nikon and fuji
    > cameras), so it obviously wouldn't make sense to actually use it for backups
    > etc, but if there is enough interest I can build it further to support other
    > raw formats, along with improved compression and speed.
    >
    > Your thoughts on this?


    For GP use? No biggie. Disk space is cheap.

    If it can be reduced to a h/w implementation (by the camera co's) that
    is more energy efficient (and faster), then it would be a great thing to
    improve in camera storage.

    Once you offload from the camera (flash) to disk, the memory is not that
    important (to me). Disk space is cheap.

    Cheers,
    Alan

    --
    -- r.p.e.35mm user resource: http://www.aliasimages.com/rpe35mmur.htm
    -- r.p.d.slr-systems: http://www.aliasimages.com/rpdslrsysur.htm
    -- [SI] gallery & rulz: http://www.pbase.com/shootin
    -- e-meil: Remove FreeLunch.
     
    Alan Browne, Mar 19, 2008
    #6
  7. Sachin Garg

    Sachin Garg Guest

    "Tony Polson" <> wrote in message
    news:...
    > "Sachin Garg" <> wrote:
    >>
    >>I just finished writing a small tool (sgraw) which can losslessly compress
    >>camera raw images, usually between 20-to-60%. (Regulars here might recall
    >>I
    >>have mentioned this 'possibility' earlier, I finally put in some hours to
    >>actually build a working demo :)
    >>
    >> http://www.sachingarg.com/compression/sgraw/
    >>
    >>That would be savings between 1GB to 3GB for every 5GB of data.

    []
    > You should patent it.


    Thanks for that tip. Like all research my work is also based on hard work
    from many other researchers, I will explore if there are any patentable
    parts.

    Sachin Garg [India]
    www.sachingarg.com | www.imagecompression.info
     
    Sachin Garg, Mar 19, 2008
    #7
  8. Sachin Garg

    Sachin Garg Guest

    "Mike Mills" <> wrote in message
    news:Xns9A65A9516DCBDefficientcanadacom@194.177.96.78...
    > "Sachin Garg" <> wrote in
    > news:frp6tm$8dv$:
    >
    >>
    >> I just finished writing a small tool (sgraw) which can losslessly
    >> compress camera raw images, usually between 20-to-60%. (Regulars
    >> here might recall I have mentioned this 'possibility' earlier, I
    >> finally put in some hours to actually build a working demo :)
    >>
    >> http://www.sachingarg.com/compression/sgraw/
    >>
    >> That would be savings between 1GB to 3GB for every 5GB of data.
    >>

    > Do you have any experience with freeware tools [preferably
    > commandline] which do Lossy Compression of jpegs?


    If you want lossy compression on commandline, you can try the opensource
    options like "libjpeg" or "libpnm". Even if you don't want to play with
    sourcecode, you should be able to find its (free) executables for any
    popular platform.

    On the other hand, if you want further 'lossless' compression of Jpeg files,
    look for "jpegoptim" (also opensource). Many popular image editors like
    photoshop and irfanview etc also allow the jpegoptim optimizations in batch
    mode (if you dont want to use a commandline tool).

    > The tool by JDO works well but it is proprietary and not freeware.
    > It can regularly wring out 30% savings in byte size and still retain
    > the "viewability" as a jpg file. Of course quality will be lost
    > depending on the quality of the input..


    You probably already know that moderate jpeg loss is less of a problem when
    'viewing', but it hurts when editing or priniting after the fact. Dont do
    'too much' of it :)

    Sachin Garg [India]
    www.sachingarg.com | www.imagecompression.info
     
    Sachin Garg, Mar 19, 2008
    #8
  9. Sachin Garg

    Sachin Garg Guest

    "Ali" <> wrote in message
    news:...
    > "Sachin Garg" <> wrote in message
    > news:frp6tm$8dv$...
    >>
    >> I just finished writing a small tool (sgraw) which can losslessly
    >> compress camera raw images, usually between 20-to-60%. (Regulars here
    >> might recall I have mentioned this 'possibility' earlier, I finally put
    >> in some hours to actually build a working demo :)
    >>
    >> http://www.sachingarg.com/compression/sgraw/
    >>
    >> That would be savings between 1GB to 3GB for every 5GB of data.

    >
    > The RAW file is already pretty low in file size compared to the PSD and
    > TIFF file size.


    Yep, that is correct. Similar savings are 'technically possible' on TIFF and
    PSD files too, but that would be another story.

    > Harddrive space is also pretty low in cost too. So, would I use it?
    > Unfortunately no.


    ok

    Sachin Garg [India]
    www.sachingarg.com | www.imagecompression.info
     
    Sachin Garg, Mar 19, 2008
    #9
  10. Sachin Garg

    Sachin Garg Guest

    "Steve" <> wrote in message
    news:...
    > On Wed, 19 Mar 2008 01:28:47 +0530, "Sachin Garg"
    > <> wrote:
    >
    >>
    >>I just finished writing a small tool (sgraw) which can losslessly compress
    >>camera raw images, usually between 20-to-60%. (Regulars here might recall
    >>I
    >>have mentioned this 'possibility' earlier, I finally put in some hours to
    >>actually build a working demo :)
    >>
    >> http://www.sachingarg.com/compression/sgraw/
    >>
    >>That would be savings between 1GB to 3GB for every 5GB of data.
    >>
    >>Even when camera has already compressed the raw image (which isn't always
    >>the case), its still possible to squeeze out extra compression as cameras
    >>have to use simple algorithms for speed.

    >
    > How much more can you squeeze out of a compressed raw image? When I
    > shoot raw, I use Nikon compressed raw. I just tried bzip2 on a few
    > files. I get about 9% further compression. I don't know exactly what
    > compression algorithm Nikon uses but I doubt it's much more
    > complicated than a requantization of the data to reduce the bits per
    > pixel. Nothing near as complicated as what bzip2 is doing.


    While Nikon does uses the requantization you mention, they also compress
    those quantized pixels further using lossless encoding. And yes, its still a
    'simple' algorithm not as complicated as bzip2, but bzip2 type methods
    (based on bwt) can't perform good on 2D data like images.

    > 9% savings isn't worth the effort for me. But 20-60% definitely would
    > be if you get that with compressed raw.


    You should be able to get atleast 20-30% even on compressed raw files using
    sgraw.

    If you try the tool on your images, I would love to see what results you
    get. In case you get anything less than that it will mean I can further
    improve it for your camera's raw files.

    Sachin Garg [India]
    www.sachingarg.com | www.imagecompression.info
     
    Sachin Garg, Mar 19, 2008
    #10
  11. Sachin Garg

    Sachin Garg Guest

    "Alan Browne" <> wrote in message
    news:...
    > Sachin Garg wrote:
    >> I just finished writing a small tool (sgraw) which can losslessly
    >> compress camera raw images, usually between 20-to-60%. (Regulars here
    >> might recall I have mentioned this 'possibility' earlier, I finally put
    >> in some hours to actually build a working demo :)
    >>
    >> http://www.sachingarg.com/compression/sgraw/
    >>
    >> That would be savings between 1GB to 3GB for every 5GB of data.
    >>
    >> Even when camera has already compressed the raw image (which isn't always
    >> the case), its still possible to squeeze out extra compression as cameras
    >> have to use simple algorithms for speed.
    >>
    >> Compression is totally lossless, we get back the exact file we start with
    >> (just like with zip), so both pixel information and meta-data are
    >> perfectly restored.
    >>
    >> This is just the first quick hack version (and supports all nikon and
    >> fuji cameras), so it obviously wouldn't make sense to actually use it for
    >> backups etc, but if there is enough interest I can build it further to
    >> support other raw formats, along with improved compression and speed.
    >>
    >> Your thoughts on this?

    >
    > For GP use? No biggie. Disk space is cheap.
    >
    > If it can be reduced to a h/w implementation (by the camera co's) that is
    > more energy efficient (and faster), then it would be a great thing to
    > improve in camera storage.


    Yep, it will be a lot more useful if it can get in camera. I can only get
    the technology ready and hopefully make it usable atleast on desktops.

    Sachin Garg [India]
    www.sachingarg.com | www.imagecompression.info
     
    Sachin Garg, Mar 19, 2008
    #11
  12. Sachin Garg

    Steve Guest

    On Wed, 19 Mar 2008 11:35:55 +0530, "Sachin Garg"
    <> wrote:

    [...]
    >You should be able to get atleast 20-30% even on compressed raw files using
    >sgraw.
    >
    >If you try the tool on your images, I would love to see what results you
    >get. In case you get anything less than that it will mean I can further
    >improve it for your camera's raw files.


    That's what I got. On one D200 compressed RAW test file, sgraw
    compressed 23.5% and took 5 seconds to do it. bzip2 compressed 7.5%
    and took 3.2 seconds to do it, on the same computer. They both
    uncompressed losslessly. sgraw taking 2.235 seconds, bunzip2 1.973
    seconds to uncompress the same file.

    Details: file size = 8,242,489. sgraw compressed = 6,304,462.

    I think that's an admirable performance, keep up the good work. Even
    with cheap disk space, it's something I'd probably use when your
    release versions are out, for archiving images I know I'm not going to
    use for a while, maybe forever.

    Steve
     
    Steve, Mar 19, 2008
    #12
  13. Steve <> wrote:

    > How much more can you squeeze out of a compressed raw image? When I
    > shoot raw, I use Nikon compressed raw. I just tried bzip2 on a few
    > files. I get about 9% further compression.


    You most probably get more if you reorder the data stream in such
    a way that similar values are next to each other, especially if
    you just record differences. That however needs a more in-depth
    understanding of the file format than a 'simple' compressor like
    bzip2 has.

    -Wolfgang
     
    Wolfgang Weisselberg, Mar 19, 2008
    #13
  14. Tony Polson <> wrote:

    > You should patent it.


    Sorry, patenting ideas is patented.

    -Wolfgang
     
    Wolfgang Weisselberg, Mar 19, 2008
    #14
  15. Sachin Garg

    ray Guest

    On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:

    > I just finished writing a small tool (sgraw) which can losslessly
    > compress camera raw images, usually between 20-to-60%. (Regulars here
    > might recall I have mentioned this 'possibility' earlier, I finally put
    > in some hours to actually build a working demo :)
    >
    > http://www.sachingarg.com/compression/sgraw/
    >
    > That would be savings between 1GB to 3GB for every 5GB of data.
    >
    > Even when camera has already compressed the raw image (which isn't
    > always the case), its still possible to squeeze out extra compression as
    > cameras have to use simple algorithms for speed.
    >
    > Compression is totally lossless, we get back the exact file we start
    > with (just like with zip), so both pixel information and meta-data are
    > perfectly restored.
    >
    > This is just the first quick hack version (and supports all nikon and
    > fuji cameras), so it obviously wouldn't make sense to actually use it
    > for backups etc, but if there is enough interest I can build it further
    > to support other raw formats, along with improved compression and speed.
    >
    > Your thoughts on this?
    >
    > Sachin Garg [India]
    > www.sachingarg.com | www.imagecompression.info


    It seems totally superfluous since there are already dozens of
    compression programs available. What makes you think you've discovered
    some 'new and improved' algorithm?
     
    ray, Mar 19, 2008
    #15
  16. Sachin Garg

    Sachin Garg Guest

    "Steve" <> wrote in message
    news:...
    > On Wed, 19 Mar 2008 11:35:55 +0530, "Sachin Garg"
    > <> wrote:
    >
    > [...]
    >>You should be able to get atleast 20-30% even on compressed raw files
    >>using
    >>sgraw.
    >>
    >>If you try the tool on your images, I would love to see what results you
    >>get. In case you get anything less than that it will mean I can further
    >>improve it for your camera's raw files.

    >
    > That's what I got. On one D200 compressed RAW test file, sgraw
    > compressed 23.5% and took 5 seconds to do it. bzip2 compressed 7.5%
    > and took 3.2 seconds to do it, on the same computer. They both
    > uncompressed losslessly. sgraw taking 2.235 seconds, bunzip2 1.973
    > seconds to uncompress the same file.


    Great, sgraw speed can be improved considerably, I will work on it.

    > Details: file size = 8,242,489. sgraw compressed = 6,304,462.
    >
    > I think that's an admirable performance, keep up the good work.


    Thanks :)

    Sachin Garg [India]
    www.sachingarg.com | www.imagecompression.info
     
    Sachin Garg, Mar 19, 2008
    #16
  17. Sachin Garg

    Sachin Garg Guest

    "Wolfgang Weisselberg" <> wrote in message
    news:...
    > Steve <> wrote:
    >
    >> How much more can you squeeze out of a compressed raw image? When I
    >> shoot raw, I use Nikon compressed raw. I just tried bzip2 on a few
    >> files. I get about 9% further compression.

    >
    > You most probably get more if you reorder the data stream in such
    > a way that similar values are next to each other, especially if
    > you just record differences. That however needs a more in-depth
    > understanding of the file format than a 'simple' compressor like
    > bzip2 has.


    Yep that is correct. By knowing that it is going to get raw files (in known
    format), sgraw can look inside the file and use specialized algorithms.
    Simply recording differences is one possibility, and there are some very
    intelligent differencing methods which are now known.

    Sachin Garg [India]
    www.sachingarg.com | www.imagecompression.info
     
    Sachin Garg, Mar 19, 2008
    #17
  18. Sachin Garg

    Sachin Garg Guest

    "ray" <> wrote in message
    news:...
    > On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:
    >
    >> I just finished writing a small tool (sgraw) which can losslessly
    >> compress camera raw images, usually between 20-to-60%. (Regulars here
    >> might recall I have mentioned this 'possibility' earlier, I finally put
    >> in some hours to actually build a working demo :)
    >>
    >> http://www.sachingarg.com/compression/sgraw/

    []
    > It seems totally superfluous since there are already dozens of
    > compression programs available. What makes you think you've discovered
    > some 'new and improved' algorithm?


    Its not just the new and improved algorithm. What makes it different is that
    it specifically focuses on raw formats, which allows for looking inside the
    file for different types of information in it, and allows for tuning the
    algorithms for the types of data found in these formats.

    And then there is also a new and improved algorithm on top of it :)

    But besides the internal technicalities, its the results that matter most.
    Its not just a hollow claim, you can try sgraw in camparison to any other
    compression program. If the results are different, I would love to know
    about it (if nothing else, it will mean that I can further improve sgraw).

    Sachin Garg [India]
    www.sachingarg.com | www.imagecompression.info
     
    Sachin Garg, Mar 19, 2008
    #18
  19. ray <> wrote:
    > On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:


    >> compress camera raw images, usually between 20-to-60%. (Regulars here


    > It seems totally superfluous since there are already dozens of
    > compression programs available. What makes you think you've discovered
    > some 'new and improved' algorithm?


    Special tuning for camera raw images and the fact of compressed
    versions that take less space than general use compressors?

    -Wolfgang
     
    Wolfgang Weisselberg, Mar 19, 2008
    #19
  20. Sachin Garg

    ray Guest

    On Wed, 19 Mar 2008 23:09:44 +0530, Sachin Garg wrote:

    > "ray" <> wrote in message
    > news:...
    >> On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:
    >>
    >>> I just finished writing a small tool (sgraw) which can losslessly
    >>> compress camera raw images, usually between 20-to-60%. (Regulars here
    >>> might recall I have mentioned this 'possibility' earlier, I finally
    >>> put in some hours to actually build a working demo :)
    >>>
    >>> http://www.sachingarg.com/compression/sgraw/

    > []
    >> It seems totally superfluous since there are already dozens of
    >> compression programs available. What makes you think you've discovered
    >> some 'new and improved' algorithm?

    >
    > Its not just the new and improved algorithm. What makes it different is
    > that it specifically focuses on raw formats, which allows for looking
    > inside the file for different types of information in it, and allows for
    > tuning the algorithms for the types of data found in these formats.
    >
    > And then there is also a new and improved algorithm on top of it :)
    >
    > But besides the internal technicalities, its the results that matter
    > most. Its not just a hollow claim, you can try sgraw in camparison to
    > any other compression program. If the results are different, I would
    > love to know about it (if nothing else, it will mean that I can further
    > improve sgraw).
    >
    > Sachin Garg [India]
    > www.sachingarg.com | www.imagecompression.info


    I'd be happy to do that if you'd provide specific information on a couple
    of images. Looking at the bar graphs is really not suitable.
     
    ray, Mar 19, 2008
    #20
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. JC Dill
    Replies:
    1
    Views:
    720
    Charles
    Jul 13, 2004
  2. Replies:
    4
    Views:
    1,468
    bugbear
    Oct 24, 2006
  3. Sachin Garg

    'extra' lossless compression for camera raw images

    Sachin Garg, Jul 5, 2008, in forum: Digital Photography
    Replies:
    12
    Views:
    612
    Sachin Garg
    Jul 8, 2008
  4. Sachin Garg

    Rawzor beta release (lossless raw compression)

    Sachin Garg, Sep 2, 2008, in forum: Digital Photography
    Replies:
    84
    Views:
    1,920
    Sachin Garg
    Sep 15, 2008
  5. Giuen
    Replies:
    0
    Views:
    983
    Giuen
    Sep 12, 2008
Loading...

Share This Page