Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Computing > Digital Photography > 'extra' lossless compression for camera raw images

Reply
Thread Tools

'extra' lossless compression for camera raw images

 
 
Sachin Garg
Guest
Posts: n/a
 
      03-19-2008

"Alan Browne" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Sachin Garg wrote:
>> I just finished writing a small tool (sgraw) which can losslessly
>> compress camera raw images, usually between 20-to-60%. (Regulars here
>> might recall I have mentioned this 'possibility' earlier, I finally put
>> in some hours to actually build a working demo
>>
>> http://www.sachingarg.com/compression/sgraw/
>>
>> That would be savings between 1GB to 3GB for every 5GB of data.
>>
>> Even when camera has already compressed the raw image (which isn't always
>> the case), its still possible to squeeze out extra compression as cameras
>> have to use simple algorithms for speed.
>>
>> Compression is totally lossless, we get back the exact file we start with
>> (just like with zip), so both pixel information and meta-data are
>> perfectly restored.
>>
>> This is just the first quick hack version (and supports all nikon and
>> fuji cameras), so it obviously wouldn't make sense to actually use it for
>> backups etc, but if there is enough interest I can build it further to
>> support other raw formats, along with improved compression and speed.
>>
>> Your thoughts on this?

>
> For GP use? No biggie. Disk space is cheap.
>
> If it can be reduced to a h/w implementation (by the camera co's) that is
> more energy efficient (and faster), then it would be a great thing to
> improve in camera storage.


Yep, it will be a lot more useful if it can get in camera. I can only get
the technology ready and hopefully make it usable atleast on desktops.

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info


 
Reply With Quote
 
 
 
 
Steve
Guest
Posts: n/a
 
      03-19-2008
On Wed, 19 Mar 2008 11:35:55 +0530, "Sachin Garg"
<(E-Mail Removed)> wrote:

[...]
>You should be able to get atleast 20-30% even on compressed raw files using
>sgraw.
>
>If you try the tool on your images, I would love to see what results you
>get. In case you get anything less than that it will mean I can further
>improve it for your camera's raw files.


That's what I got. On one D200 compressed RAW test file, sgraw
compressed 23.5% and took 5 seconds to do it. bzip2 compressed 7.5%
and took 3.2 seconds to do it, on the same computer. They both
uncompressed losslessly. sgraw taking 2.235 seconds, bunzip2 1.973
seconds to uncompress the same file.

Details: file size = 8,242,489. sgraw compressed = 6,304,462.

I think that's an admirable performance, keep up the good work. Even
with cheap disk space, it's something I'd probably use when your
release versions are out, for archiving images I know I'm not going to
use for a while, maybe forever.

Steve
 
Reply With Quote
 
 
 
 
Wolfgang Weisselberg
Guest
Posts: n/a
 
      03-19-2008
Steve <(E-Mail Removed)> wrote:

> How much more can you squeeze out of a compressed raw image? When I
> shoot raw, I use Nikon compressed raw. I just tried bzip2 on a few
> files. I get about 9% further compression.


You most probably get more if you reorder the data stream in such
a way that similar values are next to each other, especially if
you just record differences. That however needs a more in-depth
understanding of the file format than a 'simple' compressor like
bzip2 has.

-Wolfgang
 
Reply With Quote
 
Wolfgang Weisselberg
Guest
Posts: n/a
 
      03-19-2008
Tony Polson <(E-Mail Removed)> wrote:

> You should patent it.


Sorry, patenting ideas is patented.

-Wolfgang
 
Reply With Quote
 
ray
Guest
Posts: n/a
 
      03-19-2008
On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:

> I just finished writing a small tool (sgraw) which can losslessly
> compress camera raw images, usually between 20-to-60%. (Regulars here
> might recall I have mentioned this 'possibility' earlier, I finally put
> in some hours to actually build a working demo
>
> http://www.sachingarg.com/compression/sgraw/
>
> That would be savings between 1GB to 3GB for every 5GB of data.
>
> Even when camera has already compressed the raw image (which isn't
> always the case), its still possible to squeeze out extra compression as
> cameras have to use simple algorithms for speed.
>
> Compression is totally lossless, we get back the exact file we start
> with (just like with zip), so both pixel information and meta-data are
> perfectly restored.
>
> This is just the first quick hack version (and supports all nikon and
> fuji cameras), so it obviously wouldn't make sense to actually use it
> for backups etc, but if there is enough interest I can build it further
> to support other raw formats, along with improved compression and speed.
>
> Your thoughts on this?
>
> Sachin Garg [India]
> www.sachingarg.com | www.imagecompression.info


It seems totally superfluous since there are already dozens of
compression programs available. What makes you think you've discovered
some 'new and improved' algorithm?
 
Reply With Quote
 
Sachin Garg
Guest
Posts: n/a
 
      03-19-2008

"Steve" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> On Wed, 19 Mar 2008 11:35:55 +0530, "Sachin Garg"
> <(E-Mail Removed)> wrote:
>
> [...]
>>You should be able to get atleast 20-30% even on compressed raw files
>>using
>>sgraw.
>>
>>If you try the tool on your images, I would love to see what results you
>>get. In case you get anything less than that it will mean I can further
>>improve it for your camera's raw files.

>
> That's what I got. On one D200 compressed RAW test file, sgraw
> compressed 23.5% and took 5 seconds to do it. bzip2 compressed 7.5%
> and took 3.2 seconds to do it, on the same computer. They both
> uncompressed losslessly. sgraw taking 2.235 seconds, bunzip2 1.973
> seconds to uncompress the same file.


Great, sgraw speed can be improved considerably, I will work on it.

> Details: file size = 8,242,489. sgraw compressed = 6,304,462.
>
> I think that's an admirable performance, keep up the good work.


Thanks

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info


 
Reply With Quote
 
Sachin Garg
Guest
Posts: n/a
 
      03-19-2008

"Wolfgang Weisselberg" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> Steve <(E-Mail Removed)> wrote:
>
>> How much more can you squeeze out of a compressed raw image? When I
>> shoot raw, I use Nikon compressed raw. I just tried bzip2 on a few
>> files. I get about 9% further compression.

>
> You most probably get more if you reorder the data stream in such
> a way that similar values are next to each other, especially if
> you just record differences. That however needs a more in-depth
> understanding of the file format than a 'simple' compressor like
> bzip2 has.


Yep that is correct. By knowing that it is going to get raw files (in known
format), sgraw can look inside the file and use specialized algorithms.
Simply recording differences is one possibility, and there are some very
intelligent differencing methods which are now known.

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info


 
Reply With Quote
 
Sachin Garg
Guest
Posts: n/a
 
      03-19-2008

"ray" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed)...
> On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:
>
>> I just finished writing a small tool (sgraw) which can losslessly
>> compress camera raw images, usually between 20-to-60%. (Regulars here
>> might recall I have mentioned this 'possibility' earlier, I finally put
>> in some hours to actually build a working demo
>>
>> http://www.sachingarg.com/compression/sgraw/

[]
> It seems totally superfluous since there are already dozens of
> compression programs available. What makes you think you've discovered
> some 'new and improved' algorithm?


Its not just the new and improved algorithm. What makes it different is that
it specifically focuses on raw formats, which allows for looking inside the
file for different types of information in it, and allows for tuning the
algorithms for the types of data found in these formats.

And then there is also a new and improved algorithm on top of it

But besides the internal technicalities, its the results that matter most.
Its not just a hollow claim, you can try sgraw in camparison to any other
compression program. If the results are different, I would love to know
about it (if nothing else, it will mean that I can further improve sgraw).

Sachin Garg [India]
www.sachingarg.com | www.imagecompression.info


 
Reply With Quote
 
Wolfgang Weisselberg
Guest
Posts: n/a
 
      03-19-2008
ray <(E-Mail Removed)> wrote:
> On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:


>> compress camera raw images, usually between 20-to-60%. (Regulars here


> It seems totally superfluous since there are already dozens of
> compression programs available. What makes you think you've discovered
> some 'new and improved' algorithm?


Special tuning for camera raw images and the fact of compressed
versions that take less space than general use compressors?

-Wolfgang
 
Reply With Quote
 
ray
Guest
Posts: n/a
 
      03-19-2008
On Wed, 19 Mar 2008 23:09:44 +0530, Sachin Garg wrote:

> "ray" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed)...
>> On Wed, 19 Mar 2008 01:28:47 +0530, Sachin Garg wrote:
>>
>>> I just finished writing a small tool (sgraw) which can losslessly
>>> compress camera raw images, usually between 20-to-60%. (Regulars here
>>> might recall I have mentioned this 'possibility' earlier, I finally
>>> put in some hours to actually build a working demo
>>>
>>> http://www.sachingarg.com/compression/sgraw/

> []
>> It seems totally superfluous since there are already dozens of
>> compression programs available. What makes you think you've discovered
>> some 'new and improved' algorithm?

>
> Its not just the new and improved algorithm. What makes it different is
> that it specifically focuses on raw formats, which allows for looking
> inside the file for different types of information in it, and allows for
> tuning the algorithms for the types of data found in these formats.
>
> And then there is also a new and improved algorithm on top of it
>
> But besides the internal technicalities, its the results that matter
> most. Its not just a hollow claim, you can try sgraw in camparison to
> any other compression program. If the results are different, I would
> love to know about it (if nothing else, it will mean that I can further
> improve sgraw).
>
> Sachin Garg [India]
> www.sachingarg.com | www.imagecompression.info


I'd be happy to do that if you'd provide specific information on a couple
of images. Looking at the bar graphs is really not suitable.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Rawzor beta release (lossless raw compression) Sachin Garg Digital Photography 84 09-15-2008 04:43 AM
'extra' lossless compression for camera raw images Sachin Garg Digital Photography 12 07-08-2008 05:57 PM
lossless compression in hardware: what to do in case of uncompressibility? Denkedran Joe VHDL 15 12-03-2007 12:40 PM
constant bitrate approach with lossless data compression on an FPGA Kurt Kaiser VHDL 5 11-10-2006 05:50 PM
question about using jpegtran for lossless compression of jpegs ewaguespack@gmail.com Digital Photography 4 10-24-2006 09:55 AM



Advertisments