A short study on digicam's fixed jpeg compression ratio

Discussion in 'Digital Photography' started by Heikki Siltala, Jul 9, 2004.

  1. Hi all,

    I have recently purchased Nytech ND 4020 digital camera (aka Konica KD
    4000). I have shot about 500 images now and the camera seems to work
    fine. When moving the image files from memory card to PC I noticed that
    every image shot with best quality is of same file size, 1 492 992
    bytes. I would understand this if the images were uncompressed. Since
    they are JPEG comperssed the file size should vary depending on the
    image: image with lots of detail should be larger than an image with
    less detail.

    The manual says that best image quality setting uses fixed 8:1
    compression ratio. At first this seems to be correct, since the
    compressed file is 1458 kBs and uncompressed image data in RAM is 11 664
    kBs (2304x1728, 4 Mpixels). I started to wonder how does the camera
    implement this "fixed ratio jpeg compression"? I have never heard about
    that kind of JPEG compression before.

    First step towards the answer was when I rotated the images using a
    software that has lossless JPEG rotation. The rotation decreased file
    size significantly. It happened only once: when I kept rotating the file
    the size did not continue to decrease. Second step was to edit file's
    Exif data. It did the same: file was shrunk. The files that were rotated
    or Exif edited were between 414 kB and 947 kB compared to original 1458 kB.

    At this point I opened the files with hex editor and studied JPEG/EXIF
    specs. The specs told that every file should start with tag FFD8 and end
    with tag FFD9. Every rotated and Exif edited file followed this rule
    since they were saved by a software on my PC. But the unmodified images
    straight from the camera did not! The unmodified files start with FFD8
    and then follows the data. At some point comes FFD9 tag but the file
    does not end there. There are lots of data after the end tag and it
    looks like "uninitialized memory" when browsed with hex editor.

    Conclusion: the camera uses standard JPEG compression (like all other
    digicams?) and only imitates 8:1 fixed ratio compression by inflating
    the file to match 8:1 compression result. Why? Why does it fill my sD
    cards with unused data? I guess that it simplifies the user experiment:
    a regular user is happy when he gets fixed image size and can predict
    when the card will fill up. And it might have something to do with
    software development: it might be easier to write the software so that
    every time you shoot it reserves fixed 1458 kB from the card and then
    dumps image data into it starting from the beginning. Because the end
    tag is misplaced the file does not follow JPEG spec and is actually not
    a JPEG file. Happily enough all modern software seem to read and open
    these files.

    Is there someone who owns Nytech ND 4020 or Konica ND 4000 that could
    confirm this behavior? How about other Nytech/Konica models? How about
    digicams from other vendors, do they use same kind of "bogus fixed ratio
    compression"?

    Well, maybe I should spend more time shooting than analyzing the binary
    files...

    --
    Heikki@Finland
    Heikki Siltala, Jul 9, 2004
    #1
    1. Advertising

  2. "Heikki Siltala" <> wrote in message
    news:ccmo09$35s$...
    []
    > Conclusion: the camera uses standard JPEG compression (like all other
    > digicams?) and only imitates 8:1 fixed ratio compression by inflating
    > the file to match 8:1 compression result. Why? Why does it fill my sD
    > cards with unused data? I guess that it simplifies the user experiment:
    > a regular user is happy when he gets fixed image size and can predict
    > when the card will fill up. And it might have something to do with
    > software development: it might be easier to write the software so that
    > every time you shoot it reserves fixed 1458 kB from the card and then
    > dumps image data into it starting from the beginning. Because the end
    > tag is misplaced the file does not follow JPEG spec and is actually not
    > a JPEG file. Happily enough all modern software seem to read and open
    > these files.


    Heikki, that sounds like a fair conclusion to me - pad the files so that
    they are always the same size! Not clever. Not clever, at all!

    Just an aside - what size is a blank floppy disk? Let me check.
    1,457,664 bytes is the "available" space when formatting a disk. I just
    though for one minute it might be padded out to the size of a floppy disk!

    What a horrible little camera (software-wise).

    David
    David J Taylor, Jul 9, 2004
    #2
    1. Advertising

  3. Heikki Siltala

    Bob Guest

    "Heikki Siltala" <> wrote in message
    news:ccmo09$35s$...
    > digicams?) and only imitates 8:1 fixed ratio compression by inflating
    > the file to match 8:1 compression result. Why? Why does it fill my sD
    > cards with unused data? I guess that it simplifies the user experiment:



    good job, very thourough analysis:


    Based on what you report, I think the program allocates for the maximum size
    in a variable, then write to that variable...I think the FFD9 acts like a
    string terminator. I don't think it is padding or adding, it sounds like
    really sloppy programming, where they allocate space for something, but do
    not check to see the size of what they are writing to that variable
    (memory).

    for instance you get this behaviour when you write to a char[n] but write
    less than n characters to it. I find it hard to believe they would make
    such a amatuer mistake...maybe I am misunderstanding your results....

    Bob
    Bob, Jul 9, 2004
    #3
  4. Heikki Siltala <> writes:

    >Because the end
    >tag is misplaced the file does not follow JPEG spec and is actually not
    >a JPEG file.


    Are you sure? It sounds like the file does follow the JPEG spec for all
    the data between the start and end markers, and the markers are correct.
    The only anomaly is that there's garbage data after the end marker. This
    may not violate the JPEG spec if it just specifies what happens between
    markers. It's illegal only if the spec explicitly says "there shall be
    no data in the file following the end marker".

    >Is there someone who owns Nytech ND 4020 or Konica ND 4000 that could
    >confirm this behavior? How about other Nytech/Konica models? How about
    >digicams from other vendors, do they use same kind of "bogus fixed ratio
    >compression"?


    I have several Canon digicams. The file size is different every frame.
    The best quality (superfine) setting is about 7:1 compression ratio.

    Dave
    Dave Martindale, Jul 10, 2004
    #4
  5. Heikki Siltala

    Bob Guest

    "Dave Martindale" <> wrote in message
    news:ccpris$j6q$...

    > Are you sure? It sounds like the file does follow the JPEG spec for all
    > the data between the start and end markers, and the markers are correct.
    > The only anomaly is that there's garbage data after the end marker. This
    > may not violate the JPEG spec if it just specifies what happens between
    > markers. It's illegal only if the spec explicitly says "there shall be
    > no data in the file following the end marker".



    what does that matter...it's more the problem of filling up the memory card
    with uninitilized memory...due to most likely bad programming...In a
    commercial product, who would settle for such a thing?
    Bob, Jul 12, 2004
    #5
  6. Dave Martindale wrote:
    > Are you sure? It sounds like the file does follow the JPEG spec for all
    > the data between the start and end markers, and the markers are correct.
    > The only anomaly is that there's garbage data after the end marker. This
    > may not violate the JPEG spec if it just specifies what happens between
    > markers. It's illegal only if the spec explicitly says "there shall be
    > no data in the file following the end marker".


    I studied the specs again. The camera seems to output Exif 2.2 files so
    I focused on Exif 2.2 spec "JEITA CP-3451 Exchangeable image file format
    for digital still cameras: Exif Version 2.2", available at
    http://www.kodak.com/global/plugins/acrobat/en/service/digCam/exifStandard2.pdf.


    Your point is well made since I found no explicit information like
    "there shall be no data in the file following the end marker". Still,
    Page 11 Figure 6 describes "Basic Structure of Compressed Data Files"
    and indicates that the file structure is

    SOI,APP1(,APP2),DQT,DHT(,DRI),SOF,SOS,Compressed Data,EOI

    where SOI is the start marker (FFD8) and EOI is the end marker (FFD9).
    Ih the figure the file ends right after the end marker. I would
    interpret this so that there can be nothing after the end marker. At
    this light I would say the camera doesn't output valid Exif 2.2 files
    because the file continues after the end marker.

    I also studied from "JEITA CP-3461 Design rule for Camera File system
    DCF Version 2.0" available at http://tsc.jeita.or.jp/avs/data/cp3461.pdf
    but it doesn't give information on the issue.

    I examined the files output by the camera a little bit further. When I
    take photos with different resolution and compression ratio settings
    there is always at least some unnecessary data after the end marker. So
    the issue is not dependent on the resolution and compression ratio used.

    --
    H
    Heikki Siltala, Jul 13, 2004
    #6
  7. Heikki Siltala <> writes:

    >Your point is well made since I found no explicit information like
    >"there shall be no data in the file following the end marker". Still,
    >Page 11 Figure 6 describes "Basic Structure of Compressed Data Files"
    >and indicates that the file structure is


    > SOI,APP1(,APP2),DQT,DHT(,DRI),SOF,SOS,Compressed Data,EOI


    >where SOI is the start marker (FFD8) and EOI is the end marker (FFD9).
    >Ih the figure the file ends right after the end marker. I would
    >interpret this so that there can be nothing after the end marker. At
    >this light I would say the camera doesn't output valid Exif 2.2 files
    >because the file continues after the end marker.


    Generally, figures in standards are intended to illustrate what the
    standard document is describing, and make it easier to understand. They
    may illustrate a typical file structure, but not necessarily one that is
    absolutely required. It's the words of the standard that specify what
    is and is not allowed.

    I'd interpret what you describe as an illustration showing that typical
    JPEG files don't have any data after the end marker. But unless the
    standard says explicitly that extra trailing data is not allowed, it
    does not violate the standard. It is stupid, and wastes space, and
    indicates poorly-written software, but it's not "illegal".

    Dave
    Dave Martindale, Jul 13, 2004
    #7
  8. David J Taylor wrote:
    > Just an aside - what size is a blank floppy disk? Let me check.
    > 1,457,664 bytes is the "available" space when formatting a disk. I just
    > though for one minute it might be padded out to the size of a floppy disk!


    I have two wild thoughts where the 1458 kB file size comes.

    a) It has something to do with FAT16 filesystem used with sD cards.

    b) It has something to do with JPEG compression technique. Maybe the
    camera has a fixed jpeg quality factor and 1458 kb is the theoretical
    upper limit for an image that has size 2304x1728 and has been jpeg
    compressed using that quality factor. So the software can be absolutely
    sure that the image size is never more than 1458 kb and can safely
    allocate a fixed space for it. But I'm not going to study jpeg
    compression to find out if this can be true. :) The images that I have
    taken seem to be between 300 Kb and 1300 Kb of size.

    When I set the resolution lower or comperssion ratio higher, the file
    size gets smaller. But there is always some unneccessary data after the
    end marker FFD9. The camera seems to have a fixed file size for every
    resolution/compression ratio -combination.

    > What a horrible little camera (software-wise).


    Yep. Luckily it works fine and takes decent photos. A little bit high
    power consumption, some noise when using ISO 400, some barrel distortion
    etc. But compared to the one-day-price (199 eurodollars with 64 MB
    card, AC adapter, two sets of rechargeable batteries and a battery
    charger) I say that it is OUTSTANDING for the price at least here in
    Finland. It does have 4 Mpixel resolution, 3x optical zoom, video
    recording with voice, direct connection to TV etc.

    --
    H
    Heikki Siltala, Jul 13, 2004
    #8
  9. Bob wrote:
    > what does that matter...it's more the problem of filling up the memory card
    > with uninitilized memory...due to most likely bad programming...In a
    > commercial product, who would settle for such a thing?


    Yep. All software that I have tried do read and open the files just
    fine. But filling up the memory card is not nice. A short calculation:
    the file size is 1458 kilobytes and the true data size typically between
    300 and 1300 kilobytes. The average data size might be something like
    700 kB so more than 50% of the space on the card is wasted! The camera
    shoots 41 photos on 64 MB card. This could be 85 photos if no space is
    wasted! The camera shoots 83 photos on 128 MB card. This could be 172
    photos if no space is wasted!

    --
    H
    Heikki Siltala, Jul 13, 2004
    #9
  10. "Heikki Siltala" <> wrote in message
    news:cd10tl$9r2$...
    []
    > I have two wild thoughts where the 1458 kB file size comes.
    >
    > a) It has something to do with FAT16 filesystem used with sD cards.


    That sounds unlikely to me.

    > b) It has something to do with JPEG compression technique. Maybe the
    > camera has a fixed jpeg quality factor and 1458 kb is the theoretical
    > upper limit for an image that has size 2304x1728 and has been jpeg
    > compressed using that quality factor.


    Well, actually given an inappropriate image (lots of high spatial
    frequency data) it's possible that some compression techniques could
    actually increase the size of the file (compared to the uncompressed
    data). I don't know if this applies to JPEG, though.

    > When I set the resolution lower or comperssion ratio higher, the file
    > size gets smaller. But there is always some unneccessary data after the
    > end marker FFD9. The camera seems to have a fixed file size for every
    > resolution/compression ratio -combination.
    >
    > > What a horrible little camera (software-wise).

    >
    > Yep.


    It shows they don't think much about their customers!

    > Luckily it works fine and takes decent photos. A little bit high
    > power consumption, some noise when using ISO 400, some barrel distortion
    > etc. But compared to the one-day-price (199 eurodollars with 64 MB
    > card, AC adapter, two sets of rechargeable batteries and a battery
    > charger) I say that it is OUTSTANDING for the price at least here in
    > Finland. It does have 4 Mpixel resolution, 3x optical zoom, video
    > recording with voice, direct connection to TV etc.


    Eurodollars?? Do you mean Euros? I didn't know that Finland had changed
    currency but that's great it if has. All we need now is for Sweden and
    Denmark to follow suit! Norway is a lost cause.... <G>

    Cheers,
    David
    David J Taylor, Jul 13, 2004
    #10
  11. On Tue, 13 Jul 2004 15:57:47 +0000 (UTC), (Dave
    Martindale) wrote:

    >Heikki Siltala <> writes:
    >
    >>Your point is well made since I found no explicit information like
    >>"there shall be no data in the file following the end marker". Still,
    >>Page 11 Figure 6 describes "Basic Structure of Compressed Data Files"
    >>and indicates that the file structure is

    >
    >> SOI,APP1(,APP2),DQT,DHT(,DRI),SOF,SOS,Compressed Data,EOI

    >
    >>where SOI is the start marker (FFD8) and EOI is the end marker (FFD9).
    >>Ih the figure the file ends right after the end marker. I would
    >>interpret this so that there can be nothing after the end marker. At
    >>this light I would say the camera doesn't output valid Exif 2.2 files
    >>because the file continues after the end marker.

    >
    >Generally, figures in standards are intended to illustrate what the
    >standard document is describing, and make it easier to understand. They
    >may illustrate a typical file structure, but not necessarily one that is
    >absolutely required. It's the words of the standard that specify what
    >is and is not allowed.
    >
    >I'd interpret what you describe as an illustration showing that typical
    >JPEG files don't have any data after the end marker. But unless the
    >standard says explicitly that extra trailing data is not allowed, it
    >does not violate the standard. It is stupid, and wastes space, and
    >indicates poorly-written software, but it's not "illegal".


    It is also a common ambiguity in specs where someone probably meant to
    say it was not allowed, but it was not made explicit. This kind of
    thing is what leads to non-standard extensions of the spec. Someone
    will then put some other information after the EOI and someone else
    will make sure to drop that info and uses will have problems.


    --
    Matt Silberstein

    Do in order to understand.
    Matt Silberstein, Jul 13, 2004
    #11
  12. David J Taylor wrote:
    > Eurodollars?? Do you mean Euros? I didn't know that Finland had changed
    > currency but that's great it if has. All we need now is for Sweden and
    > Denmark to follow suit! Norway is a lost cause.... <G>


    Yes, I mean Euros. I have heard that in the States they say
    "eurodollars" when they mean euros. At least they did that on "The Bold
    and The Beautiful" :)

    Finland was among the countries that changed to Euro 1st January 2002.

    http://www.bof.fi/eng/4_raha/4.1_euro/index.stm
    Heikki Siltala, Jul 13, 2004
    #12
  13. Matt Silberstein wrote:
    > It is also a common ambiguity in specs where someone probably meant to
    > say it was not allowed, but it was not made explicit. This kind of
    > thing is what leads to non-standard extensions of the spec. Someone
    > will then put some other information after the EOI and someone else
    > will make sure to drop that info and uses will have problems.


    Now I started to think what if the camera uses the space after EOI for
    some internal purpose. It might be so. It looks like "uninitialized
    memory" for me when I look at it with hex editor. But it could also be
    some vendor-specific internal storage area for some important data. I
    can't tell. Still, most likely it is just a software bug or bad software
    design.

    The manual of the camera doesn't explicitly say that "this camera uses
    fixed ratio jpeg compression". It merely indicates that in a table where
    the approximate numbers of shots for each setting/card size -combination
    is given. So I can't really say the manual gives incorrect information.
    The manual can be found at

    http://www.nytech.de/cms_uk/nytech_9_569.php
    Heikki Siltala, Jul 13, 2004
    #13
  14. "Heikki Siltala" <> wrote in message
    news:cd1hld$8kq$...
    >
    > David J Taylor wrote:
    > > Eurodollars?? Do you mean Euros? I didn't know that Finland had

    changed
    > > currency but that's great it if has. All we need now is for Sweden

    and
    > > Denmark to follow suit! Norway is a lost cause.... <G>

    >
    > Yes, I mean Euros. I have heard that in the States they say
    > "eurodollars" when they mean euros. At least they did that on "The Bold
    > and The Beautiful" :)
    >
    > Finland was among the countries that changed to Euro 1st January 2002.
    >
    > http://www.bof.fi/eng/4_raha/4.1_euro/index.stm


    Thanks, Heikki. Yes, I looked it up as well and you were in the very
    first group of countries. That's great.

    For the benefit of our friends across the pond, there is no such thing as
    a EuroDollar, nor a PoundDollar, nor a CrownDollar! I suppose it's a form
    of flattery really - they expect the currency to become as widely used as
    their own!!

    JPEG files have a means of adding information, in the EXIF segment, so the
    extra stuff at the end is most likely rubbish. Try removing it, and see
    if the camera can still recognise the image if the file is written back to
    the memory card. See if anything is missing from the display such as a
    histogram which would normally be there.

    Cheers,
    David
    David J Taylor, Jul 13, 2004
    #14
  15. Hello,

    I managed to construct a test. It indicates that the camera seems to
    have no use for the "uninitialized memory" at the end of a file. So
    reserving too much space is most likely a software design flaw or a
    software bug. The test was this:

    1. Put 64 MB card into camera and took 20 random photos, 10 of
    resolution 2304x1728 & lowest compression and 10 of resolution 1280x960
    & highest compression. Browsed thru the files with the camera, used
    zooming, panning etc.

    2. Took the card out, put it into card readed and copied the files to
    computer. 2304x1728 files were of size 1 492 992 bytes, 1280x960 files
    were of size 230 400 bytes.

    3. Processed the files with custom-built Perl program. The source code
    can be found below. The program seeks the FFD9 tag starting from the end
    of the file and when it founds one, it attemps to check if there is
    "real data" before it. If there is, it cuts the file right after the end
    tag. If there is not real data after the tag, it continues the seach
    towards to the beginning of the file attempting to found an end tag that
    has real data before it. By using the program 2304x1728 files were
    truncated from 1 492 992 bytes down to 801 310 - 1 260 788 bytes.
    1280x960 files were truncated from 230 400 down to 145 792 - 228 520
    bytes. The waste of space seems to be smaller than in my earlier tests.
    It is logical because these photos were shot in the dark without flash
    using ISO 400 and so there is some noise which makes jpeg compression
    less efficient and "true file size" larger.

    4. Opened the files with a jpeg viewing software to make sure that the
    files were not corrupted due the cutting. All images opened fine and
    looked identical compared to originals.

    5. Removed the original files from the card and uploaded the truncated
    files to the card.

    6. Put the card back into camera and browsed thru the truncated files
    with camera. The camera displayed the truncated files without problems
    and zooming,panning etc. worked fine. No difference where noticed in the
    camera's behavior compared to the situation when browsing the original
    files.

    6b. An interesting sideline is that with the original files the camera
    estimated there would fit 29 more max res & min compression photos to
    the card. With the truncated files the estimate was 34. It seems that
    although the camera outputs the files using a fixed file size it still
    bases the estimate on the free memory on the card, not counting how many
    photos on what resolution & compression are already on the card.

    And now the source code, it is quite a hack but seems to work. Takes one
    argument, the file to be processed. Runs at least with Perl 5.6.1 on WinXP.

    sysopen(INFILE, $ARGV[0], 0);
    binmode INFILE;
    $inbytes = sysread INFILE, $databuf, 10000000;
    print "read $inbytes bytes from $ARGV[0]\n";
    close INFILE;
    $location=$inbytes-3;
    $end_of_file=-1;
    while($location>-1) {
    if((substr($databuf,$location,1) eq "\xff")&&(substr($databuf,$location+1,1)
    eq "\xd9")) {
    print "found FFD9 at $location, might be end of data, investigating\n";
    $scanner=$location-31;
    $found=0;
    while($scanner<$location) {
    if (substr($databuf,$scanner,1) eq "\xff") {
    $found++;
    }
    $scanner++;
    }
    if($found>9) {
    print "found $found ff's, assuming not end of data, search continues\n";
    } else {
    $end_of_file=$location+2;
    $location=0;
    print "found $found ff's, assuming end of data, placing end of file to
    $end_of_file\n";
    }
    }
    $location--;
    }
    if($end_of_file==-1) {
    $end_of_file=$inbytes+1;
    }
    # write data
    `del $ARGV[0]`;
    `copy <NUL >$ARGV[0]`;
    sysopen(OUTFILE, $ARGV[0],1);
    binmode OUTFILE;
    $outbytes = syswrite OUTFILE, $databuf, $end_of_file;
    print "wrote $outbytes bytes to $ARGV[0]\n";
    close OUTFILE;
    Heikki Siltala, Jul 21, 2004
    #15
  16. Heikki Siltala wrote:
    > Hello,
    >
    > I managed to construct a test. It indicates that the camera seems to
    > have no use for the "uninitialized memory" at the end of a file. So
    > reserving too much space is most likely a software design flaw or a
    > software bug. The test was this:

    []

    So it's just lousy software that makes the files a fixed size - I find it
    amazing that software like that can make it into the outside world. Why
    on earth would anyone write it like that?

    So I am not impressed with the Nytech ND 4020 digital camera (aka Konica
    KD 4000). The fact that the Konica/Minolta A2 has unfixed firmware
    problems first reported on the A1 suggests that the software department of
    this company is in need of some attention.

    Many thanks for reporting your observations, tests, software and results.

    Cheers,
    David
    David J Taylor, Jul 22, 2004
    #16
  17. I mailed Nytech today and reported the flaw/bug to them. It is most likely
    that the software is developed by Konica and so Nytech might not have
    anything to say on the flaw/bug.

    BTW, the software version on my camera is DSC Firmware Ver 1.47.15 1.00.

    --
    H
    Heikki Siltala, Jul 28, 2004
    #17
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Nicholas J. Coscoros

    JPEG Compression (Newbie Question)

    Nicholas J. Coscoros, Jul 29, 2003, in forum: Digital Photography
    Replies:
    5
    Views:
    514
    Bruce Chastain
    Jul 30, 2003
  2. Phillean
    Replies:
    5
    Views:
    487
    ArtKramr
    Oct 4, 2003
  3. TheChair

    Kodak JPEG Compression

    TheChair, Dec 7, 2003, in forum: Digital Photography
    Replies:
    10
    Views:
    2,226
    Ron Hunter
    Dec 10, 2003
  4. jriegle

    JPEG compression not the same?

    jriegle, Dec 12, 2003, in forum: Digital Photography
    Replies:
    5
    Views:
    534
    Andy Hewitt
    Dec 12, 2003
  5. remove the pachyderm

    How to crop w/ fixed aspect ratio?

    remove the pachyderm, Jan 19, 2004, in forum: Digital Photography
    Replies:
    49
    Views:
    10,594
    Roger Halstead
    Jan 25, 2004
Loading...

Share This Page