raw files are HUGE

Discussion in 'Digital Photography' started by Sameer, Mar 4, 2007.

  1. Sameer

    Sameer Guest

    I tried winzip and winrar but they cant compress them much. Is there
    anything else which works?
     
    Sameer, Mar 4, 2007
    #1
    1. Advertisements

  2. Sameer

    Ken Lucke Guest

    Raw files come out of the camera pretty close to as efficient as they
    are going to get. You're not going to be very sucessful in attempting
    to compress them further, regardless of the program or compression
    scheme that you try.

    You think a raw file is huge? Try converting it to .tif and see where
    it goes. My 8-9 megabyte raws go to 57.8 MB when converted to 16 bit
    ..tif files.

    Besides, what's the problem? Storage has never been cheaper - I just
    picked up [yet another] Maxtor 300 GB Firewire/USB drive from Costco
    for $149, including cables... that's 50 cents a gig. That makes 3
    250GB, 1 400GB, and 2 300 GB drives attached to my system.


    <memory lane trip>
    I remember the first commercially available hard drive [1981, the Apple
    ProFile, available for the Apple /// computer, subsequently (1983) with
    a controller card that allowed its use on the Apple //[e|c]], it
    contained a single Seagate ST-506 HD with a whoppin' 5 MEGAbytes of
    storage space (and no one at the time thought you could *ever* fill it
    up) and close to $5000 new (IIRC) - almost $1000 per MEGAbyte. These
    things were almost as big as a tower case is now, and weighed about as
    much as a tank. Later, the massive 10 megabyte model was released.
    </memory lane>

    --
    You need only reflect that one of the best ways to get yourself a
    reputation as a dangerous citizen these days is to go about repeating
    the very phrases which our founding fathers used in the struggle for
    independence.
    -- Charles A. Beard
     
    Ken Lucke, Mar 4, 2007
    #2
    1. Advertisements

  3. Sameer

    timeOday Guest

    I have surely appreciated the crazy explosion in flash memory capacity,
    but hard drives are not keeping pace. I was just looking for a drive
    today and was disappointed that hard drive prices haven't fallen more
    since I bought my last drive a few years ago.

    For instance:
    <http://www.pcworld.com/article/id,77543-page,1/article.html>

    "Less than a year ago, a $300, 80GB desktop drive was considered huge;
    today you can find a 160GB drive for the same price."

    That was in January of 02'! According to this, between January of '01
    and January of '02 capacity doubled to 160 GB while the price remained
    constant. If that trend had continued, you could now buy a 5 terabyte
    drive for $300. Instead it's $300 for 750GB.

    Consulting pricewatch from 4 years ago, I came up with the following
    annual growth rates:

    Flash: 138%
    Hard drive: 32%
    RAM: 19%

    For comparison, Moore stated his "Law" at 100% per year in 1965,
    and at 41% (doubling every other year) in 1975. It is often quoted at
    doubling every 18 months, which would be 59% per year.

    Yeah, I'm spoiled. But compared to last century, this one isn't doing
    so hot. (Except for Flash).





    For the interested, here's the data I used:

    Look at pricewatch on the internet archive from 4 years ago:
    <http://web.archive.org/web/20030128022327/http://www.pricewatch.com/>
    A 120GB drive was $117, and the maximum available size was 250GB.

    Now, 4 years later, that same $120 will buy you a 400 GB drive and the
    max available size is 750GB.
    <http://www.pricewatch.com/hard_drives/>
    So in 4 years, price is basically constant while capacity has gone up by
    a little more than a factor of 3.

    Meanwhile, using the same sources, 4 years ago a 256 MB usb flash drive
    was $75. Today for $79 you get an 8 GB flash. That's a factor of 32!

    In 2002, 1GB of PC133 RAM would set you back $144.
    In 2007, for the same price you can get 1 GB ddr2-1066 1gb or 2GB
    DDR2-400. The cheapest 1GB module now is PC100 for $60.
    So in 4 years, RAM has only doubled in capacity for the same price.


    <http://en.wikipedia.org/wiki/Moore's_law>
     
    timeOday, Mar 4, 2007
    #3
  4. Sameer

    ray Guest

    That is because, like executable files, there is much entropy there. The
    data is inherently not very compressible. I note on my 5mp camera that
    jpegs run 1-2mb; raws are around 8-9mb; and tiffs are 15mb - I guess it's
    all relative - I can still pack a lot of pictures on a $25 2gb memory card.
     
    ray, Mar 4, 2007
    #4
  5. Some cameras don't compress their raw files. For example the Pentax K10D
    doesn't compress the DNG raw files, but does compress the PEF raw files. I
    shoot DNG, when I get home I use the Adobe DNG converter to compress the DNG
    files in to DNG files. This takes their size from about 16MB to about half
    that at around 8MB. Try using Adobe DNG converter, it is free and can
    downloaded off the Adobe site and it may give you smaller files.

    TGC Ltd.
     
    The Grape Smasher Ltd., Mar 4, 2007
    #5
  6. Sameer

    Rod Speed Guest

    That isnt as true with the best after rebate prices.
     
    Rod Speed, Mar 4, 2007
    #6
  7. Sameer

    CJT Guest

    There has been a lot of consolidation in the hard drive industry,
    resulting in an oligopoly. Without competitive pressures, prices
    will not decline at the same pace.
     
    CJT, Mar 4, 2007
    #7
  8. Sameer

    Rod Speed Guest

    Have fun explaining how come you dont get the same
    effect with flash ram which has the same problem.
     
    Rod Speed, Mar 4, 2007
    #8
  9. Sameer

    Matt Ion Guest

    Kinda gotta consider where you shop as well.

    Right now, a retail-boxed internal 300GB/16MB Maxtor SATA drive (exact model
    unspecified) would cost me CDN$236(!!!) at Staples... meanwhile my regular
    retailer sells a 500GB/16MB SATA-II packed only in an antistatic bag, for
    CDN$169 (they don't even list a 300, but a 250 is $85-$95). That's a lotta
    extra cash for a cardboard box and "installation instructions".
     
    Matt Ion, Mar 4, 2007
    #9
  10. Also, check your warranties. If Maxtor is anything like Intel you get really
    poor warranties with OEM (plastic bag only) items. If you buy a retail CPU
    and a retail motherboard Intel gives you the warranty through them for 3
    years on each. If you go OEM they don't warrant anything and leave it up to
    the place you bought it. For example an OEM CPU from TigerDirect has a 1
    year warranty and it is through them.

    So make sure that the extra you pay isn't for warranty. Though honestly the
    price difference still doesn't warrant that. If a drive or CPU or
    Motherboard is going to fail it would surely do it within a year. Just make
    sure that you do get a warranty and if it is handled by the place you buy
    from that you can count on them being in business for at least the length of
    warranty.

    TGC Ltd.


     
    The Grape Smasher Ltd., Mar 5, 2007
    #10
  11. Sameer

    Sameer Guest

    My files must be compressed already or winzip would have got more than
    2% :)

    I don't like DNG, the convertor doesn't let me convert back to
    original format. I read a little bit about it and it seems not many
    people use it.
     
    Sameer, Mar 5, 2007
    #11
  12. Actually, you can embed your original file in to the DNG and pull it out of
    the DNG at any time. You of course are back to large files.

    As for Winzip and RAR neither one of them was designed for image or audio or
    video data. There were designed for documents and programs. So the fact that
    they give you next to nothing isn't a surprise and doesn't mean your images
    can't be compressed a lot. You just have to use something designed for that
    kind of data.

    TGC Ltd.
     
    The Grape Smasher Ltd., Mar 5, 2007
    #12
  13. Sameer

    Bill Funk Guest

    Moore didn't state a "law."
    He made an observation.
    And the observation was concerning chips, and had nothing to do with
    hard drives.
    Trying to compare the density of transistors on chips and aerial
    density of hardd rives is comparing apples and potatos.

    --
    John Edwards was warned by Democrats
    in Nevada Thursday that his support
    for a bill to ban betting on college
    sports will cost him the state. You
    have to love Nevada. It's the only
    place where a personal injury lawyer
    has the moral high ground.
     
    Bill Funk, Mar 5, 2007
    #13
  14. Sameer

    AZ Nomad Guest

    His observation has been valid for hard drives.
     
    AZ Nomad, Mar 5, 2007
    #14
  15. Sameer

    Arno Wagner Guest

    Indeed. The ''law'' was made out of his observations by the press, I
    beliveve.
    More like apples and fish, really.

    Arno
     
    Arno Wagner, Mar 5, 2007
    #15
  16. Sameer

    Arno Wagner Guest

    To some degree for capacities. Not at all for interface speeds.
    And they are not his observations for HDDs, since he did not
    observe HDDs at all.

    Arno
     
    Arno Wagner, Mar 5, 2007
    #16
  17. Sameer

    AZ Nomad Guest

    Know of many consumer hard drives that did 80MB/s in 1995? In 1997, a $500
    10Krpm scsi-3 drive was only good for 3-4MB/s. In 1985, consumer PC hard
    drivess were good for maybe 500KB/s. Right now I'm getting 80MB/s on a $140
    500GB drive. Sure sounds like a geometric progression to me.

    Pretty much, drive speeds have gone up in step with drive capacities.
    You get greater capacity either with more platters or a higher data density.
    Both translate to greater throughput given the same rotational speed.
     
    AZ Nomad, Mar 5, 2007
    #17
  18. I was getting 3.5 MB/s STR out of 500 MB 3600 RPM (?) IDE Quantum ($350) in
    1994. This was on a IDE card connected to VESA bus.
     
    Alexander Grigoriev, Mar 6, 2007
    #18
  19. Sameer

    Bob Willard Guest

    Uh, adding platters does nothing for STR. More throughput, but all of
    the data about speed in this thread has been about STR.

    {Long long ago, in a galazy near near to us, more platters meant higher
    STRs for a special class of expensive HDs that did parallel reads from
    multiple heads. But everything made today reads (and writes) one head
    at a time. If you want RAID0, the economics of HDs make it cheaper to
    deploy multiple HDs than multiple heads on the same HD, at least for now.}

    HD STR has never maintained a 60% CAGR over any period of a decade or so
    for mainstream HDs AFAIK.
     
    Bob Willard, Mar 6, 2007
    #19
  20. Sameer

    Bill Funk Guest

    Lead balls dropped in air acellerate at a certain rate.
    Do you, therefore, expect feathers dropped in air to acellerate at the
    same rate?
    But you *observed* the lead balls.
    Moore's observation had/has nothing to do with hard drives.

    --
    Hillary Clinton went to a civil
    rights ceremony in Selma Sunday
    and spoke to a local black Baptist
    Church congregation. She mentioned
    her husband at the top of her
    speech. Whenever you face a tough
    crowd you have to open with your
    best joke.
     
    Bill Funk, Mar 6, 2007
    #20
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.