"Digital ICE" without Digital ICE

Discussion in 'Digital Photography' started by Lorenzo J. Lucchini, Oct 26, 2004.

  1. First, sorry for cross-posting, I admit I just posted to the two
    groups where the topic of Digital ICE seems to come up most often.

    I'm a newcomer in digital photography and scanning, so the ideas I'll
    discuss below might very well be badly flawed - please bear with me.

    Ok, you have a flatbed scanner with a transparency adaptor; it's got
    two lamps, one below the glass and one above. Film is supposed to be
    scanned with the top lamp.

    But what happend if you scan film with the bottom lamp, possibly with
    the lid open and in a dark room?
    The scan will show an almost completely black film, since nearly no
    light passed through it. However, dust particles and white-ish
    scratches on the film will (or might) reflect the light from the
    bottom lamp, and thus will easily be spotted in the scan! (or will be
    after some histogram normalization).

    The rest is just a software process of applying dust-removal filters
    on the spots in the image we know dust lies.

    What do you think? I *have* actually tried this stuff and it *appears*
    to work and remove a number of defects from the final image - although
    many dust spots remains, and I haven't been able to find out whether
    the system works differently with dust that's *on the film* and dust
    that's *on the scanner*.

    However, after cleaning the scanner's glass as carefully as I could, I
    still think I'm spotting particles that do lie on the film surface.
    Then maybe it also depends which side of the film dust lies on.

    I'll post sample pictures and some sketched Bash scripts using NetPBM
    for the actual picture cleaning if anybody's interested.

    by LjL
    Lorenzo J. Lucchini, Oct 26, 2004
    1. Advertisements

  2. Lorenzo J. Lucchini

    Matt Ion Guest

    Interesting concept, and should somewhat work in theory, although it
    will only work for "noise" that's between the film/negative and the
    scanner head. Dust that's on "top" of the film (facing the upper light
    head) won't be detected by the reflective scan. Also, if the noise is
    severe enough to block light on a transmissive scan, the reflective scan
    may give you a "cancellation" image for it, but the information in the
    film itself will still be obscured by it.
    Matt Ion, Oct 26, 2004
    1. Advertisements

  3. Lorenzo J. Lucchini

    Gadgets Guest

    Interesting idea, but you'd need your object to be in the exact same place
    when you lifted the lid. Maybe it could work if you put glass over your

    Cheers, Jason (remove ... to reply)
    Video & Gaming: http://gadgetaus.com
    Gadgets, Oct 26, 2004
  4. I'm not sure. It's true that there is dust that doesn't get detected,
    but after a few experiments my opinion is that it's either because
    they're too small, too dark to reflect enough light, or out-of-focus
    (ie on the glass).

    What you say is probably true if I do the "defect scan" with the lid
    closed, which reduces contrast between image data and noise data, so
    making the particles that lie on top of the film too dim for
    detection; with the lid open, however, that doesn't seem to be the

    (Actually, it obviously *will* be the case for particles that lie on
    top of a *dark region* of the film, but those particles won't do much
    damage to the final image anyway)
    Sure, but AFAICS this applies to real Digital ICE as well - if
    information is lost, it's lost, but we can try to mask the loss by
    interpolation or some like technique.

    On the topic of the noise-masking algorithm... what I do now is simply
    - create a very blurred version of the scan
    - superimpose the blurred version onto the original version, using the
    "defect scan" as an alpha channel so that the blurred version is only
    shown where there are defects

    However, this doesn't seem to work very well, since the color of the
    final image looks quite a bit different from what you'd expect in the
    noise spots.

    What's a better idea? I've read Digital ICE simply divides every image
    pixel by the corresponding pixel in the "defect scan" (an IR scan in
    ICE's case).
    After trying this, though, it doesn't seem to work well at all: the
    features of dust spots in my "defect scans" probably differ a lot from
    ICE's IR scans.

    Now I'm entertaing myself with the idea of using a median filter,
    which seems to preserve pixel colors fairly well while removing most
    defects; what do you think?

    I'm not at home at the moment, but I'll upload some of my scans as
    soon as I'm back.

    by LjL
    Lorenzo J. Lucchini, Oct 27, 2004
  5. I've uploaded a picture cleaned using my method.

    You can find the original slide as scanned at

    The cleaned picture is at

    The "noise map" as scanned (lid open, little ambient light) is at

    There is post-processed version of the noise map, which is the one
    actually used as the alpha channel, at

    All the scans were made at 1200 dpi, 24-bit color with an Epson RX500.
    The "image scan" and the "noise scan" have been hand-aligned, since my
    scanner doesn't pick up the same image area in film mode as in flatbed
    The final images have been scaled down (Paint Shop Pro, "Pixel
    Resize") to be 800 pixels wide.

    As you can see, I didn't get the histogram stretching quite right for
    the "normalized noise map": I made the darkest 97% of pixels black,
    and the brightest 0.1% white, which seems to be definitely overkill. I
    need to experiment a bit more with this.

    You can see how I also applied a convolution (whatever a convolution
    is) to dilate the noise spots, in order to avoid the edges of the dust
    particles to show up in the final image.

    by LjL
    Lorenzo J. Lucchini, Oct 28, 2004
  6. Lorenzo J. Lucchini

    Ken Weitzel Guest

    Hi Lorenzo...

    It appeasts that slide_clean.jpg "cannot be shown because
    it contains errors"

    Perhaps upload it again?

    Ken Weitzel, Oct 28, 2004
  7. Strange, it works here, and both slide_original.jpg and
    slide_clean.jpg were converted to JPEG using PSP.
    I'll convert it again using NetPBM, err, I mean, I'll do it after The
    Simpsons :)
    Also expect a slide_original.png and a slide_clean.png, but please
    don't download these massively since my bandwidth is limited.

    by LjL
    Lorenzo J. Lucchini, Oct 28, 2004
  8. I've succesfully opened both files with Opera and MSIE on two
    computers. I guess you simply accessed them while sitecopy was
    updating the site, or while something else weird was happening on the
    web server.
    Anyway, I've now uploaded a slide_clean_b.jpg made with NetPBM.
    I didn't upload these ones, because my web server apparently forbids
    files bigger than one megabyte.

    by LjL
    Lorenzo J. Lucchini, Oct 28, 2004
  9. Lorenzo J. Lucchini

    Don Guest

    I really like your lateral thinking (well done!!!). Unfortunately,
    using the flatbed to scan first as transparency and then laying the
    film on the scanner and scanning as reflective introduces far too many
    problems to be effective.

    Besides the obvious such as flaws (both scratches and dust) on top of
    the film which will not be detected in the reflective scan, there are
    other far more serious problems, in particular alignment.

    Regarding flaws within the film itself (e.g. scratches) the alignment
    between the two scans will be way off on both axis. Indeed, there are
    bound to be all sorts of spatial distortions and I would expect the
    two images to be of totally different sizes making it impossible to
    accurately superimpose in order to identify the flaws.

    Regarding surface particles (e.g. dust), once you've moved the film
    you've dislodged some dust and introduced other. Therefore, surface
    debris between the two scans will not correspond anymore. In other
    words, you'd be "cleaning" dust which doesn't exist and, yet, leaving
    dust which does.

    So in both instances (internal and external flaws) the alignment,
    which is the cornerstone of the method, will itself be flawed.

    Nevertheless, I really must commend you again on creative thinking!

    Don, Oct 28, 2004
  10. Lorenzo J. Lucchini

    Mendel Leisk Guest

    Lorenzo, your first jpeg link appears to be to a maliscous site...

    Any idea why this should be???
    Mendel Leisk, Oct 29, 2004
  11. I still am not sure of this one. If dust (or scratches, I just say
    dust because it's one word) is on top of a transparent region of the
    film, why do you say it won't be detected? I suppose it will undergo
    some dimming, but that should be all, or should it?
    I'm afraid you're thinking about a different kind of scanner than the
    one I'm using. I don't move the film at all between the two scans - I
    just leave it on the glass, held by means of the film holder provided
    with my scanner.

    The only mechanical operation I do between the two scans is opening
    the lid for the reflective scan (or closing it for the transmittive
    scan, if I do the reflective scan first), so that as little light as
    possible is reflected down.
    Of course, slamming the lid down is not a good idea since that
    obviously *will* move something - but you just have to move it down
    It does almost correspond on my scanner. Mind you, I had to do an
    (extremely boring) manual alignment between two test scans in order to
    measure the shift my scanner introduces between reflective scanning
    and transparency scanning.
    So now I know that every transparency scan will be x=-2, y=13 pixels
    off every reflective scan, and I just have to adjust every scan
    accordingly (of course I've written a script to do that).

    Now, it's true that there *will* be some alignment problems between
    any two scans, since the motor stepping can't be 100% accurate.
    However, if it's good enough for VueScan to offer a multi-scan option,
    it can't be too bad for my purposes.

    In any case the scripts I've written dilate every noise spot found in
    the reflective scan so that these alignment problems should go
    unnoticed. Clearly, this way you lose more image data that you must,
    since the interpolated region will be somewhat larger than exactly the
    size of the dust particle / scratch.
    It all depends on whether that's a price one feels like paying.
    While I make the reflective scan, some "new" dust will deposit on the
    film, since I'm keeping the lid open. But I hope it won't affect
    things too much...
    .... on some scanners that work in a different way than mine. Perhaps
    you're thinking of that kind of flatbed+film scanners where you place
    the film in a dedicated "drawer" just below the scanner's glass?

    At least, if I understood your critics correctly, my method should
    still work for scanners that don't require you to put the film in a
    different place than you put normal paper.
    Thanks :) You see, when I buy something and then discover that more
    expensive devices have more features (duh!), I then feel I *must*
    somehow replicate those features - although using LEGO bricks is only
    one option (I mean, this RX500 thing doesn't have automatic power
    on/off! I thought every printer, scanner, or printer+scanner made
    after 2000 had that feature! What can I do now except connecting a
    LEGO-made kludge with a LEGO motor to the RS323 port, and make it push
    the Power button on request?).

    But seriously, I'm not claiming that my method works as well as real
    Digital ICE, nor that it will remove every single defect from a
    picture, nor that it will not try to correct some defects that aren't
    really there.
    I'm just suggesting that, at least on scanners that work like mine, it
    might be more effective than a software-only filter, with less risk of
    removing real image detail.

    Besides, I haven't tried it yet on negatives, since I must first write
    a script that crops negatives in the right place, and that's harder
    and more boring than with slides, so it's very possible that it won't
    work on negatives very well at all.
    By the way, a fluorescent lamp like you find them in scanners doesn't
    emit much infrared, right? If it did, I suppose one could place an
    infrared-pass filtering material between the lamp and the film,
    thereby doing the same thing Digital ICE does. That's assuming a
    scanner's CCD is sensitive to infrared light, and it looks like mine
    is (I scanned a remote control's emitter with a button pressed -
    definitely seems to pick up infrared).

    by LjL
    Lorenzo J. Lucchini, Oct 29, 2004
  12. No. Which link are you referring to?
    In case I've mispelled something, the working links are

    All they do here is load the respective pictures, with JavaScript both
    enabled and disabled.

    I've noticed that 150m.com *sometimes* opens a pop-up window, but
    usually not. If that popup window contains malicious code, I don't
    I see it connects to 0catch.com when it has to show a 404.

    Anyway, 150m.com is just one of those free web providers that filling
    your mailbox with spam when you sign for an account, whose policies
    are dubious at least.

    If I could host the files on my own machine, I would, but I can't - I
    don't even have a public IP address.

    Please avoid using your browser to load those pictures, and just use
    wget or your favorite downloader to avoid unpleasant surprises.
    If the pictures don't load, try later - the files *are* there and the
    links I posted (at least the ones in this article) are correct.

    Sorry for any inconvenience, I just don't have a better way to publish
    files on the net.

    by LjL
    Lorenzo J. Lucchini, Oct 29, 2004
  13. The film holder bundled with my scanner looks just about firm enough.
    On the other hand, putting glass over the film is something I, too, was
    thinking of... not only to keep the film better in place, but for a much
    more compelling reason: focus.
    As I understand it, my scanner has focus fixed on the glass. However, the
    film holder keeps the film something like half a millimeter above the
    glass surface, and unless I'm missing something, this would definitely
    compromise (compromit?) correct focus.
    Besides, films from one-hour photo development often comes back less than
    flat, with the section looking like this:

    _ _
    __ __
    ____ ____
    ________ ________

    (although the effect in ASCII art is definitely exaggerated).

    I bet focus doesn't benefit from this.

    However, putting glass on the film means that the film comes in contact
    with both the scanner glass and the newly added glass; I'm afraid friction
    could cause scratches on the film as soon as there is a dust particle
    moving... so I'm not sure this would be a good idea, at least for the
    film's health.

    by LjL
    Lorenzo J. Lucchini, Oct 29, 2004
  14. Lorenzo J. Lucchini

    Don Guest

    Because the reflective scan will generally only register what's on the
    underside of the film. Depending on film density some "stuff" on top
    may come through but it will be distorted and unreliable not to
    mention overwhelmed by the underside reflections. (Also, see next
    segment about focus.)

    BTW, I read below you actually open the lid for the reflective scan. I
    understand you're doing this to reduce the "transparency effect" but
    by the same token this will further reduce data from the top as the
    light from the lamp has nothing to bounce off of once it's gone
    through the film. Any reflections from the top scratches or dust
    itself will be very tiny and then further diffused by the film
    substrate by the time they reach the sensors. Finally, I expect that
    whatever weak data survives all that it will then be drowned by the
    strong reflection from the underside of the film.
    Ah, then you have a different problem: Focus!

    In general, flatbed scanners are "fixed focus" to the top of the
    glass. By placing the film holder on top you're moving the film away
    from the scanner's range.

    That's unimportant when you're performing a transparency scan because
    in that case it's the projected image on top of the glass that's being
    scanned and not the film surface.

    (Indeed, if you want to improve the resolution, you may wish to use a
    slide projector and project the image onto the glass to fill the whole
    scan area! Some scanners actually come with a holder to keep the
    scanner on its side. In my case this was undocumented and I was
    puzzled for weeks what that funny looking part was for! :)
    Failing that you can open the scanner lid at a 90 degree angle and
    have the scanner rest on the lid. Since it's hard to focus on the
    glass you may wish to place a sheet of paper on the glass to help with
    the focus and then remove it before scanning. I haven't actually tried
    all this, but it should work.)

    However, when you do a reflective scan, this time it's the bounce from
    the film itself that gets registered. And since the holder makes the
    film "float" several millimeters above the glass it will be out of

    Now then, flatbeds generally have a certain amount of depth of field
    (except some, like Canon Lido series, which use a different
    technology). However, for such finicky work like removing dust (and
    scratches) I fear this inaccuracy will impact the algorithm.

    I guess, you could improve this considerably by not using the film
    holder at all and placing the film directly on the glass.
    Unfortunately, at this level of accuracy no matter how careful you are
    there will be movement.

    But more importantly, any time you do multi-pass multi-scanning (which
    is in effect what you are doing) there will be registration problems.
    Even if the film was perfectly in place the stepper motor will never
    be in the same place as on the previous scan. Furthermore, again due
    to scanner mechanics, the sensor array is bound to move laterally as
    well. The end result will be misalignment on both axis.

    You can check this by simply scanning an image twice. You'll notice
    that even though nothing has changed between the two scans there will
    be major misalignment. When I got my flatbed this was a big revelation
    to me! I then kept reducing the resolution expecting that at some
    point the images will be in sync but even at the lowest resolution of
    50 they were still off!

    Actually, I've been wrestling with multi-pass multi-scan image
    alignment in a different context (film scanner) where the slide
    remains in the scanner between the scans and - still - there is major
    misalignment. What's worse this is on sub-pixel level, so I have to do
    sub-pixel alignment before I can merge the two scans, which opens up a
    whole new can of worms...

    BTW, on my flatbed I don't have to open the lid to switch between
    reflective and transparency scans. The only thing I have to do is
    connect the cable that provides the power to the light source in the
    lid, as well as indicate that the lamp in the scanner should be turned
    off. Couldn't you just do that? It would improve things quite a lot
    because no matter how careful you are, you are bound to move the
    holder if you open the lid. Not to mention that by opening the lid
    you're not pressing on the holder anymore so it's bound to "float"
    even further from the glass.
    I fear that, for the reasons outlined above, this will actually differ
    with each scan. Furthermore, there will also be sub-pixel

    Here's a little trick/workflow I use in Photoshop to check the
    (mis)alignment of two images:

    1. Click on the magnifying glass.
    2. Make sure "Resize Windows To Fit" checkmark is *ON*
    3. Load an image.
    4. Double-click the magnifying glass to get 100%
    5. Press Control/+ until the image is 300% or 400%
    6. Repeat for second image.
    7. Use Control/Tab to toggle between the images.

    NOTE: This is the only reliable way I found to actually have the two
    *windows* aligned. Any other method appears to skew the windows'

    To move the image within the window use Home, End, PageUp and
    PageDown, then toggle with Control/Tab and repeat for the other image
    to keep them in sync.
    Ah... VueScan... Hmmm... I'm not a fan, to say the least... ;o) Far
    too buggy for my taste.

    As I like to say, VueScan's multi-pass multi-scan option is a very
    time consuming and complicated way to blur an image... ;o)
    That's exactly it! What you perceive as spot removal is really loss of
    data. You could probably achieve a very similar effect but simply
    applying a small amount of Gaussian Blur.
    Well... Yes... But doing stuff like this is so much fun! ;o)
    I don't know the spectral characteristics of a garden variety flatbed
    lamp or CDD response but I wouldn't be surprised if some IR data is
    picked up. The problem is separating this IR data from the rest
    because that's what you need to perform dust removal. This may be
    possible by applying Fast Fourier Transformation to extract only
    certain (i.e., IR) frequencies but that's only a wild guess and way
    over my head...

    Don, Oct 29, 2004
  15. "No FT - no comment!" ;-)
    (For those unaware, the FT in that instance was Financial Times!)

    Seriously though, there is no need to FT to achieve this, just an IR
    filter - although I doubt that much IR is emitted from the light source.
    I had hoped to check this for you, but having looked in my filter box I
    seem to have mislaid my Kodak No.87 - and since I haven't used it for
    years, I probably won't bother replacing it.

    In any case, with an IR filter in place you will probably get some
    response in all colours, so just make a grey mask.
    Kennedy McEwen, Oct 29, 2004
  16. Lorenzo J. Lucchini

    Tetractys Guest

    PMFBI, and I haven't posted here before, but ...
    two things:

    1) Parallax may enter into the equation, if one scan is reflected
    from one side of the film, and the other is "through." Although
    tiny, there may be a noticeable difference in scanned image
    size -- depending on pixel density, possibly enough to affect
    your final result beyond the intended effect.

    2) Often, there is crud under the scanner glass which is difficult
    to clean. Every couple of years, I take my scanners apart to
    clean off dust which has gotten inside the works, and most
    especially the plasticizer residue sublimated off the wiring and
    electronics. While out of the focal plane, probably enough not
    to matter, it's something to consider when you're getting down
    to the (admirable!) level of detail you're chasing. BTW, all
    of the scanners I have require breaking of seals or use of Torx
    drivers and so on to get into the case. You also have to be
    very careful of ribbon cables, tiny boards, optics, plugs, etc.
    Not for the faint-hearted, but IMHO worth the time and risk
    to end up with a pristine imaging path. (I've never damaged
    a scanner doing this, but I have experience in this kind of thing.)
    Tetractys, Oct 30, 2004
  17. Lorenzo J. Lucchini

    Don Guest

    That's a good idea, although he'd then have to lift the lid in order
    to insert the filter thereby causing misalignment problems as the
    slide is bound to move. I guess a gelatinous IR filter (if such a
    thing exists) would be even better in practical terms.

    So, using Financial Times ;o) to extract only the IR component may be
    easier in the end.

    Don, Oct 30, 2004
  18. Lorenzo J. Lucchini

    Don Guest

    Not at all! The more the merrier!
    That's a very good point. Parallax is bound to contribute to his
    misalignment problems.
    I recently did this on my flatbed which is only months old but I
    already noticed the residue on the underside of the glass. In my case
    no seals needed breaking I just lifted two plastic screw protectors
    alongside the lid hinge to reveal the screws (took a while to figure
    it out!). I then lifted the back and the front just slid out.
    Furthermore, the lid came off together with the glass making it much
    easier to handle. I just turn the whole assembly over. All the
    electronics as well as the lamp assembly stay in the base which is
    very handy.

    Ever since, I've been doing this every couple of weeks because the
    residue keeps building up. The longer the scanner is on the more
    apparent it gets as, presumably, the temperature rises and evaporation

    BTW, what do you use to clean the glass? I use lens cleaner and lens
    paper but this still seems to leave a thin film/smudges behind no
    matter how thorough I am. I even tried to polish this off using a
    microfiber cloth but just can't seem to get it all off. I'm starting
    to get the feeling I'm just redistributing it.

    Don, Oct 30, 2004
  19. Lorenzo J. Lucchini

    Tetractys Guest

    Windex with ammonia works pretty well.
    Lenscrafters sells an alcohol-based glasses
    cleaner that works also. To wipe, I use
    lint-free lab wipes. You're quite right, though;
    that last bit of residue is really tough, and it
    takes multiple passes with solvents. I use lots
    of liquid and big wipes.
    Tetractys, Oct 30, 2004
  20. Lorenzo J. Lucchini

    Don Guest

    I was afraid to use household cleaners because I wasn't sure what
    effect that will have on the glass. I mean, for all I know, there may
    be some special coating that might be affected by the chemicals.

    But if this is indeed regular glass (although made with better
    tolerances, of course) I'll give it a squirt of Windex and see how it

    BTW, I'm probably overdoing it anyway because the smudges are very
    faint. In order to even see them I have to look at the glass almost
    parallel to the scanner (looking from above, I don't see any smudges).

    A good test is to do a scan with the lid open in a darkened room
    (nothing on the glass). This will show all the dust/smudges, etc.
    However, to actually see them, the image must be brightened radically.
    And I mean radically! Since that amount of extreme brightening will
    never be needed for real images, these last few imperfections will be
    drowned by image data and virtually invisible.

    Don, Oct 31, 2004
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.