Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly later versions) loading large interlaced PNGs with low memory
 

Using Java 1.6.0_13 -server -Xmx1100M what do you get if you run this code?

import java.lang.ref.SoftReference;
import java.io.File;
import javax.imageio.ImageIO;

public class Crash {
public static void main (char[] args) {
File f = new File("path-to-any-24-megapixel-RGB-PNG-goes-here");
SoftReference a = new SoftReference(ImageIO.read(f));
SoftReference b = new SoftReference(ImageIO.read(f));
SoftReference c = new SoftReference(ImageIO.read(f));
SoftReference d = new SoftReference(ImageIO.read(f));
SoftReference e = new SoftReference(ImageIO.read(f));
System.out.println("" + (a.get() == null) + (b.get() == null)
+ (c.get() == null) + (d.get() == null) + (e.get() == null));
}
}

It should be easy for any of you with a digital camera to adapt this --
just change the filename string to point to a 24-megapixel image you have
laying around. Failing that, there's one linked at the bottom left of
http://aerialphotographysandiego.com...san-diego.html

The above will work fine with jpegs and noninterlaced pngs, reporting
falsefalsefalsefalsefalse if you have more than a few hundred megs of mem
and the -server VM. Convert the image to an interlaced png and point the
above at the png, though, and it seems to behave as if System.exit was
called, at least on my system, which is clearly incorrect behavior. (I
tested it with the file from that link, converted to interlaced png with
Photoshop CS2, in case that somehow makes a difference -- with a decoder
bug, who knows? With the png created as described, it crashes with five
copies loaded, but not with four.)

Curiously, this change seems to prevent it:

import java.lang.ref.SoftReference;
import java.io.File;
import javax.imageio.ImageIO;

public class Crash {
public static void main (char[] args) {
File f = new File("path-to-a-24-megapixel-RGB-PNG-goes-here");
SoftReference a = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference b = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference c = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference d = new SoftReference(ImageIO.read(f));
System.gc();
SoftReference e = new SoftReference(ImageIO.read(f));
System.out.println("" + (a.get() == null) + (b.get() == null)
+ (c.get() == null) + (d.get() == null) + (e.get() == null));
}
}

That's clearly buggy, because System.gc() added or removed is not supposed
to alter program semantics, only maybe performance; PLUS if it was running
out of memory some SoftReferences should have been cleared to make more
room without anything else in the way of consequences; PLUS if it somehow
ran out of memory anyway it should have thrown an OOME rather than
pretended the code called System.exit.

As near as I can tell from this, the ImageIO png decoder in Java 1.6.0_13
contains a crash-inducing bug that requires the png it's decoding to be
interlaced *and* requires heap space to be running low to trigger it.

I'm curious to know what other Java versions reproduce this buggy behavior.

If it's present in 1.6.0_13 but absent in a later version, then obviously
I'd especially like to know that.

But I don't feel like going to a huge effort downloading a hundred megs of
later-Java-version, installing it, rebooting, fixing everything I'd need to
fix to make stuff use the later version, fixing broken links because the
binary pathname changed, and so forth, only to find out that the bug's
still there in the current version. :) So I'd like confirmation that it's
gone in some later version before I spend an hour or two of my life on such
a task.

Re: Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly later versions) loading large interlaced PNGs with low memory
 

On Sun, 11 Nov 2012 23:36:09 -0500, dy/dx wrote:

> Using Java 1.6.0_13 -server -Xmx1100M what do you get if you run this code?
>
> import java.lang.ref.SoftReference;
> import java.io.File;
> import javax.imageio.ImageIO;
>
> public class Crash {
> public static void main (char[] args) {
> File f = new File("path-to-any-24-megapixel-RGB-PNG-goes-here");
> SoftReference a = new SoftReference(ImageIO.read(f));
> SoftReference b = new SoftReference(ImageIO.read(f));
> SoftReference c = new SoftReference(ImageIO.read(f));
> SoftReference d = new SoftReference(ImageIO.read(f));
> SoftReference e = new SoftReference(ImageIO.read(f));
> System.out.println("" + (a.get() == null) + (b.get() == null)
> + (c.get() == null) + (d.get() == null) + (e.get() == null));
> }
> }
>
> It should be easy for any of you with a digital camera to adapt this --
> just change the filename string to point to a 24-megapixel image you have
> laying around. Failing that, there's one linked at the bottom left of
> http://aerialphotographysandiego.com...san-diego.html
>
> The above will work fine with jpegs and noninterlaced pngs, reporting
> falsefalsefalsefalsefalse if you have more than a few hundred megs of mem
> and the -server VM. Convert the image to an interlaced png and point the
> above at the png, though, and it seems to behave as if System.exit was
> called, at least on my system, which is clearly incorrect behavior. (I
> tested it with the file from that link, converted to interlaced png with
> Photoshop CS2, in case that somehow makes a difference -- with a decoder
> bug, who knows? With the png created as described, it crashes with five
> copies loaded, but not with four.)
>
> Curiously, this change seems to prevent it:
>
> import java.lang.ref.SoftReference;
> import java.io.File;
> import javax.imageio.ImageIO;
>
> public class Crash {
> public static void main (char[] args) {
> File f = new File("path-to-a-24-megapixel-RGB-PNG-goes-here");
> SoftReference a = new SoftReference(ImageIO.read(f));
> System.gc();
> SoftReference b = new SoftReference(ImageIO.read(f));
> System.gc();
> SoftReference c = new SoftReference(ImageIO.read(f));
> System.gc();
> SoftReference d = new SoftReference(ImageIO.read(f));
> System.gc();
> SoftReference e = new SoftReference(ImageIO.read(f));
> System.out.println("" + (a.get() == null) + (b.get() == null)
> + (c.get() == null) + (d.get() == null) + (e.get() == null));
> }
> }
>
> That's clearly buggy, because System.gc() added or removed is not supposed
> to alter program semantics, only maybe performance; PLUS if it was running
> out of memory some SoftReferences should have been cleared to make more
> room without anything else in the way of consequences; PLUS if it somehow
> ran out of memory anyway it should have thrown an OOME rather than
> pretended the code called System.exit.
>
> As near as I can tell from this, the ImageIO png decoder in Java 1.6.0_13
> contains a crash-inducing bug that requires the png it's decoding to be
> interlaced *and* requires heap space to be running low to trigger it.

Addendum: if the png is *either* interlaced *or* 32bpp (alpha channel) that
seems to suffice. Encoding a problem png in Photoshop as a 24bpp
non-interlaced png seems to make it "clean", i.e. non-bug-triggering for
Java use. In Photoshop CS2 that involves "flatten image" and then saving to
another directory and choosing a "none" radio button on a save options
popup. YMMV with other Photoshop versions -- you're probably all using CS4
or later. :)

Similarly, taking a non-troublesome png (or non-png) and reencoding it as a
png that's interlaced or 32bpp seems to make it crash ImageIO's decoder
*if* the heap space is low enough at the time of decoding. In particular it
makes the above code exhibit the crash. The size of the png matters, at
least insofar as how quickly the above code gets the heap space low enough
to enable the bug to strike. I pngcrushed a problem png and the number of
loads I could have without a crash went up from 3 to 5; pngcrush reported a
27% reduction in size. 5*0.73 = 3.65 so the bug enabling threshold was
somewhere between 3*original size and 3.65*original size with that png.
Moreover this was the *same image*; the BufferedImage object would have
been about 72 megs and identical down to the last byte for both cases.

So it's not the BufferedImage alone, it's also whatever temporary objects
the decoder makes that affect the bug on subsequent decodes, through their
lingering memory use as uncollected-as-yet garbage or some other mechanism,
and this effect is proportional to the problem png's file size, not its
uncompressed size, pointing to data structures created early in the
decoding -- likely, the byte arrays holding successive chunks of the file
itself.

Changing the decoder to recycle one array instead of constantly making and
discarding them might "fix" the bug, then, though it would really only be
working around it. I'd have to guess that ImageIO's png decoder contains
native code, and that native code does something to allocate memory on the
Java heap for something, likely the output's WritableRaster, in a way that
bypasses some safeguards. In particular, perhaps it doesn't check for heap
exhaustion, run a stop-the-world collection, try again, and then throw OOME
on failure like a normal allocation in non-native code, and some idiot put
if (buff == NULL) { /* Can't happen */ exit(0); } or something similar. In
any event, the bug should be found and fixed, if it hasn't been already,
and not simply papered over by finding a way to avoid as easily triggering
it. It would just end up happening with even
larger-but-should-still-fit-in-the-heap-space pngs, or even with smaller
pngs with big enough other data structures lying about.

Re: Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly later versions) loading large interlaced PNGs with low memory
 

On Mon, 12 Nov 2012 00:40:18 -0500, dy/dx wrote:

> I pngcrushed a problem png and the number of loads I could have without a
> crash went up from 3 to 5; pngcrush reported a 27% reduction in size.
> 5*0.73 = 3.65 so the bug enabling threshold was somewhere between 3*original
> size and 3.65*original size with that png.

Bah. Late night. 4*0.73 = 2.92 (no crash) while it crashed at 3, so the
threshold was between 2.92*original size and 3*original size -- a pretty
narrow range. That file was about 10MB on disk before crushing and about
7.3MB afterward, so between 29.2 and 30 megs of interlaced-or-32-bpp png
lies the triggering threshold, at least in my system's case. Again, that
might vary even on systems that have the bug: try replacing the individual
SoftReference variable initializers and println with something like List a
= new ArrayList(); for (int i = 0; i < 50; i++) { System.out.println("" +
i); a.add(ImageIO.read(f)); } and run it. If you get OOME, the bug isn't
happening for you; if Java just exits, it is.

Re: Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly laterversions) loading large interlaced PNGs with low memory
 

On 11/11/2012 9:50 PM, dy/dx wrote:
> On Mon, 12 Nov 2012 00:40:18 -0500, dy/dx wrote:
>
>> I pngcrushed a problem png and the number of loads I could have without a
>> crash went up from 3 to 5; pngcrush reported a 27% reduction in size.
>> 5*0.73 = 3.65 so the bug enabling threshold was somewhere between 3*original
>> size and 3.65*original size with that png.
>
> Bah. Late night. 4*0.73 = 2.92 (no crash) while it crashed at 3, so the
> threshold was between 2.92*original size and 3*original size -- a pretty
> narrow range. That file was about 10MB on disk before crushing and about
> 7.3MB afterward, so between 29.2 and 30 megs of interlaced-or-32-bpp png
> lies the triggering threshold, at least in my system's case. Again, that
> might vary even on systems that have the bug: try replacing the individual
> SoftReference variable initializers and println with something like List a
> = new ArrayList(); for (int i = 0; i < 50; i++) { System.out.println("" +
> i); a.add(ImageIO.read(f)); } and run it. If you get OOME, the bug isn't
> happening for you; if Java just exits, it is.
>


Could you load the offending files on a photo sharing service? I'd like
to check them out.

Re: Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly laterversions) loading large interlaced PNGs with low memory
 

In <k7pubs$rff$1@news.mixmin.net> dy/dx wrote:

> Using Java 1.6.0_13 -server -Xmx1100M what do you get if you run this code?
>
> [...] I don't feel like going to a huge effort downloading a hundred megs of
> later-Java-version, installing it, rebooting, fixing everything I'd need to
> fix to make stuff use the later version, fixing broken links because the
> binary pathname changed, and so forth, only to find out that the bug's
> still there in the current version. :)

Please confirm that your internet connection is a 110 baud telex line on
Antarctica, and that you need to copy every byte by hand from the telprinter
paper to your terminal prompt to write them to disk. I'd be happy to assist if
I knew downloading 70 - not a couple of hundred - megabytes was a substantial
chore for you.

Also, tell me you're not running a 6u13 based service that exposed to the
internet? There are remotely triggerable DOS issues that has been resolved
since u13. The latest patch release is update 37, that's a whopping 24
security and stability improving patch releases ahead of your environment.

Besides, if your source code, build system, and service configuration is so
fragile it requires several hours of work just to upgrade the JDK, I recommend
that you take some time to fix that. Changing your $PATH and $JAVA_HOME
variables shouldn't be that much work. And while you're at it, consider
upgrading to JDK 7, as JDK 6 (non-for-pay) has a scheduled EOL in November 2012.

http://www.oracle.com/technetwork/java/eol-135779.html
https://blogs.oracle.com/henrik/entr...ava_6_eol_date

--
Fredrik Jonson

Re: Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly laterversions) loading large interlaced PNGs with low memory
 

Fredrik Jonson wrote:

> And while you're at it, consider upgrading to JDK 7, as JDK 6
> (non-for-pay) has a scheduled EOL in November 2012.
>
> http://www.oracle.com/technetwork/java/eol-135779.html
> https://blogs.oracle.com/henrik/entr...ava_6_eol_date

I apologise. You've got another three month of time to upgrade to
JDK 7. Public releases of JDK 6 wont go away until February 2013.

--
Fredrik Jonson

Re: Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly later versions) loading large interlaced PNGs with low memory
 

On Sun, 11 Nov 2012 22:35:55 -0800, markspace wrote:

> On 11/11/2012 9:50 PM, dy/dx wrote:
>> On Mon, 12 Nov 2012 00:40:18 -0500, dy/dx wrote:
>>
>>> I pngcrushed a problem png and the number of loads I could have without a
>>> crash went up from 3 to 5; pngcrush reported a 27% reduction in size.
>>> 5*0.73 = 3.65 so the bug enabling threshold was somewhere between 3*original
>>> size and 3.65*original size with that png.
>>
>> Bah. Late night. 4*0.73 = 2.92 (no crash) while it crashed at 3, so the
>> threshold was between 2.92*original size and 3*original size -- a pretty
>> narrow range. That file was about 10MB on disk before crushing and about
>> 7.3MB afterward, so between 29.2 and 30 megs of interlaced-or-32-bpp png
>> lies the triggering threshold, at least in my system's case. Again, that
>> might vary even on systems that have the bug: try replacing the individual
>> SoftReference variable initializers and println with something like List a
>> = new ArrayList(); for (int i = 0; i < 50; i++) { System.out.println("" +
>> i); a.add(ImageIO.read(f)); } and run it. If you get OOME, the bug isn't
>> happening for you; if Java just exits, it is.
>>
>
>
> Could you load the offending files on a photo sharing service? I'd like
> to check them out.

I already provided an exact recipe for creating a problem png: download the
24-megapixel image linked from

http://aerialphotographysandiego.com...san-diego.html

and use Photoshop to create a copy that is an interlaced png. (As long as
you don't /distribute/ the copy, it shouldn't be copyright infringement, as
private format-shifting of copyrighted content has been found legal. But
I'm not about to risk getting sued by uploading the results to imageshack
or wherever, and the other problem pngs I have are part of some
confidential work, so...)

Re: Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly later versions) loading large interlaced PNGs with low memory
 

On 12 Nov 2012 09:37:16 GMT, Fredrik Jonson wrote:

> In <k7pubs$rff$1@news.mixmin.net> dy/dx wrote:
>
>> Using Java 1.6.0_13 -server -Xmx1100M what do you get if you run this code?
>>
>> [...] I don't feel like going to a huge effort downloading a hundred megs of
>> later-Java-version, installing it, rebooting, fixing everything I'd need to
>> fix to make stuff use the later version, fixing broken links because the
>> binary pathname changed, and so forth, only to find out that the bug's
>> still there in the current version. :)
>
> Please confirm that your internet connection is a 110 baud telex line on
> Antarctica, and that you need to copy every byte by hand from the telprinter
> paper to your terminal prompt to write them to disk. I'd be happy to assist if
> I knew downloading 70 - not a couple of hundred - megabytes was a substantial
> chore for you.
>
> Also, tell me you're not running a 6u13 based service that exposed to the
> internet?

Nope. Private development machine. And we are making desktop apps --
shocker, I know.

> Besides, if your source code, build system, and service configuration is so
> fragile it requires several hours of work just to upgrade the JDK, I recommend
> that you take some time to fix that. Changing your $PATH and $JAVA_HOME
> variables shouldn't be that much work. And while you're at it, consider
> upgrading to JDK 7, as JDK 6 (non-for-pay) has a scheduled EOL in November 2012.
>
> http://www.oracle.com/technetwork/java/eol-135779.html
> https://blogs.oracle.com/henrik/entr...ava_6_eol_date

Nothing is ever just straightforward plug-and-play, whatever is advertised.
Simply downloading and running an installer for JDK 7 will not be
sufficient. Either stuff will just chug along merrily using 1.6.0_13 or
stuff will break. It happened before when our shop finally updated to Java
6 from Java 1.3, a few years ago. Without a compelling reason it just
doesn't seem worth the hassle.

Re: Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly later versions) loading large interlaced PNGs with low memory
 

On Mon, 12 Nov 2012 11:30:27 -0500, dy/dx wrote:

> I already provided an exact recipe for creating a problem png: download the
> 24-megapixel image linked from
>
> http://aerialphotographysandiego.com...san-diego.html
>
> and use Photoshop to create a copy that is an interlaced png.

You think people will buy (or pirate) a $700 product because you're too
lazy to find an example image for the problem you want people to spend
their time on for you ?

Good luck with that ;-)

Liebe Gruesse,
Joerg

--
Ich lese meine Emails nicht, replies to Email bleiben also leider
ungelesen.

Re: Crash in Java 1.6.0_13 ImageIO PNG decoder (and possibly later versions) loading large interlaced PNGs with low memory
 

On Mon, 12 Nov 2012 18:20:51 +0100, Joerg Meier wrote:

> On Mon, 12 Nov 2012 11:30:27 -0500, dy/dx wrote:
>
>> I already provided an exact recipe for creating a problem png: download the
>> 24-megapixel image linked from
>>
>> http://aerialphotographysandiego.com...san-diego.html
>>
>> and use Photoshop to create a copy that is an interlaced png.
>
> You think people will buy (or pirate) a $700 product because you're too
> lazy to find an example image for the problem you want people to spend
> their time on for you ?
>
> Good luck with that ;-)
>
> Liebe Gruesse,
> Joerg

Who said anything about buying or pirating anything? I gave a recipe I knew
was guaranteed to make a problem png. I doubt very much it's the only one.
Surely you have access to image conversion tools that can make an
interlaced png from a jpg.