Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Re: FileChannel.map() gives 'cannot allocate memory'

Reply
Thread Tools

Re: FileChannel.map() gives 'cannot allocate memory'

 
 
Roedy Green
Guest
Posts: n/a
 
      08-13-2003
On Wed, 13 Aug 2003 15:38:37 -0400, Ian Soboroff <(E-Mail Removed)>
wrote or quoted :

> But mapping 1MB
>'pages' causes me to get an IOException from the native map0() call
>saying 'cannot allocate memory' at some random point in my program.


The JVM has only a 2 gig address space all told. I would hazard a
guess that it allocates virtual RAM for whacking huge native map
chunks say 10+ MB in a way different from small ones. I would guess
it allocates outside the heap for huge ones, and uses the standard gc
heap for small ones.


Allocating a 1 MB contiguous object is going to tax the limits of the
JVM. Try giving it some extra virtual ram on your java.exe command
line. See http://mindprod.com/jgloss/javaexe.html

--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
 
Reply With Quote
 
 
 
 
Ian Soboroff
Guest
Posts: n/a
 
      08-13-2003
Roedy Green <(E-Mail Removed)> writes:

> Allocating a 1 MB contiguous object is going to tax the limits of the
> JVM. Try giving it some extra virtual ram on your java.exe command
> line. See http://mindprod.com/jgloss/javaexe.html


Try running the sample code a few times. It allocates a 1MB map just
fine. Allocate and re-allocate over and over and at some point the VM
says it can't allocate memory for it.

Note the program doesn't keep the old map.

But, trying your suggestion:

$ java -Xms256m -Xmx512m -cp . Test j2sdk-1_4_2-linux-i586.rpm 10000 1000000
Mapping /usr/local/src/Java/Sun/1.4/1.4.2/j2sdk-1_4_2-linux-i586.rpm
size of file: 34653233
size of page: 1000000
iterations : 10000
Exception occurred at count 1510 trying to map from 30000000 to 31000000
(1461 faults so far)
Exception in thread "main" java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.jav a:705)
at Test.main(Test.java:5


Ian
 
Reply With Quote
 
 
 
 
Ian Soboroff
Guest
Posts: n/a
 
      08-14-2003
Roedy Green <(E-Mail Removed)> writes:

> On Wed, 13 Aug 2003 17:18:38 -0400, Ian Soboroff <(E-Mail Removed)>
> wrote or quoted :
>
>>Try running the sample code a few times. It allocates a 1MB map just
>>fine. Allocate and re-allocate over and over and at some point the VM
>>says it can't allocate memory for it.

>
> What you want to find out is:
>
> 1. how much free ram is there at each stage. Is something packratting
> the old allocations?
>
> 2. Just before you die is the problem lack of ram or lack of
> contiguous ram?
>
> See http://mindprod.com/jgloss/packratting.html


Tricky to find out, but I'm running the VM on the test code with a
start size of 256MB and a max heap of 512MB. The file I'm mapping is
about 35MB, and I'm only mapping a 1MB region of it.

As for packratting, MappedByteBuffers don't have a dispose() method.
Setting the reference to null before remapping doesn't help. The
exception is thrown from a native method
(sun.nio.ch.FileChannelImpl.map0(Native Method)), so if memory leakage
is happening it's on the C level.

In fact, I'm pretty sure it's leakage in the native method side. If
I divide the map size by a factor of 10, but increase the number of
iterations by 10, I get a crash after 10x iterations (where x is the
number if iterations before I get an exception in the first case):

$ java -Xms256m -Xmx512m -cp . Test j2sdk-1_4_2-linux-i586.rpm 10000 1000000
Mapping j2sdk-1_4_2-linux-i586.rpm
size of file: 34653233
size of page: 1000000
iterations : 10000
Exception occurred at count 1500 trying to map from 0 to 1000000
(1460 faults so far)
Exception in thread "main" java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.jav a:705)
at Test.main(Test.java:59)

$ java -Xms256m -Xmx512m -cp . Test j2sdk-1_4_2-linux-i586.rpm 100000 100000
Mapping j2sdk-1_4_2-linux-i586.rpm
size of file: 34653233
size of page: 100000
iterations : 100000
Exception occurred at count 14071 trying to map from 22400000 to 22500000
(14036 faults so far)
Exception in thread "main" java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.jav a:705)
at Test.main(Test.java:59)

It's something related to map overhead, but I would have expected that
to be constant per map setup/teardown, rather than related to the size
of the mapping.

Ian
 
Reply With Quote
 
Roedy Green
Guest
Posts: n/a
 
      08-14-2003
On Thu, 14 Aug 2003 08:29:00 -0400, Ian Soboroff <(E-Mail Removed)>
wrote or quoted :

>
>It's something related to map overhead, but I would have expected that
>to be constant per map setup/teardown, rather than related to the size
>of the mapping.


Look up http://mindprod.com/jgloss/profiler.html

You need a tool that will let you find out what flavours of objects
were gobbling ram just before you die. That may be a clue if it is
your fault or Sun's.

I get the impression there may be some JNI involved. Is it possible
to temporarily turn it off or roughly simulate it with some Java code.

--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Cant get router (DHCP) to allocate ip =?Utf-8?B?bWFyYw==?= Wireless Networking 27 12-28-2005 04:06 PM
how to remove line "no scheduler allocate" Michael Cisco 1 01-21-2005 05:53 PM
How to allocate an equal share of incoming web traffic to every user Anton Panyushkin Cisco 4 11-23-2004 02:41 PM
What does "no scheduler allocate" mean? Steve Gross Cisco 1 08-11-2004 11:36 PM
Re: more than 10 seconds to allocate an array! Thomas G. Marshall Java 2 08-28-2003 07:30 PM



Advertisments