Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > MappedByteBuffer

Reply
Thread Tools

MappedByteBuffer

 
 
Alan Gutierrez
Guest
Posts: n/a
 
      07-23-2010
I'm implementing file storage for a B+Tree for deployment on Linux,
development on OS X, and wanted to use MappedByteBuffer. This seems like
an easy way to address large pages of data and not have implement
buffering myself.

I've done a little testing...

import java.io.IOException;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;
import java.nio.channels.FileChannel.MapMode;

public class MemoryMap {
public static void main(String[] args) throws IOException {
int count = Integer.parseInt(args[0]);
MappedByteBuffer[] mappedByteBuffers =
new MappedByteBuffer[count];
for (int i = 0; i < count; i++) {
mappedByteBuffers[i] = opened(i);
}
}

public static MappedByteBuffer opened(int count)
throws IOException {
int size = 1024 * 1024 * 256; // * 1024; // 1 GB
RandomAccessFile raf =
new RandomAccessFile("hello" + count + ".raf", "rw");
try {
raf.setLength(size);
FileChannel channel = raf.getChannel();
try {
MappedByteBuffer map
= channel.map(MapMode.READ_WRITE, 0, size);
channel.close();
map.put(size - 1, (byte) 7);
} finally {
channel.close();
}
} finally {
raf.close();
}
raf = new RandomAccessFile("hello" + count + ".raf", "rw");
try {
FileChannel channel = raf.getChannel();
try {
MappedByteBuffer map
= channel.map(MapMode.READ_WRITE, 0, size);
channel.close();
if (map.get(size - 1) != 7) {
throw new IllegalStateException();
}
return map;
} finally {
channel.close();
}
} finally {
raf.close();
}
}
}

On a smallish Linode running Ubuntu 10.4 I can create an array of
hundreds of MappedByteBuffer. Here are some timings.

[alan@maribor ~]$ time java MemoryMap 24

real 0m0.119s
user 0m0.067s
sys 0m0.029s
[alan@maribor ~]$ time java MemoryMap 512

real 0m0.225s
user 0m0.150s
sys 0m0.092s
[alan@maribor ~]$

On OS X running Java 1.5 I can create an array of 5, the first pass is
very slow, the second pass is faster, I assume this is because OS X
isn't being clever about creating the empty file.

[alan@postojna ~]$ time java MemoryMap 5

real 0m20.920s
user 0m0.089s
sys 0m1.267s
[alan@postojna ~]$ time java MemoryMap 5

real 0m0.301s
user 0m0.091s
sys 0m0.044s
[alan@postojna ~]$

When I attempt to create an array of 6 250 MB `MappedByteBuffer`s I get:

[alan@postojna ~]$ java MemoryMap 6
Exception in thread "main" java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.jav a:742)
at MemoryMap.opened(MemoryMap.java:23)
at MemoryMap.main(MemoryMap.java:12)
[alan@postojna ~]$

However:

[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 6

real 0m0.386s
user 0m0.354s
sys 0m0.087s
[alan@postojna ~]$

I've created up to 24 250MB buffers using Java 1.6 on OS X. It takes a
while to create them, but once created, performance seems reasonable.

[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 24

real 3m13.779s
user 0m0.358s
sys 0m5.543s
[alan@postojna ~]$ time
/System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home/bin/java
MemoryMap 24

real 0m1.646s
user 0m0.364s
sys 0m0.100s
[alan@postojna ~]$

I've tried the program running on Windows XP with JDK 1.6 running in
Virtual Box and I can't get an array of three pages.

All this leads me to the question: am I on the right track? It seems
like I'll be able to get the benefits I seek,

* paging managed by the operating system,
* a simple memory management strategy.

on Linux and OS X with JDK 1.6.

Or am I misunderstanding the usage of MappedByteBuffer?

--
Alan Gutierrez - http://www.velocityreviews.com/forums/(E-Mail Removed) - http://twitter.com/bigeasy
 
Reply With Quote
 
 
 
 
Kevin McMurtrie
Guest
Posts: n/a
 
      07-23-2010
In article <i2av27$9h7$(E-Mail Removed)-september.org>,
Alan Gutierrez <(E-Mail Removed)> wrote:

> I'm implementing file storage for a B+Tree for deployment on Linux,
> development on OS X, and wanted to use MappedByteBuffer. This seems like
> an easy way to address large pages of data and not have implement
> buffering myself.
>
> I've done a little testing...
>
> import java.io.IOException;
> import java.io.RandomAccessFile;
> import java.nio.MappedByteBuffer;

...
> Or am I misunderstanding the usage of MappedByteBuffer?



Java 1.5 is more commonly a 32 bit process while Java 1.6 is more
commonly a 64 bit process. You're running out of memory addresses in 32
bit mode.

Adding '-d64' as the first JVM option will force the 64 bit version or
fail with a message that it's not installed.
--
I won't see Google Groups replies because I must filter them as spam
 
Reply With Quote
 
 
 
 
Lew
Guest
Posts: n/a
 
      07-23-2010
Kevin McMurtrie wrote:
> Java 1.5 is more commonly a 32 bit process while Java 1.6 is more
> commonly a 64 bit process.


How do you figure that?

> You're running out of memory addresses in 32 bit mode.
>
> Adding '-d64' as the first JVM option will force the 64 bit version or
> fail with a message that it's not installed.


--
Lew
 
Reply With Quote
 
Alan Gutierrez
Guest
Posts: n/a
 
      07-23-2010
Kevin McMurtrie wrote:
> In article <i2av27$9h7$(E-Mail Removed)-september.org>,
> Alan Gutierrez <(E-Mail Removed)> wrote:
>
>> I'm implementing file storage for a B+Tree for deployment on Linux,
>> development on OS X, and wanted to use MappedByteBuffer. This seems like
>> an easy way to address large pages of data and not have implement
>> buffering myself.
>>
>> I've done a little testing...
>>
>> import java.io.IOException;
>> import java.io.RandomAccessFile;
>> import java.nio.MappedByteBuffer;

> ...
>> Or am I misunderstanding the usage of MappedByteBuffer?

>
>
> Java 1.5 is more commonly a 32 bit process while Java 1.6 is more
> commonly a 64 bit process. You're running out of memory addresses in 32
> bit mode.
>
> Adding '-d64' as the first JVM option will force the 64 bit version or
> fail with a message that it's not installed.


Adding -d64 to JDK 1.5 does allow it to address more than 6 pages. Thank
you.

Can you help me understand what is going on here? Is it that, Even
through `MappedByteBuffer` will page in and out of memory, you still
need a 64 bits to address entire region, and the addresses cannot overlap?

--
Alan Gutierrez - (E-Mail Removed) - http://twitter.com/bigeasy
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      07-23-2010
Kevin McMurtrie wrote:
> Adding '-d64' as the first JVM option will force the 64 bit version or
> fail with a message that it's not installed.
>


I see nothing in
<http://download.oracle.com/docs/cd/E...ocs/technotes/
tools/solaris/java.html>
that requires the "-d64" option to be the first option.

--
Lew
 
Reply With Quote
 
Alan Gutierrez
Guest
Posts: n/a
 
      07-23-2010
Lew wrote:
> Kevin McMurtrie wrote:
>> Adding '-d64' as the first JVM option will force the 64 bit version or
>> fail with a message that it's not installed.
>>

>
> I see nothing in
> <http://download.oracle.com/docs/cd/E...ocs/technotes/
> tools/solaris/java.html>
> that requires the "-d64" option to be the first option.


It works though:

[alan@postojna ~]$ java -d64 MemoryMap 6
[alan@postojna ~]$ java MemoryMap 6
Exception in thread "main" java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.jav a:742)
at MemoryMap.opened(MemoryMap.java:23)
at MemoryMap.main(MemoryMap.java:12)
[alan@postojna ~]$

--
Alan Gutierrez - (E-Mail Removed) - http://twitter.com/bigeasy
 
Reply With Quote
 
Nigel Wade
Guest
Posts: n/a
 
      07-23-2010
On 23/07/10 14:27, Alan Gutierrez wrote:
> Kevin McMurtrie wrote:
>> In article <i2av27$9h7$(E-Mail Removed)-september.org>,
>> Alan Gutierrez <(E-Mail Removed)> wrote:
>>
>>> I'm implementing file storage for a B+Tree for deployment on Linux,
>>> development on OS X, and wanted to use MappedByteBuffer. This seems
>>> like an easy way to address large pages of data and not have
>>> implement buffering myself.
>>>
>>> I've done a little testing...
>>>
>>> import java.io.IOException;
>>> import java.io.RandomAccessFile;
>>> import java.nio.MappedByteBuffer;

>> ...
>>> Or am I misunderstanding the usage of MappedByteBuffer?

>>
>>
>> Java 1.5 is more commonly a 32 bit process while Java 1.6 is more
>> commonly a 64 bit process. You're running out of memory addresses in
>> 32 bit mode.
>>
>> Adding '-d64' as the first JVM option will force the 64 bit version or
>> fail with a message that it's not installed.

>
> Adding -d64 to JDK 1.5 does allow it to address more than 6 pages. Thank
> you.
>
> Can you help me understand what is going on here? Is it that, Even
> through `MappedByteBuffer` will page in and out of memory, you still
> need a 64 bits to address entire region, and the addresses cannot overlap?
>


In the dim and distant past I did some work using memory mapped files in
C, and I presume that the mechanism behind Java memory mapped files is
similar. A memory mapped file is mapped onto a specific address in the
virtual memory address space of the application. Although they are not
actually occupying the memory there they still require a unique, and
contiguous, address space equal to the size of the memory mapped region.
So if you want to map a region of 256MB you need 256MB of contiguous
address space.

A value is read from a memory mapped region by accessing a particular
address in the virtual memory of the application e.g. if the file is
mapped at address 0x20000, and you want to read a value at offset 0x500
from that start of the file then the data at 0x20500 is read. This
address must be unique, so the mapped regions cannot overlap, they must
be contiguous, and all mapped regions must lie within the addressable
space of the application.

In a 32bit application your maximum address space is 4GB, and could be
a good deal less than that due to system dependent memory limitations.
Thus, your attempt to allocate multiple regions of that address space in
blocks of 256MB will have a limited amount of success, depending on what
contiguous regions of the address space are currently available.

When you run the 64bit version of the JVM the available address space
has increased by a factor of 1 billion. Therefore you are able to
allocate those regions of contiguous address space for your application,
even though there is still the same amount of physical RAM and system VM.

--
Nigel Wade
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      07-23-2010
Kevin McMurtrie wrote:
>>> Adding '-d64' as the first JVM option will force the 64 bit version or
>>> fail with a message that it's not installed.

>


Lew wrote:
>> I see nothing in
>> <http://download.oracle.com/docs/cd/E...ocs/technotes/
>> tools/solaris/java.html>
>> that requires the "-d64" option to be the first option.

>


Alan Gutierrez wrote:
> It works though:
>


Of course. That isn't the question.

--
Lew
 
Reply With Quote
 
Alan Gutierrez
Guest
Posts: n/a
 
      07-23-2010
Patricia Shanahan wrote:
> On 7/23/2010 10:42 AM, Peter Duniho wrote:
>> Nigel Wade wrote:
>>> [...]
>>> In a 32bit application your maximum address space is 4GB, and could be
>>> a good deal less than that due to system dependent memory limitations.
>>> Thus, your attempt to allocate multiple regions of that address space in
>>> blocks of 256MB will have a limited amount of success, depending on what
>>> contiguous regions of the address space are currently available.

>>
>> Indeed, it's worth pointing out that on Windows, the maximum address
>> space for Java programs is only 2GB, and really only about 1.5GB (give
>> or take) after various overhead involved in a Windows process and Java.
>> And of course, any interesting program will have to allocate other
>> memory objects, reducing that further (through both just the use of
>> memory and fragmentation of the virtual address space).

>
> I assume when you say "on Windows" you mean "on 32 bit Windows"?


I assume you're implying that 64 bit Windows has the same advantages as
64 bit JVM as noted by Nigel Wade?

--
Alan Gutierrez - (E-Mail Removed) - http://twitter.com/bigeasy
 
Reply With Quote
 
Arne Vajh°j
Guest
Posts: n/a
 
      07-23-2010
On 23-07-2010 01:30, Kevin McMurtrie wrote:
> Java 1.5 is more commonly a 32 bit process while Java 1.6 is more
> commonly a 64 bit process.


For Sun Java then:
Windows Linux Solaris not SPARC
1.6 latest both avilable both available both available
1.5 latest both avilable both available both available
1.4.2 latest only 32 bit only 32 bit only 32 bit

So I can not see why it should necessarily be that way.

Arne
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
IOException in MappedByteBuffer.force0 Me Java 9 01-09-2007 09:30 AM
MappedByteBuffer Claudio! Java 2 09-20-2006 03:38 PM
Maximum size of MappedByteBuffer in Java 5.0 64 bit The POWER of 2WO Java 2 12-02-2004 01:23 PM
Re: Reading lines of text from a MappedByteBuffer mromarkhan@rogers.com Java 0 05-03-2004 11:07 PM
MappedByteBuffer, close(), and Access Denied Tim Ward Java 2 10-02-2003 11:19 AM



Advertisments