Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Fastest way to read array of ints off disk?

Reply
Thread Tools

Fastest way to read array of ints off disk?

 
 
Chris
Guest
Posts: n/a
 
      01-17-2007
I need to read several large arrays of integers off a disk into int []
variables.

I could read them into byte[] arrays and then convert, doing lots of
masks and bitshifting. I can't help but believe there's a faster way.

I've fooled with NIO, thinking that I could create an IntBuffer and read
stuff into it using FileChannel.read(), but no go. I don't think it will
work to create a ByteBuffer, read data into it, and then call
asIntBuffer() on it, because the .array() method I need to return the
int [] is described as an "optional operation" in the Javadoc.

Any other ideas how to do it?
 
Reply With Quote
 
 
 
 
Knute Johnson
Guest
Posts: n/a
 
      01-17-2007
Chris wrote:
> I need to read several large arrays of integers off a disk into int []
> variables.
>
> I could read them into byte[] arrays and then convert, doing lots of
> masks and bitshifting. I can't help but believe there's a faster way.
>
> I've fooled with NIO, thinking that I could create an IntBuffer and read
> stuff into it using FileChannel.read(), but no go. I don't think it will
> work to create a ByteBuffer, read data into it, and then call
> asIntBuffer() on it, because the .array() method I need to return the
> int [] is described as an "optional operation" in the Javadoc.
>
> Any other ideas how to do it?


You could use DataInputStream.readInt(). It does the masking and
shifting for you . Use a big buffer with it.

--

Knute Johnson
email s/nospam/knute/
 
Reply With Quote
 
 
 
 
Tom Hawtin
Guest
Posts: n/a
 
      01-17-2007
Chris wrote:
> I need to read several large arrays of integers off a disk into int []
> variables.
>
> I could read them into byte[] arrays and then convert, doing lots of
> masks and bitshifting. I can't help but believe there's a faster way.
>
> I've fooled with NIO, thinking that I could create an IntBuffer and read
> stuff into it using FileChannel.read(), but no go. I don't think it will
> work to create a ByteBuffer, read data into it, and then call
> asIntBuffer() on it, because the .array() method I need to return the
> int [] is described as an "optional operation" in the Javadoc.


The array() method returns the array that backs the buffer. If you start
with a 'default' ByteBuffer, the backing array will be of type byte[].

The easiest thing to do is to go through the IntBuffer and copy the data
into a new int[].

I'm not entirely sure what performance results you will get. Using
allocateDirect instead of allocate should improve the time to read the
buffer, but not sure about reading the results out of it.

In order for NIO to be efficient it requires multiple levels of calls to
be inlined. Therefore, make sure the method with your inner loop is as
small as absolutely possible and use the Server VM.

You might find it faster to manually copy data from a byte[] to an int[].

Tom Hawtin
 
Reply With Quote
 
Chris Uppal
Guest
Posts: n/a
 
      01-17-2007
Chris wrote:

> I could read them into byte[] arrays and then convert, doing lots of
> masks and bitshifting. I can't help but believe there's a faster way.


I doubt if there's a way that's /much/ faster. On my machine a simple
hand-coded byte[]-int[] conversion loops is more than an order of magnitude
faster than reading the same amount of the data off-disk, so I wouldn't expect
that you can improve the total time by more than a very small
margin. Of course your hardware/software combo may be different, but I'd be
surprised if it were /that/ different...

-- chris



 
Reply With Quote
 
EJP
Guest
Posts: n/a
 
      01-21-2007
Chris wrote:
> Any other ideas how to do it?


Use a java.nio.MappedByteBuffer and get an IntBuffer view of it, then
set the appropriate ByteOrder on it. This is about as fast as you can get.
 
Reply With Quote
 
Randolf Richardson
Guest
Posts: n/a
 
      01-28-2007
On Tue, 16 Jan 2007 19:41:41 -0800, Chris <(E-Mail Removed)> wrote:

> I need to read several large arrays of integers off a disk into int []
> variables.
>
> I could read them into byte[] arrays and then convert, doing lots of
> masks and bitshifting. I can't help but believe there's a faster way.
>
> I've fooled with NIO, thinking that I could create an IntBuffer and read
> stuff into it using FileChannel.read(), but no go. I don't think it will
> work to create a ByteBuffer, read data into it, and then call
> asIntBuffer() on it, because the .array() method I need to return the
> int [] is described as an "optional operation" in the Javadoc.
>
> Any other ideas how to do it?


Have you looked at java.io.RandomAccessFile.readFully(byte[])?

--
Randolf Richardson - (E-Mail Removed)
The Lumber Cartel, local 42 (Canadian branch)
http://www.lumbercartel.ca/
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
In-place memory manager, mmap (was: Fastest way to store ints andfloats on disk) castironpi Python 5 08-24-2008 08:36 PM
Re: Fastest way to store ints and floats on disk M.-A. Lemburg Python 3 08-10-2008 04:41 AM
Fastest way to store ints and floats on disk Laszlo Nagy Python 2 08-09-2008 11:42 AM
numpy and filtering (was: Fastest way to store ints and floats ondisk) Laszlo Nagy Python 0 08-08-2008 11:06 AM
ints ints ints and ints Skybuck Flying C Programming 24 07-10-2004 04:48 AM



Advertisments