«

This should be a common need. Yet I am not aware of anything
like it in Java SE. What is the most common (pure Java)
solution to it?

I would like to have an implementation of java.util.Map,
which is constructed with an int »m« and a java.io.File »f«.

It will use no more than »m« bytes of memory, but »swap« out
(the least often used) entries to the file »f«, when they do
not fit into the given memory size anymore.

Stefan Ram a écrit :
> This should be a common need. Yet I am not aware of anything
> like it in Java SE. What is the most common (pure Java)
> solution to it?
>
> I would like to have an implementation of java.util.Map,
> which is constructed with an int »m« and a java.io.File »f«.
>
> It will use no more than »m« bytes of memory, but »swap« out
> (the least often used) entries to the file »f«, when they do
> not fit into the given memory size anymore.
>

just to make you renounce, search for a feature request about a
release() or free() method for the "MappedByteBuffer" class in the sun
bugs database.

Stefan Ram wrote:
> This should be a common need. Yet I am not aware of anything
> like it in Java SE. What is the most common (pure Java)
> solution to it?
>
> I would like to have an implementation of java.util.Map,
> which is constructed with an int »m« and a java.io.File »f«.
>
> It will use no more than »m« bytes of memory, but »swap« out
> (the least often used) entries to the file »f«, when they do
> not fit into the given memory size anymore.
>


I was going to the say the same thing as Patricia: the map is called
"Derby". Search for JavaDB.

<http://java.sun.com/developer/technicalArticles/J2SE/Desktop/javadb/>

There's no point in actually implementing such a specialized thing as a
literal Map when there's a better, more general solution already available.

On 18 Aug 2009 12:09:58 GMT, (Stefan Ram)
wrote, quoted or indirectly quoted someone who said :

> This should be a common need. Yet I am not aware of anything
> like it in Java SE. What is the most common (pure Java)
> solution to it?
>
> I would like to have an implementation of java.util.Map,
> which is constructed with an int »m« and a java.io.File »f«.
>
> It will use no more than »m« bytes of memory, but »swap« out
> (the least often used) entries to the file »f«, when they do
> not fit into the given memory size anymore.

I rolled my own something similar. It is not that much code to handle
the random quotations you see on my website.

What you could do in write a class that uses a HashMap internally just
to hold the keys and objects that are offsets in a sequential file.

When you build the Map, you write the objects out with writeUTF or
writeObject, and record the size/offset of the stream before the
write.

Then to lookup in the Map, you look up the key, get the offset, seek
and do a read. You don't even need to know the length.

This is pretty fast, especially when the drive/OS does read caching.
If you wanted to make it even faster, you could put the objects in a
NIO memory mapped file, but that limits your file size. You also
might put the file on fast flash drive.

I would write such a beast to your specs for $50 US.


--
Roedy Green Canadian Mind Products
http://mindprod.com

http://thecovemovie.com : The Cove: a documentary about Japan's secret atrocities against dolphins.

On Tue, 18 Aug 2009 05:42:09 -0700, Patricia Shanahan <>
wrote, quoted or indirectly quoted someone who said :

>
>Have you considered putting the data in a database instead, and using
>java.sql to access it? The data structures and algorithms that Java uses
>for in-memory maps are not very suitable for disk-based maps. Database
>managers use structures and algorithms designed for the job.

Other advantages:

1. They don't fail entirely when RAM gets tighter. With an in RAM key
lookup, you must have the ram for all the keys.

2. You can improve their performance just by throwing more ram at
them. They can use it for both keys and objects.

3. If your indexing needs become more complex, you can just add. You
don't have to start over from scratch.

The disadvantages:

1. Usually you must separately install, start/stop the SQL engine.
This might conflict with other apps.

2. You have the overhead of all of SQL when you may need only a tiny
fraction of it. That RAM would have been better used with more
specific code.
--
Roedy Green Canadian Mind Products
http://mindprod.com

http://thecovemovie.com : The Cove: a documentary about Japan's secret atrocities against dolphins.

Roedy Green <> writes:
>Then to lookup in the Map, you look up the key, get the offset, seek
>and do a read. You don't even need to know the length.

I have read your ideas on how to implement this beast, and
they are nearly identical to what I had in mind before I
read your suggestion but after I wrote my OP (I made it up
in the time between). So, maybe I might go this way.

Roedy Green <> writes:
>I would write such a beast to your specs for $50 US.
¯¯¯¯¯
This is funny, because just yesterday I read an article
about brain research that found that all people (even blind
people) have different areas in the brain to represent
animals versus non-animal things, independent of how they
look like.

Then, yesterday, I posted my thoughts in »de.comp.lang.misc«
about how the brain sometimes uses the animal area and
sometimes the things area to represent abstract concepts in
programming. For example, a program looks like an animal to
the brain, while data looks like a thing.

Some people then made fun of my post, but now I read your
choice of the word »beast«, which supports my idea.

Roedy Green <> writes:
>1. Usually you must separately install, start/stop the SQL engine.
>This might conflict with other apps.

I have this crazy idea to write applications that are based
on Java SE only and not require any additional library.
I know that this idea might not be very pragmatic or reasonable.

/If/ Derby would finally be included in Java SE, I would
love to use it.

On Tue, 18 Aug 2009 13:02:36 -0700, Roedy Green
<> wrote, quoted or indirectly quoted
someone who said :

>1. They don't fail entirely when RAM gets tighter. With an in RAM key
>lookup, you must have the ram for all the keys.
>
>2. You can improve their performance just by throwing more ram at
>them. They can use it for both keys and objects.
>
>3. If your indexing needs become more complex, you can just add. You
>don't have to start over from scratch.

I expound on this theme at
http://mindprod.com/jgloss/sql.html#ALTERNATIVES
--
Roedy Green Canadian Mind Products
http://mindprod.com

http://thecovemovie.com : The Cove: a documentary about Japan's secret atrocities against dolphins.

On 18 Aug 2009 20:21:08 GMT, (Stefan Ram)
wrote, quoted or indirectly quoted someone who said :

> Some people then made fun of my post, but now I read your
> choice of the word »beast«, which supports my idea.

I have a more elaborate version for frequently changing data called a
Hermit Crab file. It was inspired by the metaphor of a hermit crab
seeking out ever larger shells as it grows.

see http://mindprod.com/jgloss/hermitcrab.html

As RAM has become less expensive, the need for a light weight indexed
record lookup has waned. You can just use a big hammer -- SQL.
--
Roedy Green Canadian Mind Products
http://mindprod.com

http://thecovemovie.com : The Cove: a documentary about Japan's secret atrocities against dolphins.