Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C++ > Memory Mapped IO and Sample code for win2000

Reply
Thread Tools

Memory Mapped IO and Sample code for win2000

 
 
Peter Nolan
Guest
Posts: n/a
 
      10-08-2004
Hi All,
I have some software that currently loads database tables into sorted arrays
in memory to be binary searched by processes. (Long story as to why it does
this..)

Each process must either read the data from the in memory array or from the
database itself if the table is too large to be loaded into memory. (2GB is
usually the limit because the array is indexed by a 32 bit signed integer.
I have never had one of these tables go over 2GB, even when the client had
20M customers.)

This current set up is fine for companies with a few million customers and a
few million transactions per day. The software works just fine on win2000
and solaris/aix. None of my existing customers has large enough lookup
tables or so many transactions that it is a problem to load them into
memory.

However, one of the major 'upgrades' I want to make to this code is to
enable it to share these in-memory arrays between many processes. This
sharing of memory will make the product pretty much 'endlessly scalable',
and this is where I want to get it to.

The idea is that if the customer is a large telephone company then I can
split the Call Detail Records into say 12 input files and have 12 processes
run at the same time. By sharing the tables to be searched in memory I can
cut the memory requirement by a factor of 12. By putting all the data to be
looked up into memory the speed of processing the CDRs can be maximised
because a lookup to a database is at least 20 times slower than the lookup
to the array even if the data is in memory.

I have the code I need to do this under solaris.....but from the examples I
can see in VS.NET it looks like windows 2000 acts a little differently.

Questions....

Q1. Does anyone have a fragement of code for a producer/consumer for memory
mapped IO under win2000 that they are willing to share? Or is there a good
public example anywhere that someone here can point to? The examples I can
find with vs.net are in java or very old.

Q2. Is there a 2GB limit on creating a memory mapped file under win2000?

Q3. Will a memory mapped file like this run at the same speed as an in
memory array that is created by calloc like the current one? (Once the
producer has loaded the table into memory all other processes are
'read-only' and will be able to read the memory at the same time, no locks
required.)

Q4. Are there any other pointers the experts on this group might like to
pass along to someone going to do their very first Memory Mapped IO piece of
work? Are there any 'gotchas' for beginners in this area?

All assistance is much appreciated!!!

Best Regards

Peter Nolan
www.peternolan.com


 
Reply With Quote
 
 
 
 
Lothar Werzinger
Guest
Posts: n/a
 
      10-08-2004
Peter Nolan wrote:

> Hi All,
> I have some software that currently loads database tables into sorted
> arrays in memory to be binary searched by processes. (Long story as to why
> it does this..)

... deleted
> All assistance is much appreciated!!!
>
> Best Regards
>
> Peter Nolan
> www.peternolan.com


Hi, the ACE framework (http://www.cs.wustl.edu/~schmidt/ACE.html) provides
memory mapped files on all it's supported platforms (man unices, embedded
systems and win32)

Lothar
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
iostream and memory-mapped file wakun@wakun.com C++ 3 02-23-2006 06:29 PM
looking for asp.net sample with vb.net backend sample is there one? Jake ASP .Net 0 02-09-2006 10:44 PM
My memory mapped file challenge for CD Writing iksrazal@terra.com.br Java 2 07-24-2005 07:33 PM
Re: B+Tree and Memory Mapped Files Thomas Matthews C++ 2 04-02-2004 05:55 PM
Re: B+Tree and Memory Mapped Files Evan Carew C++ 0 04-01-2004 08:05 PM



Advertisments