Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > Large data array (operations) via disk disk files

Reply
Thread Tools

Large data array (operations) via disk disk files

 
 
geerrxin@gmail.com
Guest
Posts: n/a
 
      11-13-2006
Hi,

I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
which can't fit in physical memory (2 BG). But the nature of
computation
requires the operation on only a portion of the data, e.g. 500 MB (0.5
GB)
at a time.

The procedure is as follows:

1. Generate data and store the data in array A(N,N), N is HUGE.

2. Do computation on A in loops, e.g.

for i = 1, N
for j = 1, N
compute something using A (a portion)
end
end

How can I implement the procedure to accommodate the large memory
needs?

Thanks,
Zin

 
Reply With Quote
 
 
 
 
Ian Collins
Guest
Posts: n/a
 
      11-13-2006
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Hi,
>
> I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
> which can't fit in physical memory (2 BG). But the nature of
> computation
> requires the operation on only a portion of the data, e.g. 500 MB (0.5
> GB)
> at a time.
>
> The procedure is as follows:
>
> 1. Generate data and store the data in array A(N,N), N is HUGE.
>
> 2. Do computation on A in loops, e.g.
>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end
>
> How can I implement the procedure to accommodate the large memory
> needs?
>

Two solutions:

If performance is an issue, use a box with enough memory.

Otherwise use the memory mapped file support provided by your operating
environment and map the required portion of the matrix into memory. How
you do this will be OS specific and best asked on an OS group.

--
Ian Collins.
 
Reply With Quote
 
 
 
 
Terence
Guest
Posts: n/a
 
      11-13-2006

(E-Mail Removed) wrote:
> Hi,
>
> I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
> which can't fit in physical memory (2 BG). But the nature of
> computation
> requires the operation on only a portion of the data, e.g. 500 MB (0.5
> GB)
> at a time.
>
> The procedure is as follows:
>
> 1. Generate data and store the data in array A(N,N), N is HUGE.
>
> 2. Do computation on A in loops, e.g.
>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end
>
> How can I implement the procedure to accommodate the large memory
> needs?
>
> Thanks,
> Zin


Two possibilities
1) if the data has a lot of missing elements or inferred constants
(like zero) as elements, then you could use sparse matrix processing,
where you have a marked link to each next element. Marking the element
means you identify the value with :-
either the previous and last element coordinates
or just the element's row and column number.
This needs the value of the cell and the coordinates of the cell (three
values) per cell.
Sometimes you can get by with just the column number and process by row
and so only need to note when a column index "1" appears for the next
counted row .
This problem is often attacked with generised linked list processing
routines.

These methods will use less memory space if the needed elements occupy
less than one third of the theoretical maximum (N*N), (or one half in
the linear case) but only will be really useful if the prortion is far
less, like one fifth or lower.

2) rework the algorithm you wish to use, so that it needs less elements
in memory at one time than the available memory, for the operation to
proceed.

If that doesn't work then use virtual memory by treating the disk as a
random access file by row, with all of a row in each "record" and try
to use an algorthm that processes on a column-within-row basis.

 
Reply With Quote
 
=?ISO-8859-1?Q?Jan_Vorbr=FCggen?=
Guest
Posts: n/a
 
      11-14-2006
> 2. Do computation on A in loops, e.g.
>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end
>
> How can I implement the procedure to accommodate the large memory
> needs?


I suspect you need to look up the keyword "blocking", perhaps together with
the words "array" or "matrix". That does require breaking up the naive
sequence of operations of your double loop above, and depending on the nature
of your "compute something", might require some careful thinking to get this
reordered sequence do the correct thing.

Jan
 
Reply With Quote
 
alexei.matveev@gmail.com
Guest
Posts: n/a
 
      11-14-2006
>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end

You need to say more about how exactly
you want to "manipulate" your matrix ---
simplest manipulations (e.g. scaling) require exactly one
matrix element at a time.

If your algorithm really requires more than one matrix block
at a time --- split your matrix in manageable blocks that could
be addressed separately (swapped in or out). These blocks dont need
to reside within a single file, of course.

Alexei

 
Reply With Quote
 
Skybuck Flying
Guest
Posts: n/a
 
      11-14-2006

<(E-Mail Removed)> wrote in message
news:(E-Mail Removed) oups.com...
> Hi,
>
> I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
> which can't fit in physical memory (2 BG). But the nature of
> computation
> requires the operation on only a portion of the data, e.g. 500 MB (0.5
> GB)
> at a time.
>
> The procedure is as follows:
>
> 1. Generate data and store the data in array A(N,N), N is HUGE.
>
> 2. Do computation on A in loops, e.g.
>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end
>
> How can I implement the procedure to accommodate the large memory
> needs?


Use a Windows's 98 or XP's virtual memory/pagefile of 10 GB or so

Windows will do the rest.

Bye,
Skybuck.


 
Reply With Quote
 
J. F. Cornwall
Guest
Posts: n/a
 
      11-14-2006
Skybuck Flying wrote:
> <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed) oups.com...
>
>>Hi,
>>
>>I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
>>which can't fit in physical memory (2 BG). But the nature of
>>computation
>>requires the operation on only a portion of the data, e.g. 500 MB (0.5
>>GB)
>>at a time.
>>
>>The procedure is as follows:
>>
>>1. Generate data and store the data in array A(N,N), N is HUGE.
>>
>>2. Do computation on A in loops, e.g.
>>
>>for i = 1, N
>> for j = 1, N
>> compute something using A (a portion)
>> end
>>end
>>
>>How can I implement the procedure to accommodate the large memory
>>needs?

>
>
> Use a Windows's 98 or XP's virtual memory/pagefile of 10 GB or so
>
> Windows will do the rest.
>
> Bye,
> Skybuck.
>
>

That is a spectacularly useless suggestion if the program is not
intended to run on a Windows machine... The OP doesn't say either way,
so you have a 50/50 chance at best.

Jim
 
Reply With Quote
 
santosh
Guest
Posts: n/a
 
      11-14-2006
J. F. Cornwall wrote:
> Skybuck Flying wrote:
> > <(E-Mail Removed)> wrote in message
> > news:(E-Mail Removed) oups.com...
> >
> >>Hi,
> >>
> >>I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
> >>which can't fit in physical memory (2 BG). But the nature of
> >>computation
> >>requires the operation on only a portion of the data, e.g. 500 MB (0.5
> >>GB)
> >>at a time.
> >>
> >>The procedure is as follows:
> >>
> >>1. Generate data and store the data in array A(N,N), N is HUGE.
> >>
> >>2. Do computation on A in loops, e.g.
> >>
> >>for i = 1, N
> >> for j = 1, N
> >> compute something using A (a portion)
> >> end
> >>end
> >>
> >>How can I implement the procedure to accommodate the large memory
> >>needs?

> >
> >
> > Use a Windows's 98 or XP's virtual memory/pagefile of 10 GB or so
> >
> > Windows will do the rest.
> >
> > Bye,
> > Skybuck.
> >
> >

> That is a spectacularly useless suggestion if the program is not
> intended to run on a Windows machine... The OP doesn't say either way,
> so you have a 50/50 chance at best.


Besides, AFAIK, 32 bit versions of Windows don't support a 10 GB paging
file. Also only Windows has access to it.

 
Reply With Quote
 
Skybuck Flying
Guest
Posts: n/a
 
      11-15-2006

"santosh" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed) ups.com...
> J. F. Cornwall wrote:
>> Skybuck Flying wrote:
>> > <(E-Mail Removed)> wrote in message
>> > news:(E-Mail Removed) oups.com...
>> >
>> >>Hi,
>> >>
>> >>I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
>> >>which can't fit in physical memory (2 BG). But the nature of
>> >>computation
>> >>requires the operation on only a portion of the data, e.g. 500 MB (0.5
>> >>GB)
>> >>at a time.
>> >>
>> >>The procedure is as follows:
>> >>
>> >>1. Generate data and store the data in array A(N,N), N is HUGE.
>> >>
>> >>2. Do computation on A in loops, e.g.
>> >>
>> >>for i = 1, N
>> >> for j = 1, N
>> >> compute something using A (a portion)
>> >> end
>> >>end
>> >>
>> >>How can I implement the procedure to accommodate the large memory
>> >>needs?
>> >
>> >
>> > Use a Windows's 98 or XP's virtual memory/pagefile of 10 GB or so
>> >
>> > Windows will do the rest.
>> >
>> > Bye,
>> > Skybuck.
>> >
>> >

>> That is a spectacularly useless suggestion if the program is not
>> intended to run on a Windows machine... The OP doesn't say either way,
>> so you have a 50/50 chance at best.


Neh more 99.9 procent chance

Windows dominates >D =D

>
> Besides, AFAIK, 32 bit versions of Windows don't support a 10 GB paging
> file. Also only Windows has access to it.


Get a new PC.

I am using Windows XP 64 bit <- it totally sucks but hey it's the future

Strangly enough I have two pagefiles each 8 GB.

One one each harddisk. (I have two harddisks)

I think Windows XP moved the pagefile to one of the disks.. or maybe it was
me lol.

So I allocated two pagefiles one on each disk just in case

If I can allocate 8 GB I can probably allocate 10 GB's as well or even more


Surely other operating systems have the same virtual memory future ? <- if
not gjez get a decent OS lol.

Bye,
Skybuck.


 
Reply With Quote
 
Randy Howard
Guest
Posts: n/a
 
      11-15-2006
Skybuck Flying wrote
(in article <ejdtut$4gk$(E-Mail Removed)1.ov.home.nl>):

> Get a new PC.


A Mac Pro would be a nice choice.

> I am using Windows XP 64 bit <- it totally sucks but hey it's the future


One of the rarest of all Skybuck statements, a correct one.
There are very few of these in the wild. You are correct, the
future for Windows users really does suck.


--
Randy Howard (2reply remove FOOBAR)
"The power of accurate observation is called cynicism by those
who have not got it." - George Bernard Shaw





 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
median of large data set (from large file) friend.05@gmail.com Perl Misc 5 04-02-2009 04:06 AM
maps to hold ultra large data sets using customer allocators to allocate disk space rather than main memory CMOS C++ 15 05-17-2007 10:12 PM
[Urgent] Is there a size limit on returning a large dataset or a large typed array from web service? Ketchup ASP .Net Web Services 1 05-25-2004 10:11 AM
large files via Response.OutputStream.Write Ryan Hartman ASP .Net 6 11-11-2003 02:29 AM
Backing Up Large Files..Or A Large Amount Of Files Scott D. Weber For Unuathorized Thoughts Inc. Computer Support 1 09-19-2003 07:28 PM



Advertisments