Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   C Programming (http://www.velocityreviews.com/forums/f42-c-programming.html)
-   -   Large data array (operations) via disk disk files (http://www.velocityreviews.com/forums/t445121-large-data-array-operations-via-disk-disk-files.html)

 geerrxin@gmail.com 11-13-2006 08:22 PM

Large data array (operations) via disk disk files

Hi,

I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
which can't fit in physical memory (2 BG). But the nature of
computation
requires the operation on only a portion of the data, e.g. 500 MB (0.5
GB)
at a time.

The procedure is as follows:

1. Generate data and store the data in array A(N,N), N is HUGE.

2. Do computation on A in loops, e.g.

for i = 1, N
for j = 1, N
compute something using A (a portion)
end
end

How can I implement the procedure to accommodate the large memory
needs?

Thanks,
Zin

 Ian Collins 11-13-2006 08:35 PM

Re: Large data array (operations) via disk disk files

geerrxin@gmail.com wrote:
> Hi,
>
> I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
> which can't fit in physical memory (2 BG). But the nature of
> computation
> requires the operation on only a portion of the data, e.g. 500 MB (0.5
> GB)
> at a time.
>
> The procedure is as follows:
>
> 1. Generate data and store the data in array A(N,N), N is HUGE.
>
> 2. Do computation on A in loops, e.g.
>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end
>
> How can I implement the procedure to accommodate the large memory
> needs?
>

Two solutions:

If performance is an issue, use a box with enough memory.

Otherwise use the memory mapped file support provided by your operating
environment and map the required portion of the matrix into memory. How
you do this will be OS specific and best asked on an OS group.

--
Ian Collins.

 Terence 11-13-2006 08:50 PM

Re: Large data array (operations) via disk disk files

geerrxin@gmail.com wrote:
> Hi,
>
> I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
> which can't fit in physical memory (2 BG). But the nature of
> computation
> requires the operation on only a portion of the data, e.g. 500 MB (0.5
> GB)
> at a time.
>
> The procedure is as follows:
>
> 1. Generate data and store the data in array A(N,N), N is HUGE.
>
> 2. Do computation on A in loops, e.g.
>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end
>
> How can I implement the procedure to accommodate the large memory
> needs?
>
> Thanks,
> Zin

Two possibilities
1) if the data has a lot of missing elements or inferred constants
(like zero) as elements, then you could use sparse matrix processing,
where you have a marked link to each next element. Marking the element
means you identify the value with :-
either the previous and last element coordinates
or just the element's row and column number.
This needs the value of the cell and the coordinates of the cell (three
values) per cell.
Sometimes you can get by with just the column number and process by row
and so only need to note when a column index "1" appears for the next
counted row .
This problem is often attacked with generised linked list processing
routines.

These methods will use less memory space if the needed elements occupy
less than one third of the theoretical maximum (N*N), (or one half in
the linear case) but only will be really useful if the prortion is far
less, like one fifth or lower.

2) rework the algorithm you wish to use, so that it needs less elements
in memory at one time than the available memory, for the operation to
proceed.

If that doesn't work then use virtual memory by treating the disk as a
random access file by row, with all of a row in each "record" and try
to use an algorthm that processes on a column-within-row basis.

 =?ISO-8859-1?Q?Jan_Vorbr=FCggen?= 11-14-2006 08:08 AM

Re: Large data array (operations) via disk disk files

> 2. Do computation on A in loops, e.g.
>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end
>
> How can I implement the procedure to accommodate the large memory
> needs?

I suspect you need to look up the keyword "blocking", perhaps together with
the words "array" or "matrix". That does require breaking up the naive
sequence of operations of your double loop above, and depending on the nature
of your "compute something", might require some careful thinking to get this
reordered sequence do the correct thing.

Jan

 alexei.matveev@gmail.com 11-14-2006 09:11 AM

Re: Large data array (operations) via disk disk files

>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end

You need to say more about how exactly
you want to "manipulate" your matrix ---
simplest manipulations (e.g. scaling) require exactly one
matrix element at a time.

If your algorithm really requires more than one matrix block
at a time --- split your matrix in manageable blocks that could
be addressed separately (swapped in or out). These blocks dont need
to reside within a single file, of course.

Alexei

 Skybuck Flying 11-14-2006 05:33 PM

Re: Large data array (operations) via disk disk files

<geerrxin@gmail.com> wrote in message
> Hi,
>
> I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
> which can't fit in physical memory (2 BG). But the nature of
> computation
> requires the operation on only a portion of the data, e.g. 500 MB (0.5
> GB)
> at a time.
>
> The procedure is as follows:
>
> 1. Generate data and store the data in array A(N,N), N is HUGE.
>
> 2. Do computation on A in loops, e.g.
>
> for i = 1, N
> for j = 1, N
> compute something using A (a portion)
> end
> end
>
> How can I implement the procedure to accommodate the large memory
> needs?

Use a Windows's 98 or XP's virtual memory/pagefile of 10 GB or so ;)

Windows will do the rest.

Bye,
Skybuck.

 J. F. Cornwall 11-14-2006 06:17 PM

Re: Large data array (operations) via disk disk files

Skybuck Flying wrote:
> <geerrxin@gmail.com> wrote in message
>
>>Hi,
>>
>>I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
>>which can't fit in physical memory (2 BG). But the nature of
>>computation
>>requires the operation on only a portion of the data, e.g. 500 MB (0.5
>>GB)
>>at a time.
>>
>>The procedure is as follows:
>>
>>1. Generate data and store the data in array A(N,N), N is HUGE.
>>
>>2. Do computation on A in loops, e.g.
>>
>>for i = 1, N
>> for j = 1, N
>> compute something using A (a portion)
>> end
>>end
>>
>>How can I implement the procedure to accommodate the large memory
>>needs?

>
>
> Use a Windows's 98 or XP's virtual memory/pagefile of 10 GB or so ;)
>
> Windows will do the rest.
>
> Bye,
> Skybuck.
>
>

That is a spectacularly useless suggestion if the program is not
intended to run on a Windows machine... The OP doesn't say either way,
so you have a 50/50 chance at best.

Jim

 santosh 11-14-2006 06:25 PM

Re: Large data array (operations) via disk disk files

J. F. Cornwall wrote:
> Skybuck Flying wrote:
> > <geerrxin@gmail.com> wrote in message
> >
> >>Hi,
> >>
> >>I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
> >>which can't fit in physical memory (2 BG). But the nature of
> >>computation
> >>requires the operation on only a portion of the data, e.g. 500 MB (0.5
> >>GB)
> >>at a time.
> >>
> >>The procedure is as follows:
> >>
> >>1. Generate data and store the data in array A(N,N), N is HUGE.
> >>
> >>2. Do computation on A in loops, e.g.
> >>
> >>for i = 1, N
> >> for j = 1, N
> >> compute something using A (a portion)
> >> end
> >>end
> >>
> >>How can I implement the procedure to accommodate the large memory
> >>needs?

> >
> >
> > Use a Windows's 98 or XP's virtual memory/pagefile of 10 GB or so ;)
> >
> > Windows will do the rest.
> >
> > Bye,
> > Skybuck.
> >
> >

> That is a spectacularly useless suggestion if the program is not
> intended to run on a Windows machine... The OP doesn't say either way,
> so you have a 50/50 chance at best.

Besides, AFAIK, 32 bit versions of Windows don't support a 10 GB paging

 Skybuck Flying 11-15-2006 02:27 AM

Re: Large data array (operations) via disk disk files

"santosh" <santosh.k83@gmail.com> wrote in message
> J. F. Cornwall wrote:
>> Skybuck Flying wrote:
>> > <geerrxin@gmail.com> wrote in message
>> >
>> >>Hi,
>> >>
>> >>I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB
>> >>which can't fit in physical memory (2 BG). But the nature of
>> >>computation
>> >>requires the operation on only a portion of the data, e.g. 500 MB (0.5
>> >>GB)
>> >>at a time.
>> >>
>> >>The procedure is as follows:
>> >>
>> >>1. Generate data and store the data in array A(N,N), N is HUGE.
>> >>
>> >>2. Do computation on A in loops, e.g.
>> >>
>> >>for i = 1, N
>> >> for j = 1, N
>> >> compute something using A (a portion)
>> >> end
>> >>end
>> >>
>> >>How can I implement the procedure to accommodate the large memory
>> >>needs?
>> >
>> >
>> > Use a Windows's 98 or XP's virtual memory/pagefile of 10 GB or so ;)
>> >
>> > Windows will do the rest.
>> >
>> > Bye,
>> > Skybuck.
>> >
>> >

>> That is a spectacularly useless suggestion if the program is not
>> intended to run on a Windows machine... The OP doesn't say either way,
>> so you have a 50/50 chance at best.

Neh more 99.9 procent chance ;)

Windows dominates >D =D

>
> Besides, AFAIK, 32 bit versions of Windows don't support a 10 GB paging

Get a new PC.

I am using Windows XP 64 bit <- it totally sucks but hey it's the future ;)

Strangly enough I have two pagefiles each 8 GB.

One one each harddisk. (I have two harddisks)

I think Windows XP moved the pagefile to one of the disks.. or maybe it was
me lol.

So I allocated two pagefiles one on each disk just in case ;)

If I can allocate 8 GB I can probably allocate 10 GB's as well or even more
;)

Surely other operating systems have the same virtual memory future ? <- if
not gjez get a decent OS lol.

Bye,
Skybuck.

 Randy Howard 11-15-2006 02:48 AM

Re: Large data array (operations) via disk disk files

Skybuck Flying wrote
(in article <ejdtut\$4gk\$1@news1.zwoll1.ov.home.nl>):

> Get a new PC.

A Mac Pro would be a nice choice.

> I am using Windows XP 64 bit <- it totally sucks but hey it's the future ;)

One of the rarest of all Skybuck statements, a correct one.
There are very few of these in the wild. You are correct, the
future for Windows users really does suck.

--