(E-Mail Removed) wrote:

> Hi,

>

> I have a need to manipulate a large matrix, say, A(N,N) (of real) > 8GB

> which can't fit in physical memory (2 BG). But the nature of

> computation

> requires the operation on only a portion of the data, e.g. 500 MB (0.5

> GB)

> at a time.

>

> The procedure is as follows:

>

> 1. Generate data and store the data in array A(N,N), N is HUGE.

>

> 2. Do computation on A in loops, e.g.

>

> for i = 1, N

> for j = 1, N

> compute something using A (a portion)

> end

> end

>

> How can I implement the procedure to accommodate the large memory

> needs?

>

> Thanks,

> Zin
Two possibilities

1) if the data has a lot of missing elements or inferred constants

(like zero) as elements, then you could use sparse matrix processing,

where you have a marked link to each next element. Marking the element

means you identify the value with :-

either the previous and last element coordinates

or just the element's row and column number.

This needs the value of the cell and the coordinates of the cell (three

values) per cell.

Sometimes you can get by with just the column number and process by row

and so only need to note when a column index "1" appears for the next

counted row .

This problem is often attacked with generised linked list processing

routines.

These methods will use less memory space if the needed elements occupy

less than one third of the theoretical maximum (N*N), (or one half in

the linear case) but only will be really useful if the prortion is far

less, like one fifth or lower.

2) rework the algorithm you wish to use, so that it needs less elements

in memory at one time than the available memory, for the operation to

proceed.

If that doesn't work then use virtual memory by treating the disk as a

random access file by row, with all of a row in each "record" and try

to use an algorthm that processes on a column-within-row basis.