Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > C Programming > handling large data

Reply
Thread Tools

handling large data

 
 
Malcolm McLean
Guest
Posts: n/a
 
      11-24-2007
"Richard Heathfield" <(E-Mail Removed)> wrote in message
>

2000 * 2000 * int
> ITYM "megabytes", which isn't all *that* big really (although it's pretty
> hefty for a local array).
>
> One malloc for the row pointers, and one malloc per row, should do it
> nicely on modern desktop systems with not too much else running.
>

That's just a fairly normal sized image. How did we use to manage with
little 64k systems?

--
Free games and programming goodies.
http://www.personal.leeds.ac.uk/~bgy1mm

 
Reply With Quote
 
 
 
 
Richard Tobin
Guest
Posts: n/a
 
      11-24-2007
In article <fi80g5$d8e$(E-Mail Removed)>, a <(E-Mail Removed)> wrote:
>Would anybody please tell me how to handle array data of int 2000 * 2000?
>Even make file passes by using a supercomputer, it seems calloc fails and
>segmentation fault results during a trial run.


If calloc() fails trying to allocate 2000*2000 ints, you almost
certainly have a system limit set too low. Ask your system
administrator how to change it (presumably you have a system
administrator if you have a supercomputer). You should of course be
checking for calloc() failing so that you don't get a segmentation
fault.

Or else there's some other bug in your program and it's nothing to do
with the amount of memory. To check for that possibility use a
program such as valgrind.

-- Richard

--
"Consideration shall be given to the need for as many as 32 characters
in some alphabets" - X3.4, 1963.
 
Reply With Quote
 
 
 
 
a
Guest
Posts: n/a
 
      11-24-2007
it seems that big static *2D* array creates the problem, e.g.


a[1024][1024] but not a[1024*1024] makes the problem.

i try gcc -static to see if this helps but of no hope.

there will be too many modifications to take care when I switch to malloc or
related solutions. Is there any workaround?


 
Reply With Quote
 
a
Guest
Posts: n/a
 
      11-24-2007
Which file is to be changed? I also remember there should be a config file
to set many parameters...


"Richard Tobin" <(E-Mail Removed)> wrote in message
news:fi8vgd$2thk$(E-Mail Removed)...
> In article <fi80g5$d8e$(E-Mail Removed)>, a <(E-Mail Removed)> wrote:
>>Would anybody please tell me how to handle array data of int 2000 * 2000?
>>Even make file passes by using a supercomputer, it seems calloc fails and
>>segmentation fault results during a trial run.

>
> If calloc() fails trying to allocate 2000*2000 ints, you almost
> certainly have a system limit set too low. Ask your system
> administrator how to change it (presumably you have a system
> administrator if you have a supercomputer). You should of course be
> checking for calloc() failing so that you don't get a segmentation
> fault.
>
> Or else there's some other bug in your program and it's nothing to do
> with the amount of memory. To check for that possibility use a
> program such as valgrind.
>
> -- Richard
>
> --
> "Consideration shall be given to the need for as many as 32 characters
> in some alphabets" - X3.4, 1963.



 
Reply With Quote
 
Vitor Py
Guest
Posts: n/a
 
      11-24-2007
On 23 nov, 23:58, "a" <(E-Mail Removed)> wrote:
> Would anybody please tell me how to handle array data of int 2000 * 2000?
> Even make file passes by using a supercomputer, it seems calloc fails and
> segmentation fault results during a trial run.


If you're handling matrices, you may want study them use more
apropriate storage. Let's say, if you have a three diagonal matrix,
use 3 arrays for storage (it's going to make your code uglier, and
require more work - but it's gonna be really faster). Diagonal
matrices are fairly common from problems arising from finite
differences or finite elements. Allocating them as full matrices
usually is not worth it's price.

Vitor.
PS: I'm not a native speaker of english, so, pardon for my errors.
 
Reply With Quote
 
Barry Schwarz
Guest
Posts: n/a
 
      11-25-2007
On Sat, 24 Nov 2007 22:32:37 +0800, "a" <(E-Mail Removed)> wrote:

>it seems that big static *2D* array creates the problem, e.g.
>
>
>a[1024][1024] but not a[1024*1024] makes the problem.
>
>i try gcc -static to see if this helps but of no hope.
>
>there will be too many modifications to take care when I switch to malloc or
>related solutions. Is there any workaround?


What is the actual type that size_t is a typedef for? What is the
maximum value for that type?

Since both a[1024][1024] and a[1024*1024] require the exact same
space, if one succeeds and the other fails then something else is
going on. A likely candidate is that SIZE_MAX is less than one
million and the multiplication in the latter declaration is wrapping
to some tolerable value. You should print the size of a in this case.

In any event, changing from a defined array to an allocated one
requires replacing only the one line declaration
int a[1024][1024];
with the small loop
int *a[1024];
for (i = 0; i < 1024; i++)
a[i] = malloc(1024 * sizeof *a[i]);
From then on, all references in your code to a[i][j] should remain
valid without modification.



Remove del for email
 
Reply With Quote
 
Gordon
Guest
Posts: n/a
 
      12-08-2007
On Nov 24, 9:58 am, "a" <(E-Mail Removed)> wrote:
> Would anybody please tell me how to handle array data of int 2000 * 2000?
> Even make file passes by using a supercomputer, it seems calloc fails and
> segmentation fault results during a trial run.


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
median of large data set (from large file) friend.05@gmail.com Perl Misc 5 04-02-2009 04:06 AM
Design of a pipelined architecture/framework for handling large data sets nish Java 1 11-30-2006 07:19 PM
Handling large amounts of data Wayne Marsh C Programming 3 11-20-2004 11:40 PM
handling large data sets Martin Pirker Ruby 10 12-09-2003 07:02 PM
Backing Up Large Files..Or A Large Amount Of Files Scott D. Weber For Unuathorized Thoughts Inc. Computer Support 1 09-19-2003 07:28 PM



Advertisments