«

Hi all,

Having a problem with addressing large amounts of memory. I have a
simple piece of code here that is meant to allocate a large piece of
memory on a ppc64 machine. The code is:

/*
Test to see what happens when we try to allocate a massively huge
piece of memory.
*/

#include <iostream>
#include <string>
#include <stdexcept>
using namespace std;

int main(int argc, char** argv) {
cout << "Attemping to allocate.." << endl;

const int ROWS = 635000;
const int COLS = 2350;

// Allocate.
try {
int** test = new int*[ROWS];
for (int i = 0; i < ROWS; i++) {
test[i] = new int[COLS];
for (int j = 0; j < COLS; j++) {
test[i][j] = 0;
}
}

cout << "Allocation succeeded!" << endl;
cout << "Press a key to deallocate and continue.." << endl;
string blank;
getline(cin,blank);

// Deallocate.
for (int k = 0; k < ROWS; k++) {
delete[] test[k];
}
delete[] test;

cout << "Deallocation completed!" << endl;
cout << "Press a key to terminate.." << endl;
getline(cin,blank);
}
catch(bad_alloc& e) {
cout << "Allocation failed.." << endl;
}

return 0;
}

If I set ROWS and COLS to 5000 and 5000, it works just fine. However,
if I set ROWS to 635000 and COLS to 2350, it will give me the
following error upon deallocation:

HugeMemory.exe(2946 malloc: *** Deallocation of a pointer not
malloced: 0x20afd2000; This could be a double free(), or free() called
with the middle of an allocated block; Try setting environment
variable MallocHelp to see tools to help debug

Note that the allocation step succeeds, and that I only receive this
error after allowing the code to deallocate the array.

Any ideas?

Thanks,
Ryan

wrote:
> Hi all,
>
> Having a problem with addressing large amounts of memory. I have a
> simple piece of code here that is meant to allocate a large piece of
> memory on a ppc64 machine. The code is:
>
> /*
> Test to see what happens when we try to allocate a massively huge
> piece of memory.
> */
>
> #include <iostream>
> #include <string>
> #include <stdexcept>
> using namespace std;
>
> int main(int argc, char** argv) {
> cout << "Attemping to allocate.." << endl;
>
> const int ROWS = 635000;
> const int COLS = 2350;
>
> // Allocate.
> try {
> int** test = new int*[ROWS];
> for (int i = 0; i < ROWS; i++) {
> test[i] = new int[COLS];
> for (int j = 0; j < COLS; j++) {
> test[i][j] = 0;
> }
> }
>
> cout << "Allocation succeeded!" << endl;
> cout << "Press a key to deallocate and continue.." << endl;
> string blank;
> getline(cin,blank);
>
> // Deallocate.
> for (int k = 0; k < ROWS; k++) {
> delete[] test[k];
> }
> delete[] test;
>
> cout << "Deallocation completed!" << endl;
> cout << "Press a key to terminate.." << endl;
> getline(cin,blank);
> }
> catch(bad_alloc& e) {
> cout << "Allocation failed.." << endl;
> }
>
> return 0;
> }
>
> If I set ROWS and COLS to 5000 and 5000, it works just fine. However,
> if I set ROWS to 635000 and COLS to 2350, it will give me the
> following error upon deallocation:
>
> HugeMemory.exe(2946 malloc: *** Deallocation of a pointer not
> malloced: 0x20afd2000; This could be a double free(), or free() called
> with the middle of an allocated block; Try setting environment
> variable MallocHelp to see tools to help debug
>
> Note that the allocation step succeeds, and that I only receive this
> error after allowing the code to deallocate the array.
>
Looks suspect, has the machine got the nigh on 6GB or virtual memory
available? Make sure your operator new behaves correctly by attempting
to allocate way more than the machine can provide.

--
Ian Collins.

On May 12, 2:02 am, welch.r...@gmail.com wrote:
> Hi all,
>
> Having a problem with addressing large amounts of memory. I have a
> simple piece of code here that is meant to allocate a large piece of
> memory on a ppc64 machine. The code is:
>
> /*
> Test to see what happens when we try to allocate a massively huge
> piece of memory.
> */
>
> #include <iostream>
> #include <string>
> #include <stdexcept>
> using namespace std;
>
> int main(int argc, char** argv) {
> cout << "Attemping to allocate.." << endl;
>
> const int ROWS = 635000;
> const int COLS = 2350;
>
> // Allocate.
> try {
> int** test = new int*[ROWS];
> for (int i = 0; i < ROWS; i++) {
> test[i] = new int[COLS];
> for (int j = 0; j < COLS; j++) {
> test[i][j] = 0;
> }
> }
>
> cout << "Allocation succeeded!" << endl;
> cout << "Press a key to deallocate and continue.." << endl;
> string blank;
> getline(cin,blank);
>
> // Deallocate.
> for (int k = 0; k < ROWS; k++) {
> delete[] test[k];
> }
> delete[] test;
>
> cout << "Deallocation completed!" << endl;
> cout << "Press a key to terminate.." << endl;
> getline(cin,blank);
> }
> catch(bad_alloc& e) {
> cout << "Allocation failed.." << endl;
> }
>
> return 0;
>
> }
>
> If I set ROWS and COLS to 5000 and 5000, it works just fine. However,
> if I set ROWS to 635000 and COLS to 2350, it will give me the
> following error upon deallocation:

There are lot of allocations there.
Looks to me that malloc internaly has overflow in pointer arithmetic.
Perhaps you use 32 bit malloc on 64 bit setup somehow?

Try following and see if works:

const unsigned rows = 635000;
const unsigned cols = 2350;
int (*p)[cols] = new int[rows][cols];
for(unsigned i = 0; i<rows;++i)
{
for(unsigned j=0;j<cols;++j)
p[i][j] = 0;
}
cin.get();
delete[] p;

Greetings, Branimir

On May 11, 8:24 pm, Ian Collins <ian-n...@hotmail.com> wrote:
> welch.r...@gmail.com wrote:
> > Hi all,
>
> > Having a problem with addressing large amounts of memory. I have a
> > simple piece of code here that is meant to allocate a large piece of
> > memory on a ppc64 machine. The code is:
>
> > /*
> > Test to see what happens when we try to allocate a massively huge
> > piece of memory.
> > */
>
> > #include <iostream>
> > #include <string>
> > #include <stdexcept>
> > using namespace std;
>
> > int main(int argc, char** argv) {
> > cout << "Attemping to allocate.." << endl;
>
> > const int ROWS = 635000;
> > const int COLS = 2350;
>
> > // Allocate.
> > try {
> > int** test = new int*[ROWS];
> > for (int i = 0; i < ROWS; i++) {
> > test[i] = new int[COLS];
> > for (int j = 0; j < COLS; j++) {
> > test[i][j] = 0;
> > }
> > }
>
> > cout << "Allocation succeeded!" << endl;
> > cout << "Press a key to deallocate and continue.." << endl;
> > string blank;
> > getline(cin,blank);
>
> > // Deallocate.
> > for (int k = 0; k < ROWS; k++) {
> > delete[] test[k];
> > }
> > delete[] test;
>
> > cout << "Deallocation completed!" << endl;
> > cout << "Press a key to terminate.." << endl;
> > getline(cin,blank);
> > }
> > catch(bad_alloc& e) {
> > cout << "Allocation failed.." << endl;
> > }
>
> > return 0;
> > }
>
> > If I set ROWS and COLS to 5000 and 5000, it works just fine. However,
> > if I set ROWS to 635000 and COLS to 2350, it will give me the
> > following error upon deallocation:
>
> > HugeMemory.exe(2946 malloc: *** Deallocation of a pointer not
> > malloced: 0x20afd2000; This could be a double free(), or free() called
> > with the middle of an allocated block; Try setting environment
> > variable MallocHelp to see tools to help debug
>
> > Note that the allocation step succeeds, and that I only receive this
> > error after allowing the code to deallocate the array.
>
> Looks suspect, has the machine got the nigh on 6GB or virtual memory
> available? Make sure your operator new behaves correctly by attempting
> to allocate way more than the machine can provide.
>
> --
> Ian Collins.

Hmm.. the machine has 8 GB of RAM, so that probably isn't the issue.
I'll try maxing it out to see what happens.

On May 11, 9:15 pm, Branimir Maksimovic <b...@hotmail.com> wrote:
> On May 12, 2:02 am, welch.r...@gmail.com wrote:
>
>
>
> > Hi all,
>
> > Having a problem with addressing large amounts of memory. I have a
> > simple piece of code here that is meant to allocate a large piece of
> > memory on a ppc64 machine. The code is:
>
> > /*
> > Test to see what happens when we try to allocate a massively huge
> > piece of memory.
> > */
>
> > #include <iostream>
> > #include <string>
> > #include <stdexcept>
> > using namespace std;
>
> > int main(int argc, char** argv) {
> > cout << "Attemping to allocate.." << endl;
>
> > const int ROWS = 635000;
> > const int COLS = 2350;
>
> > // Allocate.
> > try {
> > int** test = new int*[ROWS];
> > for (int i = 0; i < ROWS; i++) {
> > test[i] = new int[COLS];
> > for (int j = 0; j < COLS; j++) {
> > test[i][j] = 0;
> > }
> > }
>
> > cout << "Allocation succeeded!" << endl;
> > cout << "Press a key to deallocate and continue.." << endl;
> > string blank;
> > getline(cin,blank);
>
> > // Deallocate.
> > for (int k = 0; k < ROWS; k++) {
> > delete[] test[k];
> > }
> > delete[] test;
>
> > cout << "Deallocation completed!" << endl;
> > cout << "Press a key to terminate.." << endl;
> > getline(cin,blank);
> > }
> > catch(bad_alloc& e) {
> > cout << "Allocation failed.." << endl;
> > }
>
> > return 0;
>
> > }
>
> > If I set ROWS and COLS to 5000 and 5000, it works just fine. However,
> > if I set ROWS to 635000 and COLS to 2350, it will give me the
> > following error upon deallocation:
>
> There are lot of allocations there.
> Looks to me that malloc internaly has overflow in pointer arithmetic.
> Perhaps you use 32 bit malloc on 64 bit setup somehow?
>
> Try following and see if works:
>
> const unsigned rows = 635000;
> const unsigned cols = 2350;
> int (*p)[cols] = new int[rows][cols];
> for(unsigned i = 0; i<rows;++i)
> {
> for(unsigned j=0;j<cols;++j)
> p[i][j] = 0;
> }
> cin.get();
> delete[] p;
>
> Greetings, Branimir

I can't seem to get that code to compile, it complains about the line
before the for loop. I'm using gcc 4.0.1 for apple/darwin. Any ideas?

I think you're probably right, it has something to do with pointer
arithmetic. I'm just not sure what. The malloc() failure is happening
on the for loop where I'm deallocating each row of the array, I've
figured out that much. Beyond that, I'm not sure.

I've tried the following compiler options too but they don't warn me
of anything:

g++ -o HugeMemory.exe -O3 -mcpu=powerpc64 -arch ppc64 -faltivec -Wall -
Wconversion HugeMemory.cpp

wrote:
> On May 11, 9:15 pm, Branimir Maksimovic <b...@hotmail.com> wrote:
>> There are lot of allocations there.
>> Looks to me that malloc internaly has overflow in pointer arithmetic.
>> Perhaps you use 32 bit malloc on 64 bit setup somehow?
>>
>> Try following and see if works:
>>
>> const unsigned rows = 635000;
>> const unsigned cols = 2350;
>> int (*p)[cols] = new int[rows][cols];
>> for(unsigned i = 0; i<rows;++i)
>> {
>> for(unsigned j=0;j<cols;++j)
>> p[i][j] = 0;
>> }
>> cin.get();
>> delete[] p;
>>
>> Greetings, Branimir
>
> I can't seem to get that code to compile, it complains about the line
> before the for loop. I'm using gcc 4.0.1 for apple/darwin. Any ideas?
>
The code is fine, with the exception of an integer overflow warning from
gcc.

> I think you're probably right, it has something to do with pointer
> arithmetic. I'm just not sure what. The malloc() failure is happening
> on the for loop where I'm deallocating each row of the array, I've
> figured out that much. Beyond that, I'm not sure.
>
Update your code to scan the rows for duplicate addresses. If you find
one, something is wrong!

--
Ian Collins.

On May 12, 5:37 am, welch.r...@gmail.com wrote:
> On May 11, 9:15 pm, Branimir Maksimovic <b...@hotmail.com> wrote:
>
>
>
> > On May 12, 2:02 am, welch.r...@gmail.com wrote:
>
> > > Hi all,
>
> > > Having a problem with addressing large amounts of memory. I have a
> > > simple piece of code here that is meant to allocate a large piece of
> > > memory on a ppc64 machine. The code is:
>
.......
> > > If I set ROWS and COLS to 5000 and 5000, it works just fine. However,
> > > if I set ROWS to 635000 and COLS to 2350, it will give me the
> > > following error upon deallocation:
>
> > There are lot of allocations there.
> > Looks to me that malloc internaly has overflow in pointer arithmetic.
> > Perhaps you use 32 bit malloc on 64 bit setup somehow?
>
> > Try following and see if works:
>
> > const unsigned rows = 635000;
> > const unsigned cols = 2350;
> > int (*p)[cols] = new int[rows][cols];
> > for(unsigned i = 0; i<rows;++i)
> > {
> > for(unsigned j=0;j<cols;++j)
> > p[i][j] = 0;
> > }
> > cin.get();
> > delete[] p;
>
> > Greetings, Branimir
>
> I can't seem to get that code to compile, it complains about the line
> before the for loop. I'm using gcc 4.0.1 for apple/darwin. Any ideas?

What is the error message?

Greetings, Branimir.

On 2007-05-12 02:02, wrote:
> Hi all,
>
> Having a problem with addressing large amounts of memory. I have a
> simple piece of code here that is meant to allocate a large piece of
> memory on a ppc64 machine. The code is:
>
> /*
> Test to see what happens when we try to allocate a massively huge
> piece of memory.
> */
>
> #include <iostream>
> #include <string>
> #include <stdexcept>
> using namespace std;
>
> int main(int argc, char** argv) {
> cout << "Attemping to allocate.." << endl;
>
> const int ROWS = 635000;
> const int COLS = 2350;
>
> // Allocate.
> try {
> int** test = new int*[ROWS];
> for (int i = 0; i < ROWS; i++) {
> test[i] = new int[COLS];
> for (int j = 0; j < COLS; j++) {
> test[i][j] = 0;
> }
> }
>
> cout << "Allocation succeeded!" << endl;
> cout << "Press a key to deallocate and continue.." << endl;
> string blank;
> getline(cin,blank);
>
> // Deallocate.
> for (int k = 0; k < ROWS; k++) {
> delete[] test[k];
> }
> delete[] test;
>
> cout << "Deallocation completed!" << endl;
> cout << "Press a key to terminate.." << endl;
> getline(cin,blank);
> }
> catch(bad_alloc& e) {
> cout << "Allocation failed.." << endl;
> }
>
> return 0;
> }
>
> If I set ROWS and COLS to 5000 and 5000, it works just fine. However,
> if I set ROWS to 635000 and COLS to 2350, it will give me the
> following error upon deallocation:
>
> HugeMemory.exe(2946 malloc: *** Deallocation of a pointer not
> malloced: 0x20afd2000; This could be a double free(), or free() called
> with the middle of an allocated block; Try setting environment
> variable MallocHelp to see tools to help debug
>
> Note that the allocation step succeeds, and that I only receive this
> error after allowing the code to deallocate the array.

I have absolutely no idea, but you could try to make the code a bit more
simple by allocating everything in one large block instead:

#include <iostream>
#include <string>
#include <stdexcept>
using namespace std;

int main() {
cout << "Attemping to allocate.." << endl;

const int ROWS = 635000;
const int COLS = 2350;

// Allocate.
try {
int* test = new int[ROWS * COLS];
for (int i = 0; i < ROWS * COLS; i++) {
test[i] = 0;
}

cout << "Allocation succeeded!" << endl;
cout << "Press a key to deallocate and continue.." << endl;

string blank;
getline(cin,blank);

// Deallocate.
delete[] test;

cout << "Deallocation completed!" << endl;
cout << "Press a key to terminate.." << endl;
getline(cin,blank);
}
catch(bad_alloc& e) {
cout << "Allocation failed.." << endl;
}

return 0;
}

Do you still get the same error (or some other)? If you do there's
probably something wrong with your standard library.

--
Erik Wikström

On May 11, 9:15 pm, Branimir Maksimovic <b...@hotmail.com> wrote:
> On May 12, 2:02 am, welch.r...@gmail.com wrote:
>
>
>
> > Hi all,
>
> > Having a problem with addressing large amounts of memory. I have a
> > simple piece of code here that is meant to allocate a large piece of
> > memory on a ppc64 machine. The code is:
>
> > /*
> > Test to see what happens when we try to allocate a massively huge
> > piece of memory.
> > */
>
> > #include <iostream>
> > #include <string>
> > #include <stdexcept>
> > using namespace std;
>
> > int main(int argc, char** argv) {
> > cout << "Attemping to allocate.." << endl;
>
> > const int ROWS = 635000;
> > const int COLS = 2350;
>
> > // Allocate.
> > try {
> > int** test = new int*[ROWS];
> > for (int i = 0; i < ROWS; i++) {
> > test[i] = new int[COLS];
> > for (int j = 0; j < COLS; j++) {
> > test[i][j] = 0;
> > }
> > }
>
> > cout << "Allocation succeeded!" << endl;
> > cout << "Press a key to deallocate and continue.." << endl;
> > string blank;
> > getline(cin,blank);
>
> > // Deallocate.
> > for (int k = 0; k < ROWS; k++) {
> > delete[] test[k];
> > }
> > delete[] test;
>
> > cout << "Deallocation completed!" << endl;
> > cout << "Press a key to terminate.." << endl;
> > getline(cin,blank);
> > }
> > catch(bad_alloc& e) {
> > cout << "Allocation failed.." << endl;
> > }
>
> > return 0;
>
> > }
>
> > If I set ROWS and COLS to 5000 and 5000, it works just fine. However,
> > if I set ROWS to 635000 and COLS to 2350, it will give me the
> > following error upon deallocation:
>
> There are lot of allocations there.
> Looks to me that malloc internaly has overflow in pointer arithmetic.
> Perhaps you use 32 bit malloc on 64 bit setup somehow?
>
> Try following and see if works:
>
> const unsigned rows = 635000;
> const unsigned cols = 2350;
> int (*p)[cols] = new int[rows][cols];
> for(unsigned i = 0; i<rows;++i)
> {
> for(unsigned j=0;j<cols;++j)
> p[i][j] = 0;
> }
> cin.get();
> delete[] p;
>
> Greetings, Branimir

This is what happens:

HugeMemory2.cpp:4: error: expected unqualified-id before 'for'
HugeMemory2.cpp:4: error: expected constructor, destructor, or type
conversion before '<' token
HugeMemory2.cpp:4: error: expected unqualified-id before '++' token
HugeMemory2.cpp:9: error: expected constructor, destructor, or type
conversion before '.' token
HugeMemory2.cpp:10: error: expected unqualified-id before 'delete'

I thought maybe it was because there's no 'int' after 'unsigned' but
that didn't help.

On May 12, 6:09 am, Erik Wikström <Erik-wikst...@telia.com> wrote:
> On 2007-05-12 02:02, welch.r...@gmail.com wrote:
>
>
>
> > Hi all,
>
> > Having a problem with addressing large amounts of memory. I have a
> > simple piece of code here that is meant to allocate a large piece of
> > memory on a ppc64 machine. The code is:
>
> > /*
> > Test to see what happens when we try to allocate a massively huge
> > piece of memory.
> > */
>
> > #include <iostream>
> > #include <string>
> > #include <stdexcept>
> > using namespace std;
>
> > int main(int argc, char** argv) {
> > cout << "Attemping to allocate.." << endl;
>
> > const int ROWS = 635000;
> > const int COLS = 2350;
>
> > // Allocate.
> > try {
> > int** test = new int*[ROWS];
> > for (int i = 0; i < ROWS; i++) {
> > test[i] = new int[COLS];
> > for (int j = 0; j < COLS; j++) {
> > test[i][j] = 0;
> > }
> > }
>
> > cout << "Allocation succeeded!" << endl;
> > cout << "Press a key to deallocate and continue.." << endl;
> > string blank;
> > getline(cin,blank);
>
> > // Deallocate.
> > for (int k = 0; k < ROWS; k++) {
> > delete[] test[k];
> > }
> > delete[] test;
>
> > cout << "Deallocation completed!" << endl;
> > cout << "Press a key to terminate.." << endl;
> > getline(cin,blank);
> > }
> > catch(bad_alloc& e) {
> > cout << "Allocation failed.." << endl;
> > }
>
> > return 0;
> > }
>
> > If I set ROWS and COLS to 5000 and 5000, it works just fine. However,
> > if I set ROWS to 635000 and COLS to 2350, it will give me the
> > following error upon deallocation:
>
> > HugeMemory.exe(2946 malloc: *** Deallocation of a pointer not
> > malloced: 0x20afd2000; This could be a double free(), or free() called
> > with the middle of an allocated block; Try setting environment
> > variable MallocHelp to see tools to help debug
>
> > Note that the allocation step succeeds, and that I only receive this
> > error after allowing the code to deallocate the array.
>
> I have absolutely no idea, but you could try to make the code a bit more
> simple by allocating everything in one large block instead:
>
> #include <iostream>
> #include <string>
> #include <stdexcept>
> using namespace std;
>
> int main() {
> cout << "Attemping to allocate.." << endl;
>
> const int ROWS = 635000;
> const int COLS = 2350;
>
> // Allocate.
> try {
> int* test = new int[ROWS * COLS];
> for (int i = 0; i < ROWS * COLS; i++) {
> test[i] = 0;
> }
>
> cout << "Allocation succeeded!" << endl;
> cout << "Press a key to deallocate and continue.." << endl;
>
> string blank;
> getline(cin,blank);
>
> // Deallocate.
> delete[] test;
>
> cout << "Deallocation completed!" << endl;
> cout << "Press a key to terminate.." << endl;
> getline(cin,blank);
> }
> catch(bad_alloc& e) {
> cout << "Allocation failed.." << endl;
> }
>
> return 0;
>
> }
>
> Do you still get the same error (or some other)? If you do there's
> probably something wrong with your standard library.
>
> --
> Erik Wikström

Nope! That code succeeds. However, that code should require around 6
GB of RAM, correct? If I use top to check the memory usage of the
process, I see:

3688 HugeMemory 0.0% 0:39.81 1 13 34 232K 6.51M 1.56G
1.63G

So it's not even coming close.. or am I reading that incorrectly?
Quite strange..