Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > OutOfMemoryException

Reply
Thread Tools

OutOfMemoryException

 
 
srinivas.veeranki@gmail.com
Guest
Posts: n/a
 
      12-20-2006
Hi All,

While I am retrieving from the data base I am getting more than 45k
Records. I am unable to stored all those records in to the List. Here I
am getting OutOfMemoryException. How Can i store all those records in
the List.

Regards,

Srinivas.

 
Reply With Quote
 
 
 
 
bugbear
Guest
Posts: n/a
 
      12-20-2006
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Hi All,
>
> While I am retrieving from the data base I am getting more than 45k
> Records. I am unable to stored all those records in to the List. Here I
> am getting OutOfMemoryException. How Can i store all those records in
> the List.


Use less memory per record. Which may involve
simple data packing or compression.

BugBear
 
Reply With Quote
 
 
 
 
Oliver Wong
Guest
Posts: n/a
 
      12-20-2006

"bugbear" <bugbear@trim_papermule.co.uk_trim> wrote in message
news:45892c1f$0$8747$(E-Mail Removed)...
> (E-Mail Removed) wrote:
>> Hi All,
>>
>> While I am retrieving from the data base I am getting more than 45k
>> Records. I am unable to stored all those records in to the List. Here I
>> am getting OutOfMemoryException. How Can i store all those records in
>> the List.

>
> Use less memory per record. Which may involve
> simple data packing or compression.


Alternatively, assign more memory to your Java program, or don't
actually store all the data in the list. See the Proxy design pattern, for
an example of the latter.

- Oliver


 
Reply With Quote
 
bugbear
Guest
Posts: n/a
 
      12-20-2006
Oliver Wong wrote:
>>Use less memory per record. Which may involve
>>simple data packing or compression.

>
>
> Alternatively, assign more memory to your Java program, or don't
> actually store all the data in the list. See the Proxy design pattern, for
> an example of the latter.


At the risk of topic drift, does any one
know of a List implementation
which doesn't suffer from ArrayList's
"growth problem", causing (IMHO) premature
out of memory reports?

The problem is that when the current memory allocation
is exceeded, an ArrayList tries to double in size;

assuming you're currently using (e.g.) 64 Mb,
it will try and allocate 128 Mb, then copy the data
from the 64 to the 128, and only then does the 64
become available for GC.

Assuming you actual data size were 65 Mb, this
means you would require 196 Mb to "survive" the handover.

A better data structure (w.r.t growth) would be a list
of BLOCKs of some size. Additional blocks can be added,
and serial (iterator) access would be fast.

Random access is also "quite" fast, if the blocks are a
"reasonable" size.

Does anyone know an implementation of this, or something like it?

BugBear
 
Reply With Quote
 
Simon Brooke
Guest
Posts: n/a
 
      12-20-2006
in message <(E-Mail Removed) m>,
(E-Mail Removed) ('(E-Mail Removed)') wrote:

> Hi All,
>
> While I am retrieving from the data base I am getting more than 45k
> Records. I am unable to stored all those records in to the List. Here I
> am getting OutOfMemoryException. How Can i store all those records in
> the List.


Records belong in the database. Retrieve them a few at a time. As a general
rule, trying to schlurp the whole result of a database query into memory
is a bad idea, because the data can be arbitrarily big.

--
(E-Mail Removed) (Simon Brooke) http://www.jasmine.org.uk/~simon/

;; Life would be much easier if I had the source code.

 
Reply With Quote
 
Daniel Pitts
Guest
Posts: n/a
 
      12-20-2006

bugbear wrote:
> Oliver Wong wrote:
> >>Use less memory per record. Which may involve
> >>simple data packing or compression.

> >
> >
> > Alternatively, assign more memory to your Java program, or don't
> > actually store all the data in the list. See the Proxy design pattern, for
> > an example of the latter.

>
> At the risk of topic drift, does any one
> know of a List implementation
> which doesn't suffer from ArrayList's
> "growth problem", causing (IMHO) premature
> out of memory reports?
>
> The problem is that when the current memory allocation
> is exceeded, an ArrayList tries to double in size;
>
> assuming you're currently using (e.g.) 64 Mb,
> it will try and allocate 128 Mb, then copy the data
> from the 64 to the 128, and only then does the 64
> become available for GC.
>
> Assuming you actual data size were 65 Mb, this
> means you would require 196 Mb to "survive" the handover.
>
> A better data structure (w.r.t growth) would be a list
> of BLOCKs of some size. Additional blocks can be added,
> and serial (iterator) access would be fast.
>
> Random access is also "quite" fast, if the blocks are a
> "reasonable" size.
>
> Does anyone know an implementation of this, or something like it?
>
> BugBear


If you know before hand that you'll have a lot of data, you can tell
the ArrayList to allocate a capacity that is as large (or larger) than
you need.

 
Reply With Quote
 
Daniel Pitts
Guest
Posts: n/a
 
      12-20-2006

(E-Mail Removed) wrote:
> Hi All,
>
> While I am retrieving from the data base I am getting more than 45k
> Records. I am unable to stored all those records in to the List. Here I
> am getting OutOfMemoryException. How Can i store all those records in
> the List.
>
> Regards,
>
> Srinivas.


If possilbe, try to retrieve only a subset (using LIMIT in MySQL)

If you go through the records, and test them for some condition, and
THEN work with them, do that test in the SQL statement, not Java.

Depending on how you are accessing the database, there may be ways to
"paginate" your result list, so that only a portion is loaded in memory
at any one time.

 
Reply With Quote
 
Mark Jeffcoat
Guest
Posts: n/a
 
      12-21-2006
Simon Brooke <(E-Mail Removed)> writes:

> in message <(E-Mail Removed) m>,
> (E-Mail Removed) ('(E-Mail Removed)') wrote:
>
>> Hi All,
>>
>> While I am retrieving from the data base I am getting more than 45k
>> Records. I am unable to stored all those records in to the List. Here I
>> am getting OutOfMemoryException. How Can i store all those records in
>> the List.

>
> Records belong in the database. Retrieve them a few at a time. As a general
> rule, trying to schlurp the whole result of a database query into memory
> is a bad idea, because the data can be arbitrarily big.



More concretely:

If you're getting a java.sql.ResultSet from your database,
you're already retrieving them a few at a time. (I have few
nice things to say about this interface, but it is handy
that you get this for free.) You can step through the results
by repeatedly calling next(), and never have to worry about
memory, as long as you can make your computation without
keeping a reference to all those rows.



(You could even wrap up this behavior into your own List
subclass, which would provide an Iterator that would call
ResultSet.next(). I think this would be a terrible idea.)

--
Mark Jeffcoat
Austin, TX
 
Reply With Quote
 
EJP
Guest
Posts: n/a
 
      12-21-2006
bugbear wrote:
> At the risk of topic drift, does any one
> know of a List implementation
> which doesn't suffer from ArrayList's
> "growth problem", causing (IMHO) premature
> out of memory reports?


java.util.LinkedList.

> The problem is that when the current memory allocation
> is exceeded, an ArrayList tries to double in size;


No. 'The details of the growth policy are not specified beyond the fact
that adding an element has constant amortized time cost.' The JDK 1.5
implementation increases the size by 50%.
 
Reply With Quote
 
Mike Schilling
Guest
Posts: n/a
 
      12-21-2006

"bugbear" <bugbear@trim_papermule.co.uk_trim> wrote in message
news:458973b5$0$8759$(E-Mail Removed)...
> Oliver Wong wrote:
>>>Use less memory per record. Which may involve
>>>simple data packing or compression.

>>
>>
>> Alternatively, assign more memory to your Java program, or don't
>> actually store all the data in the list. See the Proxy design pattern,
>> for an example of the latter.

>
> At the risk of topic drift, does any one
> know of a List implementation
> which doesn't suffer from ArrayList's
> "growth problem", causing (IMHO) premature
> out of memory reports?
>
> The problem is that when the current memory allocation
> is exceeded, an ArrayList tries to double in size;
>
> assuming you're currently using (e.g.) 64 Mb,
> it will try and allocate 128 Mb, then copy the data
> from the 64 to the 128, and only then does the 64
> become available for GC.
>
> Assuming you actual data size were 65 Mb, this
> means you would require 196 Mb to "survive" the handover.
>


No, the only thing that increases (doubling or otherwise) is the array of
references it uses; the data those references point to isn't duplicated. If
you have 64K objects and the reference array doubles in sixe, you'd add 128K
references, which, depending on the size of a reference, would probably
total between 512KB and 1MB.


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
System.OutOfMemoryException: Out of memory. Versteijn ASP .Net 7 03-07-2008 06:57 PM
Exception of type System.OutOfMemoryException is thrown =?Utf-8?B?U3Rhbg==?= ASP .Net 11 12-28-2004 07:00 PM
OutOfMemoryException Error: Garbage Collector doesn't release memory to OS Pyramis ASP .Net 0 01-25-2004 04:37 PM
OutOfMemoryException =?Utf-8?B?amF6?= ASP .Net 2 12-16-2003 12:41 AM
Base Exception:System.OutOfMemoryException error message =?Utf-8?B?UmlwdWwgSGFuZGE=?= ASP .Net 1 11-18-2003 07:52 AM



Advertisments