Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Java > Memory utilization

Reply
Thread Tools

Memory utilization

 
 
ghostwhoowalks@gmail.com
Guest
Posts: n/a
 
      01-30-2008
I was trying to profile the following piece of code using NetBeans:

public static void main(String[] args)
{
Map<String, String> map = new HashMap<String, String>();
for( int i = 0; i < Integer.MAX_VALUE; ++i )
{
map.put(Integer.valueOf(i).toString(),
Integer.valueOf(i).toString());
}
}


I set my max and min heap size to be 1GB. I find that the program dies
after having created 4M entries (at which point the heap used is at
1GB) although the memory used by the objects as indicated by the
Netbeans memory profiler is the following:

String -- 48MB
char[] -- 48 MB
HeapCharBuffer -- 45MB
Integer -- 19MB
HashMap$Entry -- 19MB


This totals to less than 200 MB and the rest of the objects are at 0%
of the total used memory at most another 200MB. How come the heap
usage is at 1GB at this point of time? I am extremely confused. Could
someone please explain what might be going on?

Thanks
A
 
Reply With Quote
 
 
 
 
tzvika.barenholz@gmail.com
Guest
Posts: n/a
 
      01-30-2008
On Jan 30, 4:54 am, (E-Mail Removed) wrote:
> I was trying to profile the following piece of code using NetBeans:
>
> public static void main(String[] args)
> {
> Map<String, String> map = new HashMap<String, String>();
> for( int i = 0; i < Integer.MAX_VALUE; ++i )
> {
> map.put(Integer.valueOf(i).toString(),
> Integer.valueOf(i).toString());
> }
>
> }
>
> I set my max and min heap size to be 1GB. I find that the program dies
> after having created 4M entries (at which point the heap used is at
> 1GB) although the memory used by the objects as indicated by the
> Netbeans memory profiler is the following:
>
> String -- 48MB
> char[] -- 48 MB
> HeapCharBuffer -- 45MB
> Integer -- 19MB
> HashMap$Entry -- 19MB
>
> This totals to less than 200 MB and the rest of the objects are at 0%
> of the total used memory at most another 200MB. How come the heap
> usage is at 1GB at this point of time? I am extremely confused. Could
> someone please explain what might be going on?
>
> Thanks
> A


The profiler itself takes up quite a bit, but doesn't show it to you.
Also, space is needed for classes and other items of overhead.

T
 
Reply With Quote
 
 
 
 
GArlington
Guest
Posts: n/a
 
      01-30-2008
On Jan 30, 2:54 am, (E-Mail Removed) wrote:
> I was trying to profile the following piece of code using NetBeans:
>
> public static void main(String[] args)
> {
> Map<String, String> map = new HashMap<String, String>();
> for( int i = 0; i < Integer.MAX_VALUE; ++i )
> {
> map.put(Integer.valueOf(i).toString(),
> Integer.valueOf(i).toString());
> }
>
> }
>
> I set my max and min heap size to be 1GB. I find that the program dies
> after having created 4M entries (at which point the heap used is at
> 1GB) although the memory used by the objects as indicated by the
> Netbeans memory profiler is the following:
>
> String -- 48MB
> char[] -- 48 MB
> HeapCharBuffer -- 45MB
> Integer -- 19MB
> HashMap$Entry -- 19MB
>
> This totals to less than 200 MB and the rest of the objects are at 0%
> of the total used memory at most another 200MB. How come the heap
> usage is at 1GB at this point of time? I am extremely confused. Could
> someone please explain what might be going on?
>
> Thanks
> A


Something is VERY wrong here:
"4M entries" in Map<String, String> will mean 8M String(s) - each
String has an OVERHEAD of 40bytes (AFAIK) - this alone should give you
320MB heap usage even when ALL String(s) are empty.
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      01-30-2008
GArlington wrote:
> each
> String has an OVERHEAD of 40bytes (AFAIK)


I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
seems ridiculously high. Where'd you get that number?

--
Lew
 
Reply With Quote
 
gwoodhouse@gmail.com
Guest
Posts: n/a
 
      01-30-2008
On Jan 30, 3:20 pm, Lew <(E-Mail Removed)> wrote:
> GArlington wrote:
> > each
> > String has an OVERHEAD of 40bytes (AFAIK)

>
> I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
> seems ridiculously high. Where'd you get that number?
>
> --
> Lew


On the machine you are on is it possible to increase your max heap
space higher? If so problem solved!

Graeme
 
Reply With Quote
 
Patricia Shanahan
Guest
Posts: n/a
 
      01-30-2008
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> On Jan 30, 3:20 pm, Lew <(E-Mail Removed)> wrote:
>
>>GArlington wrote:
>>
>>>each
>>>String has an OVERHEAD of 40bytes (AFAIK)

>>
>>I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
>>seems ridiculously high. Where'd you get that number?
>>
>>--
>>Lew

>
>
> On the machine you are on is it possible to increase your max heap
> space higher? If so problem solved!


That depends on what the problem really is. The program looks more like
a test case than an actual application. I believe the problem is
explaining the relationship between the NetBeans memory profile, the
program, and the out-of-memory condition. If so, increasing the heap
space could not possibly solve it, and might make it harder to solve.

Patricia
 
Reply With Quote
 
Eric Sosman
Guest
Posts: n/a
 
      01-30-2008
(E-Mail Removed) wrote:
> On Jan 30, 3:20 pm, Lew <(E-Mail Removed)> wrote:
>> GArlington wrote:
>>> each
>>> String has an OVERHEAD of 40bytes (AFAIK)

>> I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
>> seems ridiculously high. Where'd you get that number?


Might be high, but doesn't seem ridiculous. A String
instance has its object-ness plus three ints and a char[]
reference. The char[] will have its own object-ness plus
a length. If object-ness takes eight bytes and a reference
takes four (32-bit JVM), we're up to at least 36 bytes. If
there's an eight-byte alignment requirement, 40 bytes is
right on the money.

> On the machine you are on is it possible to increase your max heap
> space higher? If so problem solved!


He's trying to create 2G pairs of String objects (72-80
bytes of overhead per pair) plus 2G HashMap.Entry objects
(~24 bytes each, under the same set of assumptions), hence
about 192-208GB -- and that's not counting the "payload" of
the char values, which will add another 80GB or thereabouts.
272GB is well beyond what a 32-bit JVM can manage, so he'll
need to go to a 64-bit version -- and this will expand the
String overhead to at least 40 bytes, and the HashMap.Entry
size to at least 36-40, raising the total to about 312-320GB.
And that's just for this one data structure; the rest of the
Java environment needs some heap, too. I'd guess that a
machine with half a terabyte of RAM could just about manage
it, with luck and a following wind.

... but in truth he's not really trying to populate this
enormous and pointlessly wasteful data structure; he's trying
to understand how the NetBeans profiler reports the memory
used by the program, and I'm afraid I can't help much with
that. After 4M pairs he sees

> String -- 48MB
> char[] -- 48 MB
> HeapCharBuffer -- 45MB
> Integer -- 19MB
> HashMap$Entry -- 19MB


.... which works out to 6 bytes per string, 6 bytes per char[],
and <5 bytes per HashMap.Entry -- clearly, the profiler reports
some other notion of "memory used" than we've been talking
about in the last few messages. All I can suggest is RTFM.

--
(E-Mail Removed)
 
Reply With Quote
 
ghostwhoowalks@gmail.com
Guest
Posts: n/a
 
      01-31-2008
On Jan 30, 8:48 am, Eric Sosman <(E-Mail Removed)> wrote:
> (E-Mail Removed) wrote:
> > On Jan 30, 3:20 pm, Lew <(E-Mail Removed)> wrote:
> >> GArlington wrote:
> >>> each
> >>> String has an OVERHEAD of 40bytes (AFAIK)
> >> I'm not sure that's correct. Are you sure it isn't 4 bytes per String? 40
> >> seems ridiculously high. Where'd you get that number?

>
> Might be high, but doesn't seem ridiculous. A String
> instance has its object-ness plus three ints and a char[]
> reference. The char[] will have its own object-ness plus
> a length. If object-ness takes eight bytes and a reference
> takes four (32-bit JVM), we're up to at least 36 bytes. If
> there's an eight-byte alignment requirement, 40 bytes is
> right on the money.
>
> > On the machine you are on is it possible to increase your max heap
> > space higher? If so problem solved!

>
> He's trying to create 2G pairs of String objects (72-80
> bytes of overhead per pair) plus 2G HashMap.Entry objects
> (~24 bytes each, under the same set of assumptions), hence
> about 192-208GB -- and that's not counting the "payload" of
> the char values, which will add another 80GB or thereabouts.
> 272GB is well beyond what a 32-bit JVM can manage, so he'll
> need to go to a 64-bit version -- and this will expand the
> String overhead to at least 40 bytes, and the HashMap.Entry
> size to at least 36-40, raising the total to about 312-320GB.
> And that's just for this one data structure; the rest of the
> Java environment needs some heap, too. I'd guess that a
> machine with half a terabyte of RAM could just about manage
> it, with luck and a following wind.
>
> ... but in truth he's not really trying to populate this
> enormous and pointlessly wasteful data structure; he's trying
> to understand how the NetBeans profiler reports thememory
> used by the program, and I'm afraid I can't help much with
> that. After 4M pairs he sees
>
> > String -- 48MB
> > char[] -- 48 MB
> > HeapCharBuffer -- 45MB
> > Integer -- 19MB
> > HashMap$Entry -- 19MB

>
> ... which works out to 6 bytes per string, 6 bytes per char[],
> and <5 bytes per HashMap.Entry -- clearly, the profiler reports
> some other notion of "memoryused" than we've been talking
> about in the last few messages. All I can suggest is RTFM.
>
> --
> (E-Mail Removed)


Obviously my intention is not to fill 2G of pairs. If you thought so I
think you are dumber than I am. The point is to understand when the
JVM craps out how much of memory is taken up by the VM apart from the
objects used by the application. The profiler reports that the memory
used by the objects in the heap is atmost 200M. But the heap memory
used is at 800M. So why is there a 4x increase? Even if you throw in
200M used by the profiler there is an additional 400M that is
unaccounted for. I even tried MinHeapRatio and MaxHeapRatio options
but to no avail. I agree that there will be some overhead with Java
but 3x or 4x is seriously flawed. I am wondering if someone can point
me to what might be going wrong.

A
 
Reply With Quote
 
Lew
Guest
Posts: n/a
 
      02-01-2008
(E-Mail Removed) wrote:
> The point is to understand when the
> JVM craps out how much of memory is taken up by the VM apart from the
> objects used by the application. The profiler reports that the memory
> used by the objects in the heap is atmost 200M. But the heap memory
> used is at 800M. So why is there a 4x increase?


How are you determining that the heap consumption is 800MB ?

--
Lew
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
tomcat memory utilization, vinodhsri@gmail.com Java 1 03-12-2006 08:24 AM
SNMP OID for memory utilization on CISCO AS5350 ahassiotis Cisco 1 09-08-2005 01:52 PM
Cisco PIX memory utilization atif Cisco 1 02-03-2005 07:19 PM
Memory utilization for Windows services Jay Kavimandan ASP .Net 1 11-02-2004 11:41 AM
Low memory utilization sort? pt Perl 1 06-04-2004 11:09 AM



Advertisments