Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: 64 bit memory usage

Reply
Thread Tools

Re: 64 bit memory usage

 
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      12-09-2010
On Wed, 8 Dec 2010 14:44:30 +0000, Rob Randall <(E-Mail Removed)>
declaimed the following in gmane.comp.python.general:

> I am trying to understand how much memory is available to a 64 bit python
> process running under Windows XP 64 bit.
>
> When I run tests just creating a series of large dictionaries containing
> string keys and float values I do not seem to be able to grow the process
> beyond the amount of RAM present.
>
> For example, on a box with 2GB RAM and 3 GB pagefile the process stalls at
> around 2GB.
>
> On another machine with 16GB RAM and 24GB pagefile the process stalls at
> 16GB.
>


Probably need to ask M$, but I can understand this behavior as a
hypothetical...

The page file can be larger than physical memory because it contains
memory "images" for multiple processes. However, all those "images" have
to map into the physically addressable memory -- so a process is likely
limited to physical memory, but you can have multiple processes adding
up to physical + pagefile in total.
--
Wulfraed Dennis Lee Bieber AF6VN
http://www.velocityreviews.com/forums/(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
 
 
 
Nobody
Guest
Posts: n/a
 
      12-09-2010
Rob Randall <(E-Mail Removed)> wrote:

> I am trying to understand how much memory is available to a 64 bit python
> process running under Windows XP 64 bit.
>
> When I run tests just creating a series of large dictionaries containing
> string keys and float values I do not seem to be able to grow the process
> beyond the amount of RAM present.
>
> For example, on a box with 2GB RAM and 3 GB pagefile the process stalls at
> around 2GB.


What do you mean by "stalls"? Do you get an exception, or does the program
just slow to a crawl?

 
Reply With Quote
 
 
 
 
John Nagle
Guest
Posts: n/a
 
      12-09-2010
On 12/8/2010 10:42 PM, Dennis Lee Bieber wrote:
> On Wed, 8 Dec 2010 14:44:30 +0000, Rob Randall<(E-Mail Removed)>
> declaimed the following in gmane.comp.python.general:
>
>> I am trying to understand how much memory is available to a 64 bit python
>> process running under Windows XP 64 bit.
>>
>> When I run tests just creating a series of large dictionaries containing
>> string keys and float values I do not seem to be able to grow the process
>> beyond the amount of RAM present.


If you get to the point where you need multi-gigabyte Python
dictionaries, you may be using the wrong tool for the job.
If it's simply that you need to manage a large amount of data,
that's what databases are for.

If this is some super high performance application that needs to
keep a big database in memory for performance reasons, CPython
is probably too slow. For that, something like Google's BigTable
may be more appropriate, and will scale to terabytes if necessary.

John Nagle
 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      12-11-2010
On 12/10/2010 2:03 PM, Rob Randall wrote:
> I manged to get my python app past 3GB on a smaller 64 bit machine.
> On a test to check memory usage with gc disabled only an extra 6MB was
> used.
> The figures were 1693MB to 1687MB.
>
> This is great.
>
> Thanks again for the help.


Do remember, though, that with the GC turned off you will "lose" memory
if you accidentally create cyclic data structures, since they will never
be reclaimed. It doesn't sound like this is an issue, but I wanted this
to act as a warning to others who might come across your solution but
have programmed less carefully.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon 2011 Atlanta March 9-17 http://us.pycon.org/
See Python Video! http://python.mirocommunity.org/
Holden Web LLC http://www.holdenweb.com/
 
Reply With Quote
 
Steve Holden
Guest
Posts: n/a
 
      12-11-2010
On 12/10/2010 2:03 PM, Rob Randall wrote:
> I manged to get my python app past 3GB on a smaller 64 bit machine.
> On a test to check memory usage with gc disabled only an extra 6MB was
> used.
> The figures were 1693MB to 1687MB.
>
> This is great.
>
> Thanks again for the help.


Do remember, though, that with the GC turned off you will "lose" memory
if you accidentally create cyclic data structures, since they will never
be reclaimed. It doesn't sound like this is an issue, but I wanted this
to act as a warning to others who might come across your solution but
have programmed less carefully.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
PyCon 2011 Atlanta March 9-17 http://us.pycon.org/
See Python Video! http://python.mirocommunity.org/
Holden Web LLC http://www.holdenweb.com/

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
What is the point of having 16 bit colour if a computer monitor can only display 8 bit colour? How do you edit 16 bit colour when you can only see 8 bit? Scotius Digital Photography 6 07-13-2010 03:33 AM
retrieving CPU Usage and Memory Usage information in JAVA hvt Java 0 03-13-2007 01:07 PM
64 bit - Windows Liberty 64bit, Windows Limited Edition 64 Bit, Microsoft SQL Server 2000 Developer Edition 64 Bit, IBM DB2 64 bit - new ! vvcd Computer Support 0 09-17-2004 08:15 PM
64 bit - Windows Liberty 64bit, Windows Limited Edition 64 Bit,Microsoft SQL Server 2000 Developer Edition 64 Bit, IBM DB2 64 bit - new! Ionizer Computer Support 1 01-01-2004 07:27 PM
Need help on memory usage VS PF usage metfan Java 2 10-21-2003 01:58 PM



Advertisments