Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Re: Python does not take up available physical memory

Reply
Thread Tools

Re: Python does not take up available physical memory

 
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      10-19-2012
On Fri, 19 Oct 2012 12:08:53 -0500, Pradipto Banerjee
<(E-Mail Removed)> declaimed the following in
gmane.comp.python.general:

> Hi,
>
> I am trying to read a file into memory. The size of the file is around 1 GB. I have a 3GB memory PC and the Windows Task Manager shows 2.3 GB available physical memory when I was trying to read the file. I tried to read the file as follows:


1) 32-bit Windows OS can only physically address 4GB BUT
2) Unless you have a SERVER version of windows, the most allocated to a
user process is 2GB (there is a boot-time parameter which can configure
WinXP [for example] to allow 3GB to user processes -- the last GB is
always reserved for the OS)

> Is there any reason why python can't read a 1GB file in memory even when a 2.3 GB physical memory is available? Do I need to make a change in some setting or preferences?
>
> I am using python(x,y) distribution (python 2.7) and use Spyder as the IDE.
>

Are you /running/ from within the IDE -- such that the Python
interpreter is considered part of the IDE address space?

<snipped>

My first thought would be that you should NOT be loading an entire
GB file as one chunk (for all I know, Windows low-level I/O read the
file into a OS buffer [1GB] and then tries to pass it on to the Python
memory buffer [another 1GB]... Or Python is reading in chunks -- say 1MB
at a time, and appending chunks... That means it goes "whoops, I've got
a 512MB data buffer and just read another 1MB -- better allocate a 768MB
buffer [total now is 512 + 768 + 1 => 1281MB], copy the 512MB into the
new buffer, append the 1MB, NOW free the older 512MB... You might have
the raw memory, but it may be fragmented such that the system can not
allocate a contiguous 1MB chunk).

If .readlines() works when .read() fails, it is memory fragmentation
(each parsed line is an individual chunk of memory since the list of
lines is just a list of addresses to the lines)
--
Wulfraed Dennis Lee Bieber AF6VN
http://www.velocityreviews.com/forums/(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
RE: Python does not take up available physical memory Pradipto Banerjee Python 6 10-21-2012 02:14 PM
Re: Python does not take up available physical memory Emile van Sebille Python 0 10-20-2012 06:46 AM
RE: Python does not take up available physical memory Pradipto Banerjee Python 1 10-19-2012 09:37 PM
Re: Python does not take up available physical memory MRAB Python 0 10-19-2012 07:13 PM
Re: Python does not take up available physical memory Ian Kelly Python 0 10-19-2012 06:48 PM



Advertisments