Velocity Reviews

Velocity Reviews (http://www.velocityreviews.com/forums/index.php)
-   Python (http://www.velocityreviews.com/forums/f43-python.html)
-   -   Memory footpring of python objects (http://www.velocityreviews.com/forums/t680891-memory-footpring-of-python-objects.html)

BlueBird 04-22-2009 09:24 AM

Memory footpring of python objects
 

Hi,

I have a program that manages several thousands instances of one
object. To reduce memory
consumption, I want of course that specific object to have the
smallest memory footpring possible.

I have a few ideas that I want to experiment with, like using
__slots__, using a tuple or using a dict. My
question is: how do I know the memory footprint of a given python
object ? I could not find any
builtin functions for this in the documentation.

cheers,

Philippe

Chris Rebert 04-22-2009 09:30 AM

Re: Memory footpring of python objects
 
On Wed, Apr 22, 2009 at 2:24 AM, BlueBird <phil@freehackers.org> wrote:
>
> Hi,
>
> I have a program that manages several thousands instances of one
> object. To reduce memory
> consumption, I want of course that specific object to have the
> smallest memory footpring possible.
>
> I have a few ideas that I want to experiment with, like using
> __slots__, using a tuple or using a dict. My
> question is: how do I know the memory footprint of a given python
> object ? I could not find any
> builtin functions for this in the documentation.


sys.getsizeof() - http://docs.python.org/library/sys.html#sys.getsizeof

Cheers,
Chris
--
I have a blog:
http://blog.rebertia.com

Steven D'Aprano 04-22-2009 09:56 AM

Re: Memory footpring of python objects
 
On Wed, 22 Apr 2009 02:30:32 -0700, Chris Rebert wrote:

> On Wed, Apr 22, 2009 at 2:24 AM, BlueBird <phil@freehackers.org> wrote:
>>
>> Hi,
>>
>> I have a program that manages several thousands instances of one
>> object. To reduce memory
>> consumption, I want of course that specific object to have the smallest
>> memory footpring possible.
>>
>> I have a few ideas that I want to experiment with, like using
>> __slots__, using a tuple or using a dict. My question is: how do I know
>> the memory footprint of a given python object ? I could not find any
>> builtin functions for this in the documentation.

>
> sys.getsizeof() - http://docs.python.org/library/sys.html#sys.getsizeof


Only in Python 2.6.

But if you search Activestate, there's a recipe to do the same thing. Ah,
here it is:

http://code.activestate.com/recipes/546530/



--
Steven

Marco Mariani 04-22-2009 09:56 AM

Re: Memory footpring of python objects
 
BlueBird wrote:

> I have a program that manages several thousands instances of one
> object. To reduce memory
> consumption, I want of course that specific object to have the
> smallest memory footpring possible.


Have you thought of using something like the flyweight pattern and a
compact data representation like a numpy array?


BlueBird 04-22-2009 12:09 PM

Re: Memory footpring of python objects
 
On 22 avr, 11:56, Steven D'Aprano
<ste...@REMOVE.THIS.cybersource.com.au> wrote:
> On Wed, 22 Apr 2009 02:30:32 -0700, Chris Rebert wrote:
> > On Wed, Apr 22, 2009 at 2:24 AM, BlueBird <p...@freehackers.org> wrote:

>
> >> Hi,

>
> >> I have a program that manages several thousands instances of one
> >> object. To reduce memory
> >> consumption, I want of course that specific object to have the smallest
> >> memory footpring possible.

>
> >> I have a few ideas that I want to experiment with, like using
> >> __slots__, using a tuple or using a dict. My question is: how do I know
> >> the memory footprint of a given python object ? I could not find any
> >> builtin functions for this in the documentation.

>
> > sys.getsizeof() -http://docs.python.org/library/sys.html#sys.getsizeof

>
> Only in Python 2.6.
>
> But if you search Activestate, there's a recipe to do the same thing. Ah,
> here it is:
>
> http://code.activestate.com/recipes/546530/


Thanks. "There is always a tool that makes you realise how you are
less smart than you thought" - Andrew Morton, about git.

I had a bit of hard time getting through the different sizes reported
and the vocabulary. From what I understood, basicsize is the size of
the object alone, without the size of his references. And asizeof( x,
code=False, limit=100 ) will give me the memory footprint of a new
instance of an object.

In my case, I just want an object with named attributes.

My two attempts for comparison:
import sys, asizeof

def print_sizeof( v_name, v ):
print 'Size of %s: %d, %d' % (v_name,
asizeof.flatsize(v),
asizeof.asizeof(v, code=False, limit=100) )

def test_impl( v ):
assert v.a == 1
assert v.b == 'bbb'
assert v.c == False

### The reference, a dictionnary
ref_dict = {'a':1, 'b':'bbb', 'c':False }
print_sizeof( 'ref_dict', ref_dict )

### A modified dictionnary
class AttrDict (dict):
"""Dictionary allowing attribute access to its elements if they
are valid attribute names and not already existing methods."""

def __getattr__ (self, name):
return self[name]

attrDict = AttrDict( ref_dict )
test_impl( attrDict )
print_sizeof( 'attrDict', attrDict )

### Using __slots__
class ObjWithSlots(object):
__slots__ = [ 'a', 'b', 'c' ]

objWithSlots = ObjWithSlots()
for k in ref_dict:
setattr( ObjWithSlots, k, ref_dict[k] )
test_impl( objWithSlots )
print_sizeof( 'ObjectWithSlots', objWithSlots )

is giving me:
Size of ref_dict: 140, 304
Size of attrDict: 140,
304
Size of ObjectWithSlots: 36,
144

So, it looks like I could shrink down the structure to 144 bytes in a
simple case.

Marco, my objects have varying length data structures (strings) so I
don't think that would be possible. And it would clearly be overkill
in my case.





All times are GMT. The time now is 01:01 PM.

Powered by vBulletin®. Copyright ©2000 - 2014, vBulletin Solutions, Inc.
SEO by vBSEO ©2010, Crawlability, Inc.