Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > py3k concerns. An example

Reply
Thread Tools

py3k concerns. An example

 
 
Aaron Watters
Guest
Posts: n/a
 
      04-18-2008
Why is the migration to py3k a concern?
For example I have libraries which use string%dictionary
substitution where the dictionary is actually an object
which emulates a dictionary. The __getitem__ for
the object can be very expensive and is only called when
needed by the string substitution.

In py3k string%dictionary is going away. Why?
I have no idea.

The replacement is a string.format(...) method
which supports dictionary calling.
string.format(**dictionary)
But dictionary
calling doesn't support dictionary emulation.
So in the example below the substitution works
but the call fails.

=== code

class fdict(dict):
def __getitem__(self, item):
return "got("+item+")"

def fn(**d):
print d["boogie"]

if __name__=="__main__":
fd = fdict()
print "attempting string substitution with fake dictionary"
print
print "hello there %(boogie)s" % fd # <-- works
print
print "now attempting function call with fake dictionary"
print
fn(**fd) # <-- fails

=== output

% python2.6 dtest.py
attempting string substitution with fake dictionary

hello there got(boogie)

now attempting function call with fake dictionary

Traceback (most recent call last):
File "dtest.py", line 17, in <module>
fn(**fd)
File "dtest.py", line 7, in fn
print d["boogie"]
KeyError: 'boogie'

==== end of output

Consequently there is no simple way to translate
my code, I think. I suspect you will find this kind of subtle
issue in many places. Or worse, you won't find it
until after your program has been installed
in production.

It's a damn shame because
if string%dict was just left in it wouldn't be an issue.

Also, if making f(**d) support dict emulation
has any negative performance implications
then I don't want it please.

sigh. -- Aaron Watters

===
http://www.xfeedme.com/nucular/pydis...EXT=crack+open
 
Reply With Quote
 
 
 
 
J. Cliff Dyer
Guest
Posts: n/a
 
      04-18-2008
On Fri, 2008-04-18 at 08:58 -0700, Aaron Watters wrote:
> Why is the migration to py3k a concern?
> For example I have libraries which use string%dictionary
> substitution where the dictionary is actually an object
> which emulates a dictionary. The __getitem__ for
> the object can be very expensive and is only called when
> needed by the string substitution.
>
> In py3k string%dictionary is going away. Why?
> I have no idea.
>
> The replacement is a string.format(...) method
> which supports dictionary calling.
> string.format(**dictionary)
> But dictionary
> calling doesn't support dictionary emulation.
> So in the example below the substitution works
> but the call fails.
>
> === code
>
> class fdict(dict):
> def __getitem__(self, item):
> return "got("+item+")"
>
> def fn(**d):
> print d["boogie"]
>
> if __name__=="__main__":
> fd = fdict()
> print "attempting string substitution with fake dictionary"
> print
> print "hello there %(boogie)s" % fd # <-- works
> print
> print "now attempting function call with fake dictionary"
> print
> fn(**fd) # <-- fails
>
> === output
>
> % python2.6 dtest.py
> attempting string substitution with fake dictionary
>
> hello there got(boogie)
>
> now attempting function call with fake dictionary
>
> Traceback (most recent call last):
> File "dtest.py", line 17, in <module>
> fn(**fd)
> File "dtest.py", line 7, in fn
> print d["boogie"]
> KeyError: 'boogie'
>
> ==== end of output
>
> Consequently there is no simple way to translate
> my code, I think. I suspect you will find this kind of subtle
> issue in many places. Or worse, you won't find it
> until after your program has been installed
> in production.
>
> It's a damn shame because
> if string%dict was just left in it wouldn't be an issue.
>
> Also, if making f(**d) support dict emulation
> has any negative performance implications
> then I don't want it please.
>
> sigh. -- Aaron Watters
>
> ===
> http://www.xfeedme.com/nucular/pydis...EXT=crack+open
>


I was with you on this issue right up until that last paragraph. You
want it, but only if its free. That's ridiculous. Every thing a
computer does requires processor cycles.

Do you really mean to tell me that string interpolation has been a major
bottleneck for you? Now I think you're just whining because you like to
hear yourself whine. Try coming up with a real standard for evaluation.
How much of a performance hit will actually cause you trouble? 1% extra
on string interpolation? 10%? 50%? 200%?

You do provide a link to a website called xfeedme.com. And I just fed
you. IHBT. HAND. :-/

--
Oook,
J. Cliff Dyer
Carolina Digital Library and Archives
UNC Chapel Hill

 
Reply With Quote
 
 
 
 
Aaron Watters
Guest
Posts: n/a
 
      04-18-2008

> Try coming up with a real standard for evaluation.
> How much of a performance hit will actually cause you trouble? 1% extra
> on string interpolation? 10%? 50%? 200%?


You misread my comment. I don't want function calls to support
dictionary emulation if there is a speed penalty because this is
such a special case and function calls are so important.
I don't really know, but I think "fixing" the above issue for
string.format(...) might involve changing the representation of
every python stack frame... not worth it in this case if there
is any penalty (afaik there isn't).

An alternative is to provide an alternate interface to string.format
so
that you could pass in an object which might emulate a dictionary,
like string.formatDict(D) -- or you could even adopt a shortcut
notation like string % D -- hey there's an idea!
-- Aaron Watters

===
http://www.xfeedme.com/nucular/pydis...gnore+warnings
 
Reply With Quote
 
Kay Schluehr
Guest
Posts: n/a
 
      04-18-2008
On 18 Apr., 20:07, Aaron Watters <(E-Mail Removed)> wrote:

> I don't really know, but I think "fixing" the above issue for
> string.format(...) might involve changing the representation of
> every python stack frame... not worth it in this case if there
> is any penalty (afaik there isn't).


In Python 2.6 the format function simply calls PyDict_GetItem on the
dictionary object and raises a KeyError if it fails. A dict emulation
would require one more lookup if the fast lookup has no result. It
would be a speed penalty in the general case if this lookup would be
implemented in PyDict_GetItem but handling a special case in the
format implementation might not look that bad.

 
Reply With Quote
 
Matimus
Guest
Posts: n/a
 
      04-18-2008
On Apr 18, 8:58 am, Aaron Watters <(E-Mail Removed)> wrote:
> Why is the migration to py3k a concern?
> For example I have libraries which use string%dictionary
> substitution where the dictionary is actually an object
> which emulates a dictionary. The __getitem__ for
> the object can be very expensive and is only called when
> needed by the string substitution.
>
> In py3k string%dictionary is going away. Why?
> I have no idea.
>
> The replacement is a string.format(...) method
> which supports dictionary calling.
> string.format(**dictionary)
> But dictionary
> calling doesn't support dictionary emulation.
> So in the example below the substitution works
> but the call fails.
>
> === code
>
> class fdict(dict):
> def __getitem__(self, item):
> return "got("+item+")"
>
> def fn(**d):
> print d["boogie"]
>
> if __name__=="__main__":
> fd = fdict()
> print "attempting string substitution with fake dictionary"
> print
> print "hello there %(boogie)s" % fd # <-- works
> print
> print "now attempting function call with fake dictionary"
> print
> fn(**fd) # <-- fails
>
> === output
>
> % python2.6 dtest.py
> attempting string substitution with fake dictionary
>
> hello there got(boogie)
>
> now attempting function call with fake dictionary
>
> Traceback (most recent call last):
> File "dtest.py", line 17, in <module>
> fn(**fd)
> File "dtest.py", line 7, in fn
> print d["boogie"]
> KeyError: 'boogie'
>
> ==== end of output
>
> Consequently there is no simple way to translate
> my code, I think. I suspect you will find this kind of subtle
> issue in many places. Or worse, you won't find it
> until after your program has been installed
> in production.
>
> It's a damn shame because
> if string%dict was just left in it wouldn't be an issue.
>
> Also, if making f(**d) support dict emulation
> has any negative performance implications
> then I don't want it please.
>
> sigh. -- Aaron Watters
>
> ===http://www.xfeedme.com/nucular/pydistro.py/go?FREETEXT=crack+open


The reason it doesn't work is that you are unpacking the dictionary
with **, and you have done nothing to define any keys or define a
length. I would describe the way you are using dict as a hack, and not
part of any standard feature. You make a good point that it breaks
your code, but at the same time the format facility gives you the
ability to do something similar but in a standard way. You simply
define a class with a __format__ method and pass that in instead of
your dict.


class MyClass:
def __format__(self, spec):
return "got({0})".format(spec)

c = MyClass()
print ("hey everybody {0:some words}".format(c))
print ("lets try this {0:more words} {0:even more words}".format(c))

should produce:

hey everybody got(some words)
lets try this got(more words) got(even more words)

My point is that instead of exploiting the string formatting of
dictionaries feature to create a custom format specifier, you actually
get to define your own format specifiers, something which is much more
powerful.

And actually, you can do this too... which is even simpler and allows
you to use your a portion of your existing solution:

class fdict(dict):
def __getitem__(self, item):
return "got("+item+")"

fd = fdict()
print ("hello there {0[boogie]} hello there {0[george])".format(fd))

Which should result in:

hello there got(boogie) hello there got(george)

* Keep in mind that I have not tested any of this code, there may be
bugs. I don't have Py3k or 2.6 installed locally.

I think this is a good trade-off.

Adding to that... don't worry about py3k. Nobody is forcing you to
switch. In fact, you are encouraged not to until you are comfortable.
Py3k won't _break_ your code. You wrote the code for Python 2.x use it
in 2.x. Python 2.x probably has a good 5-10 years remaining.

Matt
 
Reply With Quote
 
Kay Schluehr
Guest
Posts: n/a
 
      04-18-2008
On 18 Apr., 23:09, Matimus <(E-Mail Removed)> wrote:
> The reason it doesn't work is that you are unpacking the dictionary
> with **, and you have done nothing to define any keys or define a
> length.


This is a non-issue. The class derives from dict; it has all the
desired attributes. It is also not a problem in particular because
these properties are not requested by format ( at least not in the
code I have examined which was admittedly just a critical section that
caused the exception ).

> Adding to that... don't worry about py3k. Nobody is forcing you to
> switch. In fact, you are encouraged not to until you are comfortable.
> Py3k won't _break_ your code. You wrote the code for Python 2.x use it
> in 2.x. Python 2.x probably has a good 5-10 years remaining.


These advices start to get annoying.

Software hardly ever exists in isolation for the sake of the beauty of
the algorithm but is supplementary to a large framework/engine/
library. So if e.g. Django switches to 3 everyone who works with it
has to switch sooner or later as well or lose track otherwise, no
matter how long Python 1.5.2 or Python 2.5.2 or whatever version will
be maintained. If Pythons code base becomes fragmented it will be
harmful and affect almost everyones work.

 
Reply With Quote
 
Ivan Illarionov
Guest
Posts: n/a
 
      04-19-2008
On Fri, 18 Apr 2008 16:19:38 -0700, Kay Schluehr wrote:

> On 18 Apr., 23:09, Matimus <(E-Mail Removed)> wrote:
>> The reason it doesn't work is that you are unpacking the dictionary
>> with **, and you have done nothing to define any keys or define a
>> length.

>
> This is a non-issue. The class derives from dict; it has all the desired
> attributes. It is also not a problem in particular because these
> properties are not requested by format ( at least not in the code I have
> examined which was admittedly just a critical section that caused the
> exception ).
>
>> Adding to that... don't worry about py3k. Nobody is forcing you to
>> switch. In fact, you are encouraged not to until you are comfortable.
>> Py3k won't _break_ your code. You wrote the code for Python 2.x use it
>> in 2.x. Python 2.x probably has a good 5-10 years remaining.

>
> These advices start to get annoying.
>
> Software hardly ever exists in isolation for the sake of the beauty of
> the algorithm but is supplementary to a large framework/engine/ library.
> So if e.g. Django switches to 3 everyone who works with it has to switch
> sooner or later as well or lose track otherwise, no matter how long
> Python 1.5.2 or Python 2.5.2 or whatever version will be maintained. If
> Pythons code base becomes fragmented it will be harmful and affect
> almost everyones work.


This Py3k-Django-related FUD starts to get annoying. AFAIK Django is
going to support everything from 2.3 to 3.0 from single codebase. 2to3
tool will be called from setup.py and code will have few 'if
sys.version_info < (3, 0)/else' tricks. Proof of concept already exists:
http://wiki.python.org/moin/PortingDjangoTo3k

I work with Django as well as *on* Django and 3rd party Django addons and
don't see any reason to worry.

Your particular problem may be solved by using a real dictionary with
real keys which will probably have positive performance impact on your
code and make it more clean.

Even if Python code base will become fragmented why is it harmfull? This
is a way our life work - you rise and get better or die. It is essential
part of progress and evolution. IMHO, we'll only get better Python and
better Python libraries.

--
Ivan
 
Reply With Quote
 
Jason Scheirer
Guest
Posts: n/a
 
      04-19-2008
On Apr 18, 4:19 pm, Kay Schluehr <(E-Mail Removed)> wrote:
> On 18 Apr., 23:09, Matimus <(E-Mail Removed)> wrote:
>
> > The reason it doesn't work is that you are unpacking the dictionary
> > with **, and you have done nothing to define any keys or define a
> > length.

>
> This is a non-issue. The class derives from dict; it has all the
> desired attributes. It is also not a problem in particular because
> these properties are not requested by format ( at least not in the
> code I have examined which was admittedly just a critical section that
> caused the exception ).
>
> > Adding to that... don't worry about py3k. Nobody is forcing you to
> > switch. In fact, you are encouraged not to until you are comfortable.
> > Py3k won't _break_ your code. You wrote the code for Python 2.x use it
> > in 2.x. Python 2.x probably has a good 5-10 years remaining.

>
> These advices start to get annoying.
>
> Software hardly ever exists in isolation for the sake of the beauty of
> the algorithm but is supplementary to a large framework/engine/
> library. So if e.g. Django switches to 3 everyone who works with it
> has to switch sooner or later as well or lose track otherwise, no
> matter how long Python 1.5.2 or Python 2.5.2 or whatever version will
> be maintained. If Pythons code base becomes fragmented it will be
> harmful and affect almost everyones work.


This has happened before, though -- I remember the pain of moving to
2.0 from the 1.5 branch way back when, and it wasn't getting my 1.5
code to work in 2.0, it was being jealous of all the cool features of
2.0 that I had to wait to get. I was working in production with 1.5 in
2000 and all the libraries available for Python gradually stopped
supporting 1.5 to pick up interesting 2.0 features that actually made
them easier to work with, and new libraries all began to assume you
were using a 2.0+ Python version because that's what was current. By
2003-2004 everyone I knew had switched over to 2.0, but by then I had
had nearly 5 years to port my legacy 1.5 code to 2.0, take advantage
of the 2.0 version's features, and do plenty of testing of my 1.5
codebase in 2.0 before I switched my production systems over. Not to
mention the fact that plenty of warning was offered BEFORE 2.0 was
released and 1.5 was not abruptly ended, but gradually phased out
until only the teeniest far ends of the long tail were using it. The
2.6->3.0 process is going to be even less of a pain than the 1.5->2.0
conversion, which was not hard at all going forward into it. You may
not want to switch, but by the time you decide to it will be pretty
easy to move on -- the extreme opposite reality being your application
will be so frozen that both your Python version and your codebase will
be fossils, left to hum on completely unchanged on some forgotten
server like so much other legacy code.
 
Reply With Quote
 
Carl Banks
Guest
Posts: n/a
 
      04-19-2008
On Apr 18, 11:58 am, Aaron Watters <(E-Mail Removed)> wrote:
> Why is the migration to py3k a concern?
> For example I have libraries which use string%dictionary
> substitution where the dictionary is actually an object
> which emulates a dictionary. The __getitem__ for
> the object can be very expensive and is only called when
> needed by the string substitution.
>
> In py3k string%dictionary is going away. Why?
> I have no idea.
>
> The replacement is a string.format(...) method
> which supports dictionary calling.
> string.format(**dictionary)
> But dictionary
> calling doesn't support dictionary emulation.
> So in the example below the substitution works
> but the call fails.
>
> === code
>
> class fdict(dict):
> def __getitem__(self, item):
> return "got("+item+")"
>
> def fn(**d):
> print d["boogie"]
>
> if __name__=="__main__":
> fd = fdict()
> print "attempting string substitution with fake dictionary"
> print
> print "hello there %(boogie)s" % fd # <-- works
> print
> print "now attempting function call with fake dictionary"
> print
> fn(**fd) # <-- fails
>
> === output
>
> % python2.6 dtest.py
> attempting string substitution with fake dictionary
>
> hello there got(boogie)
>
> now attempting function call with fake dictionary
>
> Traceback (most recent call last):
> File "dtest.py", line 17, in <module>
> fn(**fd)
> File "dtest.py", line 7, in fn
> print d["boogie"]
> KeyError: 'boogie'
>
> ==== end of output
>
> Consequently there is no simple way to translate
> my code, I think. I suspect you will find this kind of subtle
> issue in many places. Or worse, you won't find it
> until after your program has been installed
> in production.
>
> It's a damn shame because
> if string%dict was just left in it wouldn't be an issue.
>
> Also, if making f(**d) support dict emulation
> has any negative performance implications
> then I don't want it please.
>
> sigh. -- Aaron Watters



If you don't like Python 3, DON'T USE IT.

It's been stated repeatedly that 2.x and 3.x are going to be supported
in parallel for years.

Refusing to use 3, thus casting your brain-share vote against it, is
far more likely to have an effect than you coming here and making
everyone's life miserable with your pitiful whining.


Carl Banks
 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      04-21-2008
En Fri, 18 Apr 2008 12:58:56 -0300, Aaron Watters <(E-Mail Removed)> escribió:

> Why is the migration to py3k a concern?
> For example I have libraries which use string%dictionary
> substitution where the dictionary is actually an object
> which emulates a dictionary. The __getitem__ for
> the object can be very expensive and is only called when
> needed by the string substitution.
>
> In py3k string%dictionary is going away. Why?
> I have no idea.


But not soon. It's not listed in PEP 3100 and according to this message
http://mail.python.org/pipermail/pyt...il/013094.html
%s formatting will not disappear until Python 3.3
You have plenty of time to evaluate alternatives. Your code may become obsolete even before 3.3 is shipped.

--
Gabriel Genellina

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
py3k feature proposal: field auto-assignment in constructors coldpizza Python 45 01-29-2008 11:55 AM
logic programming in python (was something about py3k) Jean-Paul Calderone Python 0 10-30-2007 02:33 PM
no more reload() in py3k timaranz@gmail.com Python 1 09-19-2007 03:00 AM
Py3K: Ensuring future compatibility between function annotation basedtools Ferenczi Viktor Python 0 09-04-2007 08:09 PM
py3k - format specifier for strings Carl Trachte Python 3 09-04-2007 07:55 AM



Advertisments