Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > PEP on path module for standard library

Reply
Thread Tools

PEP on path module for standard library

 
 
Michael Hoffman
Guest
Posts: n/a
 
      07-21-2005
Many of you are familiar with Jason Orendorff's path module
<http://www.jorendorff.com/articles/python/path/>, which is frequently
recommended here on c.l.p. I submitted an RFE to add it to the Python
standard library, and Reinhold Birkenfeld started a discussion on it in
python-dev
<http://mail.python.org/pipermail/python-dev/2005-June/054438.html>.

The upshot of the discussion was that many python-dev'ers wanted path
added to the stdlib, but Guido was not convinced and said it must have a
PEP. So Reinhold and I are going to work on one. Reinhold has already
made some changes to the module to fit the python-dev discussion and put
it in CPython CVS at nondist/sandbox/path.

For the PEP, do any of you have arguments for or against including path?
Code samples that are much easier or more difficult with this class
would also be most helpful.

I use path in more of my modules and scripts than any other third-party
module, and I know it will be very helpful when I no longer have to
worry about deploying it.

Thanks in advance,
--
Michael Hoffman
 
Reply With Quote
 
 
 
 
Peter Hansen
Guest
Posts: n/a
 
      07-21-2005
Michael Hoffman wrote:
> For the PEP, do any of you have arguments for or against including path?
> Code samples that are much easier or more difficult with this class
> would also be most helpful.


I believe the strongest argument for "path" can be made for how it
integrates functionality which, although closely related conceptually,
is currently distributed across a half dozen or more different modules
in the standard library. Especially for newbies (I can only imagine, at
this stage) it would make working with files much easier in a many ways.

"Easier" or "more difficult" is a subjective thing, of course, but one
can't argue with the fact that path can sometimes do through a single
object what would otherwise require several imports and a bunch of calls
into things like open(), os.path, grep, and shutil.

Examples showing effective uses of path that simplify those cases would
probably merit the label "easier" even in Guido's mind, though
unfortunately that's not certain. "Easier" in some minds might simply
translate to "many lines less code", and while path can sometimes do
that, aside from the ease of splitting and joining stuff without
multiple calls to os.path.this-and-that, it really doesn't often reduce
code size _that_ much, in my experience. (Postings to c.l.p showing a
50% reduction in code size for contrived examples notwithstanding.)

A related thoughts: since paths are objects, they have attributes or
properties, and having things like ".basename" and ".parent" readily
available without having to do obscure things like
os.path.split(somepath)[0] makes things much easier to read (therefore
more maintainable). In fact, I'd propose that as another strong
argument in path's favour: it makes code much more readable, even if not
"easier" to write.

Hmm... does "easier" or "more difficult" apply to the writing of the
code or the reading of it? I find it self-evident that code written
using "path" is much easier to read, not necessarily much easier to
write (for non-newbies).

I'd summarize this by saying that the integration of "path" in the
stdlib would make it easier for newbies to write code (that might not be
obvious to a non-newbie... shall we ask some to help?), and easier for
everyone to read code (self-evident, no?), and if that's not a
sufficient condition for inclusion I don't know what is.

-Peter
 
Reply With Quote
 
 
 
 
Fernando Perez
Guest
Posts: n/a
 
      07-21-2005
Peter Hansen wrote:

> Michael Hoffman wrote:
>> For the PEP, do any of you have arguments for or against including path?
>> Code samples that are much easier or more difficult with this class
>> would also be most helpful.

>
> I believe the strongest argument for "path" can be made for how it
> integrates functionality which, although closely related conceptually,
> is currently distributed across a half dozen or more different modules
> in the standard library. Especially for newbies (I can only imagine, at
> this stage) it would make working with files much easier in a many ways.


+10

One of the few things that annoys me about the stdlib is what one could call
performing 'shell-scripting-like' tasks, and precisely because of the problem
you point out. A number of conceptually related and common tasks are
scattered all over, and every time I need to write this kind of code, I find
myself paging over the docs for multiple modules, with no real intuition as to
where I could even guess where to find things. This is very unusual for
python, where in most cases things are so well organized, that blind guessing
tends to work remarkably well.

Personally I like the path module _a lot_, though I'm sure a thorough once-over
from c.l.py and python-dev, via a PEP, can only make it better and smooth out
hidden rough edges and corner cases. But I'll be very happy if it does go
into the stdlib in the future.

Just my .02.

Best,

f

 
Reply With Quote
 
mkent@webmd.net
Guest
Posts: n/a
 
      07-21-2005
I really love Jason's 'path' module. Sadly, I've encountered a serious
problem with using it. When you try to 'freeze' an application module,
and Jason's 'path' module is present in any of the directories that are
looked at by freeze's module finder (your app doesn't have to import
it), freeze goes into an infinite loop of imports, eventually getting a
'maximum recursion depth' exception. This seems to be related to
freeze getting confused between 'os.path' and Jason's 'path'.

I encountered this using Jason's latest 'path' module and Python 2.3.2.
I was able to solve it for my use by renaming path.py to newpath.py
and using 'from newpath import path' in my modules.

I've just notified Jason about this. I presume a solution like mine
will be used, and look forward to seeing Jason's module in stdlib.

 
Reply With Quote
 
Reinhold Birkenfeld
Guest
Posts: n/a
 
      07-21-2005
FYI: I modified the path module a bit so that it fits many of the suggestions
from python-dev, and put the result in the Python CVS tree under
nondist/sandbox/path.

Most prominent change is that it doesn't inherit from str/unicode anymore.
I found this distinction important, because as a str subclass the Path object
has many methods that don't make sense for it.

Peter Hansen wrote:
> Michael Hoffman wrote:
>> For the PEP, do any of you have arguments for or against including path?
>> Code samples that are much easier or more difficult with this class
>> would also be most helpful.

>
> I believe the strongest argument for "path" can be made for how it
> integrates functionality which, although closely related conceptually,
> is currently distributed across a half dozen or more different modules
> in the standard library. Especially for newbies (I can only imagine, at
> this stage) it would make working with files much easier in a many ways.
>
> "Easier" or "more difficult" is a subjective thing, of course, but one
> can't argue with the fact that path can sometimes do through a single
> object what would otherwise require several imports and a bunch of calls
> into things like open(), os.path, grep, and shutil.


Correct.

> Examples showing effective uses of path that simplify those cases would
> probably merit the label "easier" even in Guido's mind, though
> unfortunately that's not certain. "Easier" in some minds might simply
> translate to "many lines less code", and while path can sometimes do
> that, aside from the ease of splitting and joining stuff without
> multiple calls to os.path.this-and-that, it really doesn't often reduce
> code size _that_ much, in my experience. (Postings to c.l.p showing a
> 50% reduction in code size for contrived examples notwithstanding.)


Well, these examples are the ones we'd like to see here. So, people:
If you posted examples to c.l.py in the past, please try to collect
them here!

> A related thoughts: since paths are objects, they have attributes or
> properties, and having things like ".basename" and ".parent" readily
> available without having to do obscure things like
> os.path.split(somepath)[0] makes things much easier to read (therefore
> more maintainable). In fact, I'd propose that as another strong
> argument in path's favour: it makes code much more readable, even if not
> "easier" to write.
>
> Hmm... does "easier" or "more difficult" apply to the writing of the
> code or the reading of it? I find it self-evident that code written
> using "path" is much easier to read, not necessarily much easier to
> write (for non-newbies).


And it is much more "Pythonic" in my eyes. Though that word may be inaccurate
when it comes from someone else that Guido, I feel that endless chains of
'[os.path.join(os.path.join(z, "a"), x) for x in os.path.listdir(os.path.join(z, "a") if os.path.isfile(os.path.join(....
are not qualified as being Pythonic.

> I'd summarize this by saying that the integration of "path" in the
> stdlib would make it easier for newbies to write code (that might not be
> obvious to a non-newbie... shall we ask some to help?), and easier for
> everyone to read code (self-evident, no?), and if that's not a
> sufficient condition for inclusion I don't know what is.


Reinhold
 
Reply With Quote
 
Terry Reedy
Guest
Posts: n/a
 
      07-21-2005

"Reinhold Birkenfeld" <(E-Mail Removed)> wrote in
message news:(E-Mail Removed)...
> Most prominent change is that it doesn't inherit from str/unicode
> anymore.
> I found this distinction important, because as a str subclass the Path
> object
> has many methods that don't make sense for it.


While I am - on adding more to learn, I am + on collecting scattered
filesystem functions into methods of coherent classes for eventually
replacement of the former. And I understand that this will mean a period
of duplication. But it is important to get the replacement right.

My only knowledge of the path module is what has been posted. However, it
seems to me that a path is conceptually a sequence of strings, rather than
the single joined-string representation thereof. If so, then of course it
is not and should not be a subclass of single strings. But that Path was
so defined makes me wonder, in my ignorance, whether the current
implementation is the best we can do for the future.

One advantage, for instance, of a split-up list implementation is that a
set of paths with a common prefix could be represented by replacing the
last string with a set of strings.

A more OO-friendly OS than the dominant ones today would accept a path as a
list (sequence) instead of requiring that the list be joined (by an
artifactual character) just to be split back into a list again.

My thoughts anyway.

Terry J. Reedy



 
Reply With Quote
 
Michael Hoffman
Guest
Posts: n/a
 
      07-21-2005
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> I really love Jason's 'path' module. Sadly, I've encountered a serious
> problem with using it. When you try to 'freeze' an application module,
> and Jason's 'path' module is present in any of the directories that are
> looked at by freeze's module finder (your app doesn't have to import
> it), freeze goes into an infinite loop of imports, eventually getting a
> 'maximum recursion depth' exception. This seems to be related to
> freeze getting confused between 'os.path' and Jason's 'path'.
>
> I encountered this using Jason's latest 'path' module and Python 2.3.2.
> I was able to solve it for my use by renaming path.py to newpath.py
> and using 'from newpath import path' in my modules.
>
> I've just notified Jason about this. I presume a solution like mine
> will be used, and look forward to seeing Jason's module in stdlib.


This sounds like a bug in "freeze" rather than something that should be
worked around in the standard library. Although there have already been
people opposed to naming it path because the duplication with os.path
might confuse humans.
--
Michael Hoffman
 
Reply With Quote
 
Michael Hoffman
Guest
Posts: n/a
 
      07-21-2005
Reinhold Birkenfeld wrote:
> FYI: I modified the path module a bit so that it fits many of the suggestions
> from python-dev, and put the result in the Python CVS tree under
> nondist/sandbox/path.
>
> Most prominent change is that it doesn't inherit from str/unicode anymore.
> I found this distinction important, because as a str subclass the Path object
> has many methods that don't make sense for it.


Having path descend from str/unicode is extremely useful since I can
then pass a path object to any function someone else wrote without
having to worry about whether they were checking for basestring. I think
there is a widely used pattern of accepting either a basestring[1] or a
file-like object as a function argument, and using isinstance() to
figure out which it is.

What do you gain from removing these methods? A smaller dir()?

[1] Probably str in actuality.
--
Michael Hoffman
 
Reply With Quote
 
Reinhold Birkenfeld
Guest
Posts: n/a
 
      07-21-2005
Michael Hoffman wrote:
> Reinhold Birkenfeld wrote:
>> FYI: I modified the path module a bit so that it fits many of the suggestions
>> from python-dev, and put the result in the Python CVS tree under
>> nondist/sandbox/path.
>>
>> Most prominent change is that it doesn't inherit from str/unicode anymore.
>> I found this distinction important, because as a str subclass the Path object
>> has many methods that don't make sense for it.

>
> Having path descend from str/unicode is extremely useful since I can
> then pass a path object to any function someone else wrote without
> having to worry about whether they were checking for basestring. I think
> there is a widely used pattern of accepting either a basestring[1] or a
> file-like object as a function argument, and using isinstance() to
> figure out which it is.


Where do you see that pattern? IIRC it's not in the stdlib.

> What do you gain from removing these methods? A smaller dir()?


It made sense to me at the time I changed this, although at the moment
I can't exactly recall the reasons.

Probably as Terry said: a path is both a list and a string.

Reinhold
 
Reply With Quote
 
Reinhold Birkenfeld
Guest
Posts: n/a
 
      07-21-2005
Michael Hoffman wrote:
> (E-Mail Removed) wrote:
>> I really love Jason's 'path' module. Sadly, I've encountered a serious
>> problem with using it. When you try to 'freeze' an application module,
>> and Jason's 'path' module is present in any of the directories that are
>> looked at by freeze's module finder (your app doesn't have to import
>> it), freeze goes into an infinite loop of imports, eventually getting a
>> 'maximum recursion depth' exception. This seems to be related to
>> freeze getting confused between 'os.path' and Jason's 'path'.
>>
>> I encountered this using Jason's latest 'path' module and Python 2.3.2.
>> I was able to solve it for my use by renaming path.py to newpath.py
>> and using 'from newpath import path' in my modules.
>>
>> I've just notified Jason about this. I presume a solution like mine
>> will be used, and look forward to seeing Jason's module in stdlib.

>
> This sounds like a bug in "freeze" rather than something that should be
> worked around in the standard library. Although there have already been
> people opposed to naming it path because the duplication with os.path
> might confuse humans.


As the most likely placement will be a class named "Path" inside the "os.path"
module, that bug with freeze won't apply to the "stdlib version" of Path.

Reinhold
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: PEP 3143: Standard daemon process library (was: Writing awell-behaved daemon) Jean-Paul Calderone Python 0 03-20-2009 01:02 PM
Path as a dictionary tree key? (was Re: PEP on path module forstandard library) Ron Adam Python 3 08-02-2005 04:27 AM
Re: PEP on path module for standard library Mike Orr Python 7 07-31-2005 07:28 AM
Proposed PEP: Treating Builtins as Constants in the Standard Library Raymond Hettinger Python 8 04-21-2004 11:31 AM
PEP 329: Treating Builtins as Constants in the Standard Library Raymond Hettinger Python 0 04-19-2004 12:54 PM



Advertisments