Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > loops -> list/generator comprehensions

Reply
Thread Tools

loops -> list/generator comprehensions

 
 
jamesthiele.usenet@gmail.com
Guest
Posts: n/a
 
      02-06-2005
I wrote this little piece of code to get a list of relative paths of
all files in or below the current directory (*NIX):

walkList = [(x[0], x[2]) for x in os.walk(".")]
filenames = []
for dir, files in walkList:
filenames.extend(["/".join([dir, f]) for f in files])

It works fine, I don't need to change it, but I know there is a one
liner list/generator comprehension to do this - I'm just not well
enough versed in comprehensions to figure it out. Can someone please
show me what it is?

Even better, is there a generalized way to transform simple loops into
comprehensions that someone can point me to?

james

 
Reply With Quote
 
 
 
 
Steven Bethard
Guest
Posts: n/a
 
      02-06-2005
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> I wrote this little piece of code to get a list of relative paths of
> all files in or below the current directory (*NIX):
>
> walkList = [(x[0], x[2]) for x in os.walk(".")]
> filenames = []
> for dir, files in walkList:
> filenames.extend(["/".join([dir, f]) for f in files])
>
> It works fine, I don't need to change it, but I know there is a one
> liner list/generator comprehension to do this - I'm just not well
> enough versed in comprehensions to figure it out. Can someone please
> show me what it is?


I've used os.path.join instead of "/".join since it's more general, but
other than that, this should be eqivalent:

filenames = [os.path.join(dirpath, filename)
for dirpath, _, filenames in os.walk('.')
for filename in filenames]


> Even better, is there a generalized way to transform simple loops into
> comprehensions that someone can point me to?


Well, generally, you need to write your loops to use an append, and then
the translation to LC is simpler.

filenames = []
for dirpath, _, filenames in os.walk('.'):
for filename in filenames:
filenames.append(os.path.join(dirpath, filename))

Now that you know what it looks like with an append, you simply move the
expression in the append to the top, and leave the fors in the same order:

filenames = [os.path.join(dirpath, filename)
for dirpath, _, filenames in os.walk('.')
for filename in filenames]

HTH,

STeVe
 
Reply With Quote
 
 
 
 
jamesthiele.usenet@gmail.com
Guest
Posts: n/a
 
      02-06-2005
> HTH,

It does. Thanks.

 
Reply With Quote
 
Steven Bethard
Guest
Posts: n/a
 
      02-06-2005
On 6 Feb 2005 11:28:37 -0800, <(E-Mail Removed)> wrote:
>
> walkList = [(x[0], x[2]) for x in os.walk(".")]
> filenames = []
> for dir, files in walkList:
> filenames.extend(["/".join([dir, f]) for f in files])


Caleb Hattingh top-posted:
> I would be interested to see an example of code that is more concise
> but yet as *clear* as the one you already have.


Out of curiosity, do you find:

filenames = [os.path.join(dirpath, filename)
for dirpath, _, filenames in os.walk('.')
for filename in filenames]

harder to read?

Steve
 
Reply With Quote
 
Steven Bethard
Guest
Posts: n/a
 
      02-06-2005
Caleb Hattingh wrote:
>> filenames = [os.path.join(dirpath, filename)

> # This is cool
> for dirpath, _, filenames in os.walk('.')
> # This is getting tricky, whats the '_' for?


Nothing to do with the list comprehension really. '_' is a commonly
used variable name for an object that we don't care about. If you look
at the OP's original code, the line:

[(x[0], x[2]) for x in os.walk(".")]

is the equivalent of:

[dirpath, filenames for dirpath, dirnames, filenames in os.walk('.')]

I prefer to give names to the values produced by os.walk -- I think it
makes the usage much clearer. However, since I don't use 'dirnames', I
use '_' to indicate this:

[dirpath, filenames for dirpath, _, filenames in os.walk('.')]

Would

filenames = [os.path.join(dirpath, filename)
for dirpath, dirnames, filenames in os.walk('.')
for filename in filenames]

have been clearer for you? Then all you have to do is remember the
order of the for-loop execution:

> # Which thing goes where again in a comprehension?
> for filename in filenames]


As mentioned in my other post, the order is identical to that of
for-loops, e.g.

result = []
for dirpath, _, filenames in os.walk('.'):
for filename in filenames:
result.append(os.path.join(dirpath, filename))

is basically equivalent to:

[os.path.join(dirpath, filename)
for dirpath, _, filenames in os.walk('.')
for filename in filenames]

Does that help?

Steve
 
Reply With Quote
 
Alex Martelli
Guest
Posts: n/a
 
      02-06-2005
Steven Bethard <(E-Mail Removed)> wrote:

> at the OP's original code, the line:
>
> [(x[0], x[2]) for x in os.walk(".")]
>
> is the equivalent of:
>
> [dirpath, filenames for dirpath, dirnames, filenames in os.walk('.')]


Just a nit: you need parentheses in your second LC too, i.e.:

[(dirpath, filenames) for dirpath, dirnames, filenames in os.walk('.')]

> I prefer to give names to the values produced by os.walk -- I think it
> makes the usage much clearer.


Full agreement here -- those numeric indices are nasty.


Alex
 
Reply With Quote
 
Steven Bethard
Guest
Posts: n/a
 
      02-06-2005
Alex Martelli wrote:
> Steven Bethard <(E-Mail Removed)> wrote:
>
>
>>at the OP's original code, the line:
>>
>> [(x[0], x[2]) for x in os.walk(".")]
>>
>>is the equivalent of:
>>
>> [dirpath, filenames for dirpath, dirnames, filenames in os.walk('.')]

>
>
> Just a nit: you need parentheses in your second LC too, i.e.:
>
> [(dirpath, filenames) for dirpath, dirnames, filenames in os.walk('.')]


Yup, you're right (of course) Thanks for the catch!

Steve
 
Reply With Quote
 
Caleb Hattingh
Guest
Posts: n/a
 
      02-07-2005
I would be interested to see an example of code that is more concise but
yet as *clear* as the one you already have. I can actually read and
understand what you've got there. That's cool


On 6 Feb 2005 11:28:37 -0800, <(E-Mail Removed)> wrote:

> I wrote this little piece of code to get a list of relative paths of
> all files in or below the current directory (*NIX):
>
> walkList = [(x[0], x[2]) for x in os.walk(".")]
> filenames = []
> for dir, files in walkList:
> filenames.extend(["/".join([dir, f]) for f in files])
>
> It works fine, I don't need to change it, but I know there is a one
> liner list/generator comprehension to do this - I'm just not well
> enough versed in comprehensions to figure it out. Can someone please
> show me what it is?
>
> Even better, is there a generalized way to transform simple loops into
> comprehensions that someone can point me to?
>
> james
>


 
Reply With Quote
 
Caleb Hattingh
Guest
Posts: n/a
 
      02-07-2005
Sure Steve

Lemme see ... (indentation changed so comments associate with correct bits)

> Out of curiosity, do you find:
>
> filenames = [os.path.join(dirpath, filename)

# This is cool
for dirpath, _, filenames in os.walk('.')
# This is getting tricky, whats the '_' for? Which thing
goes where again in a comprehension?
for filename in filenames]
# The hierarchy has nailed me by this point. I
will have to start over...

Yes, yes I do find it more difficult to read. Maybe I just don't know
python as well as I should (certainly not as well as others here!). I
guess it is just familiarity. A single comprehension is ok, nesting them
gets tricky, and 3 times is a strike for me. I will practise

keep well
Caleb



 
Reply With Quote
 
Steven Bethard
Guest
Posts: n/a
 
      02-07-2005
Caleb Hattingh wrote:
>> Would
>>
>> filenames = [os.path.join(dirpath, filename)
>> for dirpath, dirnames, filenames in os.walk('.')
>> for filename in filenames]
>>
>> have been clearer for you? Then all you have to do is remember the
>> order of the for-loop execution:

>
> Bizarre as this may sound, it was the '_' that was throwing me off the
> whole thing (at the 'grok' level I generally read the newsgroup,
> anyway). For some weird reason, I can read *this* comprehension
> pretty easily! Does that make sense at all? I figure a little bit of
> uncertainty along the way probably derails understanding of the whole
> thing a little bit


Yup, actually, that's what I kinda suspected happened. But if it
happened for you, it probably happened for a dozen other folks who read
the same thing, so I'm glad to hear rewriting it and explaining the '_'
was helpful.

STeVe
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Loops with loops using html-template Me Perl Misc 2 01-12-2006 05:07 PM
Can I simulate list comprehensions? seguso C++ 9 12-22-2004 11:51 PM
Simple db using list comprehensions Frank Millman Python 1 04-16-2004 03:29 AM
Generator comprehensions -- patch for compiler module Jeff Epler Python 2 08-27-2003 12:50 PM
Re: Style in list comprehensions Dave Kuhlman Python 1 08-16-2003 12:03 PM



Advertisments