Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Do you have real-world use cases for map's None fill-in feature?

Reply
Thread Tools

Do you have real-world use cases for map's None fill-in feature?

 
 
Raymond Hettinger
Guest
Posts: n/a
 
      01-08-2006
I am evaluating a request for an alternate version of itertools.izip()
that has a None fill-in feature like the built-in map function:

>>> map(None, 'abc', '12345') # demonstrate map's None fill-in feature

[('a', '1'), ('b', '2'), ('c', '3'), (None, '4'), (None, '5')]

The movitation is to provide a means for looping over all data elements
when the input lengths are unequal. The question of the day is whether
that is both a common need and a good approach to real-world problems.
The answer to the question can likely be found in results from other
programming languages or from real-world Python code that has used
map's None fill-in feature.

I scanned the docs for Haskell, SML, and Perl and found that the norm
for map() and zip() is to truncate to the shortest input or raise an
exception for unequal input lengths. I scanned the standard library
and found no instances where map's fill-in feature was used. Likewise,
I found no examples in all of the code I've ever written.

The history of Python's current zip() function serves as another
indicator that the proposal is weak. PEP 201 contemplated and rejected
the idea as one that likely had unintended consequences. In the years
since zip() was introduced in Py2.0, SourceForge has shown no requests
for a fill-in version of zip().

My request for readers of comp.lang.python is to search your own code
to see if map's None fill-in feature was ever used in real-world code
(not toy examples). I'm curious about the context, how it was used,
and what alternatives were rejected (i.e. did the fill-in feature
improve the code).

Also, I'm curious as to whether someone has seen a zip fill-in feature
employed to good effect in some other programming language, perhaps
LISP or somesuch?

Maybe a few real-word code examples and experiences from other
languages will shed light on the question of whether lock-step
iteration has meaning beyond the length of the shortest matching
elements. If ordinal position were considered as a record key, then
the proposal equates to a database-style outer join operation (where
data elements with unmatched keys are included) and order is
significant. Does an outer-join have anything to do with lock-step
iteration? Is this a fundamental looping construct or just a
theoretical wish-list item? IOW, does Python really need
itertools.izip_longest() or would that just become a distracting piece
of cruft?


Raymond Hettinger


P.S. FWIW, the OP's use case involved printing files in multiple
columns:

for f, g in itertools.izip_longest(file1, file2, fillin_value=''):
print '%-20s\t|\t%-20s' % (f.rstrip(), g.rstrip())

The alternative was straight-forward but not as terse:

while 1:
f = file1.readline()
g = file2.readline()
if not f and not g:
break
print '%-20s\t|\t%-20s' % (f.rstrip(), g.rstrip())

 
Reply With Quote
 
 
 
 
Paul Rubin
Guest
Posts: n/a
 
      01-09-2006
"Raymond Hettinger" <(E-Mail Removed)> writes:
> I am evaluating a request for an alternate version of itertools.izip()
> that has a None fill-in feature like the built-in map function:
>
> >>> map(None, 'abc', '12345') # demonstrate map's None fill-in feature


I think finding different ways to write it was an entertaining
exercise but it's too limited in usefulness to become a standard
feature.

I do think some idiom ought to develop to allow checking whether an
iterator is empty, without consuming an item. Here's an idea:
introduce something like

iterator = check_empty(iterator)

where check_empty would work roughly like (untested):

def check_empty(iterator):
iclass = iterator.__class__
class buffered(iclass):
def __init__(self):
n = iter((self.next(),)) # might raise StopIteration
self.__save = chain(n, self)
def next(self):
return self.__save.next()
# all other operations are inherited from iclass

return buffered(iterator)

The idea is you get back a new iterator which yields the same stream
and supports the same operations as the old one, if the old one is
non-empty. Otherwise it raises StopIteration.

There are some obvious problems with the above:

1) the new iterator should support all of the old one's attributes,
not just inherit its operations
2) In the case where the old iterator is already buffered, the
constructor should just peek at the lookahead instead of making
a new object. That means that checking an iterator multiple times
won't burn more and more memory.

Maybe there is some way of doing the above with metaclasses but I've
never been able to wrap my head around those.
 
Reply With Quote
 
 
 
 
Bengt Richter
Guest
Posts: n/a
 
      01-10-2006
On 7 Jan 2006 23:19:41 -0800, "Raymond Hettinger" <(E-Mail Removed)> wrote:

>I am evaluating a request for an alternate version of itertools.izip()
>that has a None fill-in feature like the built-in map function:
>
>>>> map(None, 'abc', '12345') # demonstrate map's None fill-in feature

>[('a', '1'), ('b', '2'), ('c', '3'), (None, '4'), (None, '5')]
>

I don't like not being able to supply my own sentinel. None is too common
a value. Hm, ... <bf warning> unless maybe it can also be a type that we can instantiate with
really-mean-it context level like None(5)

>>> map(None(5), 'abc', '12345') # demonstrate map's None fill-in feature

[('a', '1'), ('b', '2'), ('c', '3'), (None(5), '4'), (None(5), '5')]

But seriously, standard sentinels for "missing data" and "end of data" might be nice to have,
and to have produced in appropriate standard contexts. Singleton string subclass
instances "NOD" and "EOD"? Doesn't fit with EOF=='' though.
</bf warning>

>The movitation is to provide a means for looping over all data elements
>when the input lengths are unequal. The question of the day is whether
>that is both a common need and a good approach to real-world problems.
>The answer to the question can likely be found in results from other
>programming languages or from real-world Python code that has used
>map's None fill-in feature.
>

What about some semantics like my izip2 in
http://groups.google.com/group/comp....1ddb1f46?hl=en

(which doesn't even need a separate name, since it would be backwards compatible)

Also, what about factoring sequence-related stuff into being methods or attributes
of iter instances? And letting iter take multiple sequences or callable/sentinel pairs,
which could be a substitute for izip and then some? Methods could be called via a returned
iterator before or after the first .next() call, to control various features, such as
sentinel testing by 'is' instead of '==' for callable/sentinel pairs, or buffering n
steps of lookahead supported by a .peek(n) method defaulting to .peek(1), etc. etc.
The point being to have a place to implement universal sequence stuff.

>I scanned the docs for Haskell, SML, and Perl and found that the norm
>for map() and zip() is to truncate to the shortest input or raise an
>exception for unequal input lengths. I scanned the standard library
>and found no instances where map's fill-in feature was used. Likewise,
>I found no examples in all of the code I've ever written.
>
>The history of Python's current zip() function serves as another
>indicator that the proposal is weak. PEP 201 contemplated and rejected
>the idea as one that likely had unintended consequences. In the years
>since zip() was introduced in Py2.0, SourceForge has shown no requests
>for a fill-in version of zip().
>
>My request for readers of comp.lang.python is to search your own code
>to see if map's None fill-in feature was ever used in real-world code
>(not toy examples). I'm curious about the context, how it was used,
>and what alternatives were rejected (i.e. did the fill-in feature
>improve the code).
>
>Also, I'm curious as to whether someone has seen a zip fill-in feature
>employed to good effect in some other programming language, perhaps
>LISP or somesuch?

ISTM in general there is a chicken-egg problem where workarounds are easy.
I.e., the real question is how many workaround situations there are
that would have been handled conveniently with a builtin feature,
and _then_ to see whether the convenience would be worth enough.
>
>Maybe a few real-word code examples and experiences from other
>languages will shed light on the question of whether lock-step
>iteration has meaning beyond the length of the shortest matching
>elements. If ordinal position were considered as a record key, then
>the proposal equates to a database-style outer join operation (where
>data elements with unmatched keys are included) and order is
>significant. Does an outer-join have anything to do with lock-step
>iteration? Is this a fundamental looping construct or just a
>theoretical wish-list item? IOW, does Python really need
>itertools.izip_longest() or would that just become a distracting piece
>of cruft?

Even if there is little use for continuing in correct code, IWT getting
at the state of the iterator in an erroroneous situation would be a benefit.
Being able to see the result of the last attempt at gathering tuple elements
could help. (I can see reasons for wanting variations of trying all streams
vs shortcutting on the first to exhaust though).

Regards,
Bengt Richter
 
Reply With Quote
 
Raymond Hettinger
Guest
Posts: n/a
 
      01-10-2006
[Bengt Richter]
> What about some semantics like my izip2 in
> http://groups.google.com/group/comp....1ddb1f46?hl=en
>
> (which doesn't even need a separate name, since it would be backwards compatible)
>
> Also, what about factoring sequence-related stuff into being methods or attributes
> of iter instances? And letting iter take multiple sequences or callable/sentinel pairs,
> which could be a substitute for izip and then some? Methods could be called via a returned
> iterator before or after the first .next() call, to control various features, such as
> sentinel testing by 'is' instead of '==' for callable/sentinel pairs, or buffering n
> steps of lookahead supported by a .peek(n) method defaulting to .peek(1), etc. etc.
> The point being to have a place to implement universal sequence stuff.


ISTM, these cures are worse than the disease


> Even if there is little use for continuing in correct code, IWT getting
> at the state of the iterator in an erroroneous situation would be a benefit.
> Being able to see the result of the last attempt at gathering tuple elements
> could help. (I can see reasons for wanting variations of trying all streams
> vs shortcutting on the first to exhaust though).


On the one hand, that seems reasonable. On the other hand, I can't see
how to use it without snarling the surrounding code in which case it is
probably better to explicitly manage individual iterators within a
while loop.


Raymond

 
Reply With Quote
 
Raymond Hettinger
Guest
Posts: n/a
 
      01-10-2006
[Raymond Hettinger]
> > I am evaluating a request for an alternate version of itertools.izip()
> > that has a None fill-in feature like the built-in map function:
> >
> > >>> map(None, 'abc', '12345') # demonstrate map's None fill-in feature


[Paul Rubin]
> I think finding different ways to write it was an entertaining
> exercise but it's too limited in usefulness to become a standard
> feature.


Makes sense.

> I do think some idiom ought to develop to allow checking whether an
> iterator is empty, without consuming an item. Here's an idea:
> introduce something like
>
> iterator = check_empty(iterator)


There are so many varieties of iterator that it's probably not workable
to alter the iterator API for all of the them. In any case, a broad
API change like this would need its own PEP.


> There are some obvious problems with the above:
>
> 1) the new iterator should support all of the old one's attributes,
> not just inherit its operations
> 2) In the case where the old iterator is already buffered, the
> constructor should just peek at the lookahead instead of making
> a new object. That means that checking an iterator multiple times
> won't burn more and more memory.
>
> Maybe there is some way of doing the above with metaclasses but I've
> never been able to wrap my head around those.


Metaclasses are unlikely to be of help because there are so many,
unrelated kinds of iterator -- most do not inherit from a common
parent.


Raymond

 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      01-10-2006
"Raymond Hettinger" <(E-Mail Removed)> writes:
> > iterator = check_empty(iterator)

>
> There are so many varieties of iterator that it's probably not workable
> to alter the iterator API for all of the them. In any case, a broad
> API change like this would need its own PEP.


The hope was that it wouldn't be an API change, but rather just a new
function dropped into the existing library, that could wrap any
existing iterator without having to change or break anything that's
already been written. Maybe the resulting iterator couldn't support
every operation, or maybe it could have a __getattr__ that delegates
everything except "next" to the wrapped iterator, or something. The
obvious implementation methods that I can see are very kludgy but
maybe something better is feasible. I defer to your knowledge about
this.
 
Reply With Quote
 
Tim Peters
Guest
Posts: n/a
 
      01-10-2006
[Raymond Hettinger]
> ...
> I scanned the docs for Haskell, SML, and Perl and found that the norm
> for map() and zip() is to truncate to the shortest input or raise an
> exception for unequal input lengths.
> ...
> Also, I'm curious as to whether someone has seen a zip fill-in feature
> employed to good effect in some other programming language, perhaps
> LISP or somesuch?


FYI, Common Lisp's `pairlis` function requires that its first two
arguments be lists of the same length. It's a strain to compare to
Python's zip() though, as the _intended_ use of `pairlis` is to add
new pairs to a Lisp association list. For that reason, `pairlis`
accepts an optional third argument; if present, this should be an
association list, and pairs from zipping the first two arguments are
prepended to it. Also for this reason, the _order_ in which pairs are
taken from the first two arguments isn't defined(!).

http://www.lispworks.com/documentati...li.htm#pairlis

For its intended special-purpose use, it wouldn't make sense to allow
arguments of different lengths.
 
Reply With Quote
 
Bengt Richter
Guest
Posts: n/a
 
      01-10-2006
On 10 Jan 2006 00:47:36 -0800, "Raymond Hettinger" <(E-Mail Removed)> wrote:

>[Bengt Richter]
>> What about some semantics like my izip2 in
>> http://groups.google.com/group/comp....1ddb1f46?hl=en
>>
>> (which doesn't even need a separate name, since it would be backwards compatible)
>>
>> Also, what about factoring sequence-related stuff into being methods or attributes
>> of iter instances? And letting iter take multiple sequences or callable/sentinel pairs,
>> which could be a substitute for izip and then some? Methods could be called via a returned
>> iterator before or after the first .next() call, to control various features, such as
>> sentinel testing by 'is' instead of '==' for callable/sentinel pairs, or buffering n
>> steps of lookahead supported by a .peek(n) method defaulting to .peek(1), etc. etc.
>> The point being to have a place to implement universal sequence stuff.

>
>ISTM, these cures are worse than the disease

Are you reacting to my turgidly rambling post, or to

>>> from ut.izip2 import izip2 as izip
>>> it = izip('abc','12','ABCD')
>>> for t in it: print t

...
('a', '1', 'A')
('b', '2', 'B')

Then after a backwards-compatible izip, if the iterator has
been bound, it can be used to continue, with sentinel sustitution:

>>> for t in it.rest('<sentinel>'): print t

...
('c', '<sentinel>', 'C')
('<sentinel>', '<sentinel>', 'D')

or optionally in sentinel substitution mode from the start:

>>> for t in izip('abc','12','ABCD').rest('<sentinel>'): print t

...
('a', '1', 'A')
('b', '2', 'B')
('c', '<sentinel>', 'C')
('<sentinel>', '<sentinel>', 'D')

Usage-wise, this seems not too diseased to me, so I guess I want to make sure
this is what you were reacting to

(Implementation was just to hack together a working demo. I'm sure it can be improved upon

>
>
>> Even if there is little use for continuing in correct code, IWT getting
>> at the state of the iterator in an erroroneous situation would be a benefit.
>> Being able to see the result of the last attempt at gathering tuple elements
>> could help. (I can see reasons for wanting variations of trying all streams
>> vs shortcutting on the first to exhaust though).

>
>On the one hand, that seems reasonable. On the other hand, I can't see
>how to use it without snarling the surrounding code in which case it is
>probably better to explicitly manage individual iterators within a
>while loop.
>

The above would seem to allow separation of concerns. I.e., if you care why
a normal iteration terminated, you can test after the fact. I.e., if all sequences
were the same length, the .rest() iterator will be empty. And if you don't care at
all about possible data, you can just try: it.rest().next() and catch StopIteration
to check.

BTW, is there any rule against passing information with StopIteration?

Regards,
Bengt Richter
 
Reply With Quote
 
Szabolcs Nagy
Guest
Posts: n/a
 
      01-10-2006
> There are so many varieties of iterator that it's probably not workable
> to alter the iterator API for all of the them.


i always wondered if it can be implemented:

there are iterators which has length:
>>> i = iter([1,2,3])
>>> len(i)

3

now isn't there a way to make this length inheritible?
eg. generators could have length in this case:
>>> g = (x for x in [1,2,3])
>>> # len(g) == len([1,2,3]) == 3


of course in special cases length would remain undefined:
>>> f = (x for x in [1,2,3] if x>2)
>>> # len(f) == ?


IMHO there are special cases when this is useful:
L=list(it)
here if it has length, then list creation can be more effective
(required memory is known in advance)

nsz

 
Reply With Quote
 
Szabolcs Nagy
Guest
Posts: n/a
 
      01-10-2006
> There are so many varieties of iterator that it's probably not workable
> to alter the iterator API for all of the them.


i always wondered if it can be implemented:

there are iterators which has length:
>>> i = iter([1,2,3])
>>> len(i)

3

now isn't there a way to make this length inheritible?
eg. generators could have length in this case:
>>> g = (x for x in [1,2,3])
>>> # len(g) == len([1,2,3]) == 3


of course in special cases length would remain undefined:
>>> f = (x for x in [1,2,3] if x>2)
>>> # len(f) == ?


IMHO there are special cases when this is useful:
L=list(it)
here if it has length, then list creation can be more effective
(required memory is known in advance)

nsz

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
RE;Kontki if you delete kontiki any program you loaded with it in it 'will not work I have tried it with three programs and none work anymore (if you se it just stop download) 1-Twitch Computer Support 5 04-23-2009 02:45 PM
Re: ATTN: LOOKOUT, YOU NEED FRIENDS ON THE INTERNET? IT'S A SURE SIGN YOU HAVE NONE IN REAL LIFE. Wadjebkhte XXV Computer Support 0 01-23-2009 09:58 PM
imported method from module evaluates to None in some cases Andrew Python 5 11-23-2008 11:23 AM
Home server cases and external firewire cases thingy@nowhere.commy NZ Computing 5 03-14-2006 07:56 AM
Real-world use cases for map's None fill-in feature? Raymond Hettinger Python 30 01-25-2006 07:30 AM



Advertisments