Velocity Reviews > Pre-PEP: reverse iteration methods

# Pre-PEP: reverse iteration methods

Raymond Hettinger
Guest
Posts: n/a

 09-24-2003
Here is a discussion draft of a potential PEP.
The ideas grew out of the discussion on pep-284.
Comments are invited. Dart throwing is optional.

Raymond Hettinger

-------------------------------------------------------------

PEP: 323
Version: \$Revision: 1.1 \$
Last-Modified: \$Date: 2003/03/11 04:49:44 \$
Author: Raymond Hettinger <python at rcn.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 23-Sep-2003
Python-Version: 2.4
Post-History: 23-Sep-2003

Abstract
========

This proposal is to extend the API of several sequence types
to include methods for iterating over the sequence in reverse.

Motivation
==========

For indexable objects, current methods for reverse iteration are
error prone, unnatural, and not especially readable::

for i in xrange(n-1, -1, -1):
print seqn[i]

One other current approach involves reversing a list before iterating
over it. That technique wastes computer cycles, memory, and lines of
code. Also, it only works with lists (strings, for example, do not define
a reverse method)::

rseqn = list(seqn)
rseqn.reverse()
for elem in rseqn:
print elem

Reverse iteration is much less common than forward iteration, but it
does arise regularly in practice.

Proposal
========

Add a method called iter_backwards() to sequence objects that can benefit
from it. The above examples then simplify to::

for i in xrange(n).iter_backwards():
print seqn[i]

for elem in seqn.iter_backwards():
print elem

The new protocol would be applied to lists, strings, xranges objects,
and possibly other sequence objects as well (depending on use cases
and implementation issues). It would not apply to unordered collections
like dicts and sets.

No language syntax changes are needed.

Open Issues
===========

* Should tuples be included? In the past they have been denied some list
like behaviors such as count() and index().

* Should file objects be included? Implementing reverse iteration may not
be easy though it would be useful on occasion.

* Should enumerate() be included? It would only provide reverse iteration
whenever the underlying sequence supported it.

=========

This document has been placed in the public domain.

Andrew Dalke
Guest
Posts: n/a

 09-24-2003
Raymond Hettinger:
> Proposal
> ========
>
> Add a method called iter_backwards() to sequence objects that can benefit
> from it. The above examples then simplify to::
>
> for i in xrange(n).iter_backwards():
> print seqn[i]
>
> for elem in seqn.iter_backwards():
> print elem

What about letting 'iter_backwards' be a builtin which
looks for __riter__ and if it doesn't exist, get the length
and do old-fashioned offset indexing?

Something like this untested code

def iter_backwards(obj):
try:
riter = getattr(obj, "__riter__")
except AttributeError:
n = len(obj)
while n != 0:
n = n -1
yield obj[n]
else:
for term in riter():
yield term

which would be used like

for i in iter_backwards(xrange(n)):
print seqn[i]

for elem in iter_backwards(seqn):
print elem

Andrew
http://www.velocityreviews.com/forums/(E-Mail Removed)

John Roth
Guest
Posts: n/a

 09-24-2003
Overall impression: I like it.

John Roth

"Raymond Hettinger" <(E-Mail Removed)> wrote in message
news:HA5cb.185\$(E-Mail Removed)...
> Here is a discussion draft of a potential PEP.
> The ideas grew out of the discussion on pep-284.
> Comments are invited. Dart throwing is optional.
>
>
> Raymond Hettinger
>
> -------------------------------------------------------------
>
> PEP: 323
> Title: Add Reverse Iteration Methods
> Version: \$Revision: 1.1 \$
> Last-Modified: \$Date: 2003/03/11 04:49:44 \$
> Author: Raymond Hettinger <python at rcn.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 23-Sep-2003
> Python-Version: 2.4
> Post-History: 23-Sep-2003
>
>
> Abstract
> ========
>
> This proposal is to extend the API of several sequence types
> to include methods for iterating over the sequence in reverse.
>
>
> Motivation
> ==========
>
> For indexable objects, current methods for reverse iteration are
> error prone, unnatural, and not especially readable::
>
> for i in xrange(n-1, -1, -1):
> print seqn[i]
>
> One other current approach involves reversing a list before iterating
> over it. That technique wastes computer cycles, memory, and lines of
> code.

It also has the "returns None instead of an object" wart.

> Also, it only works with lists (strings, for example, do not define
> a reverse method)::
>
> rseqn = list(seqn)
> rseqn.reverse()
> for elem in rseqn:
> print elem
>
> Reverse iteration is much less common than forward iteration, but it
> does arise regularly in practice.

Absolutely.

>
> Proposal
> ========
>
> Add a method called iter_backwards() to sequence objects that can benefit
> from it. The above examples then simplify to::
>
> for i in xrange(n).iter_backwards():
> print seqn[i]
>
> for elem in seqn.iter_backwards():
> print elem
>
> The new protocol would be applied to lists, strings, xranges objects,
> and possibly other sequence objects as well (depending on use cases
> and implementation issues). It would not apply to unordered collections
> like dicts and sets.

I presume that the result of iter_backwards() is an iterator
object with the desired behavior. In other words, it's not
the original object that has methods to handle the iteration
protocol.

>
> No language syntax changes are needed.
>
>
> Open Issues
> ===========
>
> * Should tuples be included? In the past they have been denied some list
> like behaviors such as count() and index().

I'd say yes, but I don't know the reason why tuples are missing methods.

>
> * Should file objects be included? Implementing reverse iteration may not
> be easy though it would be useful on occasion.

Are we talking text files or binary files here? Offhand, I'm not even
sure there is a binary file iterator, although if there is reverse iteration
shouldn't be difficult. Reverse iteration for text files simply means
implementing
the reverse scan in C, since the standard library doesn't support it. For
small enough files, it may be easier to simply internally use getlines() and
then use iter_reverse() on the list.

>
> * Should enumerate() be included? It would only provide reverse iteration
> whenever the underlying sequence supported it.

I don't see why not.

In general, I support the notion that a concept should be implemented
pervasively, unless there's a clear reason not to do it in a special case.
Special cases are just that - if they really are special, you trip over them
once and then remember them. What gets messy is something that's
partially implemented, where there is no clear rule to determine when
it is and when it isn't.

John Roth
>
>
> =========
>
> This document has been placed in the public domain.
>
>
>

Terry Reedy
Guest
Posts: n/a

 09-24-2003

"Raymond Hettinger" <(E-Mail Removed)> wrote in message
news:HA5cb.185\$(E-Mail Removed)...
> for i in xrange(n).iter_backwards():
> print seqn[i]
>
> for elem in seqn.iter_backwards():
> print elem

I would prefer a much shorter name, such as riter for reverse/right
iter. This goes along with current nomenclature such as rfind, etc.

Terry J. Reedy

John Roth
Guest
Posts: n/a

 09-24-2003

"Andrew Dalke" <(E-Mail Removed)> wrote in message
news:hJ6cb.1252\$(E-Mail Removed) nk.net...
> Raymond Hettinger:
> > Proposal
> > ========
> >
> > Add a method called iter_backwards() to sequence objects that can

benefit
> > from it. The above examples then simplify to::
> >
> > for i in xrange(n).iter_backwards():
> > print seqn[i]
> >
> > for elem in seqn.iter_backwards():
> > print elem

>
> What about letting 'iter_backwards' be a builtin which
> looks for __riter__ and if it doesn't exist, get the length
> and do old-fashioned offset indexing?

There are objects that support iteration that
don't support len(), such as file objects.
This has got the advantage, of course, that it would
automatically work on all objects that support
an sequence-like protocol, though.

Frankly, I prefer the notion of a method.
Maybe make it a method of object that
automatically works on all new style
objects that support a sequence-like
protocol?

John Roth

Andrew Dalke
Guest
Posts: n/a

 09-24-2003
John Roth:
> There are objects that support iteration that
> don't support len(), such as file objects.

Sure. Then it'll given an exception. The implementation
should turn around and raise the proper exception there
instead of a "len doesn't exist" one.

There's also a problem that len works on a dict but
__getitem__(int) will only work if the dict stores 0->n-1
as keys.

> This has got the advantage, of course, that it would
> automatically work on all objects that support
> an sequence-like protocol, though.

Yup.

> Frankly, I prefer the notion of a method.

While I don't. There are many, many ways to
iterate through a list. (What about all evens
followed by all odds?) Right now they are all
done by using a function which creates an iterator
across a container. If it's a method then you've
said that reverse_iter is special enough that it
must be a method, which seems strange for something
which is so infrequently used.

Andrew
(E-Mail Removed)

Sean Ross
Guest
Posts: n/a

 09-24-2003
"Raymond Hettinger" <(E-Mail Removed)> wrote in message
news:HA5cb.185\$(E-Mail Removed)...
> Here is a discussion draft of a potential PEP.

[snip]
> Proposal
> ========
>
> Add a method called iter_backwards() to sequence objects that can benefit
> from it. The above examples then simplify to::
>
> for i in xrange(n).iter_backwards():
> print seqn[i]
>
> for elem in seqn.iter_backwards():
> print elem
>

Hi.
This will mostly just be some alternate naming suggestions:

that returns a generator for reverse iteration?

for i in xrange(n).backwards:
print seqn[i]

for elem in seqn.backwards:
print elem

It's not as explicit as iter_backwards(), but it's shorter, cleaner, and
appears
obvious (to me, anyway). Or, perhaps, 'ibackwards', or 'ireverse' ; or
ibackwards() or ireverse(). (being reminiscent of imap() and izip())
'.reverse' would be good, but, of course, it's already taken...

If one could be made, a fast** general purpose "reverse(iterable)" (or
ireverse(iterable)) function would be my preference (at the very least, it
would
avoid having to add iter_backwards() to several type definitions).

# e.g.
for i in reverse(xrange(n)):
print seqn[i]

for index, item in reverse(enumerate(seqn)):
print "index:%s item:%s"%(index, item)

# that sort of thing...

** Yep, fast. Something where xrange(n-1, -1, -1) is not exorbitantly
faster than reverse(xrange(n)). Similarly, for sequence types, you'd
want reverse(list) to be atleast somewhere near as quick as:

# python-faq, entry 4.6
list.reverse()
try:
for x in list:
"do something with x"
finally:
list.reverse()

Raymond Hettinger
Guest
Posts: n/a

 09-24-2003
[Raymond Hettinger]
> > Reverse iteration is much less common than forward iteration, but it
> > does arise regularly in practice.

[John Roth]
> Absolutely.

Because the question will arise, I did a quick use case analysis from
the standard library. I'll attach the following to the PEP. Feel free
to comment or add other use cases.

Raymond Hettinger

--------------------------------------------------------------------------

Use Case Analysis
=================

Here are some instances of reverse iteration taken from the standard
library and comments on why reverse iteration was necessary:

* atexit.exit_handlers() uses::
while _exithandlers:
func, targs, kargs = _exithandlers.pop()
. . .
The application dictates the need to run exit handlers in the
reverse order they were built. The "while alist: alist.pop()"
form is readable and clean; however, it would be slightly faster
and clearer with:
for func, target, kargs in _exithandlers.iter_backwards():
. . .
del _exithandlers

* difflib.get_close_matches() uses::
result.sort() # Retain only the best n.
result = result[-n:] # Move best-scorer to head of list.
result.reverse() # Strip scores.
return [x for score, x in result]
The need for reverse iteration arises from a requirement to return
a portion of a sort in an order opposite of the sort criterion. The
list comprehension is incidental (the third step of a Schwartzian
transform). This particular use case can met with extended slicing,
but the code is somewhat unattractive and hard to visually verify::
result.sort()
return result[:-n-1:-1]

* heapq.heapify() uses "for i in xrange(n//2 - 1, -1, -1)" because
higher-level orderings are more easily formed from pairs of
lower-level orderings. A forward version of this algorithm is
possible; however, that would complicate the rest of the heap code
which iterates over the underlying list in the opposite direction.

* mhlib.test() uses::
testfolders.reverse();
for t in testfolders:
do('mh.deletefolder(%s)' % `t`)
The need for reverse iteration arises because the tail of the underlying
list is altered during iteration.

* platform._dist_try_harder() uses "for n in range(len(verfiles)-1, -1, -1)"
because the loop deletes selected elements from "verfiles" but needs to
leave the rest of the list intact for further iteration. This use
case could be more efficiently met with itertools.ifilter but the
lambda condition and functional style would render it less readable.

* random.shuffle uses "for i in xrange(len(x)-1, 0, -1)" because the
algorithm is most easily understood as randomly selecting elements
from an ever diminishing pool. In fact, the algorithm can be run in
a forward direction but is less intuitive and rarely presented that
way in literature.

* rfc822.Message.__delitem__() uses:
list.reverse()
for i in list:
The need for reverse iteration arises because the tail of the underlying
list is altered during iteration.

Andrew Dalke
Guest
Posts: n/a

 09-24-2003
Sean Ross:
> This will mostly just be some alternate naming suggestions:
>

property
> that returns a generator for reverse iteration?

-1 from me. 'backwards' looks to me like something which changes
the list ordering in place, like what 'reverse' does. Or something
which returns a new list but in reverse order. It's an adjective,
when it should be a verb.

ibackwords would work, but so would riter or reviter or
variations thereof.

Andrew
(E-Mail Removed)

Stephen Horne
Guest
Posts: n/a

 09-24-2003
On Wed, 24 Sep 2003 00:30:31 GMT, "Raymond Hettinger"
<(E-Mail Removed)> wrote:

>Proposal
>========
>
>Add a method called iter_backwards() to sequence objects that can benefit
>from it. The above examples then simplify to::
>
> for i in xrange(n).iter_backwards():
> print seqn[i]
>
> for elem in seqn.iter_backwards():
> print elem

That is a pretty long name. Can we use the convention from itertools
where an 'i' prefix is sufficient to suggest an iterator, giving
'ibackwards' or 'ireverse' or similar.

Second, I'm not quite ready to drop my property idea

An advantage is that the object returned by the property can sensibly
support more than just iteration - e.g. slicing (as touched on in the

>>> x = range(10)
>>> list(x.backward)

[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
>>> list(x.backward [::2])

[8, 6, 4, 2, 0]

>* Should file objects be included? Implementing reverse iteration may not
> be easy though it would be useful on occasion.

IMO no - doing this essentially needs the whole file to be read into
memory, in which case you may as well read the whole file into a list
and then iterate the list backwards.

>* Should enumerate() be included? It would only provide reverse iteration
> whenever the underlying sequence supported it.

Why not...

for i, j in enumerate (listname.iter_backwards ()) :

in other words, as enumerate can handle the already-reversed
sequence/iteration, I don't see the point.

--
Steve Horne

steve at ninereeds dot fsnet dot co dot uk