Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > with statement and context managers

Reply
Thread Tools

with statement and context managers

 
 
Steven D'Aprano
Guest
Posts: n/a
 
      08-03-2011
I'm not greatly experienced with context managers and the with statement, so
I would like to check my logic.

Somebody (doesn't matter who, or where) stated that they frequently use this
idiom:

spam = MyContextManager(*args)
for ham in my_iter:
with spam:
# do stuff


but to me that looks badly wrong. Surely the spam context manager object
will exit after the first iteration, and always raise an exception on the
second? But I don't quite understand context managers enough to be sure.


I've tested it with two examples:

# Simple example using built-in file context manager.

>>> spam = open('aaa')
>>> for ham in range(5):

.... with spam:
.... print ham
....
0
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
ValueError: I/O operation on closed file


# Slightly more complex example.

>>> from contextlib import closing
>>> import urllib
>>> spam = closing(urllib.urlopen('http://www.python.org'))
>>> for ham in range(5):

.... with spam as page:
.... print ham, sum(len(line) for line in page)
....
0 18486
1
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
File "<stdin>", line 3, in <genexpr>
File "/usr/local/lib/python2.7/socket.py", line 528, in next
line = self.readline()
File "/usr/local/lib/python2.7/socket.py", line 424, in readline
recv = self._sock.recv
AttributeError: 'NoneType' object has no attribute 'recv'




Am I right to expect that the above idiom cannot work? If not, what sort of
context managers do work as shown?




--
Steven

 
Reply With Quote
 
 
 
 
Jack Diederich
Guest
Posts: n/a
 
      08-03-2011
On Tue, Aug 2, 2011 at 10:15 PM, Steven D'Aprano
<(E-Mail Removed)> wrote:
> I'm not greatly experienced with context managers and the with statement,so
> I would like to check my logic.
>
> Somebody (doesn't matter who, or where) stated that they frequently use this
> idiom:
>
> spam = MyContextManager(*args)
> for ham in my_iter:
> * *with spam:
> * * * * # do stuff
>

[snip]
> # Simple example using built-in file context manager.
>
>>>> spam = open('aaa')
>>>> for ham in range(5):

> ... * * with spam:
> ... * * * * * * print ham
> ...
> 0
> Traceback (most recent call last):
> *File "<stdin>", line 2, in <module>
> ValueError: I/O operation on closed file


file_context = lambda: open('aaa')
for i in range(3):
with file_context():
print "hello"

... but if the context is short it is clearer and time saving to _not_
alias it. If the context is sufficiently complicated then it is worth
it to make the complex code into a first class context manager -
contextlib.contextmanager makes this very easy and extremely readable.

-Jack
 
Reply With Quote
 
 
 
 
Nobody
Guest
Posts: n/a
 
      08-03-2011
On Wed, 03 Aug 2011 12:15:44 +1000, Steven D'Aprano wrote:

> I'm not greatly experienced with context managers and the with statement, so
> I would like to check my logic.
>
> Somebody (doesn't matter who, or where) stated that they frequently use this
> idiom:
>
> spam = MyContextManager(*args)
> for ham in my_iter:
> with spam:
> # do stuff
>
>
> but to me that looks badly wrong. Surely the spam context manager object
> will exit after the first iteration, and always raise an exception on the
> second? But I don't quite understand context managers enough to be sure.


It depends upon the implementation of MyContextManager. If it's
implemented using the contextlib.contextmanager decorator, then you're
correct: you can only use it once. OTOH, if you implement your own class
with __enter__ and __exit__ methods, you can use the same context manager
object multiple times.

 
Reply With Quote
 
Thomas Rachel
Guest
Posts: n/a
 
      08-03-2011
Am 03.08.2011 04:15 schrieb Steven D'Aprano:

> I'm not greatly experienced with context managers and the with
> statement, so I would like to check my logic.
>
> Somebody (doesn't matter who, or where) stated that they frequently
> use this idiom:
>
> spam = MyContextManager(*args)
> for ham in my_iter:
> with spam:
> # do stuff
>
>
> but to me that looks badly wrong. Surely the spam context manager
> object will exit after the first iteration, and always raise an
> exception on the second? But I don't quite understand context
> managers enough to be sure.


Depends on the implementation. As already stated, a
contextlib.contextmanager will only run once. But it is easy to turn it
into a persistent one - which internally initializes as often as needed.

class PersistentCM(object):
def __init__(self, target):
self.target = target
self.current = []
def __enter__(self):
cm = self.target()
cm.__enter__()
self.current.append(cm)
def __exit__(self, *e):
return self.current.pop(-1).__exit__(*e)

(untested)

This would allow for a CM to be used multiple times. It internally
generates a new one on every __enter__() call and disposes of it on
every __exit__(). As __enter__() and __exit__() are supposed to be used
symmetrical, it should(!) work this way.

Many CMs don't allow your shown use, though, partly due to obvious reasons:

* The ...ing named ones do the action they are named after in the
__exit__() part and do nothing in the __enter__() part. (E.g. closing.)

* The ...ed named ones do the named action in __enter__() and undo it in
__exit__(). They may or may not work multiple times.

For example, take threading.Lock et al., which you can have locked
several times.


>>>> spam = open('aaa')
>>>> for ham in range(5):

> ... with spam:
> ... print ham
> ...
> 0
> Traceback (most recent call last):
> File "<stdin>", line 2, in<module>
> ValueError: I/O operation on closed file


To be honest, this one I don't understand as well. __exit__() closes the
file spam, but even before "print ham" can be executed for a second
time, there is a check if the file is still open. What does __enter__()
do here? Just check if it is closed, or is it more? Nevertheless, it
makes sense.


> # Slightly more complex example.
>
>>>> from contextlib import closing
>>>> import urllib
>>>> spam = closing(urllib.urlopen('http://www.python.org'))
>>>> for ham in range(5):

> ... with spam as page:
> ... print ham, sum(len(line) for line in page)
> ...
> 0 18486
> 1
> Traceback (most recent call last):
> File "<stdin>", line 3, in<module>
> File "<stdin>", line 3, in<genexpr>
> File "/usr/local/lib/python2.7/socket.py", line 528, in next
> line = self.readline()
> File "/usr/local/lib/python2.7/socket.py", line 424, in readline
> recv = self._sock.recv
> AttributeError: 'NoneType' object has no attribute 'recv'


Here the above said applies: after closing happens in __exit__() and
_sock gets set to None, the object is not usable any longer.


> Am I right to expect that the above idiom cannot work? If not, what sort of
> context managers do work as shown?


As said, take threading.Lock as an example: they can be acquire()d and
release()d as often as you want, and these actions happen in __enter__()
and __exit__(), respectively.


HTH,

Thomas
 
Reply With Quote
 
Steven D'Aprano
Guest
Posts: n/a
 
      08-05-2011
Thomas Rachel wrote:

> Am 03.08.2011 04:15 schrieb Steven D'Aprano:

[...]
> > but to me that looks badly wrong. Surely the spam context manager
> > object will exit after the first iteration, and always raise an
> > exception on the second? But I don't quite understand context
> > managers enough to be sure.

>
> Depends on the implementation. As already stated, a
> contextlib.contextmanager will only run once. But it is easy to turn it
> into a persistent one - which internally initializes as often as needed.


Thanks, that's exactly the sort of information I was after.

Thank you to everyone who answered.


--
Steven

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How does GC affect generator context managers? Jason Python 0 11-30-2010 02:16 PM
Re: bug? context managers vs ImportErrors Ethan Furman Python 2 08-19-2010 06:25 PM
bug? context managers vs ImportErrors Chris Withers Python 1 08-19-2010 05:38 PM
Generating nested code with context managers Terry Reedy Python 1 05-05-2010 12:38 AM
context managers and co-routines Bob.Sidebotham@gmail.com Python 3 01-12-2007 04:58 PM



Advertisments