«

Does anyone have an example of utilisation?

On 09/03/2012 08:32 AM, Marco wrote:
> Does anyone have an example of utilisation?

As of Python 3.2.3, there is no "opener" parameter in the open() function.
http://docs.python.org/py3k/library/functions.html

I don't know of any such parameter in earlier or later versions, but I
could be wrong there.

--

DaveA

On 09/03/2012 03:05 PM, Dave Angel wrote:

>> Does anyone have an example of utilisation?

> As of Python 3.2.3, there is no "opener" parameter in the open() function.
> http://docs.python.org/py3k/library/functions.html
>
> I don't know of any such parameter in earlier or later versions, but I
> could be wrong there.

It's new in Python 3.3:

http://docs.python.org/dev/library/functions.html#open

On Mon, 03 Sep 2012 15:29:05 +0200, Christian Heimes wrote:

> Am 03.09.2012 14:32, schrieb Marco:
>> Does anyone have an example of utilisation?
>
> The opener argument is a new 3.3 feature. For example you can use the
> feature to implement exclusive creation of a file to avoid symlink
> attacks.
>
> import os
>
> def opener(file, flags):
> return os.open(file, flags | os.O_EXCL)
>
> open("newfile", "w", opener=opener)


Why does the open builtin need this added complexity? Why not just call
os.open directly? Or for more complex openers, just call the opener
directly?

What is the rationale for complicating open instead of telling people to
just call their opener directly?


--
Steven

On 04 Sep 2012 01:13:09 GMT, Steven D'Aprano
<steve+> declaimed the following in
gmane.comp.python.general:


>
> Why does the open builtin need this added complexity? Why not just call
> os.open directly? Or for more complex openers, just call the opener
> directly?
>
Because os.open() returns a low-level file descriptor, not a Python
file object?

> What is the rationale for complicating open instead of telling people to
> just call their opener directly?

To avoid the new syntax would mean coding the example as

f = os.fdopen(os.open("newfile", flags | os.O_EXCL), "w")

which does NOT look any cleaner to me... Especially not if "opener" is
to be used in more than one location. Furthermore, using "opener" could
allow for a localized change to affect all open statements in the module
-- change file path, open for string I/O rather than file I/O, etc.
--
Wulfraed Dennis Lee Bieber AF6VN
HTTP://wlfraed.home.netcom.com/

On Mon, 03 Sep 2012 23:19:51 -0400, Dennis Lee Bieber wrote:

> On 04 Sep 2012 01:13:09 GMT, Steven D'Aprano
> <steve+> declaimed the following in
> gmane.comp.python.general:
>
>
>
>> Why does the open builtin need this added complexity? Why not just call
>> os.open directly? Or for more complex openers, just call the opener
>> directly?
>>
> Because os.open() returns a low-level file descriptor, not a
> Python file object?

Good point.

But you can wrap the call to os.open, as you mention below. The only
complication is that you have to give the mode twice, converting between
low-level O_* integer modes and high-level string modes:

a = os.open('/tmp/foo', os.O_WRONLY | os.O_CREAT)
b = os.fdopen(a, 'w')


But to some degree, you still have to do that with the opener argument,
at least in your own head.


>> What is the rationale for complicating open instead of telling people
>> to just call their opener directly?
>
> To avoid the new syntax would mean coding the example as
>
> f = os.fdopen(os.open("newfile", flags | os.O_EXCL), "w")
>
> which does NOT look any cleaner to me...

Well, I don't know about that. Once you start messing about with low-
level O_* flags, it's never going to exactly be clean no matter what you
do. But I think a one-liner like the above *is* cleaner than a three-
liner like the original:

def opener(file, flags):
return os.open(file, flags | os.O_EXCL)

open("newfile", "w", opener=opener)

although I accept that this is a matter of personal taste.

Particularly if the opener is defined far away from where you eventually
use it. A lambda is arguably better from that perspective:

open("newfile", "w",
opener=lambda file, flags: os.open(file, flags | os.O_EXCL)
)

but none of these solutions are exactly neat or clean. You still have to
mentally translate between string modes and int modes, and make sure
you're not passing the wrong mode:

py> open('junk', 'w').write('hello world')
11
py> open('junk', 'r', opener=lambda file, flags: os.open(file, flags |
os.O_TRUNC)).read() # oops
''

so it's not exactly a high-level interface.

In my opinion, a cleaner, more Pythonic interface would be either:

* allow built-in open to take numeric modes:

open(file, os.O_CREAT | os.O_WRONLY | os.O_EXCL)

* or even more Pythonic, expose those numeric modes using strings:

open(file, 'wx')


That's not as general as an opener, but it covers the common use-case and
for everything else, write a helper function.


> Especially not if "opener" is to be used in more than one location.

The usual idiom for fixing the "used more than once" is "write a helper",
not "add a callback function to a builtin"


> Furthermore, using "opener" could
> allow for a localized change to affect all open statements in the module
> -- change file path, open for string I/O rather than file I/O, etc.


A common idiom for that is to shadow open in the module, like this:

_open = open
def open(file, *args):
file = file.lowercase()
return _open(file, *args)



--
Steven

On 04.09.12 04:13, Steven D'Aprano wrote:
> Why does the open builtin need this added complexity? Why not just call
> os.open directly? Or for more complex openers, just call the opener
> directly?
>
> What is the rationale for complicating open instead of telling people to
> just call their opener directly?

See http://bugs.python.org/issue12797.

On 03.09.12 15:32, Marco wrote:
> Does anyone have an example of utilisation?

http://bugs.python.org/issue13424

On Tue, 04 Sep 2012 15:12:48 +1000, Ben Finney
<ben+> declaimed the following in
gmane.comp.python.general:

>
> why not call that directly?
>
> f = opener(file, flags)
>
> It certainly is cleaner than either of the alternatives so far, and it
> doesn't add a parameter to the builtin.
>
But it returns an OS file descriptor... It doesn't return a Python
file object. From what I can tell, (I've just upgraded to Python 2.7
<G>) the opener is meant to replace the low-level function normally used
by Python's open(), and supplies an fd which gets wrapped by Python's
open().

>>> import os
>>> fd = os.open("somefile.txt", os.O_CREAT)
>>> type(fd)
<type 'int'>
>>> fo = open("somefile2.txt", "w")
>>> type (fo)
<type 'file'>

The two are not compatible except by using os.fdopen(fd) to get a
file object, or fo.fileno() to get the low-level file descriptor

>>> fo.fileno()
4
>>> fo1 = os.fdopen(fd, "w")
>>> type (fo1)
<type 'file'>
>>>



> > Furthermore, using "opener" could allow for a localized change to
> > affect all open statements in the module -- change file path, open for
> > string I/O rather than file I/O, etc.
>
> I don't know of any real-life code which would be significantly improved
> by that. Can you point us to some?
>
Not really -- but if they went one step further and supplied
"reader" and "writer" operations too, they'd get close to what I once
had to do in FORTRAN 77 under DEC VMS (by hooking in code to do double
buffering when reading data from magtape, while keeping the program
using regular F77 I/O statements; the open statement would do a pre-read
of one buffer and return; subsequent read statements would find a
pre-filled buffer, and issue an non-blocking read to fill the other
buffer -- cut the runtime for the program into a third or less as it was
no longer stuck waiting for slow mag-tape operations each time it did a
read). Implementing something like this in Python would likely require
"opener" to spawn a reader thread to do the I/O asynchronously, using a
limited Queue (1 buffer worth -- the reader thread would be the second
buffer, blocked on Q.put()), and a "reader" that would do Q.get() and
return the result to the Python read() logic for any parsing.

Okay, in Python, one could probably subclass "file", and override
the read methods -- but one would not be able to use the Python
open()... You'd have to do something like f = myFile(normal, open, args)
instead...


> --
> \ “I find the whole business of religion profoundly interesting. |
> `\ But it does mystify me that otherwise intelligent people take |
> _o__) it seriously.” —Douglas Adams |
> Ben Finney
--
Wulfraed Dennis Lee Bieber AF6VN
HTTP://wlfraed.home.netcom.com/

On 9/4/2012 8:58 AM, Serhiy Storchaka wrote:
> On 04.09.12 04:13, Steven D'Aprano wrote:
>> Why does the open builtin need this added complexity? Why not just call
>> os.open directly? Or for more complex openers, just call the opener
>> directly?
>>
>> What is the rationale for complicating open instead of telling people to
>> just call their opener directly?
>
> See http://bugs.python.org/issue12797.

io.open depends on a function the returns an open file descriptor.
opener exposes that dependency so it can be replaced. (Obviously, one
could go crazily overboard with this idea.) I believe this is a simple
form of dependency injection, though it might be hard to discern from
the Java-inspired verbiage of the Wikipedia article. Part of the
rationale in the issue is to future-proof io.open from any future needs
for alternate fd fetching. It could also be used to decouple a test of
io.open from os.open

--
Terry Jan Reedy