Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > subprocess.Popen inheriting

Reply
Thread Tools

subprocess.Popen inheriting

 
 
Aaron Brady
Guest
Posts: n/a
 
      12-16-2008
Hi,

I have a file handle I want to inherit in a child process. I am
looking at '_make_inheritable' in 'Popen', but it needs an instance,
and by the time I have one, the subprocess is already running.

Can't I call 'Popen._make_inheritable( None, handle )'? The method
does not use 'self'.
 
Reply With Quote
 
 
 
 
Gabriel Genellina
Guest
Posts: n/a
 
      12-16-2008
En Tue, 16 Dec 2008 07:29:19 -0200, Aaron Brady <(E-Mail Removed)>
escribió:

> I have a file handle I want to inherit in a child process. I am
> looking at '_make_inheritable' in 'Popen', but it needs an instance,
> and by the time I have one, the subprocess is already running.
>
> Can't I call 'Popen._make_inheritable( None, handle )'? The method
> does not use 'self'.


File handles are inherited by default, I think. What's your specific
problem?

--
Gabriel Genellina

 
Reply With Quote
 
 
 
 
Aaron Brady
Guest
Posts: n/a
 
      12-16-2008
On Dec 16, 4:15*am, "Gabriel Genellina" <(E-Mail Removed)>
wrote:
> En Tue, 16 Dec 2008 07:29:19 -0200, Aaron Brady <(E-Mail Removed)> *
> escribió:
>
> > I have a file handle I want to inherit in a child process. *I am
> > looking at '_make_inheritable' in 'Popen', but it needs an instance,
> > and by the time I have one, the subprocess is already running.

>
> > Can't I call 'Popen._make_inheritable( None, handle )'? *The method
> > does not use 'self'.

>
> File handles are inherited by default, I think. What's your specific *
> problem?
>
> --
> Gabriel Genellina


I want a dual-console application for some multi-threaded output. The
main process spawns a second process in a second window, and directs
its readout through a pipe. That is, the second process is just a
dummy, print pipe directly to console.

It worked when I used the 'stdin= PIPE' keyword in Popen, but passing
my own pipe handle on the command line isn't working. The keyword is
a workaround.

> File handles are inherited by default, I think.


I thought so too. The web seems to say that on Linux they are, and on
Windows, you need to call DuplicateHandle for it.

By the way, there are a few solutions to the inheritance problem.

1. Override __init__ and __del__ in a subclass.
2. Assign DummyClass= Popen._make_inheritable
3. Override Popen._execute_child to call _make_inheritable, then call
the super method.
 
Reply With Quote
 
Jeremy Sanders
Guest
Posts: n/a
 
      12-17-2008
Aaron Brady wrote:

> I thought so too. The web seems to say that on Linux they are, and on
> Windows, you need to call DuplicateHandle for it.


I hit this problem - it looks like pipes aren't very versatile on Windows.
There's also the complicating factor that the handles in windows aren't the
same as the file numbers that Python uses, so you have to convert between
them.

It would be nice if Python created pipes that are properly inheritable by
default by child processes, as they're mostly used for IPC.

It was so painful that I converted my code to use sockets instead, which
seem much more portable between Windows and Unix (though you don't get to
use socketpair and AF_UNIX in Windows).

Jeremy

--
Jeremy Sanders
http://www.jeremysanders.net/
 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      12-17-2008
En Wed, 17 Dec 2008 12:21:38 -0200, Jeremy Sanders
<(E-Mail Removed)> escribió:

> Aaron Brady wrote:
>
>> I thought so too. The web seems to say that on Linux they are, and on
>> Windows, you need to call DuplicateHandle for it.


Or set bInheritHandle=True when creating the pipe initially. os.pipe()
doesn't do that.

> I hit this problem - it looks like pipes aren't very versatile on
> Windows.
> There's also the complicating factor that the handles in windows aren't
> the
> same as the file numbers that Python uses, so you have to convert between
> them.
>
> It would be nice if Python created pipes that are properly inheritable by
> default by child processes, as they're mostly used for IPC.


I'd say it is a bug in os.pipe implementation; they should be inheritable
by default, as in posix (after all, the code is in "posixmodule.c").

--
Gabriel Genellina

 
Reply With Quote
 
Aaron Brady
Guest
Posts: n/a
 
      12-18-2008
On Dec 17, 5:05*pm, "Gabriel Genellina" <(E-Mail Removed)>
wrote:
> En Wed, 17 Dec 2008 12:21:38 -0200, Jeremy Sanders *
> <(E-Mail Removed)> escribió:
>
> > Aaron Brady wrote:

>
> >> I thought so too. *The web seems to say that on Linux they are, and on
> >> Windows, you need to call DuplicateHandle for it.

>
> Or set bInheritHandle=True when creating the pipe initially. os.pipe() *
> doesn't do that.
>
> > I hit this problem - it looks like pipes aren't very versatile on *
> > Windows.
> > There's also the complicating factor that the handles in windows aren't *
> > the
> > same as the file numbers that Python uses, so you have to convert between
> > them.

>
> > It would be nice if Python created pipes that are properly inheritable by
> > default by child processes, as they're mostly used for IPC.

>
> I'd say it is a bug in os.pipe implementation; they should be inheritable *
> by default, as in posix (after all, the code is in "posixmodule.c").


The code looks like this:

ok = CreatePipe(&read, &write, NULL, 0);
Py_END_ALLOW_THREADS
if (!ok)
return win32_error("CreatePipe", NULL);
read_fd = _open_osfhandle((Py_intptr_t)read, 0);
write_fd = _open_osfhandle((Py_intptr_t)write, 1);

'If lpPipeAttributes is NULL, the handle cannot be inherited.' You
could populate a 'SECURITY_ATTRIBUTES' structure, or call
DuplicateHandle on both of them.

A patch would look like this:

SECURITY_ATTRIBUTES sattribs;
sattribs.nLength = sizeof(sattribs);
sattribs.lpSecurityDescriptor = NULL;
sattribs.bInheritHandle = TRUE;
ok = CreatePipe(&read, &write, &sattribs, 0);

This still doesn't answer whether the file descriptor return by
'_open_osfhandle' can be inherited too.
 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      12-18-2008
En Wed, 17 Dec 2008 22:46:32 -0200, Aaron Brady <(E-Mail Removed)>
escribió:
> On Dec 17, 5:05*pm, "Gabriel Genellina" <(E-Mail Removed)>
> wrote:
>> En Wed, 17 Dec 2008 12:21:38 -0200, Jeremy Sanders *
>> <(E-Mail Removed)> escribió:
>>
>> > It would be nice if Python created pipes that are properly

>> inheritable by
>> > default by child processes, as they're mostly used for IPC.

>>
>> I'd say it is a bug in os.pipe implementation; they should be
>> inheritable *
>> by default, as in posix (after all, the code is in "posixmodule.c").

>
> The code looks like this:
>
> ok = CreatePipe(&read, &write, NULL, 0);
> Py_END_ALLOW_THREADS
> if (!ok)
> return win32_error("CreatePipe", NULL);
> read_fd = _open_osfhandle((Py_intptr_t)read, 0);
> write_fd = _open_osfhandle((Py_intptr_t)write, 1);
>
> 'If lpPipeAttributes is NULL, the handle cannot be inherited.' You
> could populate a 'SECURITY_ATTRIBUTES' structure, or call
> DuplicateHandle on both of them.
>
> A patch would look like this:
>
> SECURITY_ATTRIBUTES sattribs;
> sattribs.nLength = sizeof(sattribs);
> sattribs.lpSecurityDescriptor = NULL;
> sattribs.bInheritHandle = TRUE;
> ok = CreatePipe(&read, &write, &sattribs, 0);


Yes, that's exactly how os.popen does it (in posixmodule.c)

> This still doesn't answer whether the file descriptor return by
> '_open_osfhandle' can be inherited too.


It doesn't matter. The OS only cares about file handles, not C RTL
structures.

--
Gabriel Genellina

 
Reply With Quote
 
Aaron Brady
Guest
Posts: n/a
 
      12-18-2008
On Dec 17, 7:16*pm, "Gabriel Genellina" <(E-Mail Removed)>
wrote:
> En Wed, 17 Dec 2008 22:46:32 -0200, Aaron Brady <(E-Mail Removed)> *
> escribió:
>
>
>
> > On Dec 17, 5:05*pm, "Gabriel Genellina" <(E-Mail Removed)>
> > wrote:
> >> En Wed, 17 Dec 2008 12:21:38 -0200, Jeremy Sanders *
> >> <(E-Mail Removed)> escribió:

>
> >> > It would be nice if Python created pipes that are properly *
> >> inheritable by
> >> > default by child processes, as they're mostly used for IPC.

>
> >> I'd say it is a bug in os.pipe implementation; they should be *
> >> inheritable *
> >> by default, as in posix (after all, the code is in "posixmodule.c").

>
> > The code looks like this:

>
> > * *ok = CreatePipe(&read, &write, NULL, 0);
> > * *Py_END_ALLOW_THREADS
> > * *if (!ok)
> > * * * * * *return win32_error("CreatePipe", NULL);
> > * *read_fd = _open_osfhandle((Py_intptr_t)read, 0);
> > * *write_fd = _open_osfhandle((Py_intptr_t)write, 1);

>
> > 'If lpPipeAttributes is NULL, the handle cannot be inherited.' *You
> > could populate a 'SECURITY_ATTRIBUTES' structure, or call
> > DuplicateHandle on both of them.

>
> > A patch would look like this:

>
> > SECURITY_ATTRIBUTES sattribs;
> > sattribs.nLength = sizeof(sattribs);
> > sattribs.lpSecurityDescriptor = NULL;
> > sattribs.bInheritHandle = TRUE;
> > ok = CreatePipe(&read, &write, &sattribs, 0);

>
> Yes, that's exactly how os.popen does it (in posixmodule.c)
>
> > This still doesn't answer whether the file descriptor return by
> > '_open_osfhandle' can be inherited too.

>
> It doesn't matter. The OS only cares about file handles, not C RTL *
> structures.
>
> --
> Gabriel Genellina


Ah, I see. Was it an executive decision about what is Pythonic, or
just a bug? Do you think the patch would be accepted? I probably
ought to mimic a small Python embedding to see if it needs anything
else.
 
Reply With Quote
 
Aaron Brady
Guest
Posts: n/a
 
      12-18-2008
On Dec 17, 7:16*pm, "Gabriel Genellina" <(E-Mail Removed)>
wrote:
> En Wed, 17 Dec 2008 22:46:32 -0200, Aaron Brady <(E-Mail Removed)> *
> escribió:
>
>
>
> > On Dec 17, 5:05*pm, "Gabriel Genellina" <(E-Mail Removed)>
> > wrote:
> >> En Wed, 17 Dec 2008 12:21:38 -0200, Jeremy Sanders *
> >> <(E-Mail Removed)> escribió:

>
> >> > It would be nice if Python created pipes that are properly *
> >> inheritable by
> >> > default by child processes, as they're mostly used for IPC.

>
> >> I'd say it is a bug in os.pipe implementation; they should be *
> >> inheritable *
> >> by default, as in posix (after all, the code is in "posixmodule.c").

>
> > The code looks like this:

>
> > * *ok = CreatePipe(&read, &write, NULL, 0);
> > * *Py_END_ALLOW_THREADS
> > * *if (!ok)
> > * * * * * *return win32_error("CreatePipe", NULL);
> > * *read_fd = _open_osfhandle((Py_intptr_t)read, 0);
> > * *write_fd = _open_osfhandle((Py_intptr_t)write, 1);

>
> > 'If lpPipeAttributes is NULL, the handle cannot be inherited.' *You
> > could populate a 'SECURITY_ATTRIBUTES' structure, or call
> > DuplicateHandle on both of them.

>
> > A patch would look like this:

>
> > SECURITY_ATTRIBUTES sattribs;
> > sattribs.nLength = sizeof(sattribs);
> > sattribs.lpSecurityDescriptor = NULL;
> > sattribs.bInheritHandle = TRUE;
> > ok = CreatePipe(&read, &write, &sattribs, 0);

>
> Yes, that's exactly how os.popen does it (in posixmodule.c)
>
> > This still doesn't answer whether the file descriptor return by
> > '_open_osfhandle' can be inherited too.

>
> It doesn't matter. The OS only cares about file handles, not C RTL *
> structures.


Sorry for the multiple posts. File handles are inheritable by child
processes, if the permissions are right. File descriptors are not.
Is there a way that we can get the handles of a pipe into code, so
that we can pass them to a subprocess? Will it take calling
'CreatePipe' from ctypes directly if on Windows? Or can 'os.pipe' be
made to abstract that? If Windows can't inherit descriptors,
'os.pipe' should return handles, and 'os.read' &co. should accept
them.

It is a fairly large patch.
 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      12-19-2008
En Thu, 18 Dec 2008 08:35:58 -0200, Aaron Brady <(E-Mail Removed)>
escribió:

> On Dec 17, 7:16*pm, "Gabriel Genellina" <(E-Mail Removed)>
> wrote:
>> En Wed, 17 Dec 2008 22:46:32 -0200, Aaron Brady <(E-Mail Removed)> *
>> escribió:
>>
>>
>>
>> > On Dec 17, 5:05*pm, "Gabriel Genellina" <(E-Mail Removed)>
>> > wrote:
>> >> En Wed, 17 Dec 2008 12:21:38 -0200, Jeremy Sanders *
>> >> <(E-Mail Removed)> escribió:

>>
>> >> > It would be nice if Python created pipes that are properly *
>> >> inheritable by
>> >> > default by child processes, as they're mostly used for IPC.

>>
>> >> I'd say it is a bug in os.pipe implementation; they should be *
>> >> inheritable *
>> >> by default, as in posix (after all, the code is in "posixmodule.c").

>>
>> > The code looks like this:

>>
>> > * *ok = CreatePipe(&read, &write, NULL, 0);
>> > * *Py_END_ALLOW_THREADS
>> > * *if (!ok)
>> > * * * * * *return win32_error("CreatePipe", NULL);
>> > * *read_fd = _open_osfhandle((Py_intptr_t)read, 0);
>> > * *write_fd = _open_osfhandle((Py_intptr_t)write, 1);

>>
>> > 'If lpPipeAttributes is NULL, the handle cannot be inherited.' *You
>> > could populate a 'SECURITY_ATTRIBUTES' structure, or call
>> > DuplicateHandle on both of them.

>>
>> > A patch would look like this:

>>
>> > SECURITY_ATTRIBUTES sattribs;
>> > sattribs.nLength = sizeof(sattribs);
>> > sattribs.lpSecurityDescriptor = NULL;
>> > sattribs.bInheritHandle = TRUE;
>> > ok = CreatePipe(&read, &write, &sattribs, 0);

>>
>> Yes, that's exactly how os.popen does it (in posixmodule.c)
>>
>> > This still doesn't answer whether the file descriptor return by
>> > '_open_osfhandle' can be inherited too.

>>
>> It doesn't matter. The OS only cares about file handles, not C RTL *
>> structures.
>>
>> --
>> Gabriel Genellina

>
> Ah, I see. Was it an executive decision about what is Pythonic, or
> just a bug? Do you think the patch would be accepted? I probably
> ought to mimic a small Python embedding to see if it needs anything
> else.


I don't know - I guess someone (years ago) blindly just replaced the
pipe() system call by a CreatePipe call without further analysis.

This is how I would summarize the issue:

Pros (of changing os.pipe() to return inheritable pipes):

- it isn't explicitely documented whether os.pipe() returns inheritable
pipes or not, so both versions are "right" according to the documentation.
- if someone relies on pipes being non-inheritable on Windows, that is
undocumented behaviour, and Python has the right to change it.
- the change would improve POSIX compatibility, it mimics what os.pipe()
does on those OS.
- inheritable pipes are less surprising for guys coming from other OS
- inheritable pipes are a lot more useful than non-inheritable ones when
doing IPC (probably its main usage).

Cons:

- os.pipe has behaved that way since long time ago.
- some programs *might* break, if they relied on pipes being
non-inheritable on Windows, even if that was undocumented behaviour.

--
Gabriel Genellina

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Inheriting class with private constructor Michael Carr ASP .Net 1 12-29-2003 04:59 AM
Inheriting System.UI.ControlCollection for a WebControl - FindControl doesn't work??? Harry F. Harrison ASP .Net 0 12-18-2003 06:28 PM
Inheriting from base form Andy Breward ASP .Net 1 11-07-2003 06:15 AM
inheriting from a class that inherits from the UserControl class Joel Barsotti ASP .Net 4 10-30-2003 08:05 PM
Inheriting local variables from a common source? Wayne J ASP .Net 2 10-24-2003 05:59 PM



Advertisments