Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Doubts related to subprocess.Popen()

Reply
Thread Tools

Doubts related to subprocess.Popen()

 
 
srinivasan srinivas
Guest
Posts: n/a
 
      01-20-2009
Hi,
Does*subprocess.Popen() count a new open file for each suprocess? I mean does it occupy an entry in file descriptor table of parent process?
If so, wat is each file descriptor connected to?

Thanks,
Srini


Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
 
Reply With Quote
 
 
 
 
Diez B. Roggisch
Guest
Posts: n/a
 
      01-20-2009
srinivasan srinivas wrote:

> Hi,
> DoesÂ*subprocess.Popen() count a new open file for each suprocess? I mean
> does it occupy an entry in file descriptor table of parent process? If so,
> wat is each file descriptor connected to?


Usually, each new process has three file-descriptors associated with it -
stdin,stdout and stderr.

So when you span a process, the overall count of FDs should increase by
three.

Additionally, there are more FDs created if you chose to pipe communication
between the child-process and the parent.

This is a unix-thing btw, nothing to do with subprocess per se.

Diez
 
Reply With Quote
 
 
 
 
Mark Wooding
Guest
Posts: n/a
 
      01-20-2009
srinivasan srinivas <(E-Mail Removed)> writes:

> Does subprocess.Popen() count a new open file for each suprocess? I
> mean does it occupy an entry in file descriptor table of parent
> process? If so, wat is each file descriptor connected to?


On Unix, subprocess.Popen will use up a file descriptor in the parent
for each use of subprocess.PIPE. The descriptor in question is one end
of a pipe; the child process holds the other end.

I guess the situation is similar on Windows, but I don't know for sure.

-- [mdw]
 
Reply With Quote
 
srinivasan srinivas
Guest
Posts: n/a
 
      01-20-2009
Do*parent process will have different file descriptor in it for each subprocesses or*paprent uses a single file descriptor for all?
I really want to know creation of each subprocess will occupy an entry in parents'file descriptor table. B'cos if i create more than 200 subprocesses, i am getting 'Too many open files' error.

Thanks,
Srini
----- Original Message ----
From: Mark Wooding <(E-Mail Removed)>
To: python-list@python..org
Sent: Tuesday, 20 January, 2009 6:16:17 PM
Subject: Re: Doubts related to subprocess.Popen()

srinivasan srinivas <(E-Mail Removed)> writes:

> Does subprocess.Popen() count a new open file for each suprocess? I
> mean does it occupy an entry in file descriptor table of parent
> process?* If so, wat is each file descriptor connected to?


On Unix, subprocess.Popen will use up a file descriptor in the parent
for each use of subprocess.PIPE.* The descriptor in question is one end
of a pipe; the child process holds the other end.

I guess the situation is similar on Windows, but I don't know for sure.

-- [mdw]
--
http://mail.python.org/mailman/listinfo/python-list



Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
 
Reply With Quote
 
Philip Semanchuk
Guest
Posts: n/a
 
      01-20-2009

On Jan 20, 2009, at 9:19 AM, srinivasan srinivas wrote:

> Do parent process will have different file descriptor in it for each
> subprocesses or paprent uses a single file descriptor for all?
> I really want to know creation of each subprocess will occupy an
> entry in parents'file descriptor table. B'cos if i create more than
> 200 subprocesses, i am getting 'Too many open files' error.


If you're on Unix, a command like lsof can be useful at a time like
this.

lsof = "ls open files"

If you're on Linux, there's the /proc tree to investigate.





>
> ----- Original Message ----
> From: Mark Wooding <(E-Mail Removed)>
> To: http://www.velocityreviews.com/forums/(E-Mail Removed)
> Sent: Tuesday, 20 January, 2009 6:16:17 PM
> Subject: Re: Doubts related to subprocess.Popen()
>
> srinivasan srinivas <(E-Mail Removed)> writes:
>
>> Does subprocess.Popen() count a new open file for each suprocess? I
>> mean does it occupy an entry in file descriptor table of parent
>> process? If so, wat is each file descriptor connected to?

>
> On Unix, subprocess.Popen will use up a file descriptor in the parent
> for each use of subprocess.PIPE. The descriptor in question is one
> end
> of a pipe; the child process holds the other end.
>
> I guess the situation is similar on Windows, but I don't know for
> sure.
>
> -- [mdw]
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
>
> Add more friends to your messenger and enjoy! Go to http://messenger.yahoo.com/invite/
> --
> http://mail.python.org/mailman/listinfo/python-list


 
Reply With Quote
 
Jeff McNeil
Guest
Posts: n/a
 
      01-20-2009
On Jan 20, 9:19 am, srinivasan srinivas <(E-Mail Removed)>
wrote:
> Do parent process will have different file descriptor in it for each subprocesses or paprent uses a single file descriptor for all?
> I really want to know creation of each subprocess will occupy an entry in parents'file descriptor table. B'cos if i create more than 200 subprocesses, i am getting 'Too many open files' error.
>
> Thanks,
> Srini
>
> ----- Original Message ----
> From: Mark Wooding <(E-Mail Removed)>
> To: (E-Mail Removed)
> Sent: Tuesday, 20 January, 2009 6:16:17 PM
> Subject: Re: Doubts related to subprocess.Popen()
>
> srinivasan srinivas <(E-Mail Removed)> writes:
>
> > Does subprocess.Popen() count a new open file for each suprocess? I
> > mean does it occupy an entry in file descriptor table of parent
> > process? If so, wat is each file descriptor connected to?

>
> On Unix, subprocess.Popen will use up a file descriptor in the parent
> for each use of subprocess.PIPE. The descriptor in question is one end
> of a pipe; the child process holds the other end.
>
> I guess the situation is similar on Windows, but I don't know for sure.
>
> -- [mdw]
> --http://mail.python.org/mailman/listinfo/python-list
>
> Add more friends to your messenger and enjoy! Go tohttp://messenger.yahoo.com/invite/


Have you upped your open files limit? My test script:

import subprocess

procs = []
for i in xrange(400):
procs.append(subprocess.Popen("/bin/cat",
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.PIPE))

By default, 'ulimit -n' returns 1024, which is the number of open
files allowed. Running the test script without changing it results
in:

Traceback (most recent call last):
File "test.py", line 9, in <module>
stderr=subprocess.PIPE))
File "/usr/lib/python2.5/subprocess.py", line 593, in __init__
errread, errwrite)
File "/usr/lib/python2.5/subprocess.py", line 1002, in
_execute_child
errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

Now, up that limit to 8192 via ulimit -n, and run the script again:

[root@marvin jeff]# ulimit -n 8192
[root@marvin jeff]# python test.py
[root@marvin jeff]#

HTH,

Jeff

 
Reply With Quote
 
Mark Wooding
Guest
Posts: n/a
 
      01-20-2009
"Diez B. Roggisch" <(E-Mail Removed)> writes:

> Usually, each new process has three file-descriptors associated with
> it - stdin,stdout and stderr.
>
> So when you span a process, the overall count of FDs should increase
> by three.


Yes, but that's irrelevant. There are two file limits which are
relevant:

* the per-process limit on file descriptors -- basically, the largest
number which can be a file descriptor -- and

* the overall number of files which can be open at a time.

Note that, in Unix, the same file can be referred to by many different
descriptors in many different processes, and fork(2), exec(2) and dup(2)
don't change the number of files open. However, dup(2) allocates a new
descriptor in the calling process, so it may hit the per-process limit.

> Additionally, there are more FDs created if you chose to pipe
> communication between the child-process and the parent.


And these are the ones I mentioned.

OK, in more detail: each pipe(2) call allocates two files (one for each
end) and two file descriptors (one for each file). If you call Popen
with PIPE specified for each of stdin, stdout and stderr, then that's a
total of six descriptors and six files. But the parent will close half
of them after calling fork(2) (freeing three descriptors), and the child
will close all six after dup2(2)-ing them over the descriptors 0, 1, and
2. The net result is:

* six new files in the global file table, and
* three new descriptors in the parent.

(The child ends up with no new descriptors at the end of all this.)

-- [mdw]
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
doubts related to structure somenath C Programming 4 02-16-2007 07:24 PM
How should multiple (related) projects be arranged (structured) and configured so that they can share code, have a related package structure and enable proper unittesting, and ensuring no namespace collisions ToddLMorgan@gmail.com Python 14 04-21-2006 04:03 PM
How should threads be terminated? (related to 'Help with thread related tracebacks') Maxwell Hammer Python 7 06-18-2005 04:20 PM
doubts related to disk files in C Ameya C Programming 3 11-30-2004 04:56 AM
Help, .NET doubts..... MS ASP .Net 1 06-04-2004 01:16 PM



Advertisments