Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > socket's strange behavior with subprocesses

Reply
Thread Tools

socket's strange behavior with subprocesses

 
 
Jane Austine
Guest
Posts: n/a
 
      11-12-2003
Running Python 2.3 on Win XP

It seems like socket is working interdependently with subprocesses of
the process which created socket.

------------------------------------
#the server side
>>> import socket
>>> s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
>>> s.bind(('localhost',9000))
>>> s.listen(5)
>>> z=s.accept()
>>> import os
>>> f=os.popen('notepad.exe') #notepad appears on the screen
>>> z[0]

<socket._socketobject object at 0x0096C390>
>>> z[0].recv(6)

'foobar'
>>> z[0].send('hello world')

11

#the client side
>>> s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
>>> s.connect(('localhost',9000))
>>> s.send('foobar')

4
>>> print s.recv(1024)

hello world

------------------------------
Now when the client requests to recv 1024 bytes, and since there is no
more to read from the socket it blocks.

#client side
>>> s.recv(1024) #it hangs


and the server side tries to close the socket:

#server side
>>> z[0].close()
>>> #yes, it seems to have worked.


Alas, the client side doesn't wake up! It doesn't wake up unless the
notepad is exited first; only after that, 'Connection reset by peer'
is raised. What does the socket has to do with subprocesses?
 
Reply With Quote
 
 
 
 
Irmen de Jong
Guest
Posts: n/a
 
      11-12-2003
Jane Austine wrote:

> and the server side tries to close the socket:
>
> #server side
>
>>>>z[0].close()
>>>>#yes, it seems to have worked.

>
>
> Alas, the client side doesn't wake up! It doesn't wake up unless the
> notepad is exited first; only after that, 'Connection reset by peer'
> is raised. What does the socket has to do with subprocesses?


Nothing, I guess... try to shutdown the socket explicitly before closing it:


z[0].shutdown(2)
z[0].close()

does that work?

--Irmen

 
Reply With Quote
 
 
 
 
Colin Brown
Guest
Posts: n/a
 
      11-12-2003
"Jane Austine" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed) om...
> Running Python 2.3 on Win XP
>
> It seems like socket is working interdependently with subprocesses of
> the process which created socket.
> and the server side tries to close the socket:

......
> Alas, the client side doesn't wake up! It doesn't wake up unless the
> notepad is exited first; only after that, 'Connection reset by peer'
> is raised. What does the socket has to do with subprocesses?


Hi Jane

I think it may be tied up with the way subprocesses are made under Windows.
Below is a copy of part of a post I made some months back when inexplicable
file-locking was causing me problems. It took me weeks to track down the
reason.

The Python default subprocess inherits open handles from the parent.
Your options are:
1. Open the subprocess before the socket.
2. Create the subprocess using win32 API primitives (this is what I had to
do).
Let me know if want details.

Colin Brown
PyNZ

----------------------------------------------------------------------------
------
I have been struggling to solve why an occasional "Permission Denied" error
popped up from the following code fragment in one of my Win2K program
threads:

....
input_file = preprocess(raw_file)
os.system('third_party input_file output_file > error_file')
if os.path.exists(saved_file):
os.remove(saved_file)
os.rename(input_file,saved_file)
....

The error occurs on the os.rename(). The third_party executable was closing
input_file before terminating and anyway the subshell process has finished
before the os.rename is called! Most baffling.

With the aid of handle.exe from www.sysinternals.com I finally resolved the
problem. To see the problem at first hand try the following script:

import time, thread, os

def rm(file):
print 'delete x.x'
os.remove(file)

def wait():
if os.name == 'nt':
os.system('pause')
elif os.name == 'posix':
os.system('sleep 3')
print 'end wait'

print 'create x.x'
f = open('x.x','w')
thread.start_new_thread(wait,())
time.sleep(1)
print '\nclose x.x'
f.close()
rm('x.x')

Although this works fine on Linux, I get a Permission Denied error on
Windows. I surmise that an os.system call uses a C fork to create the
subshell - an exact duplicate of the current process environment including
OPEN FILE HANDLES. Because the os.system call has not returned before the
os.remove is called a "Permission Denied" error occurs. Quite simple really.

Now going back to my original problem, I had other threads doing os.system
calls. When one of these calls happens during the preprocess file write of
my above thread AND takes longer to complete than the os.system call in the
above thread then the file will still be open and the os.rename will return
the Permission Denied error.



 
Reply With Quote
 
Jane Austine
Guest
Posts: n/a
 
      11-13-2003
"Colin Brown" <(E-Mail Removed)> wrote in message news:<3fb28c56$(E-Mail Removed)>...
> "Jane Austine" <(E-Mail Removed)> wrote in message
> news:(E-Mail Removed) om...
> > Running Python 2.3 on Win XP
> >
> > It seems like socket is working interdependently with subprocesses of
> > the process which created socket.
> > and the server side tries to close the socket:

> .....
> > Alas, the client side doesn't wake up! It doesn't wake up unless the
> > notepad is exited first; only after that, 'Connection reset by peer'
> > is raised. What does the socket has to do with subprocesses?

>
> Hi Jane
>
> I think it may be tied up with the way subprocesses are made under Windows.
> Below is a copy of part of a post I made some months back when inexplicable
> file-locking was causing me problems. It took me weeks to track down the
> reason.
>
> The Python default subprocess inherits open handles from the parent.
> Your options are:
> 1. Open the subprocess before the socket.
> 2. Create the subprocess using win32 API primitives (this is what I had to
> do).
> Let me know if want details.
>
> Colin Brown
> PyNZ
>


Thank you very much, first of all.

I tried win32 API primitives for creating subprocesses: win32all's
CreateProcess. I used the Process class in winprocess.py in the
"demos" directory. However, it didn't work with sockets perfectly.

Say, a socket server python script is launched via CreateProcess. It
then launches sub-processes. And I kill(via TerminateProcess) the
socket server process. The subprocesses remain alive(as expected). I
try to connect to the 'dead server port'. I expect almost immediate
"connect refused" error, but it doesn't come up until the subprocesses
are all gone and client hangs forever.

Jane
 
Reply With Quote
 
Colin Brown
Guest
Posts: n/a
 
      11-13-2003

"Jane Austine" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed) om...
> "Colin Brown" <(E-Mail Removed)> wrote in message

news:<3fb28c56$(E-Mail Removed)>...
> > "Jane Austine" <(E-Mail Removed)> wrote in message
> > news:(E-Mail Removed) om...

....
> I tried win32 API primitives for creating subprocesses: win32all's
> CreateProcess. I used the Process class in winprocess.py in the
> "demos" directory. However, it didn't work with sockets perfectly.
>
> Say, a socket server python script is launched via CreateProcess. It
> then launches sub-processes. And I kill(via TerminateProcess) the
> socket server process. The subprocesses remain alive(as expected). I
> try to connect to the 'dead server port'. I expect almost immediate
> "connect refused" error, but it doesn't come up until the subprocesses
> are all gone and client hangs forever.
>
> Jane


If I am interpreting what you are saying here correctly you have:

Main_process
=> [create_process]
=> Sub_process1
socket_server_connection
subprocess2 (of subprocess1) started

Sub_process1 terminated, but socket connection held until subprocess2
terminated.

This is what I would expect based on my findings. Subprocess2 has inherited
the socket handle when it was created (assuming you used os.system,
os.spawn* or os.popen*). You would have to use the correct incantation of
create_process to launch subprocess2.

I have attached the code I used in place of os.system.

Colin

--[Win32.py]---------------------------------------------------------------
# perform equivalent of os.system without open file handles

import win32process,win32event

def system(cmd):
handles = win32process.CreateProcess \
(None,cmd,None,None,0,0,None,None,win32process.STA RTUPINFO())
status = win32event.WaitForSingleObject(handles[0],win32event.INFINITE)
if status == win32event.WAIT_ABANDONED:
raise 'win32.system WAIT_ABANDONED'
elif status == win32event.WAIT_FAILED:
raise 'win32.system WAIT_FAILED'
elif status == win32event.WAIT_IO_COMPLETION:
raise 'win32.system WAIT_IO_COMPLETION'
elif status == win32event.WAIT_OBJECT_0:
pass
elif status == win32event.WAIT_TIMEOUT:
raise 'win32.system WAIT_TIMEOUT'
else:
raise 'win32.system - unknown event status = '+str(status)



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Reading a subprocesses stdout on Windows brolewis Python 3 11-21-2005 08:50 PM
page faults when spawning subprocesses Dave Kirby Python 2 11-09-2005 12:17 PM
managing multiple subprocesses Marcos Python 4 02-07-2005 08:04 PM
managing multiple subprocesses Marcos Python 0 02-03-2005 01:00 AM
IDLE Problems - Subprocesses Under Windows Fuzzyman Python 0 12-22-2003 10:14 AM



Advertisments