Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Best way to spawn process on back end computer

Reply
Thread Tools

Best way to spawn process on back end computer

 
 
sophie_newbie
Guest
Posts: n/a
 
      10-16-2008
Hi,

I'm running a python cgi script on a frontend web server and I want it
to spawn another script (that takes a long time to run) on a backend
number crunching server thats connected to the same network. What do
you think is the best way to do this? I have a few ideas but I'm sure
there is a "best" way to go about this.

Thanks.
 
Reply With Quote
 
 
 
 
Simon Brunning
Guest
Posts: n/a
 
      10-16-2008
2008/10/16 sophie_newbie <(E-Mail Removed)>:

> I'm running a python cgi script on a frontend web server and I want it
> to spawn another script (that takes a long time to run) on a backend
> number crunching server thats connected to the same network. What do
> you think is the best way to do this? I have a few ideas but I'm sure
> there is a "best" way to go about this.


http://edit.kamaelia.org/ might be qworth a look.

--
Cheers,
Simon B.
http://www.velocityreviews.com/forums/(E-Mail Removed)
 
Reply With Quote
 
 
 
 
Robin Becker
Guest
Posts: n/a
 
      10-16-2008
sophie_newbie wrote:
> Hi,
>
> I'm running a python cgi script on a frontend web server and I want it
> to spawn another script (that takes a long time to run) on a backend
> number crunching server thats connected to the same network. What do
> you think is the best way to do this? I have a few ideas but I'm sure
> there is a "best" way to go about this.



The main problem here is that you'll probably need to detach the job to allow
the current cgi request to return a response to the client.

The implication of that is that the job either has to be anonymous and requires
no further attention or you need to provide some means of making the job
responsive to requests about its status so that a periodic request can be made
by the web page. That implies that the job can be identified and the creation
reponse returns the identity. One of the major problems is that the normal www
user has few privileges and cannot normally write to disk.

I have done this using both external shell scripts to do the main processing and
detaching and or python scripts that know how to detach. It was not terribly
easy or obvious.

Another alternative, as Simon's Kamaelia might indicate, is that you might
consider running a job server to service the cgi script requests on the remote
host. I have also done this as part of a web application. One of the advantages
was that the jobserver can run as any user and thus gets access to whatever the
owner has; additionally by providing a suitable protocol eg XMLRPC you can test
the jobserver without going through the web.
--
Robin Becker

 
Reply With Quote
 
Paul Boddie
Guest
Posts: n/a
 
      10-16-2008
On 16 Okt, 15:51, Robin Becker <(E-Mail Removed)> wrote:
> sophie_newbie wrote:
> > I'm running a python cgi script on a frontend web server and I want it
> > to spawn another script (that takes a long time to run) on a backend
> > number crunching server thats connected to the same network. What do
> > you think is the best way to do this? I have a few ideas but I'm sure
> > there is a "best" way to go about this.

>
> The main problem here is that you'll probably need to detach the job to allow
> the current cgi request to return a response to the client.


I've added support for background processes to the pprocess library;
this attempts to address the problems around detaching from worker
processes and re-attaching to them later in order to collect the
results:

http://www.boddie.org.uk/python/ppro...groundCallable

Arguably, this is more complicated than the most basic approach, which
would involve having separate, spawned processes just writing to files
whose contents would then be passed back to the user or processed in
the CGI script, but it's the notification that's the most difficult
part, not the data transfer: efficiently getting a notification event,
rather than polling stuff frequently, is the main problem.

> The implication of that is that the job either has to be anonymous and requires
> no further attention or you need to provide some means of making the job
> responsive to requests about its status so that a periodic request can be made
> by the web page. That implies that the job can be identified and the creation
> reponse returns the identity. One of the major problems is that the normal www
> user has few privileges and cannot normally write to disk.


I've used UNIX sockets as the means of communication between creating/
collecting processes (the CGI script in this case) and the created/
worker processes. Someone suggested an alternative method of binding
to kernel-managed namespaces, if I recall the nature of the suggestion
correctly, but I haven't looked into this yet.

More details here:

http://www.boddie.org.uk/python/pprocess.html

And for the impatient, a repository is here:

https://hg.boddie.org.uk/pprocess

Paul
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Spawn a process, then exit, whilst leaving process running? Victor Hooi Python 1 02-10-2013 03:50 AM
Logging stdout/stderr/stdin of an spawn process (Open4::spawn) Edgardo Hames Ruby 1 05-06-2008 08:17 PM
Logging stdout/stderr/stdin of an spawn process (Open4::spawn) Ed Hames Ruby 0 04-16-2008 04:21 PM
spawn syntax + os.P_WAIT mode behavior + spawn stdout redirection Derek Basch Python 2 01-21-2005 05:37 AM
Architecture question: should I use a web service as back end to my web front end? Karen Sundquist ASP .Net Web Services 1 12-01-2003 06:28 AM



Advertisments