Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Advice on long running processes

Reply
Thread Tools

Advice on long running processes

 
 
commander_coder@hotmail.com
Guest
Posts: n/a
 
      10-11-2007
Hello,

I write a lot of CGI scripts, in Python of course. Now I need to
convert some to long-running processes. I'm having trouble finding
resources about the best practices to do that.

I've found a lot of email discussions that say something like, "You
need to educate yourself about the differences when you have long-
running processes" but I've not had a lot of luck with finding things
that explain the differences. I've seen some talk about database
timeouts, for instance, but I'm not sure I understand the problems.
Can anyone here suggest some resources? I'd be happy with web sites,
with buying a book, anything.

I ask here because I write in Python and so if those resources used
Python then that would be super.

Thanks,
Jim

 
Reply With Quote
 
 
 
 
Roy Smith
Guest
Posts: n/a
 
      10-11-2007
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> Hello,
>
> I write a lot of CGI scripts, in Python of course. Now I need to
> convert some to long-running processes. I'm having trouble finding
> resources about the best practices to do that.
>
> I've found a lot of email discussions that say something like, "You
> need to educate yourself about the differences when you have long-
> running processes" but I've not had a lot of luck with finding things
> that explain the differences.


The biggest differences between run-and-exit vs. long running processes are
resource management and error recovery. Let's take them one at a time.

Resource management. In a short-lived process, you really don't have to
worry about this at all. Snarf as much memory as you need, open as many
files as you want, and when you exit, the operating system cleans it all up
for you. With a long running process, you have to worry about stuff like
that.

In Python, you're isolate from the low-level details of memory management,
but still need to think about it a bit. Imagine you had code that looked
like this in your main loop:

for request in getNextRequest():
requestList.append (request)
processRequest(request)

requestList is going to keep growing without bounds and eventually will eat
up all available memory in the system and your process will crash.
Everything you store, you also need to delete when you're done with it.

Same with files. In a short-lived process, you can generally open as many
files as you want and never worry about closing them. It unlikely you will
ever run out of file descriptors. In a long running process, that's not
the case. If you open a new file each time you get a request and never
close it, after a few hundred requests (depending on the operating system,
maybe even a few thousand), you'll run out of file descriptors.

The other big thing is error recovery. In a short lived process, if
something fails, you print an error message and exit. In a long running
process, you need to somehow recover from the error and keep going as best
you can. This can be tricky.
 
Reply With Quote
 
 
 
 
Bruno Barberi Gnecco
Guest
Posts: n/a
 
      10-11-2007
Roy Smith wrote:
> (E-Mail Removed) wrote:
>
>
>>Hello,
>>
>>I write a lot of CGI scripts, in Python of course. Now I need to
>>convert some to long-running processes. I'm having trouble finding
>>resources about the best practices to do that.
>>
>>I've found a lot of email discussions that say something like, "You
>>need to educate yourself about the differences when you have long-
>>running processes" but I've not had a lot of luck with finding things
>>that explain the differences.

>
>
> The biggest differences between run-and-exit vs. long running processes are
> resource management and error recovery. Let's take them one at a time.
>
> Resource management. In a short-lived process, you really don't have to
> worry about this at all. Snarf as much memory as you need, open as many
> files as you want, and when you exit, the operating system cleans it all up
> for you. With a long running process, you have to worry about stuff like
> that.
>
> In Python, you're isolate from the low-level details of memory management,
> but still need to think about it a bit. Imagine you had code that looked
> like this in your main loop:
>
> for request in getNextRequest():
> requestList.append (request)
> processRequest(request)
>
> requestList is going to keep growing without bounds and eventually will eat
> up all available memory in the system and your process will crash.
> Everything you store, you also need to delete when you're done with it.


In particular, it is a good idea to call gc.collect() every now
and then, specially if you are in such a loop. I don't know what is the
gc policy in python, but an application of mine that seemed to eat as much
memory as it was available was reduced to a constant small amount of
memory after I started to call the gc directly.

> The other big thing is error recovery. In a short lived process, if
> something fails, you print an error message and exit. In a long running
> process, you need to somehow recover from the error and keep going as best
> you can. This can be tricky.


You should have your main loop inside a try/except, to catch any
exceptions that were not otherwise caught without exiting the application.
Log the exceptions, of course, but in most long running applications
work in a loop like Roy Smith's code above, so if one of them fails it
won't disrupt the others to come.

--
Bruno Barberi Gnecco <brunobg_at_users.sourceforge.net>
The earth is like a tiny grain of sand, only much, much heavier.
 
Reply With Quote
 
Bruno Desthuilliers
Guest
Posts: n/a
 
      10-11-2007
(E-Mail Removed) a écrit :
> Hello,
>
> I write a lot of CGI scripts, in Python of course. Now I need to
> convert some to long-running processes. I'm having trouble finding
> resources about the best practices to do that.
>
> I've found a lot of email discussions that say something like, "You
> need to educate yourself about the differences when you have long-
> running processes" but I've not had a lot of luck with finding things
> that explain the differences. I've seen some talk about database
> timeouts, for instance, but I'm not sure I understand the problems.
> Can anyone here suggest some resources? I'd be happy with web sites,
> with buying a book, anything.
>
> I ask here because I write in Python and so if those resources used
> Python then that would be super.


As far as I'm concerned, I'd go for one of the available wsgi frameworks.
 
Reply With Quote
 
commander_coder@hotmail.com
Guest
Posts: n/a
 
      10-12-2007
Thank you to folks for the replies.

Jim

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Best pratices for long(ish) running processes BillAtWork ASP .Net 1 10-12-2009 07:21 PM
Controlling processes and what to "feed" other processes Marc Heiler Ruby 1 05-24-2009 05:37 PM
Having compilation error: no match for call to ‘(const __gnu_cxx::hash<long long int>) (const long long int&)’ veryhotsausage C++ 1 07-04-2008 05:41 PM
What's with "long running processes" ? Bruno Desthuilliers Python 1 09-18-2007 05:44 PM
How do I: Main thread spawn child threads, which child processes...control those child processes? Jeff Rodriguez C Programming 23 12-09-2003 11:06 PM



Advertisments