Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > how to handle network failures

Reply
Thread Tools

how to handle network failures

 
 
harryos
Guest
Posts: n/a
 
      10-08-2010
hi
I am trying to write a DataGrabber which reads some data from given
url..I made DataGrabber as a Thread and want to wait for some interval
of time in case there is a network failure that prevents read().
I am not very sure how to implement this

class DataGrabber(threading.Thread):
def __init__(self,url):
threading.Thread.__init__(self)
self.url=url
def run(self):
data=self.get_page_data()
process_data(data)

def get_page_data():
try:
f=urllib.urlopen(self.url)
data=f.read(1024)
except IOError:
#wait for some time and try again
time.sleep(120)
data=self.get_page_data()
return data

Is this the way to implement the part where the thread waits and
reads the data again? Will this handle network failures?Can somebody
please help?

thanks
harry


 
Reply With Quote
 
 
 
 
Diez B. Roggisch
Guest
Posts: n/a
 
      10-09-2010
harryos <(E-Mail Removed)> writes:

> hi
> I am trying to write a DataGrabber which reads some data from given
> url..I made DataGrabber as a Thread and want to wait for some interval
> of time in case there is a network failure that prevents read().
> I am not very sure how to implement this
>
> class DataGrabber(threading.Thread):
> def __init__(self,url):
> threading.Thread.__init__(self)
> self.url=url
> def run(self):
> data=self.get_page_data()
> process_data(data)
>
> def get_page_data():
> try:
> f=urllib.urlopen(self.url)
> data=f.read(1024)
> except IOError:
> #wait for some time and try again
> time.sleep(120)
> data=self.get_page_data()
> return data
>
> Is this the way to implement the part where the thread waits and
> reads the data again? Will this handle network failures?Can somebody
> please help?


This only works if your page is always 1024 bytes long. Which I
doubt. So don't pass the 1024 to read.

Also, you need a loop to re-read the data. Like this:


for n in xrange(max_number_of_retries):
try:
f=urllib.urlopen(self.url)
data = f.read()
break # exist the loop if all
except IOError:
pass


self.process_data(data)


Diez
 
Reply With Quote
 
 
 
 
Lawrence D'Oliveiro
Guest
Posts: n/a
 
      10-10-2010
In message <(E-Mail Removed)>, Diez B. Roggisch wrote:

> for n in xrange(max_number_of_retries):
> try:
> f=urllib.urlopen(self.url)
> data = f.read()
> break # exist the loop if all
> except IOError:
> pass


Is it worth delaying before retrying? In case of a transient routing error,
that kind of thing.
 
Reply With Quote
 
Jorgen Grahn
Guest
Posts: n/a
 
      10-10-2010
On Fri, 2010-10-08, harryos wrote:
> hi
> I am trying to write a DataGrabber which reads some data from given
> url..I made DataGrabber as a Thread and want to wait for some interval
> of time in case there is a network failure that prevents read().
> I am not very sure how to implement this
>
> class DataGrabber(threading.Thread):
> def __init__(self,url):
> threading.Thread.__init__(self)
> self.url=url
> def run(self):
> data=self.get_page_data()
> process_data(data)
>
> def get_page_data():
> try:
> f=urllib.urlopen(self.url)
> data=f.read(1024)
> except IOError:
> #wait for some time and try again
> time.sleep(120)
> data=self.get_page_data()
> return data
>
> Is this the way to implement the part where the thread waits and
> reads the data again? Will this handle network failures?Can somebody
> please help?


You are using TCP sockets. When you get an error on one of those, the
TCP connection is dead (except for a few special cases like EAGAIN,
EINTR).

But you also risk *not* getting told and hanging forever, or anyway
for far longer than your application is likely to want to wait. For
example if the peer host is suddenly disconnected from the network --
TCP will keep trying, in case a connection suddenly reappears much
later.

Try provoking that situation and see what happens.

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .
 
Reply With Quote
 
Aahz
Guest
Posts: n/a
 
      11-07-2010
In article <(E-Mail Removed)>,
harryos <(E-Mail Removed)> wrote:
>
>class DataGrabber(threading.Thread):
> def __init__(self,url):
> threading.Thread.__init__(self)
> self.url=url
> def run(self):
> data=self.get_page_data()
> process_data(data)
>
> def get_page_data():
> try:
> f=urllib.urlopen(self.url)
> data=f.read(1024)
> except IOError:
> #wait for some time and try again
> time.sleep(120)
> data=self.get_page_data()
> return data


Use urllib2 so that you can set a timeout (Python 2.6+).
--
Aahz ((E-Mail Removed)) <*> http://www.pythoncraft.com/

"If you think it's expensive to hire a professional to do the job, wait
until you hire an amateur." --Red Adair
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: How include a large array? Edward A. Falk C Programming 1 04-04-2013 08:07 PM
Package to handle table text render (handle space or tab betweenthe columns) ? =?ISO-8859-1?Q?KLEIN_St=E9phane?= Python 3 10-06-2006 08:46 AM
Possible to handle web requests without an ASPX page? i.e. have DLL handle request. jdlwright@shaw.ca ASP .Net 2 05-31-2005 05:42 PM
how to handle command line output(not terminal handle) Leon Python 2 11-04-2004 05:16 AM
File Handle Reading Blues: Rereading a File Handle for Input Dietrich Perl 1 07-22-2004 10:02 AM



Advertisments