Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > urllib2 and transfer-encoding = chunked

Reply
Thread Tools

urllib2 and transfer-encoding = chunked

 
 
jdvolz@gmail.com
Guest
Posts: n/a
 
      01-18-2007
I am having errors which appear to be linked to a previous bug in
urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
Has anyone established a standard workaround? I keep finding old
posts about it, that basically give up and say "well it's a known bug."
Any help would be greatly appreciated.

 
Reply With Quote
 
 
 
 
Gabriel Genellina
Guest
Posts: n/a
 
      01-18-2007
<(E-Mail Removed)> escribió en el mensaje
news:(E-Mail Removed) oups.com...
>I am having errors which appear to be linked to a previous bug in
> urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
> Has anyone established a standard workaround? I keep finding old
> posts about it, that basically give up and say "well it's a known bug."
> Any help would be greatly appreciated.


Perhaps if you said what the supposed error is...

--
Gabriel Genellina


 
Reply With Quote
 
 
 
 
jdvolz@gmail.com
Guest
Posts: n/a
 
      01-18-2007
Haha! My mistake.

The error is that when a web server is chunking a web page only the
first chunk appears to be acquired by the urllib2.urlopen call. If you
check the headers, there is no 'Content-length' (as expected) and
instead there is 'transfer-encoding' = 'chunked'. I am getting about
the first 30Kb, and then nothing else.

I don't get a ValueError like described at the following post:

http://groups.google.com/group/comp....3ea9ca84b28669

Here's the code that I think is failing, it's basically textbook Python
for accessing a url:

file = urllib2.urlopen(url)
contenttype = file.info().type
if contenttype and contenttype.find('text') > -1:
return file.read()
#
# I am checking the content type because I don't want to download
..jpegs and the like
#

I have typed similar commands into the interpreter, which also produces
only about the first 30KB of the url.

Sorry for the confusion.

Gabriel Genellina wrote:
> <(E-Mail Removed)> escribió en el mensaje
> news:(E-Mail Removed) oups.com...
> >I am having errors which appear to be linked to a previous bug in
> > urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
> > Has anyone established a standard workaround? I keep finding old
> > posts about it, that basically give up and say "well it's a known bug."
> > Any help would be greatly appreciated.

>
> Perhaps if you said what the supposed error is...
>
> --
> Gabriel Genellina


 
Reply With Quote
 
Nikita the Spider
Guest
Posts: n/a
 
      01-20-2007
In article <(E-Mail Removed). com>,
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:

> Haha! My mistake.
>
> The error is that when a web server is chunking a web page only the
> first chunk appears to be acquired by the urllib2.urlopen call. If you
> check the headers, there is no 'Content-length' (as expected) and
> instead there is 'transfer-encoding' = 'chunked'. I am getting about
> the first 30Kb, and then nothing else.
>
> I don't get a ValueError like described at the following post:


Hi jdvolz,
What error *do* you get? Or is it that no error is raised; you're just
not getting all of the data? If it is the latter, then the sending
server might be at fault for not properly following the chunked transfer
protocol. One way to find out would be to fire up Ethereal and see
what's coming down the wire.

> I am having errors which appear to be linked to a previous bug in
> urllib2 (and urllib) for v2.4 and v2.5 of Python. Has this been fixed?
> Has anyone established a standard workaround? I keep finding old
> posts about it, that basically give up and say "well it's a known bug."


Can you give us some pointers to some of these old posts? And tell us
what version of Python you're using.

--
Philip
http://NikitaTheSpider.com/
Whole-site HTML validation, link checking and more
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Chunked GZIP processing using Java Sockets aztechnology@gmail.com Java 10 03-29-2006 07:26 AM
Problem with: urllib2.build_opener(urllib2.HTTPCookieProcessor(cj)) Josef Cihal Python 0 09-05-2005 11:26 AM
HttpURLConnection receiving chunked encoded responses d_cymbal@hotmail.com Java 0 04-13-2005 09:02 PM
transfer encoding: chunked CW ASP .Net 0 02-15-2005 09:41 AM
SOAP and Chunked Transfer Eric ASP .Net Web Services 3 07-28-2004 01:14 AM



Advertisments