Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > http pipelining

Reply
Thread Tools

http pipelining

 
 
swq22@yahoo.com
Guest
Posts: n/a
 
      04-26-2007
Which python module is capable of pipelining http requests?

(I know httplib can send mulitple requests per tcp connection, but in
a strictly serial way. )


 
Reply With Quote
 
 
 
 
Steve Holden
Guest
Posts: n/a
 
      04-27-2007
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Which python module is capable of pipelining http requests?
>
> (I know httplib can send mulitple requests per tcp connection, but in
> a strictly serial way. )
>
>

There's nothing in the standard library, I believe, that includes both
client and server functionality in the same module. So you would need to
glue them together.

If you want a simple net proxy server I seem to remember the chameleon
system allows you to write one in about twelve lines. If it has to be
HTTP-specific, with header parsing and the like, you might want to think
about Twisted, which supports both client and server functionality and
tries to make it easy to plumb things together in pipelines.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

 
Reply With Quote
 
 
 
 
Steve Holden
Guest
Posts: n/a
 
      04-27-2007
(E-Mail Removed) wrote:
> Which python module is capable of pipelining http requests?
>
> (I know httplib can send mulitple requests per tcp connection, but in
> a strictly serial way. )
>

Oops, sorry, you meant sending requests in parallel, right?

You'll need to use either urllib or urllib2 for the web, and the
threading module is one way to run parallel requests. It's fairly easy
to use as long as you keep your tasks properly isolated form each other.

regards
Steve
--
Steve Holden +1 571 484 6266 +1 800 494 3119
Holden Web LLC/Ltd http://www.holdenweb.com
Skype: holdenweb http://del.icio.us/steve.holden
Recent Ramblings http://holdenweb.blogspot.com

 
Reply With Quote
 
swq22@yahoo.com
Guest
Posts: n/a
 
      04-27-2007
On Fri, 27 Apr 2007 13:50:21 -0400, Steve Holden <(E-Mail Removed)>
wrote:

>(E-Mail Removed) wrote:
>> Which python module is capable of pipelining http requests?
>>
>> (I know httplib can send mulitple requests per tcp connection, but in
>> a strictly serial way. )
>>
>>

>There's nothing in the standard library, I believe, that includes both
>client and server functionality in the same module. So you would need to
>glue them together.
>
>If you want a simple net proxy server I seem to remember the chameleon
>system allows you to write one in about twelve lines. If it has to be
>HTTP-specific, with header parsing and the like, you might want to think
>about Twisted, which supports both client and server functionality and
>tries to make it easy to plumb things together in pipelines.


Sorry for my confused question.

What I'm looking for is to emulate what a modern HTTP 1.1 browser like
Firefox does(when network.http.pipelining is enabled)

1.Open TCP connetion.

send multiple requests without waiting for a reply:

2.GET /url1
3.GET /url2
4.GET /url3
5.read 1st reply
6.read 2st reply
7.read 3st reply

8.Close TCP connection.

I'll be using Apache or IIS as my web server.


 
Reply With Quote
 
John J. Lee
Guest
Posts: n/a
 
      04-29-2007
Steve Holden <(E-Mail Removed)> writes:

> (E-Mail Removed) wrote:
> > Which python module is capable of pipelining http requests?
> > (I know httplib can send mulitple requests per tcp connection, but in
> > a strictly serial way. )
> >

> Oops, sorry, you meant sending requests in parallel, right?
>
> You'll need to use either urllib or urllib2 for the web, and the
> threading module is one way to run parallel requests. It's fairly easy
> to use as long as you keep your tasks properly isolated form each
> other.


No, he means "HTTP pipelining", which means sending multiple requests
down a single TCP connection, without waiting for the first response.

httplib's module-level docstring says (reformatted here):

"""
.... The HTTPResponse class does not enforce this state machine, which
implies sophisticated clients may accelerate the request/response
pipeline. Caution should be taken, though: accelerating the states
beyond the above pattern may imply knowledge of the server's
connection-close behavior for certain requests. For example, it is
impossible to tell whether the server will close the connection UNTIL
the response headers have been read; this means that further requests
cannot be placed into the pipeline until it is known that the server
will NOT be closing the connection.
"""

So, sort-of-yes, if you know what you're doing and you're lucky.

Certainly urllib and urllib2 don't support pipelining. There were
plans for a new HTTP client in Twisted with pipelining support, but I
don't know if that ever came about. AFAIK not many libraries (in any
language) support it -- e.g. none of "Jakarta commons HTTPClient",
libwww-perl, and libcurl currently support it. libwww (without the
"-perl") does claim to support it (I say "claim" merely because I
haven't used it or read the source -- no FUD intended).


Side note: As the OP mentions in a followup, by default firefox does
NOT do pipelining (to the disbelief of the people I told about this
when it came up in a previous job -- but I just tried running tcpdump
and indeed about:config seems to be telling the truth; fortunately, in
response to the limitations imposed by RFC 2616, somebody has
thoughtfully arranged for the speed of light to be fast enough for
this not to be a major problem when using websites on the other side
of the planet). The firefox people say:

http://www.mozilla.org/support/firef...oth_pipelining

"""
Pipelining is an experimental feature, designed to improve page-load
performance, that is unfortunately not well supported by some web
servers and proxies.
"""

Instead of pipelining, it uses multiple connections (2 when I tried
it, which is the maximum RFC 2616 says SHOULD be used by a
"single-user client"). I didn't try IE, but apparently it has the
same behaviour (2 connections, no pipelining):

http://blogs.msdn.com/ie/archive/2005/04/11/407189.aspx


I wonder if the right economic pressures are there for SCTP ever to
get used for everyday web HTTP stuff...


John
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Ruby http pipelining ? Googy Ruby 0 05-23-2008 02:55 PM
Question about http pipelining panatha Java 2 08-17-2007 08:13 PM
http pipelining Crouchez Java 4 08-10-2007 07:32 PM
enable pipelining on Firefox D Cheung Firefox 1 02-07-2005 12:42 PM
HTTP 1.1 pipelining noviceUser Python 3 05-18-2004 10:59 AM



Advertisments