«

I am trying to develop a special putpose crawler using htmllib & urllib.
How do you tell the server application that you are a modern browser and can
handle frames?

Thanks,

john Bradbury

John Bradbury:
>I am trying to develop a special putpose crawler using htmllib & urllib.
>How do you tell the server application that you are a modern browser and can
>handle frames?

I don't know of any "I can handle frames" header and I don't see why the
server would care, but you could mimic the User-agent header sent by a
modern browser.

--
René Pijlman

I don't know what is causing the problem, but the site I am accessing is
sending out forms for a browser that has a low resolution and does not
support frames. Excuse my ignorance, but where do you set up the User-agent
header you suggested.

Many thanks for your prompt reply.

John Bradbury

"Rene Pijlman" <> wrote in
message news:...
> John Bradbury:
> >I am trying to develop a special putpose crawler using htmllib & urllib.
> >How do you tell the server application that you are a modern browser and
can
> >handle frames?
>
> I don't know of any "I can handle frames" header and I don't see why the
> server would care, but you could mimic the User-agent header sent by a
> modern browser.
>
> --
> René Pijlman

John Bradbury:
>where do you set up the User-agent header you suggested.

Its an HTTP header:
http://www.ietf.org/rfc/rfc2616.txt

See also:
http://www.google.com/search?q=urlli...ser%2Dagent%22

which leads you to... tada... the documentation!
http://www.python.org/doc/current/li...le-urllib.html

--
René Pijlman

"John Bradbury" <john_bradbury@___cableinet.co.uk> writes:

> "Rene Pijlman" <> wrote in
> message news:...
> > John Bradbury:
> > >I am trying to develop a special putpose crawler using htmllib & urllib.
> > >How do you tell the server application that you are a modern browser
> > >and can handle frames?
[...]
> > server would care, but you could mimic the User-agent header sent by a
[...]
> I don't know what is causing the problem, but the site I am accessing is
> sending out forms for a browser that has a low resolution and does not
> support frames. Excuse my ignorance, but where do you set up the
> User-agent header you suggested.

For urllib2 (well, almost):

http://wwwsearch.sourceforge.net/Cli...c.html#headers


John