Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > view http info when browsing

Reply
Thread Tools

view http info when browsing

 
 
ben
Guest
Posts: n/a
 
      09-06-2004
I'm trying to write a web client script in python to log onto a web
page and pull some information off of it. The page has quite a few
behind the scenes http things going on that are making it difficult to
write the python script. I think if I could see the raw http data
that comes to my browser life would be much easier. Does anybody know
of programs that would run alongside my browser and show the http data
that is send back and forth?

I've looked at plugins like httpwatch and iewatch. httpwatch looks
good but it costs quite a bit of money, and the trial is not fully
functional. iewatch has a free trial, but they don't show cookie
info, which is important.

Does anybody know a good way of view the raw http?

Thanks for your help,

Ben
 
Reply With Quote
 
 
 
 
Peter van Kampen
Guest
Posts: n/a
 
      09-06-2004
In article <(E-Mail Removed) >, ben wrote:
> I'm trying to write a web client script in python to log onto a web
> page and pull some information off of it. The page has quite a few
> behind the scenes http things going on that are making it difficult to
> write the python script. I think if I could see the raw http data
> that comes to my browser life would be much easier. Does anybody know
> of programs that would run alongside my browser and show the http data
> that is send back and forth?
>
> I've looked at plugins like httpwatch and iewatch. httpwatch looks
> good but it costs quite a bit of money, and the trial is not fully
> functional. iewatch has a free trial, but they don't show cookie
> info, which is important.


from urllib import urlopen
r = urlopen("http://www.python.org/")
print r.headers

You can also take a look at mozilla firefox and download the excellent
webdeveloper extension. It also allows you to view headers and lots of other
stuff.

Hth,

PterK


--
Peter van Kampen
pterk -- at -- datatailors.com
 
Reply With Quote
 
 
 
 
Paul McGuire
Guest
Posts: n/a
 
      09-06-2004
"ben" <(E-Mail Removed)> wrote in message
news:(E-Mail Removed) om...
> I'm trying to write a web client script in python to log onto a web
> page and pull some information off of it. The page has quite a few
> behind the scenes http things going on that are making it difficult to
> write the python script. I think if I could see the raw http data
> that comes to my browser life would be much easier. Does anybody know
> of programs that would run alongside my browser and show the http data
> that is send back and forth?
>
> I've looked at plugins like httpwatch and iewatch. httpwatch looks
> good but it costs quite a bit of money, and the trial is not fully
> functional. iewatch has a free trial, but they don't show cookie
> info, which is important.
>
> Does anybody know a good way of view the raw http?
>
> Thanks for your help,
>
> Ben


Try HTTPLook at www.httpsniffer.com . We've used it some on our current
project, to monitor SOAP message contents. I don't know about cookie
viewing, but it's free for 15-day (full featured) trial, less than $30 if
you want to keep it.

-- Paul


 
Reply With Quote
 
Paul Rubin
Guest
Posts: n/a
 
      09-06-2004
http://www.velocityreviews.com/forums/(E-Mail Removed) (ben) writes:
> Does anybody know a good way of view the raw http?


I always just use a proxy server for that. www.junkbuster.org has a
free and convenient one that has options for logging the http stream.
 
Reply With Quote
 
Michael Foord
Guest
Posts: n/a
 
      09-06-2004
(E-Mail Removed) (ben) wrote in message news:<(E-Mail Removed). com>...
> I'm trying to write a web client script in python to log onto a web
> page and pull some information off of it. The page has quite a few
> behind the scenes http things going on that are making it difficult to
> write the python script. I think if I could see the raw http data
> that comes to my browser life would be much easier. Does anybody know
> of programs that would run alongside my browser and show the http data
> that is send back and forth?
>
> I've looked at plugins like httpwatch and iewatch. httpwatch looks
> good but it costs quite a bit of money, and the trial is not fully
> functional. iewatch has a free trial, but they don't show cookie
> info, which is important.
>
> Does anybody know a good way of view the raw http?
>
> Thanks for your help,
>
> Ben



TCPwatch should do what you want. It's a proxy serverso you'll have to
point your browser at that - it shows headers and page content
etc.....

If you're handling cookies then ClientCookie (or cookielib) could well
do what you want without you having to worry about it (if you're using
python why do you need a browser - why can't you just print the
headers ?)

Regards,

Fuzzy

http://www.voidspace.org.uk/atlantib...thonutils.html
 
Reply With Quote
 
Lad
Guest
Posts: n/a
 
      09-06-2004
(E-Mail Removed) (ben) wrote in message news:<(E-Mail Removed). com>...
> I'm trying to write a web client script in python to log onto a web
> page and pull some information off of it. The page has quite a few
> behind the scenes http things going on that are making it difficult to
> write the python script. I think if I could see the raw http data
> that comes to my browser life would be much easier. Does anybody know
> of programs that would run alongside my browser and show the http data
> that is send back and forth?
>
> I've looked at plugins like httpwatch and iewatch. httpwatch looks
> good but it costs quite a bit of money, and the trial is not fully
> functional. iewatch has a free trial, but they don't show cookie
> info, which is important.
>
> Does anybody know a good way of view the raw http?
>
> Thanks for your help,
>
> Ben


I downloaded a "sniffer" in Python. Let me know if you are interested
 
Reply With Quote
 
[ EuGeNe ]
Guest
Posts: n/a
 
      09-06-2004
give www.debugbar.com a bash

the trial version is fully functional and it is cheap
 
Reply With Quote
 
John J. Lee
Guest
Posts: n/a
 
      09-07-2004
(E-Mail Removed) (ben) writes:

> I'm trying to write a web client script in python to log onto a web
> page and pull some information off of it. The page has quite a few
> behind the scenes http things going on that are making it difficult to
> write the python script. I think if I could see the raw http data
> that comes to my browser life would be much easier. Does anybody know
> of programs that would run alongside my browser and show the http data
> that is send back and forth?
>
> I've looked at plugins like httpwatch and iewatch. httpwatch looks
> good but it costs quite a bit of money, and the trial is not fully
> functional. iewatch has a free trial, but they don't show cookie
> info, which is important.
>
> Does anybody know a good way of view the raw http?

[...]

from http://wwwsearch.sourceforge.net/bits/GeneralFAQ.html :

| I want to see what my web browser is doing, but standard network
| sniffers like ethereal or netcat (nc) don't work for HTTPS. How do I
| sniff HTTPS traffic?
|
| Three good options:
|
| Mozilla plugin: livehttpheaders.
| ieHTTPHeaders does the same for MSIE.
| Use lynx -trace, and filter out the junk with a script.
|
| I'm told you can also use a proxy like proxomitron (never tried it
| myself). There's also a commercial MSIE plugin.

URLs:

http://www.ethereal.com/

http://livehttpheaders.mozdev.org/

http://www.blunck.info/iehttpheaders.html

http://lynx.browser.org/

http://www.proxomitron.info/

http://www.simtec.ltd.uk/


There are plugins that show what session & permanent cookies your
browser has stored, too. Both Mozilla and Konqueror (& Safari and
Opera too, I guess?) do this as standard, but for IE:

http://www.codeproject.com/shell/coo...0&select=21591


John
 
Reply With Quote
 
Michael
Guest
Posts: n/a
 
      09-09-2004
Ben,

I am using iewatch (iewatch.com) quite a bit and it works great in
showing me the cookie information, both for request cookies and
response cookies (look for the Cookie and Set-Cookie headers). There
is a fully functional 21 day trial version available and it only costs
a fraction of what httpwatch is charging.

Michael

(E-Mail Removed) (ben) wrote in message news:<(E-Mail Removed). com>...
> I'm trying to write a web client script in python to log onto a web
> page and pull some information off of it. The page has quite a few
> behind the scenes http things going on that are making it difficult to
> write the python script. I think if I could see the raw http data
> that comes to my browser life would be much easier. Does anybody know
> of programs that would run alongside my browser and show the http data
> that is send back and forth?
>
> I've looked at plugins like httpwatch and iewatch. httpwatch looks
> good but it costs quite a bit of money, and the trial is not fully
> functional. iewatch has a free trial, but they don't show cookie
> info, which is important.
>
> Does anybody know a good way of view the raw http?
>
> Thanks for your help,
>
> Ben

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
HTTP SOAP/HTTP GET/HTTP POST milan_9211 Software 0 01-10-2011 02:10 PM
Persisting authentication info when browsing to a subdomain Greg ASP .Net 0 06-25-2007 09:28 AM
What info is stored on your hard drive when you view info from a memory stick? JamesAddison2@aol.com Computer Support 0 02-19-2007 12:04 PM
How to make a week view and day view calendar just like month view calendar in .NET ? Parthiv Joshi ASP .Net Web Controls 1 07-06-2004 03:15 PM
error when browsing tomcat without http William Java 2 09-12-2003 01:20 PM



Advertisments