Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Trying to make a spider using mechanize

Reply
Thread Tools

Trying to make a spider using mechanize

 
 
tedpottel@gmail.com
Guest
Posts: n/a
 
      09-08-2008
Hi,

I can read the home page using the mechanize lib. Is there a way to
load in web pages using filename.html instad of servername/
filename.html. Lots of time the links just have the file name. I'm
trying to read in the links name and then vsit those pages.

here is the sample code I am ussing.


import ClientForm
import mechanize


#get home page
request = mechanize.Request("http://www.activetechconsulting.com")
response = mechanize.urlopen(request)
print response.read()

#sub page (this does note work)
request = mechanize.Request("service.html")
response = mechanize.urlopen(request)
print response.read-Ted
 
Reply With Quote
 
 
 
 
James Mills
Guest
Posts: n/a
 
      09-08-2008
Hi,

Perhaps you might want to
try out using a sample spider
I wrote and base your code of
this ?

See: http://hg.shortcircuit.net.au/index....ples/spider.py

cheers
James

On Tue, Sep 9, 2008 at 2:24 AM, <> wrote:
> Hi,
>
> I can read the home page using the mechanize lib. Is there a way to
> load in web pages using filename.html instad of servername/
> filename.html. Lots of time the links just have the file name. I'm
> trying to read in the links name and then vsit those pages.
>
> here is the sample code I am ussing.
>
>
> import ClientForm
> import mechanize
>
>
> #get home page
> request = mechanize.Request("http://www.activetechconsulting.com")
> response = mechanize.urlopen(request)
> print response.read()
>
> #sub page (this does note work)
> request = mechanize.Request("service.html")
> response = mechanize.urlopen(request)
> print response.read-Ted
> --
> http://mail.python.org/mailman/listinfo/python-list
>




--
--
-- "Problems are solved by method"
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
SEO spider-spider ANY website prasadpelluri5@gmail.com Java 0 10-30-2008 07:04 PM
I am trying to install the mechanize lib so I can use python to dowebbrowseing tedpottel@gmail.com Python 1 08-24-2008 04:40 AM
Trying to solve a python/mechanize "error 500" http error bruce Python 0 07-21-2008 08:19 PM
Trying to solve a python/mechanize "error 500" http error bruce Python 0 07-21-2008 08:18 PM
Mechanize: Can't get it to work. Can I help make it work next week? Xeno Campanoli Ruby 1 07-01-2005 10:32 PM



Advertisments
 



1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57