Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Web crawler on python

Reply
Thread Tools

Web crawler on python

 
 
sonich
Guest
Posts: n/a
 
      10-26-2008
I need simple web crawler,
I found Ruya, but it's seems not currently maintained.
Does anybody know good web crawler on python or with python interface?
 
Reply With Quote
 
 
 
 
Mr.SpOOn
Guest
Posts: n/a
 
      10-26-2008
On Sun, Oct 26, 2008 at 9:54 PM, sonich <(E-Mail Removed)> wrote:
> I need simple web crawler,
> I found Ruya, but it's seems not currently maintained.
> Does anybody know good web crawler on python or with python interface?


What about BeautifulSoup?

http://www.crummy.com/software/BeautifulSoup/
 
Reply With Quote
 
 
 
 
James Mills
Guest
Posts: n/a
 
      10-26-2008
On Mon, Oct 27, 2008 at 6:54 AM, sonich <(E-Mail Removed)> wrote:
> I need simple web crawler,
> I found Ruya, but it's seems not currently maintained.
> Does anybody know good web crawler on python or with python interface?


Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.w...ples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

--
--
-- "Problems are solved by method"
 
Reply With Quote
 
Support Desk
Guest
Posts: n/a
 
      10-27-2008


-----Original Message-----
From: James Mills [mailto(E-Mail Removed)]
Sent: Sunday, October 26, 2008 5:26 PM
To: sonich
Cc: http://www.velocityreviews.com/forums/(E-Mail Removed)
Subject: Re: Web crawler on python

On Mon, Oct 27, 2008 at 6:54 AM, sonich <(E-Mail Removed)> wrote:
> I need simple web crawler,
> I found Ruya, but it's seems not currently maintained.
> Does anybody know good web crawler on python or with python interface?


Simple, but it works. Extend it all you like.

http://hg.softcircuit.com.au/index.w...330d047ff663/e
xamples/spider.py

$ spider.py --help
Usage: spider.py [options] <url>

Options:
--version show program's version number and exit
-h, --help show this help message and exit
-q, --quiet Enable quiet mode
-l, --links Get links for specified url only
-d DEPTH, --depth=DEPTH
Maximum depth to traverse

cheers
James

--
--
-- "Problems are solved by method"


 
Reply With Quote
 
Alex
Guest
Posts: n/a
 
      10-28-2008
On Oct 26, 9:54*pm, sonich <(E-Mail Removed)> wrote:
> I need simple web crawler,
> I found Ruya, but it's seems not currently maintained.
> Does anybody know good web crawler on python or with python interface?


You should try Orchid http://pypi.python.org/pypi/Orchid/1.1
or you can have a look at my project on launchpad
https://code.launchpad.net/~esaurito...r/experimental.
It's a single site crawler but you can easily modified it.

Bye.

Alex
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Re: web crawler in python Philip Semanchuk Python 0 12-10-2009 01:24 PM
Web crawler on python yura Python 1 10-30-2008 10:25 PM
Web Crawler - Python or Perl? disappearedng@gmail.com Python 11 06-22-2008 05:47 PM
web crawler in python or C? abhinav Python 13 02-20-2006 09:07 PM
web crawler in python or C? abhinav C Programming 1 02-16-2006 08:33 AM



Advertisments