Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Ruby > Web spider

Reply
Thread Tools

Web spider

 
 
Lucas Carlson
Guest
Posts: n/a
 
      05-04-2005
Does anyone know of a web spider built in Ruby?

 
Reply With Quote
 
 
 
 
leon breedt
Guest
Posts: n/a
 
      05-04-2005
On 5/4/05, Lucas Carlson <(E-Mail Removed)> wrote:
> Does anyone know of a web spider built in Ruby?

I couldn't find one either, so I wrote one based on http-access2 &
htmltokenizer (on RAA). Its really domain-specific though, I'm not
sure the code would be that useful to you.

I could send you the code if you want to get an idea of whats
involved, but be warned its really ugly (needed it fast).

Leon



 
Reply With Quote
 
 
 
 
Mark Hubbart
Guest
Posts: n/a
 
      05-04-2005
On 5/4/05, Lucas Carlson <(E-Mail Removed)> wrote:
> Does anyone know of a web spider built in Ruby?


Here's an interesting link:
http://www.linux-magazine.com/issue/...eb_Spiders.pdf

It's not a full-blown lib, but seems to be a pretty good tutorial for
it. Regular expressions and simple http fetch methods make this sort
of thing fairly easy to do in Ruby.



 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
SEO spider-spider ANY website prasadpelluri5@gmail.com Java 0 10-30-2008 07:04 PM
how google spider access my web site? baroque Chou ASP .Net 7 02-02-2006 09:25 PM
Web Crawler / Spider Commercial Software Info Request Gray Ghost Computer Support 1 11-07-2004 01:48 PM
Web Spider Thomas Lindgaard Python 3 07-07-2004 01:29 PM
Suggestions on a Web Spider that .... JeepGary Java 2 10-21-2003 02:53 AM



Advertisments