Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Not necessarily related to python Web Crawlers

Reply
Thread Tools

Not necessarily related to python Web Crawlers

 
 
disappearedng@gmail.com
Guest
Posts: n/a
 
      07-05-2008
Hi
Does anyone here have a good recommendation for an open source crawler
that I could get my hands on? It doesn't have to be python based. I am
interested in learning how crawling works. I think python based
crawlers will ensure a high degree of flexibility but at the same time
I am also torn between looking for open source crawlers in python vs C
++ because the latter is much more efficient(or so I heard. I will be
crawling on very cheap hardware.)

I am definitely open to suggestions.

Thx
 
Reply With Quote
 
 
 
 
defn noob
Guest
Posts: n/a
 
      07-05-2008
just crawling is supereasy. its how to index and search that is hard.
just start at yahoo.com, scrape out all the links and then for every
site visit every link.
i wrote a crawler in 15 lines of code. but then it all it did was
visit the sites, not indexing them or anything.

you could write a faster one in C++ probably but if you are new to it
doing it in python will let you experiment and learn faster.

some links:
http://infolab.stanford.edu/~backrub/google.html
http://www-csli.stanford.edu/~hinric...eval-book.html



http://www.example-code.com/python/pythonspider.asp
http://www.example-code.com/python/s...pleCrawler.asp
 
Reply With Quote
 
 
 
 
subeen
Guest
Posts: n/a
 
      07-06-2008
On Jul 5, 2:31*pm, (E-Mail Removed) wrote:
> Hi
> Does anyone here have a good recommendation for an open source crawler
> that I could get my hands on? It doesn't have to be python based. I am
> interested in learning how crawling works. I think python based
> crawlers will ensure a high degree of flexibility but at the same time
> I am also torn between looking for open source crawlers in python vs C
> ++ because the latter is much more efficient(or so I heard. I will be
> crawling on very cheap hardware.)
>
> I am definitely open to suggestions.
>
> Thx


You can check my python blog. There are some tips and codes on
crawlers.
http://love-python.blogspot.com/

regards,
Subeen
http://love-python.blogspot.com/
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Setting properties (not necessarily "Properties") in Java 5.0 Wardie Java 4 11-08-2006 09:46 AM
Is Python good for web crawlers? Tempo Python 18 02-10-2006 12:33 PM
UK Social research job to suit Excel programmer (not necessarily VBA) Duncan Smith Perl Misc 2 06-03-2004 07:40 AM
UK Social research job to suit Excel programmer (not necessarily VBA) Duncan Smith Ruby 1 05-30-2004 04:15 AM
UK Social research job to suit Excel programmer (not necessarily VBA) Duncan Smith Python 4 05-29-2004 10:36 PM



Advertisments