Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > ASP .Net > ASP General > Crawl a site

Reply
Thread Tools

Crawl a site

 
 
Dave
Guest
Posts: n/a
 
      06-23-2004
Is it possible to crawl a site using ASP & XML HTTP?

I know you can hit one link, but how can you go through each link in a
page and validate that it returns 200?
 
Reply With Quote
 
 
 
 
Eric Gibson
Guest
Posts: n/a
 
      06-27-2004
Dave wrote:
> Is it possible to crawl a site using ASP & XML HTTP?
>
> I know you can hit one link, but how can you go through each link in a
> page and validate that it returns 200?


Using PerlScript and the Robot module.

http://search.cpan.org/~awrigley/WWW...b/WWW/Robot.pm

Eric


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Can Google crawl .aspx pages? John ASP .Net 2 01-31-2007 01:54 AM
The 3-minutes RESET CRAWL Zolaris Computer Support 1 09-07-2004 07:21 PM
trying to collect links on my site, how to crawl it? Danny ASP General 1 05-27-2004 02:33 AM
router (SMC Barricade 7004ABR) slows internet to a crawl until unplugged gene Computer Information 7 03-07-2004 06:09 PM
Internet Explorer slows to a crawl chthonia@hotmail.com Computer Support 3 12-22-2003 02:42 AM



Advertisments