Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > ASP .Net > Preventing Loops

Reply
Thread Tools

Preventing Loops

 
 
Brad Baker
Guest
Posts: n/a
 
      01-08-2007
We have a custom 404 error page setup so that when a 404 is generated the
user gets sent to redirect.asp. Redirect.asp sends a user back to
default.asp (or any other page specified)

One problem we have been experiencing is with poorly written webbot/spiders
getting hung up as follows:

bot indexes default.asp, gets a bad link, generates a 404
IIS sends bot to redirect.asp
redirect.asp sends bot back to default asp
bot reindexes links on default.asp until it gets to the bad link, generates
a 404, etc etc.
Loop forever

I'm looking for recommendations on how we can enhance redirect.asp to detect
loop conditions such as this. It goes without saying that we are trying to
come up with an approach that is semi-easy to implement and doesn't require
a lot of overhead.

Any thoughts would be greatly appreciated.

Thanks,
Brad


 
Reply With Quote
 
 
 
 
Rad [Visual C# MVP]
Guest
Posts: n/a
 
      01-08-2007
On Mon, 8 Jan 2007 11:08:44 -0500, Brad Baker wrote:

> We have a custom 404 error page setup so that when a 404 is generated the
> user gets sent to redirect.asp. Redirect.asp sends a user back to
> default.asp (or any other page specified)
>
> One problem we have been experiencing is with poorly written webbot/spiders
> getting hung up as follows:
>
> bot indexes default.asp, gets a bad link, generates a 404
> IIS sends bot to redirect.asp
> redirect.asp sends bot back to default asp
> bot reindexes links on default.asp until it gets to the bad link, generates
> a 404, etc etc.
> Loop forever
>
> I'm looking for recommendations on how we can enhance redirect.asp to detect
> loop conditions such as this. It goes without saying that we are trying to
> come up with an approach that is semi-easy to implement and doesn't require
> a lot of overhead.
>
> Any thoughts would be greatly appreciated.
>
> Thanks,
> Brad


Hey Brad,

In your 404 asp page you could check the user agent. Most crawlers/spiders
usually have a descriptive user agent name. You can use that for your
conditional logic

--
Bits.Bytes
http://bytes.thinkersroom.com
 
Reply With Quote
 
 
 
 
Steven Cheng[MSFT]
Guest
Posts: n/a
 
      01-09-2007
Hi Brad,

I agree with Rad. For your scenario, you need to define an exit condition
to end such recursive redirection. And for web crawler or other robot, one
thing to distinguish them from normal web clients is the "UserAgent" http
header. Different web client(browsers or crawlers will set different value
for this field). Here is web article describing detect browser in ASP.NET

http://aspnet.4guysfromrolla.com/articles/120402-1.aspx

in classic ASP, you can directly check the UserAgent field from
Request.Headers collection.

Hope this also helps.

Sincerely,

Steven Cheng

Microsoft MSDN Online Support Lead


This posting is provided "AS IS" with no warranties, and confers no rights.

 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Loops with loops using html-template Me Perl Misc 2 01-12-2006 05:07 PM
configuring Cisco Router to preventing assigning DHCP address Rami Rosen Cisco 13 10-24-2004 01:46 PM
preventing users from dropping wireless onto the lan jim Wireless Networking 3 08-31-2004 07:33 PM
Preventing Routing Matt Cisco 10 06-09-2004 09:20 PM
Preventing users from accessing Cisco PDM lombardi Cisco 1 04-13-2004 06:00 PM



Advertisments