Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Javascript > JS Web Robot

Reply
Thread Tools

JS Web Robot

 
 
Paul Dennis
Guest
Posts: n/a
 
      01-10-2004
Hi,

I'm trying to write a web robot using JavaScript.
It's objective would be to surf around and look
for patterns in the way web pages link to each
other or in the text they contain. Data would be
returned in a web box which could later be copied
into another application.

That's not to tough a challenge. I can make a
JS application surf around my hard drive or
web site with ease. I simply click an html into
a second window and wait for the document
readyState to be complete, then grab the
document.links array and point the window
at a new location. Off it goes.

But when it tries to surf from my drive to
my web site, or from my web site to another
web site, it gets an error. It crashes the first
time it tries to check the readyState of a
document from a different server.

I think that maybe JS has been designed to foil
attempts to build web robots with it. If so, is there
any way around it? Or maybe I'm just missing a
critical JS detail or two. So, does anyone know
what's going on here? Can anyone help me out?

-Paul Dennis.


 
Reply With Quote
 
 
 
 
Lasse Reichstein Nielsen
Guest
Posts: n/a
 
      01-10-2004
"Paul Dennis" <(E-Mail Removed)> writes:

> But when it tries to surf from my drive to
> my web site, or from my web site to another
> web site, it gets an error. It crashes the first
> time it tries to check the readyState of a
> document from a different server.
>
> I think that maybe JS has been designed to foil
> attempts to build web robots with it.


The browser security model has. If you try to access the content of a
page from a different domain, you are stopped - the hard way.

> If so, is there any way around it?


Not in any browser, but if it is just your own browser you might be
able to give it extended permissions. If the browser is IE, you can
look into HTML Applications (google for "HTML application HTA").

/L
--
Lasse Reichstein Nielsen - http://www.velocityreviews.com/forums/(E-Mail Removed)
DHTML Death Colors: <URL:http://www.infimum.dk/HTML/rasterTriangleDOM.html>
'Faith without judgement merely degrades the spirit divine.'
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
web robot dickymak@yahoo.com Java 4 11-04-2007 11:35 PM
User_agent and web robot names Desmond HTML 3 06-10-2007 07:56 PM
Writing a Web Robot in Python mudgen@gmail.com Python 3 02-13-2006 08:46 PM
1993 Van Rossum Python Web Robot Jonathan Vance Python 0 04-19-2005 10:19 PM
web crawl /robot Marlo Brandon ASP General 0 08-03-2004 09:31 PM



Advertisments