Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Newbie LWP question - simulate browser?

Reply
Thread Tools

Newbie LWP question - simulate browser?

 
 
philthym
Guest
Posts: n/a
 
      05-10-2004
Hi

As the title suggests, I am a Perl newbie. I am trying to monitor a
remote site and would like to time it in returning all objects on the
page, ie the HTML and all the associated GIFs, bits of JavaScript,
Java applets and so on.

Here is the code as it stands today:

#!/usr/bin/perl
use CGI;
use LWP::Simple;
use Time::HiRes qw(gettimeofday);

$URL="http://www.xyz.com/index.html";

$usec1 = gettimeofday;
$timenow = localtime();

$HomePage=get($URL);

if
($HomePage =~ /String/)
{ $usec2 = gettimeofday;
$elapsed = $usec2-$usec1;
print "$timenow Page retrieved in $elapsed seconds\n" }
else
{ print "$timenow Page not retrieved\n"; }

I'm not sure I understand the whole lwp/get thing! What I'm wondering
is does this request effectively initiate the web server to return all
objects or just the HTML itself? If it's returning everything, then
does the second timer occur after all objects have been returned? In
other words, does this code do what I want it to? If not, any ideas
how I would achieve my aim, please?

Any help would be gratefully appreciated.

Thanks

Phil
 
Reply With Quote
 
 
 
 
Sherm Pendley
Guest
Posts: n/a
 
      05-10-2004
philthym wrote:

> I'm not sure I understand the whole lwp/get thing! What I'm wondering
> is does this request effectively initiate the web server to return all
> objects or just the HTML itself?


It does *exactly* what you ask it to, no more - it fetches index.html.
Parsing the HTML, extracting the <img ...> elements from it, and making
additional requests to the server to fetch the images they point to, will
require additional code.

Have a look at HTML:arser - it's a good place to start.

sherm--

--
Cocoa programming in Perl: http://camelbones.sourceforge.net
Hire me! My resume: http://www.dot-app.org
 
Reply With Quote
 
 
 
 
Joe Smith
Guest
Posts: n/a
 
      05-10-2004
philthym wrote:

> As the title suggests, I am a Perl newbie. I am trying to monitor a
> remote site and would like to time it in returning all objects on the
> page, ie the HTML and all the associated GIFs, bits of JavaScript,
> Java applets and so on.


It's one thing to fetch a Javascript. It is quite another to fetch
the things that would be requested had the Javascript been executed.
For that, you need a proxy that logs the requests from a real browser.

There are several, including the "Web Scrapting Proxy"
http://www.research.att.com/~hpk/wsp/

-Joe
 
Reply With Quote
 
philthym
Guest
Posts: n/a
 
      05-11-2004
Sherm Pendley <(E-Mail Removed)> wrote in message news:<(E-Mail Removed)>...
> philthym wrote:
>
> > I'm not sure I understand the whole lwp/get thing! What I'm wondering
> > is does this request effectively initiate the web server to return all
> > objects or just the HTML itself?

>
> It does *exactly* what you ask it to, no more - it fetches index.html.
> Parsing the HTML, extracting the <img ...> elements from it, and making
> additional requests to the server to fetch the images they point to, will
> require additional code.
>
> Have a look at HTML:arser - it's a good place to start.
>
> sherm--


Thanks Sherm, I thought it would be something like that. I'll check out HTMLarser.

Regards

Phil
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
any LWP::UserAgent equivalent in java && a java regex question z.m_wu@att.net Java 13 03-18-2008 01:33 AM
Newbie LWP Question on multi-stage forms... TB Perl Misc 1 12-07-2004 05:37 AM
LWP::Request technical question? at Perl Misc 2 01-20-2004 12:33 PM
Newbie LWP Question Bumble Perl Misc 7 01-11-2004 08:07 PM
Newbie Problem/Question: LWP and SSLEAY Andrew Perl Misc 3 11-24-2003 10:04 AM



Advertisments