Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > lwp::simple get (why it would stop working along with wget when fetch still works)

Reply
Thread Tools

lwp::simple get (why it would stop working along with wget when fetch still works)

 
 
rockerd@gmail.com
Guest
Posts: n/a
 
      07-18-2007
Hi Perl People,
Something recently changed on a site that I was fetching and parsing
from with lwp::simple.
Here is the thing: For the longest time I was using get() to grab a
http: site and store it in a scalar which I parsed later. Suddenly I
get an empty but defined scalar with: $html = get($url);

More: when I use fetch on a freebsd system it pulls the page to text
without any problems but when I use wget on a linux system I get a
blank file. Everything used to work. I tried changing my user-agent
headers and have had no luck. The only thing I can see is that the
file has an unknown length.. but I don't know what to do.

Thanks for the advice,
Rocker

 
Reply With Quote
 
 
 
 
Gunnar Hjalmarsson
Guest
Posts: n/a
 
      07-18-2007
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
> Something recently changed on a site that I was fetching and parsing
> from with lwp::simple.
> Here is the thing: For the longest time I was using get() to grab a
> http: site and store it in a scalar which I parsed later. Suddenly I
> get an empty but defined scalar with: $html = get($url);


Maybe the web server doesn't like requests that are generated by Perl.
You may want to try without sending a client identifier:

use LWP::UserAgent;
my $ua = LWP::UserAgent->new;
$ua->agent(''); # <- This line may make a difference
my $response = $ua->get('http://www.perl.org/');
print $response->content;

--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
 
Reply With Quote
 
 
 
 
Peter J. Holzer
Guest
Posts: n/a
 
      07-21-2007
On 2007-07-18 02:17, (E-Mail Removed) <(E-Mail Removed)> wrote:
> Something recently changed on a site that I was fetching and parsing
> from with lwp::simple.
> Here is the thing: For the longest time I was using get() to grab a
> http: site and store it in a scalar which I parsed later. Suddenly I
> get an empty but defined scalar with: $html = get($url);


Use LWP::Simple only if you are absolutely sure that you never need the
return code or headers. LWP::UserAgent is almost always the better
choice, especially if you have to handle errors or strange behaviour.


> More: when I use fetch on a freebsd system it pulls the page to text
> without any problems but when I use wget on a linux system I get a
> blank file. Everything used to work. I tried changing my user-agent
> headers and have had no luck.


Is "a linux system" the system where the script normally runs and "a
freebsd system" a different system? It might be that the owner of the
site noticed that you are automatically retrieving data and blocking
your IP address.

hp


--
_ | Peter J. Holzer | I know I'd be respectful of a pirate
|_|_) | Sysadmin WSR | with an emu on his shoulder.
| | | (E-Mail Removed) |
__/ | http://www.hjp.at/ | -- Sam in "Freefall"
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
get wget log message JuergenRiemer Python 1 02-13-2009 12:02 PM
lwp::simple get (why it would stop working along with wget when fetch still works) rockerd@gmail.com Perl Misc 0 07-18-2007 02:17 AM
How to use wget to get information of a jsp page Yang Java 6 04-20-2007 02:39 PM
How can I use wget to get the result of jsp page Yang Perl Misc 1 04-18-2007 01:38 PM
Web get command (wget) to download all icons/pics on a web page (too large or too small) barb Digital Photography 5 08-04-2006 07:06 PM



Advertisments