Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > HTTP::Request, trailing slash

Reply
Thread Tools

HTTP::Request, trailing slash

 
 
Sebastian Bauer
Guest
Posts: n/a
 
      06-29-2004
Hi,

I try to wirte a script that downloads images from a webpage. I use

my $uagent**=*LWP::UserAgent->new();
my $request = HTTP::Request->new(GET => "$url");
my $result**=*$uagent->request($request);

but the problem is that - even if $url does not contain a trailing slash -
some a slash is added at the end. This behaviour keeps me from downloading
the file because the download is refused this way. I think it must be this
slash, if i try to get the file via Mozilla (without the slash) it works,
with Mozilla with the slash it does not.

It would be great if anyone could help me out and tell me how to get rid of
the slash
Thx, Sebastian
 
Reply With Quote
 
 
 
 
Gunnar Hjalmarsson
Guest
Posts: n/a
 
      06-29-2004
Sebastian Bauer wrote:
> I try to wirte a script that downloads images from a webpage. I use
>
> my $uagent = LWP::UserAgent->new();
> my $request = HTTP::Request->new(GET => "$url");
> my $result = $uagent->request($request);
>
> but the problem is that - even if $url does not contain a trailing
> slash - some a slash is added at the end.


Where/when is the slash appended? Isn't $url what it is, and if it
contains a valid URL, the request is successful?

To make it easier to understand what you mean, please post a
*complete* program with a couple of valid URLs that your code fails to
get.

--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
 
Reply With Quote
 
 
 
 
Sebastian Bauer
Guest
Posts: n/a
 
      06-30-2004
Thx a lot for answering, here comes the program that fails.
It extracts the url of an image out of a webpage and then should download
this file. If you try to load the $img_url in mozilla it works. If you
append a slash or open the url in konqueror it fails the same way the
script does...

#!/usr/bin/perl

use strict;
use warnings;
use LWP::UserAgent;

my $taz_url = "http://www.taz.de";
my $filename = "tom.gif";

my $uagent = LWP::UserAgent->new();
my $request = HTTP::Request->new(GET => $taz_url
.."/pt/2004/06/30.nf/tomnf");

my $result = $uagent->request($request);
my $img_url;


if($result->content() =~
/<img src="(.*)" alt="TOM">\s+<br \/><b>Tom Touch&eacute; vom/)
{
$img_url = $1;
} else {
die "url of todays image cannot be determined\n";
}

print "${taz_url}${img_url}\n";

$request = HTTP::Request->new(GET => "${taz_url}${img_url}");
$result = $uagent->simple_request($request,$filename);
if($result->is_success) {
print "todays image stored in $filename\n";
} else {
die "could not store todays image\n";
}


Thx for your help
 
Reply With Quote
 
Gunnar Hjalmarsson
Guest
Posts: n/a
 
      06-30-2004
Sebastian Bauer wrote:
> Thx a lot for answering, here comes the program that fails. It
> extracts the url of an image out of a webpage and then should
> download this file. If you try to load the $img_url in mozilla it
> works.


$img_url is assigned the absolute URL
'/pt/.nf/gif.t,tom.d,1088589600', and that's not enough for any
browser to find the image. I don't understand what you mean by that.

But your script concatenates $taz_url and $img_url to
'http://www.taz.de/pt/.nf/gif.t,tom.d,1088589600'
which seems to be a valid URL to a (copyright protected) image.

> If you append a slash or open the url in konqueror it fails the
> same way the script does...


The script you posted does not fail for me. It prints "todays image
stored in tom.gif", and no slash is appended.

Sorry, but I still don't understand what the problem is.

<program snipped>

--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
 
Reply With Quote
 
Sebastian Bauer
Guest
Posts: n/a
 
      06-30-2004
the script downloads something, but if you make a

less tom.gif

you'll see that it did not download an image but plain text. If you follow
the link http://www.taz.de/pt/.nf/gif.t,tom.d,1088589600 in mozilla you'll
get an image. if you follow
http://www.taz.de/pt/.nf/gif.t,tom.d,1088589600/
you'll get the same text as the tom.gif file contains. That's the reason why
i thought that there might an additional slash

>which seems to be a valid URL to a (copyright protected) image.

this is just for personal use (i collect those images and its hard work to
to it manually)

thx sebastian
 
Reply With Quote
 
Gunnar Hjalmarsson
Guest
Posts: n/a
 
      06-30-2004
Sebastian Bauer wrote:
> the script downloads something, but if you make a
>
> less tom.gif
>
> you'll see that it did not download an image but plain text.


Aha, I see that now. Actually it downloads an HTML error page.

> If you follow the link
> http://www.taz.de/pt/.nf/gif.t,tom.d,1088589600 in mozilla you'll
> get an image. if you follow
> http://www.taz.de/pt/.nf/gif.t,tom.d,1088589600/ you'll get the
> same text as the tom.gif file contains. That's the reason why i
> thought that there might an additional slash


I see. Well, that error page is returned whichever incorrect URL you
are using, so why would it be caused by an appended slash?

The URL is not exactly the standard kind of URL. Maybe its special
nature makes LWP misinterpret it in some way. Maybe the site owner has
taken actions to prevent that people do what you are trying to do (you
can't view the image directly any longer, with or without the slash,
so I'd guess that the latter is the case).

--
Gunnar Hjalmarsson
Email: http://www.gunnar.cc/cgi-bin/contact.pl
 
Reply With Quote
 
Sebastian Bauer
Guest
Posts: n/a
 
      06-30-2004
i would accept that they dont allow me to download the image via a script,
but what drives me crazy is that i can obviously get the image via mozilla
but NO other way. I tried telnet 5 mins ago -> didnt work. So it seems i
will need to download the images by myself or i get it working any other
way.
Thx a lot, if you have any ideas please let me know
Sebastian
 
Reply With Quote
 
gnari
Guest
Posts: n/a
 
      06-30-2004
"Sebastian Bauer" <(E-Mail Removed)> wrote in message
news:cbv80l$a8e$01$(E-Mail Removed)-online.com...
> i would accept that they dont allow me to download the image via a script,
> but what drives me crazy is that i can obviously get the image via mozilla
> but NO other way. I tried telnet 5 mins ago -> didnt work. So it seems i
> will need to download the images by myself or i get it working any other
> way.
> Thx a lot, if you have any ideas please let me know


sessions, referer, cookies, useragent ...

gnari



 
Reply With Quote
 
Gisle Aas
Guest
Posts: n/a
 
      07-01-2004
"gnari" <(E-Mail Removed)> writes:

> "Sebastian Bauer" <(E-Mail Removed)> wrote in message
> news:cbv80l$a8e$01$(E-Mail Removed)-online.com...
> > i would accept that they dont allow me to download the image via a script,
> > but what drives me crazy is that i can obviously get the image via mozilla
> > but NO other way. I tried telnet 5 mins ago -> didnt work. So it seems i
> > will need to download the images by myself or i get it working any other
> > way.
> > Thx a lot, if you have any ideas please let me know

>
> sessions, referer, cookies, useragent ...


They check the 'referer'. This program works for me:

#!/usr/bin/perl

use strict;
use LWP::UserAgent;

my $ua = LWP::UserAgent->new(keep_alive => 1);
my $res = $ua->get("http://www.taz.de/pt/.nf/gif.t,tom.d,1088676000",
referer => "http://www.taz.de/pt/2004/07/01.nf/tomnf");
print $res->as_string;
 
Reply With Quote
 
Sebastian Bauer
Guest
Posts: n/a
 
      07-01-2004
Gisle Aas wrote:

> "gnari" <(E-Mail Removed)> writes:
>
>> "Sebastian Bauer" <(E-Mail Removed)> wrote in message
>> news:cbv80l$a8e$01$(E-Mail Removed)-online.com...
>> > i would accept that they dont allow me to download the image via a
>> > script, but what drives me crazy is that i can obviously get the image
>> > via mozilla but NO other way. I tried telnet 5 mins ago -> didnt work.
>> > So it seems i will need to download the images by myself or i get it
>> > working any other way.
>> > Thx a lot, if you have any ideas please let me know

>>
>> sessions, referer, cookies, useragent ...

>
> They check the 'referer'. This program works for me:
>
> #!/usr/bin/perl
>
> use strict;
> use LWP::UserAgent;
>
> my $ua = LWP::UserAgent->new(keep_alive => 1);
> my $res = $ua->get("http://www.taz.de/pt/.nf/gif.t,tom.d,1088676000",
> referer => "http://www.taz.de/pt/2004/07/01.nf/tomnf");
> print $res->as_string;


Thx a lot, this one worked, i found out the same today but thank you for the
solution
Sebastian
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
ASP.NET inserts trailing slash in tag Harlan Messinger ASP .Net 4 02-19-2008 01:15 PM
Multiline code - trailing slash usage abcd Python 7 03-15-2007 04:27 PM
path.py and directory naming: trailing slash automatic? metaperl.mogd@gmail.com Python 1 11-10-2006 05:03 PM
Finding out if a string has a trailing slash Stan Brown Perl Misc 6 10-29-2003 03:09 PM
trailing slash issue in Find.find Jeff Mitchell Ruby 0 08-23-2003 11:30 PM



Advertisments