Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > Download Files With a perl script

Reply
Thread Tools

Download Files With a perl script

 
 
heli0s
Guest
Posts: n/a
 
      08-04-2005
Hi,

I'm having a bit of a problem with a script I'm writing. I want the script
to read in a txt file with a list of files to download and then go through
the list of files to get them from the net.

it should be something like this:

use LWP::Simple;

open (FILE,"all.m3u");
@file = <FILE>;
close FILE;

foreach $line (@file){
get ( $line );
}

As seen in the above code, i've been looking at LWP::Simple, but what i
have here is no good. Maybe i'm doing something wrong but i havent figured
it out yet.

Greets Christiaan
 
Reply With Quote
 
 
 
 
Paul Lalli
Guest
Posts: n/a
 
      08-04-2005
heli0s wrote:
> Hi,
>
> I'm having a bit of a problem with a script I'm writing. I want the script
> to read in a txt file with a list of files to download and then go through
> the list of files to get them from the net.
>
> it should be something like this:
>
> use LWP::Simple;


Please always put these lines in your script:
use strict;
use warnings;

> open (FILE,"all.m3u");


Always check the return value of open()
open my $file, '<', 'all.m3u' or die "Cannot open file: $!";

> @file = <FILE>;


Why are you storing all lines of this file in memory? Why not process
them one at a time?

> close FILE;
>
> foreach $line (@file){


Each $line contains a "\n" on the end. Do you want that newline? I'm
betting not.
perldoc -f chomp

> get ( $line );


>From perldoc LWP::Simple, get() returns the data fetched. Here, you're

calling get() in a void context (ie, not assigning its return value to
any variable), so you're throwing the data away.

You're also not checking to see if this is actually succeeding or not.

my $html = get ($line) or die "Could not fetch from $line: $!";

(note: I don't know for a fact that $! contains anything useful if
get() fails)

> }
>
> As seen in the above code, i've been looking at LWP::Simple, but what i
> have here is no good. Maybe i'm doing something wrong but i havent figured
> it out yet.


Please fix up your code, making it strict- and warnings-compliant, and
check the return values of your calls, and then post the results here
if you still haven't gotten it figured out.

Paul Lalli

 
Reply With Quote
 
 
 
 
Sisyphus
Guest
Posts: n/a
 
      08-04-2005

"heli0s" <(E-Mail Removed)> wrote in message

>
> use LWP::Simple;


use warnings;

>
> open (FILE,"all.m3u");
> @file = <FILE>;
> close FILE;
>
> foreach $line (@file){


chomp($line);

> get ( $line );
> }
>


Hth.

Cheers,
Rob


 
Reply With Quote
 
axel@white-eagle.invalid.uk
Guest
Posts: n/a
 
      08-04-2005
heli0s <(E-Mail Removed)> wrote:
> I'm having a bit of a problem with a script I'm writing. I want the script
> to read in a txt file with a list of files to download and then go through
> the list of files to get them from the net.
>
> it should be something like this:
>
> use LWP::Simple;
>
> open (FILE,"all.m3u");
> @file = <FILE>;
> close FILE;
>
> foreach $line (@file){
> get ( $line );
> }


> As seen in the above code, i've been looking at LWP::Simple, but what i
> have here is no good. Maybe i'm doing something wrong but i havent figured
> it out yet.


Do you have a sample of a couple of lines from all.m3u?

Axel

 
Reply With Quote
 
heli0s
Guest
Posts: n/a
 
      08-04-2005
On Thu, 04 Aug 2005 17:23:45 +0000, axel wrote:

> heli0s <(E-Mail Removed)> wrote:
>> I'm having a bit of a problem with a script I'm writing. I want the script
>> to read in a txt file with a list of files to download and then go through
>> the list of files to get them from the net.
>>
>> it should be something like this:
>>
>> use LWP::Simple;
>>
>> open (FILE,"all.m3u");
>> @file = <FILE>;
>> close FILE;
>>
>> foreach $line (@file){
>> get ( $line );
>> }

>
>> As seen in the above code, i've been looking at LWP::Simple, but what i
>> have here is no good. Maybe i'm doing something wrong but i havent figured
>> it out yet.

>
> Do you have a sample of a couple of lines from all.m3u?
>
> Axel


there something like this:

http://my.site.org/Albums/Asian%20Du...0-%20Community
%20Music/Asian%20Dub%20Foundation%20-%2001%20-%20real%20great%20britain.mp3

(with all the spaces. Getting them out is kinda out of the question
unfortunately)

Christiaan

 
Reply With Quote
 
sopan.shewale@gmail.com
Guest
Posts: n/a
 
      08-05-2005
Hello,

See if the following stuff helps you :

Following is content of all.m3u file.
$ cat all.m3u
http://www.manning-source.com/books/...cher2_chp1.pdf
http://www.manning-source.com/books/...cher2_chp3.pdf
http://www.manning-source.com/books/...cher2_chp4.pdf
$

Following is the script which can be used to download the above files.
$ cat download.pl
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
my $filelist = 'all.m3u';
open (FILE, $filelist) or die "Not able to open $filelist : $!";

foreach my $file (<FILE>) {
chomp($file); ## remove the last \n
my @filearray = split /\//, $file;
my $filename = pop @filearray;
print "Downloading : $filename" . "\n";
my $status = getstore($file, $filename);
if(is_success($status)) { print ("Download of $filename
complete\n");}
else { print ("Download of $filename failed\n"); }
}
close (FILE);


If you want to keep the directory structure consitent, you have to
workout on $filename variable which is constucted from $file, the line
of all.m3u file.
you can add some stuff there so that directories are created and the
files are stored in those directories.

Regards,

--sopan shewale



heli0s wrote:
> Hi,
>
> I'm having a bit of a problem with a script I'm writing. I want the script
> to read in a txt file with a list of files to download and then go through
> the list of files to get them from the net.
>
> it should be something like this:
>
> use LWP::Simple;
>
> open (FILE,"all.m3u");
> @file = <FILE>;
> close FILE;
>
> foreach $line (@file){
> get ( $line );
> }
>
> As seen in the above code, i've been looking at LWP::Simple, but what i
> have here is no good. Maybe i'm doing something wrong but i havent figured
> it out yet.
>
> Greets Christiaan


 
Reply With Quote
 
J. Gleixner
Guest
Posts: n/a
 
      08-05-2005
http://www.velocityreviews.com/forums/(E-Mail Removed) wrote:
[...]
> foreach my $file (<FILE>) {
> chomp($file); ## remove the last \n


> my @filearray = split /\//, $file;
> my $filename = pop @filearray;


No need to store everything, only to pop one item.

More succinct:

my $filename = ( split '/', $file )[-1];

> print "Downloading : $filename" . "\n";
> my $status = getstore($file, $filename);
> if(is_success($status)) { print ("Download of $filename
> complete\n");}
> else { print ("Download of $filename failed\n"); }


As with open(), it's a good to show why get() failed.

else {
print "Error downloading $filename: ",
$status->as_string, "\n",
$status->status_line, "\n";
}
> }
> close (FILE);

 
Reply With Quote
 
axel@white-eagle.invalid.uk
Guest
Posts: n/a
 
      08-05-2005
heli0s <(E-Mail Removed)> wrote:
>>> I'm having a bit of a problem with a script I'm writing. I want the script
>>> to read in a txt file with a list of files to download and then go through
>>> the list of files to get them from the net.


>>> it should be something like this:


>>> use LWP::Simple;
>>>
>>> open (FILE,"all.m3u");
>>> @file = <FILE>;
>>> close FILE;
>>>
>>> foreach $line (@file){
>>> get ( $line );
>>> }


>>> As seen in the above code, i've been looking at LWP::Simple, but what i
>>> have here is no good. Maybe i'm doing something wrong but i havent figured
>>> it out yet.


>> Do you have a sample of a couple of lines from all.m3u?


> there something like this:


> http://my.site.org/Albums/Asian%20Du...0-%20Community
> %20Music/Asian%20Dub%20Foundation%20-%2001%20-%20real%20great%20britain.mp3


> (with all the spaces. Getting them out is kinda out of the question
> unfortunately)


The spaces should not be causing a problem.
Nor should not chomping the input from the file as in your original script,
although it is very bad practice not to.

Maybe the URLs are not correct or that you are getting an error
response. A quick way of checking is to use lynx or
another browser to see if you can pull one up.

One possible reason for getting an error response is a site
refusing to return documents based on the User Agent. In
that case you could use LWP::UserAgent and give an agent
name such as 'Mozilla' which should fool the site.

Axel




 
Reply With Quote
 
sopan.shewale@gmail.com
Guest
Posts: n/a
 
      08-08-2005

J. Gleixner wrote:
> (E-Mail Removed) wrote:
> [...]
> > foreach my $file (<FILE>) {
> > chomp($file); ## remove the last \n

>
> > my @filearray = split /\//, $file;
> > my $filename = pop @filearray;

>
> No need to store everything, only to pop one item.
>
> More succinct:
>
> my $filename = ( split '/', $file )[-1];
>
> > print "Downloading : $filename" . "\n";
> > my $status = getstore($file, $filename);
> > if(is_success($status)) { print ("Download of $filename
> > complete\n");}
> > else { print ("Download of $filename failed\n"); }

>
> As with open(), it's a good to show why get() failed.
>
> else {
> print "Error downloading $filename: ",
> $status->as_string, "\n",
> $status->status_line, "\n";
> }
> > }
> > close (FILE);


 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
CGI: Execute a perl script inside another perl script xdarcos@hotmail.com Perl Misc 20 01-18-2005 12:33 PM
Execute another perl script from my perl script Petterson Mikael Perl Misc 3 01-05-2005 01:31 PM
problem calling perl script from SOAP server perl script pj Perl Misc 3 04-09-2004 10:23 PM
Perl Help - Windows Perl script accessing a Unix perl Script dpackwood Perl 3 09-30-2003 02:56 AM
How to make Perl Script "POST" call from another Perl Script??? Wet Basement Perl 1 07-15-2003 10:25 PM



Advertisments