«

I am trying to gunzip a file while downloading it.

it is 11 gigs in total, so it is an important task.

what is the proper command format? everything I tried did not work. I
tried a large number of combinations

open (my $fh, "gunzip -c");
$ftp->get("outfile.csv.gz", $fh);
close $fh;

"none" <> wrote in news:1140316587.609593.223100
@g47g2000cwa.googlegroups.com:

> I am trying to gunzip a file while downloading it.
>
> it is 11 gigs in total, so it is an important task.

It maybe important to you.

> what is the proper command format?

What command format? Perl has statements, functions, operators,
variables, modules etc but no commands.

> everything I tried did not work. I tried a large number of
> combinations
>
> open (my $fh, "gunzip -c");

This tries to open a file named "gunzip -c" for reading. See for
yourself:

open my $fh, 'gunzip -c' or die "Cannot open 'gunzip -c': $!";

What you probably want is to open a pipe to gunzip.

Please read the posting guidelines for this group. Post a short but
complete script which others can compile and run.


Sinan
--
A. Sinan Unur <>
(reverse each component and remove .invalid for email address)

comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/cl...uidelines.html

"none" <> wrote:
> I am trying to gunzip a file while downloading it.
>
> it is 11 gigs in total, so it is an important task.

Is 11 gigs supposed to have some magic significance to us? It doesn't. If
it is important, then tell us why it is important.

Aside from which, if the data is so large, then wouldn't it be important to
*not* unzip it?


>
> what is the proper command format? everything I tried did not work. I
> tried a large number of combinations
>
> open (my $fh, "gunzip -c");

###perhaps you are trying for this?

open (my $fh, "| gunzip -c") or die $!;

> $ftp->get("outfile.csv.gz", $fh);
> close $fh;

close $fh or die $!;

Xho

--
-------------------- http://NewsReader.Com/ --------------------
Usenet Newsgroup Service $9.95/Month 30GB

wrote:
> "none" <> wrote:
>> I am trying to gunzip a file while downloading it.
>>
>> it is 11 gigs in total, so it is an important task.
>
> Is 11 gigs supposed to have some magic significance to us? It doesn't. If
> it is important, then tell us why it is important.

What hardware do you use, where gunzipping an 11GB file takes negligible
time?

--
John W. Kennedy
Half an hour into unzipping the OpenOffice.org 2.0 SDK.

wrote:
> "John W. Kennedy" <> wrote:
>> wrote:
>>> "none" <> wrote:
>>>> I am trying to gunzip a file while downloading it.
>>>>
>>>> it is 11 gigs in total, so it is an important task.
>>> Is 11 gigs supposed to have some magic significance to us? It doesn't.
>>> If it is important, then tell us why it is important.
>> What hardware do you use, where gunzipping an 11GB file takes negligible
>> time?
>
> What hardware do you use where gunzipping a file while you are downloading
> requires a nonneglibly different amount of time than gunzipping it after
> you download it?

Since the bottleneck will normally be the download, downloading while
gunzipping will normally take only a fraction of a second more than the
time needed for the download alone.

--
John W. Kennedy
"But now is a new thing which is very old--
that the rich make themselves richer and not poorer,
which is the true Gospel, for the poor's sake."
-- Charles Williams. "Judgement at Chelmsford"

"John W. Kennedy" <> writes:

> What hardware do you use, where gunzipping an 11GB file takes
> negligible time?

The point is that long uncompression time has nothing to do with
whether your perl code is correct, so you're confusing the issue by
even bringing it up. Even so, if your program will be run
interactively (especially if as a CGI), you would be better off *not*
adding that delay to your program. Better would be to spawn off a
background process to decompress it, or dump it into a directory where
a crontask occasionally decompresses any new files found there.

Many FTP servers will decompress files on the fly as you download
them, but that will cost CPU at the server end and bandwidth at both
ends, so I wouldn't do that unless you own the FTP server.


--
Aaron --
http://360.yahoo.com/aaron_baugher

"John W. Kennedy" <> wrote in
news:BzaKf.71$_:

> wrote:
>> "John W. Kennedy" <> wrote:
>>> wrote:
>>>> "none" <> wrote:
>>>>> I am trying to gunzip a file while downloading it.
>>>>>
>>>>> it is 11 gigs in total, so it is an important task.
>>>> Is 11 gigs supposed to have some magic significance to us? It
>>>> doesn't. If it is important, then tell us why it is important.
>>> What hardware do you use, where gunzipping an 11GB file takes
>>> negligible time?
>>
>> What hardware do you use where gunzipping a file while you are
>> downloading requires a nonneglibly different amount of time than
>> gunzipping it after you download it?
>
> Since the bottleneck will normally be the download, downloading while
> gunzipping will normally take only a fraction of a second more than
> the time needed for the download alone.

Are you claiming that the de-compression process can begin before the
whole file is downloaded? I have to admit that I do not know if it can
or cannot. I am just curious to know. I would have thought that kind of
decompression (one that does not require the compressed file to be
stored on the filesystem first) could only done with RLE.

Sinan

--
A. Sinan Unur <>
(reverse each component and remove .invalid for email address)

comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/cl...uidelines.html

A. Sinan Unur <> wrote:
> "John W. Kennedy" <> wrote in

>> Since the bottleneck will normally be the download, downloading while
>> gunzipping will normally take only a fraction of a second more than
>> the time needed for the download alone.

> Are you claiming that the de-compression process can begin before the
> whole file is downloaded? I have to admit that I do not know if it can
> or cannot. I am just curious to know. I would have thought that kind of
> decompression (one that does not require the compressed file to be
> stored on the filesystem first) could only done with RLE.

Why not? If the download can be piped into gunzip there should be
no problem.

Some FTP sites enable downloading compressed files with decompression
on the fly but I have never bothered experimenting with this feature.

Axel

wrote in
news:LvjKf.23610$ k:

> A. Sinan Unur <> wrote:
>> "John W. Kennedy" <> wrote in
>
>>> Since the bottleneck will normally be the download, downloading
>>> while gunzipping will normally take only a fraction of a second more
>>> than the time needed for the download alone.
>
>> Are you claiming that the de-compression process can begin before the
>> whole file is downloaded? I have to admit that I do not know if it
>> can or cannot. I am just curious to know. I would have thought that
>> kind of decompression (one that does not require the compressed file
>> to be stored on the filesystem first) could only done with RLE.
>
> Why not? If the download can be piped into gunzip there should be
> no problem.

Can gunzip start decompressing before it has seen the whole file? I
don't know the format very well.

> Some FTP sites enable downloading compressed files with decompression
> on the fly but I have never bothered experimenting with this feature.

In that case, the original file already exists on the FTP server. That
is not the same as the client piping the input stream through gunzip.

I am not claiming to know. I guess I should run a couple of experiments
instead of taking up bandwidth here.

Sinan

--
A. Sinan Unur <>
(reverse each component and remove .invalid for email address)

comp.lang.perl.misc guidelines on the WWW:
http://mail.augustmail.com/~tadmc/cl...uidelines.html

"A. Sinan Unur" <> writes:

> Can gunzip start decompressing before it has seen the whole file? I
> don't know the format very well.

Yes. That's why you can do either of these:

gunzip file.gz
gunzip -c <file.gz >file

The first one eliminates file.gz after creating file; the second one
does not, since gunzip is getting file.gz streamed from stdin and
doesn't even know it exists as a file.


--
Aaron --
http://360.yahoo.com/aaron_baugher