Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Perl > Perl Misc > bulk flush input

Reply
Thread Tools

bulk flush input

 
 
John Kelly
Guest
Posts: n/a
 
      06-26-2010

#!/usr/bin/perl

use strict;
use warnings;

my $data;

while (<>) {
chomp;
if (!$data && $_) {
$data = $_;
}
}

print "data=$data\n";


This code reads STDIN and remembers the first non-empty line. That's
all it cares about.

But it also keeps reading till EOF, acting like the "cat" utility, to
flush the extra input and avoid broken pipe errors.

But reading line by line, just to throw away the unwanted garbage, is
inefficient. I would like to jump out of the loop and "bulk flush" the
remaining input stream.

I don't think

>$io->flush


>flush causes perl to flush any buffered data at the perlio
>api level. Any unread data in the buffer will be discarded,


will work, because that it flushes the buffer, not the entire input
stream. I want to flush the whole file. And keep in mind, I don't want
a broken pipe either.

Suggestions?



--
Web mail, POP3, and SMTP
http://www.beewyz.com/freeaccounts.php

 
Reply With Quote
 
 
 
 
John Kelly
Guest
Posts: n/a
 
      06-26-2010
On Sat, 26 Jun 2010 03:21:08 +0100, Ben Morrow <(E-Mail Removed)> wrote:

>
>Quoth John Kelly <(E-Mail Removed)>:
>>
>> #!/usr/bin/perl
>>
>> use strict;
>> use warnings;
>>
>> my $data;
>>
>> while (<>) {
>> chomp;
>> if (!$data && $_) {
>> $data = $_;
>> }
>> }
>>
>> print "data=$data\n";
>>
>>
>> This code reads STDIN and remembers the first non-empty line. That's
>> all it cares about.

>
>No, it remembers the first line that doesn't evaluate to boolean false.
>Since you are chomping the lines, a line containing only "0" will be
>considered 'empty'. You want to check length $data.


Yeah, shot myself in the foot again.



>> But it also keeps reading till EOF, acting like the "cat" utility, to
>> flush the extra input and avoid broken pipe errors.
>>
>> But reading line by line, just to throw away the unwanted garbage, is
>> inefficient. I would like to jump out of the loop and "bulk flush" the
>> remaining input stream.
>>
>> I don't think
>>
>> >$io->flush

>>
>> >flush causes perl to flush any buffered data at the perlio
>> >api level. Any unread data in the buffer will be discarded,

>>
>> will work, because that it flushes the buffer, not the entire input
>> stream. I want to flush the whole file. And keep in mind, I don't want
>> a broken pipe either.
>>
>> Suggestions?

>
> my $data;
> while (<>) {
> chomp;
> length or next;
> $data = $_;
> last;
> }
> {
> local $/ = \2048;
> 1 while <>;
> }
>
>Ben


That looks interesting, I get the first part, the rest will give me
something to chew on ...

Thanks.




--
Web mail, POP3, and SMTP
http://www.beewyz.com/freeaccounts.php

 
Reply With Quote
 
 
 
 
C.DeRykus
Guest
Posts: n/a
 
      06-26-2010
On Jun 25, 7:47*pm, John Kelly <(E-Mail Removed)> wrote:
> On Sat, 26 Jun 2010 03:21:08 +0100, Ben Morrow <(E-Mail Removed)> wrote:
>
> >Quoth John Kelly <(E-Mail Removed)>:

>
> >> #!/usr/bin/perl

>
> >> use strict;
> >> use warnings;

>
> >> my $data;

>
> >> while (<>) {
> >> * * chomp;
> >> * * if (!$data && $_) {
> >> * * * * $data = $_;
> >> * * }
> >> }

>
> >> print "data=$data\n";

>
> >> This code reads STDIN and remembers the first non-empty line. *That's
> >> all it cares about.

>
> >No, it remembers the first line that doesn't evaluate to boolean false.
> >Since you are chomping the lines, a line containing only "0" will be
> >considered 'empty'. You want to check length $data.

>
> Yeah, shot myself in the foot again.
>
>
>
> >> But it also keeps reading till EOF, acting like the "cat" utility, to
> >> flush the extra input and avoid broken pipe errors.

>
> >> But reading line by line, just to throw away the unwanted garbage, is
> >> inefficient. *I would like to jump out of the loop and "bulk flush" the
> >> remaining input stream.

>
> >> I don't think

>
> >> >$io->flush

>
> >> >flush causes perl to flush any buffered data at the perlio
> >> >api level. Any unread data in the buffer will be discarded,

>
> >> will work, because that it flushes the buffer, not the entire input
> >> stream. *I want to flush the whole file. *And keep in mind, I don't want
> >> a broken pipe either.

>
> >> Suggestions?

>
> > * *my $data;
> > * *while (<>) {
> > * * * *chomp;
> > * * * *length or next;
> > * * * *$data = $_;
> > * * * *last;
> > * *}
> > * *{
> > * * * *local $/ = \2048;
> > * * * *1 while <>;
> > * *}

>
> >Ben

>
> That looks interesting, *I get the first part, the rest will give me
> something to chew on ...
>


But, this'll just read/toss 2048 byte chunks
till EOF. Alternatively, if you want to toss
the entire stream after exiting the loop, a
single statement:

<>;

does what you want due to the list context. I
suspect there's little to gain by changing $/.


--
Charles DeRykus
 
Reply With Quote
 
C.DeRykus
Guest
Posts: n/a
 
      06-26-2010
On Jun 25, 10:08*pm, "C.DeRykus" <(E-Mail Removed)> wrote:
> On Jun 25, 7:47*pm, John Kelly <(E-Mail Removed)> wrote:
>
>
>
> > On Sat, 26 Jun 2010 03:21:08 +0100, Ben Morrow <(E-Mail Removed)> wrote:

>
> > >Quoth John Kelly <(E-Mail Removed)>:

>
> > >> #!/usr/bin/perl

>
> > >> use strict;
> > >> use warnings;

>
> > >> my $data;

>
> > >> while (<>) {
> > >> * * chomp;
> > >> * * if (!$data && $_) {
> > >> * * * * $data = $_;
> > >> * * }
> > >> }

>
> > >> print "data=$data\n";

>
> > >> This code reads STDIN and remembers the first non-empty line. *That's
> > >> all it cares about.

>
> > >No, it remembers the first line that doesn't evaluate to boolean false..
> > >Since you are chomping the lines, a line containing only "0" will be
> > >considered 'empty'. You want to check length $data.

>
> > Yeah, shot myself in the foot again.

>
> > >> But it also keeps reading till EOF, acting like the "cat" utility, to
> > >> flush the extra input and avoid broken pipe errors.

>
> > >> But reading line by line, just to throw away the unwanted garbage, is
> > >> inefficient. *I would like to jump out of the loop and "bulk flush" the
> > >> remaining input stream.

>
> > >> I don't think

>
> > >> >$io->flush

>
> > >> >flush causes perl to flush any buffered data at the perlio
> > >> >api level. Any unread data in the buffer will be discarded,

>
> > >> will work, because that it flushes the buffer, not the entire input
> > >> stream. *I want to flush the whole file. *And keep in mind, I don't want
> > >> a broken pipe either.

>
> > >> Suggestions?

>
> > > * *my $data;
> > > * *while (<>) {
> > > * * * *chomp;
> > > * * * *length or next;
> > > * * * *$data = $_;
> > > * * * *last;
> > > * *}
> > > * *{
> > > * * * *local $/ = \2048;
> > > * * * *1 while <>;
> > > * *}

>
> > >Ben

>
> > That looks interesting, *I get the first part, the rest will give me
> > something to chew on ...

>
> But, this'll just read/toss 2048 byte chunks
> till EOF. Alternatively, if you want to toss
> the entire stream after exiting the loop, a
> single statement:
>
> * * *<>;

^^^^^^^

() = <>;

--
Charles DeRykus
Charles DeRykus


 
Reply With Quote
 
Ilya Zakharevich
Guest
Posts: n/a
 
      06-26-2010
On 2010-06-26, C.DeRykus <(E-Mail Removed)> wrote:
> () = <>;


Try to do it with a terabyte file...

Yours,
Ilya
 
Reply With Quote
 
John Kelly
Guest
Posts: n/a
 
      06-26-2010
On Fri, 25 Jun 2010 22:08:54 -0700 (PDT), "C.DeRykus"
<(E-Mail Removed)> wrote:

>> > * *my $data;
>> > * *while (<>) {
>> > * * * *chomp;
>> > * * * *length or next;
>> > * * * *$data = $_;
>> > * * * *last;
>> > * *}
>> > * *{
>> > * * * *local $/ = \2048;
>> > * * * *1 while <>;
>> > * *}

>>
>> >Ben

>>
>> That looks interesting, *I get the first part, the rest will give me
>> something to chew on ...
>>

>
>But, this'll just read/toss 2048 byte chunks
>till EOF. Alternatively, if you want to toss
>the entire stream after exiting the loop, a
>single statement:
>
> <>;
>
>does what you want due to the list context. I
>suspect there's little to gain by changing $/.


That doesn't look like list context to me. Testing a small data set
containing:

>
>
>
>one
>two
>
>three
>
>four
>


with this code:

-----------------------

#!/usr/bin/perl

use strict;
use warnings;

my $data = '';

while (<>) {
chomp;
/^\s*$/ and next;
$data = $_;
print "data=\"$data\"\n";
last;
}

<> or die "1 EOF\n";
<> or die "2 EOF\n";
<> or die "3 EOF\n";
<> or die "4 EOF\n";
<> or die "5 EOF\n";
<> or die "6 EOF\n";
<> or die "7 EOF\n";
<> or die "8 EOF\n";
<> or die "9 EOF\n";

close STDIN;


--------------------------------

produces output:

>data="one"
>6 EOF



Which seems to prove that a bare <> is scalar context.



--
Web mail, POP3, and SMTP
http://www.beewyz.com/freeaccounts.php

 
Reply With Quote
 
C.DeRykus
Guest
Posts: n/a
 
      06-26-2010
On Jun 25, 11:16*pm, Ilya Zakharevich <(E-Mail Removed)> wrote:
> On 2010-06-26, C.DeRykus <(E-Mail Removed)> wrote:
>
> > * * * () = <>;

>
> Try to do it with a terabyte file...
>


Hm, sounds like I need to look more closely...

So a humongous temp array gets built with only
the resulting assignment being optimized away...?

--
Charles DeRykus

perl -MO=Concise -e "()=<>"
8 <@> leave[1 ref] vKP/REFC ->(end)
1 <0> enter ->2
2 <;> nextstate(main 1 -e:1) v:{ ->3
7 <2> aassign[t3] vKS ->8
- <1> ex-list lK ->6
3 <0> pushmark s ->4
5 <1> readline[t2] lK/1 ->6
4 <#> gv[*ARGV] s ->5
- <1> ex-list lK ->7
6 <0> pushmark s ->7
- <0> stub lPRM* ->-
 
Reply With Quote
 
John Kelly
Guest
Posts: n/a
 
      06-26-2010
On Sat, 26 Jun 2010 06:16:29 +0000 (UTC), Ilya Zakharevich
<(E-Mail Removed)> wrote:

>On 2010-06-26, C.DeRykus <(E-Mail Removed)> wrote:
>> () = <>;

>
>Try to do it with a terabyte file...


Aim gun at foot. Pull trigger. Not my concept of fun.


--
Web mail, POP3, and SMTP
http://www.beewyz.com/freeaccounts.php

 
Reply With Quote
 
John Kelly
Guest
Posts: n/a
 
      06-26-2010

Thanks for the ideas, wrong ones too. They taught me more Perl, and got
me pointed in the right direction.


--
Web mail, POP3, and SMTP
http://www.beewyz.com/freeaccounts.php

 
Reply With Quote
 
Willem
Guest
Posts: n/a
 
      06-26-2010
John Kelly wrote:
<snip>
)<> or die "1 EOF\n";
)<> or die "2 EOF\n";
)<> or die "3 EOF\n";
)<> or die "4 EOF\n";
)<> or die "5 EOF\n";
)<> or die "6 EOF\n";
)<> or die "7 EOF\n";
)<> or die "8 EOF\n";
)<> or die "9 EOF\n";
<snip>
) produces output:
)
)>data="one"
)>6 EOF
)
)
) Which seems to prove that a bare <> is scalar context.

No, it proves that the left-hand side of 'or' has scalar context.

I'm pretty sure that a bare <> has void context, (which usually
translates to scalar context).

It is really a lot faster to change the line separator.
Setting it to \2048 means that it will always read that many bytes,
and undef'ing it would mean it will read the whole rest of the file.
I don't know if <> is smart enough to recognize void context though,
can probably be tested with a large file and a memory checker tool.


SaSW, Willem
--
Disclaimer: I am in no way responsible for any of the statements
made in the above text. For all I know I might be
drugged or something..
No I'm not paranoid. You all think I'm paranoid, don't you !
#EOT
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
How can I flush file input buffers? Anon Anon VHDL 4 05-29-2007 07:49 PM
Re: Suggestion for Cache flush in Moz Caffeine Junkie Firefox 1 01-20-2004 06:06 AM
Response.flush question Luiz Vianna ASP .Net 5 11-06-2003 10:47 PM
Re: Response.Flush() does not work William F. Robertson, Jr. ASP .Net 0 08-05-2003 03:45 PM
Re: Response.Flush() in code-behind Marina ASP .Net 1 07-01-2003 06:44 PM



Advertisments