Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > Reducing cache/buffer for faster display

Reply
Thread Tools

Reducing cache/buffer for faster display

 
 
Rikishi42
Guest
Posts: n/a
 
      09-27-2012
I have these 2 scripts that are very heavy on the file i/o, consume a very
reasonable amount of cpu and output their counters at a - very - relaxed
pace to the console. The output is very simply done using something like:

print "files:", nFiles, "\r",


Yet alltough there is no real reason for it, even a pace of a print every
10-30 secs will be cached, only to actually show an output update every 1-2
min or so.

When I run the scripts with "python -u myscript.py", the output is complete,
very speedy and without any kind of impact on the actual work being one.


Can this option be called from within the script? Or is there another option
to make the display "a bit" speedier ?


Runnning Python 2.7.3, but it seems to me I've allready had this problem a
long, long time ago with other releases.



--
When in doubt, use brute force.
-- Ken Thompson
 
Reply With Quote
 
 
 
 
Chris Angelico
Guest
Posts: n/a
 
      09-27-2012
On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 <(E-Mail Removed)> wrote:
> I have these 2 scripts that are very heavy on the file i/o, consume a very
> reasonable amount of cpu and output their counters at a - very - relaxed
> pace to the console. The output is very simply done using something like:
>
> print "files:", nFiles, "\r",
>
>
> Yet alltough there is no real reason for it, even a pace of a print every
> 10-30 secs will be cached, only to actually show an output update every 1-2
> min or so.


Yup! Just add a call to sys.stdout.flush() after each print.

ChrisA
 
Reply With Quote
 
 
 
 
John Gordon
Guest
Posts: n/a
 
      09-27-2012
In <(E-Mail Removed)> Chris Angelico <(E-Mail Removed)> writes:

> On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 <(E-Mail Removed)> wrote:
> > I have these 2 scripts that are very heavy on the file i/o, consume a very
> > reasonable amount of cpu and output their counters at a - very - relaxed
> > pace to the console. The output is very simply done using something like:
> >
> > print "files:", nFiles, "\r",
> >
> >
> > Yet alltough there is no real reason for it, even a pace of a print every
> > 10-30 secs will be cached, only to actually show an output update every 1-2
> > min or so.


> Yup! Just add a call to sys.stdout.flush() after each print.


Isn't terminal output line-buffered? I don't understand why there would
be an output delay. (Unless the "\r" is messing things up...)

--
John Gordon A is for Amy, who fell down the stairs
http://www.velocityreviews.com/forums/(E-Mail Removed) B is for Basil, assaulted by bears
-- Edward Gorey, "The Gashlycrumb Tinies"

 
Reply With Quote
 
Chris Angelico
Guest
Posts: n/a
 
      09-27-2012
On Fri, Sep 28, 2012 at 8:25 AM, John Gordon <(E-Mail Removed)> wrote:
> Isn't terminal output line-buffered? I don't understand why there would
> be an output delay. (Unless the "\r" is messing things up...)


This is a classic progress-indication case, which does indeed mess up
line-buffering. The carriage return (and no line feed, done in the
Python 2 style of a trailing comma) puts the cursor back to the
beginning of the line, ready to overwrite, and ripe for one of those
old favorite incomplete overwrite errors - if nFiles monotonically
increases, it's fine, but if it decreases, the display can get ugly.

ChrisA
 
Reply With Quote
 
Rikishi42
Guest
Posts: n/a
 
      09-28-2012
On 2012-09-27, Chris Angelico <(E-Mail Removed)> wrote:
> On Fri, Sep 28, 2012 at 8:25 AM, John Gordon <(E-Mail Removed)> wrote:
>> Isn't terminal output line-buffered? I don't understand why there would
>> be an output delay. (Unless the "\r" is messing things up...)

>
> This is a classic progress-indication case, which does indeed mess up
> line-buffering. The carriage return (and no line feed, done in the
> Python 2 style of a trailing comma) puts the cursor back to the
> beginning of the line, ready to overwrite, and ripe for one of those
> old favorite incomplete overwrite errors - if nFiles monotonically
> increases, it's fine, but if it decreases, the display can get ugly.


True, but that wasn't the problem here. The updates where. Thanks for the
given answer, I'll try it.

The scripts in question only increase numbers. But should that not be the
case, solutions are simple enough. The numbers can be formatted to have a
fixed size. In the case of random line contents (a list of filesnames, say)
it's enough to create an output function that is aware of the length of the
previously printed line, and add enough spaces to the current one to wipe
exess content.


Thanks again for the suggestion.


--
When in doubt, use brute force.
-- Ken Thompson
 
Reply With Quote
 
Rikishi42
Guest
Posts: n/a
 
      09-28-2012
On 2012-09-27, Chris Angelico <(E-Mail Removed)> wrote:
> On Fri, Sep 28, 2012 at 7:57 AM, Rikishi42 <(E-Mail Removed)> wrote:
>> I have these 2 scripts that are very heavy on the file i/o, consume a very
>> reasonable amount of cpu and output their counters at a - very - relaxed
>> pace to the console. The output is very simply done using something like:
>>
>> print "files:", nFiles, "\r",
>>
>>
>> Yet alltough there is no real reason for it, even a pace of a print every
>> 10-30 secs will be cached, only to actually show an output update every 1-2
>> min or so.

>
> Yup! Just add a call to sys.stdout.flush() after each print.


Update: tried it, ran it, I love it.

Thanks !


--
When in doubt, use brute force.
-- Ken Thompson
 
Reply With Quote
 
Dennis Lee Bieber
Guest
Posts: n/a
 
      09-28-2012
On Thu, 27 Sep 2012 22:25:39 +0000 (UTC), John Gordon <(E-Mail Removed)>
declaimed the following in gmane.comp.python.general:

>
> Isn't terminal output line-buffered? I don't understand why there would
> be an output delay. (Unless the "\r" is messing things up...)


It's the trailing , The \r is being used to reset to the
beginning of the console line, but the comma "says" more output for
/this/ line will be coming... So no output until explicitly flushed, or
a new-line is issued.
--
Wulfraed Dennis Lee Bieber AF6VN
(E-Mail Removed) HTTP://wlfraed.home.netcom.com/

 
Reply With Quote
 
Chris Angelico
Guest
Posts: n/a
 
      09-28-2012
On Fri, Sep 28, 2012 at 10:05 AM, Rikishi42 <(E-Mail Removed)> wrote:
> The scripts in question only increase numbers. But should that not be the
> case, solutions are simple enough. The numbers can be formatted to have a
> fixed size. In the case of random line contents (a list of filesnames, say)
> it's enough to create an output function that is aware of the length of the
> previously printed line, and add enough spaces to the current one to wipe
> exess content.


Yep, that's a pretty effective way to do it. One simple method to it
is to format the whole string as a single whole, then left justify it
in a field of (say) 79 characters, and output that:

msg = "Progress: %d%% (%d/%d)... %s" % (done*100/total, done, total,
current_file)
print msg.ljust(79)+"\r",
sys.stdout.flush()

ChrisA
 
Reply With Quote
 
Rikishi42
Guest
Posts: n/a
 
      09-29-2012
On 2012-09-28, Dennis Lee Bieber <(E-Mail Removed)> wrote:
> On Thu, 27 Sep 2012 22:25:39 +0000 (UTC), John Gordon <(E-Mail Removed)>
> declaimed the following in gmane.comp.python.general:
>
>>
>> Isn't terminal output line-buffered? I don't understand why there would
>> be an output delay. (Unless the "\r" is messing things up...)

>
> It's the trailing , The \r is being used to reset to the
> beginning of the console line, but the comma "says" more output for
> /this/ line will be coming... So no output until explicitly flushed, or
> a new-line is issued.


Well, the \r seems to be the problem, allright.
But output was not completely blocked, just delayed a very long time.

So perhaps flushing and a sending a newline aren't the only triggers for
output. Perhaps there's a maximum delay or a maximum cumulated size, and
the output is flushed when such a limit is reached.

Anyway, that's mainly academic. I doubt there will be a correction to
that behaviour.

--
When in doubt, use brute force.
-- Ken Thompson
 
Reply With Quote
 
Rikishi42
Guest
Posts: n/a
 
      09-29-2012
On 2012-09-28, Chris Angelico <(E-Mail Removed)> wrote:
> On Fri, Sep 28, 2012 at 10:05 AM, Rikishi42 <(E-Mail Removed)> wrote:
>> The scripts in question only increase numbers. But should that not be the
>> case, solutions are simple enough. The numbers can be formatted to have a
>> fixed size. In the case of random line contents (a list of filesnames, say)
>> it's enough to create an output function that is aware of the length of the
>> previously printed line, and add enough spaces to the current one to wipe
>> exess content.

>
> Yep, that's a pretty effective way to do it. One simple method to it
> is to format the whole string as a single whole, then left justify it
> in a field of (say) 79 characters, and output that:
>
> msg = "Progress: %d%% (%d/%d)... %s" % (done*100/total, done, total,
> current_file)
> print msg.ljust(79)+"\r",
> sys.stdout.flush()


Mmm, I allmost went for that. It's elegant, simple and clear. But there's
one drawback: I usually reduce the terminal's window to take up less desktop
surface during those long runs.
So fixing it to 79 chars won't do. And I'm not even tempted to go for a
detection of the width of the terminal from within the script. The idea is
after all to keep the scripts simple (syntax) and light (execution).

Well, good night everyone.

--
When in doubt, use brute force.
-- Ken Thompson
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
Reducing master password check? Gusvaldo Firefox 1 04-18-2005 10:31 PM
Reducing cost for Cisco memory upgrade Johnson Ng Cisco 4 05-26-2004 03:49 AM
Reducing Spam Associated with Posting to Newsgroups Microsoft Communities Team [MSFT] MCSD 1 10-15-2003 12:10 PM
Reducing Spam Associated with Posting to Newsgroups Microsoft Communities Team [MSFT] ASP .Net 0 10-14-2003 10:26 PM
Reducing Spam Associated with Posting to Newsgroups Microsoft Communities Team [MSFT] Microsoft Certification 0 10-14-2003 10:25 PM



Advertisments