Velocity Reviews - Computer Hardware Reviews

Velocity Reviews > Newsgroups > Programming > Python > get the size of a dynamically changing file fast ?

Reply
Thread Tools

get the size of a dynamically changing file fast ?

 
 
Stef Mientki
Guest
Posts: n/a
 
      01-17-2008
hello,

I've a program (not written in Python) that generates a few thousands
bytes per second,
these files are dumped in 2 buffers (files), at in interval time of 50 msec,
the files can be read by another program, to do further processing.

A program written in VB or delphi can handle the data in the 2 buffers
perfectly.
Sometimes Python is also able to process the data correctly,
but often it can't

I keep one of the files open en test the size of the open datafile each
50 msec.
I have tried
os.stat ( ....) [ ST_SIZE]
os.path.getsize ( ... )
but they both have the same behaviour, sometimes it works, and the data
is collected each 50 .. 100 msec,
sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.

I'm using python 2.4 on winXP.

Is there a solution for this problem ?

thanks,
Stef Mientki
 
Reply With Quote
 
 
 
 
Mike Driscoll
Guest
Posts: n/a
 
      01-18-2008
On Jan 17, 3:56 pm, Stef Mientki <(E-Mail Removed)> wrote:
> hello,
>
> I've a program (not written in Python) that generates a few thousands
> bytes per second,
> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
> the files can be read by another program, to do further processing.
>
> A program written in VB or delphi can handle the data in the 2 buffers
> perfectly.
> Sometimes Python is also able to process the data correctly,
> but often it can't
>
> I keep one of the files open en test the size of the open datafile each
> 50 msec.
> I have tried
> os.stat ( ....) [ ST_SIZE]
> os.path.getsize ( ... )
> but they both have the same behaviour, sometimes it works, and the data
> is collected each 50 .. 100 msec,
> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
>
> I'm using python 2.4 on winXP.
>
> Is there a solution for this problem ?
>
> thanks,
> Stef Mientki


Tim Golden has a method to watch for changes in a directory on his
website:

http://tgolden.sc.sabren.com/python/...r_changes.html

This old post also mentions something similar:

http://mail.python.org/pipermail/pyt...er/463065.html

And here's a cookbook recipe that claims to do it as well using
decorators:

http://aspn.activestate.com/ASPN/Coo.../Recipe/426620

Hopefully that will get you going.

Mike
 
Reply With Quote
 
 
 
 
Stef Mientki
Guest
Posts: n/a
 
      01-22-2008
Mike Driscoll wrote:
> On Jan 17, 3:56 pm, Stef Mientki <(E-Mail Removed)> wrote:
>
>> hello,
>>
>> I've a program (not written in Python) that generates a few thousands
>> bytes per second,
>> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
>> the files can be read by another program, to do further processing.
>>
>> A program written in VB or delphi can handle the data in the 2 buffers
>> perfectly.
>> Sometimes Python is also able to process the data correctly,
>> but often it can't
>>
>> I keep one of the files open en test the size of the open datafile each
>> 50 msec.
>> I have tried
>> os.stat ( ....) [ ST_SIZE]
>> os.path.getsize ( ... )
>> but they both have the same behaviour, sometimes it works, and the data
>> is collected each 50 .. 100 msec,
>> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
>>
>> I'm using python 2.4 on winXP.
>>
>> Is there a solution for this problem ?
>>
>> thanks,
>> Stef Mientki
>>

>
> Tim Golden has a method to watch for changes in a directory on his
> website:
>
> http://tgolden.sc.sabren.com/python/...r_changes.html
>
> This old post also mentions something similar:
>
> http://mail.python.org/pipermail/pyt...er/463065.html
>
> And here's a cookbook recipe that claims to do it as well using
> decorators:
>
> http://aspn.activestate.com/ASPN/Coo.../Recipe/426620
>
> Hopefully that will get you going.
>
> Mike
>

thanks Mike,
sorry for the late reaction.
I've it working perfect now.
After all, os.stat works perfectly well,
the problem was in the program that generated the file with increasing
size,
by truncating it after each block write, it apperently garantees that
the file is flushed to disk and all problems are solved.

cheers,
Stef Mientki


 
Reply With Quote
 
Mike Driscoll
Guest
Posts: n/a
 
      01-22-2008
On Jan 22, 3:35 pm, Stef Mientki <(E-Mail Removed)> wrote:
> Mike Driscoll wrote:
> > On Jan 17, 3:56 pm, Stef Mientki <(E-Mail Removed)> wrote:

>
> >> hello,

>
> >> I've a program (not written in Python) that generates a few thousands
> >> bytes per second,
> >> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
> >> the files can be read by another program, to do further processing.

>
> >> A program written in VB or delphi can handle the data in the 2 buffers
> >> perfectly.
> >> Sometimes Python is also able to process the data correctly,
> >> but often it can't

>
> >> I keep one of the files open en test the size of the open datafile each
> >> 50 msec.
> >> I have tried
> >> os.stat ( ....) [ ST_SIZE]
> >> os.path.getsize ( ... )
> >> but they both have the same behaviour, sometimes it works, and the data
> >> is collected each 50 .. 100 msec,
> >> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.

>
> >> I'm using python 2.4 on winXP.

>
> >> Is there a solution for this problem ?

>
> >> thanks,
> >> Stef Mientki

>
> > Tim Golden has a method to watch for changes in a directory on his
> > website:

>
> >http://tgolden.sc.sabren.com/python/...h_directory_fo...

>
> > This old post also mentions something similar:

>
> >http://mail.python.org/pipermail/pyt...er/463065.html

>
> > And here's a cookbook recipe that claims to do it as well using
> > decorators:

>
> >http://aspn.activestate.com/ASPN/Coo.../Recipe/426620

>
> > Hopefully that will get you going.

>
> > Mike

>
> thanks Mike,
> sorry for the late reaction.
> I've it working perfect now.
> After all, os.stat works perfectly well,
> the problem was in the program that generated the file with increasing
> size,
> by truncating it after each block write, it apperently garantees that
> the file is flushed to disk and all problems are solved.
>
> cheers,
> Stef Mientki


I almost asked if you were making sure you had flushed the data to the
file...oh well.

Mike
 
Reply With Quote
 
Stef Mientki
Guest
Posts: n/a
 
      01-22-2008
Mike Driscoll wrote:
> On Jan 22, 3:35 pm, Stef Mientki <(E-Mail Removed)> wrote:
>
>> Mike Driscoll wrote:
>>
>>> On Jan 17, 3:56 pm, Stef Mientki <(E-Mail Removed)> wrote:
>>>
>>>> hello,
>>>>
>>>> I've a program (not written in Python) that generates a few thousands
>>>> bytes per second,
>>>> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
>>>> the files can be read by another program, to do further processing.
>>>>
>>>> A program written in VB or delphi can handle the data in the 2 buffers
>>>> perfectly.
>>>> Sometimes Python is also able to process the data correctly,
>>>> but often it can't
>>>>
>>>> I keep one of the files open en test the size of the open datafile each
>>>> 50 msec.
>>>> I have tried
>>>> os.stat ( ....) [ ST_SIZE]
>>>> os.path.getsize ( ... )
>>>> but they both have the same behaviour, sometimes it works, and the data
>>>> is collected each 50 .. 100 msec,
>>>> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.
>>>>
>>>> I'm using python 2.4 on winXP.
>>>>
>>>> Is there a solution for this problem ?
>>>>
>>>> thanks,
>>>> Stef Mientki
>>>>
>>> Tim Golden has a method to watch for changes in a directory on his
>>> website:
>>>
>>> http://tgolden.sc.sabren.com/python/...h_directory_fo...
>>>
>>> This old post also mentions something similar:
>>>
>>> http://mail.python.org/pipermail/pyt...er/463065.html
>>>
>>> And here's a cookbook recipe that claims to do it as well using
>>> decorators:
>>>
>>> http://aspn.activestate.com/ASPN/Coo.../Recipe/426620
>>>
>>> Hopefully that will get you going.
>>>
>>> Mike
>>>

>> thanks Mike,
>> sorry for the late reaction.
>> I've it working perfect now.
>> After all, os.stat works perfectly well,
>> the problem was in the program that generated the file with increasing
>> size,
>> by truncating it after each block write, it apperently garantees that
>> the file is flushed to disk and all problems are solved.
>>
>> cheers,
>> Stef Mientki
>>

>
> I almost asked if you were making sure you had flushed the data to the
> file...oh well.
>

Yes, that's a small disadavantage of using a "high-level" language,
where there's no flush available, and you assume it'll done
automatically

cheers,
Stef

 
Reply With Quote
 
Jason
Guest
Posts: n/a
 
      01-23-2008
On Jan 22, 3:22 pm, Stef Mientki <(E-Mail Removed)> wrote:
> Mike Driscoll wrote:
> > On Jan 22, 3:35 pm, Stef Mientki <(E-Mail Removed)> wrote:

>
> >> Mike Driscoll wrote:

>
> >>> On Jan 17, 3:56 pm, Stef Mientki <(E-Mail Removed)> wrote:

>
> >>>> hello,

>
> >>>> I've a program (not written in Python) that generates a few thousands
> >>>> bytes per second,
> >>>> these files are dumped in 2 buffers (files), at in interval time of 50 msec,
> >>>> the files can be read by another program, to do further processing.

>
> >>>> A program written in VB or delphi can handle the data in the 2 buffers
> >>>> perfectly.
> >>>> Sometimes Python is also able to process the data correctly,
> >>>> but often it can't

>
> >>>> I keep one of the files open en test the size of the open datafile each
> >>>> 50 msec.
> >>>> I have tried
> >>>> os.stat ( ....) [ ST_SIZE]
> >>>> os.path.getsize ( ... )
> >>>> but they both have the same behaviour, sometimes it works, and the data
> >>>> is collected each 50 .. 100 msec,
> >>>> sometimes 1 .. 1.5 seconds is needed to detect a change in filesize.

>
> >>>> I'm using python 2.4 on winXP.

>
> >>>> Is there a solution for this problem ?

>
> >>>> thanks,
> >>>> Stef Mientki

>
> >>> Tim Golden has a method to watch for changes in a directory on his
> >>> website:

>
> >>>http://tgolden.sc.sabren.com/python/...h_directory_fo...

>
> >>> This old post also mentions something similar:

>
> >>>http://mail.python.org/pipermail/pyt...er/463065.html

>
> >>> And here's a cookbook recipe that claims to do it as well using
> >>> decorators:

>
> >>>http://aspn.activestate.com/ASPN/Coo.../Recipe/426620

>
> >>> Hopefully that will get you going.

>
> >>> Mike

>
> >> thanks Mike,
> >> sorry for the late reaction.
> >> I've it working perfect now.
> >> After all, os.stat works perfectly well,
> >> the problem was in the program that generated the file with increasing
> >> size,
> >> by truncating it after each block write, it apperently garantees that
> >> the file is flushed to disk and all problems are solved.

>
> >> cheers,
> >> Stef Mientki

>
> > I almost asked if you were making sure you had flushed the data to the
> > file...oh well.

>
> Yes, that's a small disadavantage of using a "high-level" language,
> where there's no flush available, and you assume it'll done
> automatically
>
> cheers,
> Stef


Uhm, there is a flush method for Python's files. From "http://
docs.python.org/lib/bltin-file-objects.html":
flush()
Flush the internal buffer, like stdio's fflush(). This may
be a no-op on some file-like objects.

As for an example:

>>> import os
>>> f = open('vikings.txt', 'wb')
>>> os.stat('vikings.txt').st_size

0L
>>> f.write('Spam, spam, spam, spam! ' * 1000) # Bloody vikings...
>>> os.stat('vikings.txt').st_size

24576L
>>> f.flush()
>>> os.stat('vikings.txt').st_size

25000L
>>>


Is there something that I'm missing here?

--Jason
 
Reply With Quote
 
Stef Mientki
Guest
Posts: n/a
 
      01-23-2008

>>>> thanks Mike,
>>>> sorry for the late reaction.
>>>> I've it working perfect now.
>>>> After all, os.stat works perfectly well,
>>>> the problem was in the program that generated the file with increasing
>>>> size,
>>>> by truncating it after each block write, it apperently garantees that
>>>> the file is flushed to disk and all problems are solved.
>>>>
>>>> cheers,
>>>> Stef Mientki
>>>>
>>> I almost asked if you were making sure you had flushed the data to the
>>> file...oh well.
>>>

>> Yes, that's a small disadavantage of using a "high-level" language,
>> where there's no flush available, and you assume it'll done
>> automatically
>>
>> cheers,
>> Stef
>>

>
> Uhm, there is a flush method for Python's files. From "http://
> docs.python.org/lib/bltin-file-objects.html":
> flush()
> Flush the internal buffer, like stdio's fflush(). This may
> be a no-op on some file-like objects.
>
> As for an example:
>
>
>>>> import os
>>>> f = open('vikings.txt', 'wb')
>>>> os.stat('vikings.txt').st_size
>>>>

> 0L
>
>>>> f.write('Spam, spam, spam, spam! ' * 1000) # Bloody vikings...
>>>> os.stat('vikings.txt').st_size
>>>>

> 24576L
>
>>>> f.flush()
>>>> os.stat('vikings.txt').st_size
>>>>

> 25000L
>
>
> Is there something that I'm missing here?
>
>

hi Jason,
I was talking about a "high-level" language, in which the sending
program was written,
(Delphi, not about Python
cheers,
Stef

 
Reply With Quote
 
Gabriel Genellina
Guest
Posts: n/a
 
      01-23-2008
En Wed, 23 Jan 2008 16:16:58 -0200, Stef Mientki <(E-Mail Removed)>
escribió:

>>> Yes, that's a small disadavantage of using a "high-level" language,
>>> where there's no flush available, and you assume it'll done
>>> automatically

>>
>> Uhm, there is a flush method for Python's files. From "http://
>>

> I was talking about a "high-level" language, in which the sending
> program was written,
> (Delphi, not about Python


In Delphi, flush(filevar) does work. Or are you using a TFileStream or
similar?

--
Gabriel Genellina

 
Reply With Quote
 
Stef Mientki
Guest
Posts: n/a
 
      01-24-2008
Gabriel Genellina wrote:
> En Wed, 23 Jan 2008 16:16:58 -0200, Stef Mientki
> <(E-Mail Removed)> escribió:
>
>>>> Yes, that's a small disadavantage of using a "high-level" language,
>>>> where there's no flush available, and you assume it'll done
>>>> automatically
>>>
>>> Uhm, there is a flush method for Python's files. From "http://
>>>

>> I was talking about a "high-level" language, in which the sending
>> program was written,
>> (Delphi, not about Python

>
> In Delphi, flush(filevar) does work.

flush is only valid for textfiles, not for binary files (at least in D7)

anyway thanks, cheers,
Stef
 
Reply With Quote
 
 
 
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off


Similar Threads
Thread Thread Starter Forum Replies Last Post
snapshot reading a fast changing file Ralf C++ 4 02-24-2009 05:50 PM
Preferred Size, Minimum Size, Size Jason Cavett Java 5 05-25-2008 08:32 AM
A solution for a fast std::list::size() *and* a fast std::list::splice() Juha Nieminen C++ 22 10-12-2007 07:31 PM
mega pixels, file size, image size, and print size - Adobe Evangelists Frank ess Digital Photography 0 11-14-2006 05:08 PM
[Xpert] Changing the size of a JFrame dynamically Pierre Vigneras Java 1 01-24-2004 03:14 PM



Advertisments