«

I am aware, that it is maybe the wrong group to ask this question, but
as I would like to know the history of past file operations from within
a Python script I see a chance, that someone in this group was into it
already and is so kind to share here his experience.

I have put already much efforts into this subject googling around, but
up to now in vain. Best option I encountered yet is usage of
the Greyware 'System Change Log' service which monitors disks for
changes (http://www.greyware.com/software/sys...log/index.asp),
but in own tests it turned out, that the created log file does not cover
all file events as e.g. it is not possible to detect when a file is
moved to a new directory (creation of a new file is logged, but deletion
is not, not mentioning I would expect a file 'move' event).
The own Windows logging service rejected to start on my XP SP2 system
for unknown to me reasons - I don't know how to get it to work (yes, I
have used the administrator account).

I can't believe, that using NTFS file system in Microsoft Windows 2000
or XP it is not possible to track file events as:

- updating/modifying of an existing file/directory
- deleting an existing file/directory
- creating a new file/directory
- _moving_ an existing file/directory (should _NOT_ be covered by the
event duo of deleting an existing and creating a new file/directory)

Any hints towards enlightenment?

Claudio Grondi

Claudio Grondi wrote:
> I am aware, that it is maybe the wrong group to ask this question, but
> as I would like to know the history of past file operations from within
> a Python script I see a chance, that someone in this group was into it
> already and is so kind to share here his experience.
>
> I can't believe, that using NTFS file system in Microsoft Windows 2000
> or XP it is not possible to track file events as:
>
> - updating/modifying of an existing file/directory
> - deleting an existing file/directory
> - creating a new file/directory
> - _moving_ an existing file/directory (should _NOT_ be covered by the
> event duo of deleting an existing and creating a new file/directory)
>
> Any hints towards enlightenment?
>
> Claudio Grondi

On the offchance that you haven't seen it, you might
look at this:

http://timgolden.me.uk/python/win32_...rectorychanges

but since it doesn't fulfil your criterion of *not*
representing renames by a delete and an add, it may
well not be suitable. Apart from that, I think it does
what you want.

TJG

you want a directory watching daemon. it isn't hard at all to build
from scratch.
first, determine which directories should be watched.
then, os.walk each directory, building a mapping from filename to mtime
[modified time; os.path.getmtime].
next is your main event loop. this while loop consists of os.walk-ing
each directory again, comparing the current mtime to the corresponding
entry in the mapping. if they differ, or if a filename isn't in the
mapping, something happened, at which point you can logick out whether
a file was moved, deleted, changed, or created.

so many folks have looked for this that i'll just write a generic one
and put it in the cheeseshop. look for "dirmon" in about a week.


Claudio Grondi wrote:
> I am aware, that it is maybe the wrong group to ask this question, but
> as I would like to know the history of past file operations from within
> a Python script I see a chance, that someone in this group was into it
> already and is so kind to share here his experience.
>
> I have put already much efforts into this subject googling around, but
> up to now in vain. Best option I encountered yet is usage of
> the Greyware 'System Change Log' service which monitors disks for
> changes (http://www.greyware.com/software/sys...log/index.asp),
> but in own tests it turned out, that the created log file does not cover
> all file events as e.g. it is not possible to detect when a file is
> moved to a new directory (creation of a new file is logged, but deletion
> is not, not mentioning I would expect a file 'move' event).
> The own Windows logging service rejected to start on my XP SP2 system
> for unknown to me reasons - I don't know how to get it to work (yes, I
> have used the administrator account).
>
> I can't believe, that using NTFS file system in Microsoft Windows 2000
> or XP it is not possible to track file events as:
>
> - updating/modifying of an existing file/directory
> - deleting an existing file/directory
> - creating a new file/directory
> - _moving_ an existing file/directory (should _NOT_ be covered by the
> event duo of deleting an existing and creating a new file/directory)
>
> Any hints towards enlightenment?
>
> Claudio Grondi

"faulkner" <> wrote in message
news: oups.com...
> you want a directory watching daemon. it isn't hard at all to build
> from scratch.
> first, determine which directories should be watched.
> then, os.walk each directory, building a mapping from filename to mtime
> [modified time; os.path.getmtime].
> next is your main event loop. this while loop consists of os.walk-ing
> each directory again, comparing the current mtime to the corresponding
> entry in the mapping. if they differ, or if a filename isn't in the
> mapping, something happened, at which point you can logick out whether
> a file was moved, deleted, changed, or created.
>
> so many folks have looked for this that i'll just write a generic one
> and put it in the cheeseshop. look for "dirmon" in about a week.
>
>

While I am a fan of "brute force"

"faulkner" <> wrote in message
news: oups.com...
> you want a directory watching daemon. it isn't hard at all to build
> from scratch.
> first, determine which directories should be watched.
> then, os.walk each directory, building a mapping from filename to mtime
> [modified time; os.path.getmtime].
> next is your main event loop. this while loop consists of os.walk-ing
> each directory again, comparing the current mtime to the corresponding
> entry in the mapping. if they differ, or if a filename isn't in the
> mapping, something happened, at which point you can logick out whether
> a file was moved, deleted, changed, or created.
>
> so many folks have looked for this that i'll just write a generic one
> and put it in the cheeseshop. look for "dirmon" in about a week.
>
>
Ahem... (sorry for premature usenet-post-ication...)

While I am a big fan of "brute force", there are OS services (at least on
Windows) for doing just this function, with asynchronous callbacks when
files are created, deleted, etc.

Here is a link that does a much better comparison of several options than I
could (including your brute force version):
http://tgolden.sc.sabren.com/python/...r_changes.html

Good luck!
-- Paul

faulkner wrote:
> you want a directory watching daemon. it isn't hard at all to build
> from scratch.
> first, determine which directories should be watched.
> then, os.walk each directory, building a mapping from filename to mtime
> [modified time; os.path.getmtime].
> next is your main event loop. this while loop consists of os.walk-ing
> each directory again, comparing the current mtime to the corresponding
> entry in the mapping. if they differ, or if a filename isn't in the
> mapping, something happened, at which point you can logick out whether
> a file was moved, deleted, changed, or created.
>
> so many folks have looked for this that i'll just write a generic one
> and put it in the cheeseshop. look for "dirmon" in about a week.

Yes, I _know_ about it and exactly this knowledge is the reason I am
looking for tracking single file system related _events_ as I expect a
professional operating system like Windows to provide such service. If
there is none, this will be sure a severe reason to go for Linux if it
has such one instead of going for a SVN server or special file systems
if there are any.

Has someone experience with SVN handling million(s) of files?

The problem is, that brute force applied to large amount of
files/directories is not a convenient way to backup/synchronize the few
new/changed/deleted/moved files/directories multiple times a day as the
brute force approach just makes the hard drive(s) unnecessary wasting
much energy and getting hot.

Claudio Grondi
>
>
> Claudio Grondi wrote:
>
>>I am aware, that it is maybe the wrong group to ask this question, but
>>as I would like to know the history of past file operations from within
>>a Python script I see a chance, that someone in this group was into it
>>already and is so kind to share here his experience.
>>
>>I have put already much efforts into this subject googling around, but
>>up to now in vain. Best option I encountered yet is usage of
>>the Greyware 'System Change Log' service which monitors disks for
>>changes (http://www.greyware.com/software/sys...log/index.asp),
>>but in own tests it turned out, that the created log file does not cover
>>all file events as e.g. it is not possible to detect when a file is
>>moved to a new directory (creation of a new file is logged, but deletion
>>is not, not mentioning I would expect a file 'move' event).
>>The own Windows logging service rejected to start on my XP SP2 system
>>for unknown to me reasons - I don't know how to get it to work (yes, I
>>have used the administrator account).
>>
>>I can't believe, that using NTFS file system in Microsoft Windows 2000
>>or XP it is not possible to track file events as:
>>
>>- updating/modifying of an existing file/directory
>>- deleting an existing file/directory
>>- creating a new file/directory
>>- _moving_ an existing file/directory (should _NOT_ be covered by the
>>event duo of deleting an existing and creating a new file/directory)
>>
>>Any hints towards enlightenment?
>>
>>Claudio Grondi
>
>

Tim Golden wrote:
> Claudio Grondi wrote:
>
>> I am aware, that it is maybe the wrong group to ask this question, but
>> as I would like to know the history of past file operations from
>> within a Python script I see a chance, that someone in this group was
>> into it already and is so kind to share here his experience.
>>
>> I can't believe, that using NTFS file system in Microsoft Windows 2000
>> or XP it is not possible to track file events as:
>>
>> - updating/modifying of an existing file/directory
>> - deleting an existing file/directory
>> - creating a new file/directory
>> - _moving_ an existing file/directory (should _NOT_ be covered by the
>> event duo of deleting an existing and creating a new file/directory)
>>
>> Any hints towards enlightenment?
>>
>> Claudio Grondi
>
>
> On the offchance that you haven't seen it, you might
> look at this:
>
> http://timgolden.me.uk/python/win32_...rectorychanges
>
>
> but since it doesn't fulfil your criterion of *not*
> representing renames by a delete and an add, it may
> well not be suitable. Apart from that, I think it does
> what you want.
>
> TJG

It seems, that it will be necessary to use some logic based on the
sequence of events to exactly detect rename and move changes done to
files/directories, but in principle it is the best approach I know about
yet.

Thank you!

By the way:
Is there something similar/same available for Linux?

Claudio Grondi

Claudio Grondi wrote:
> Here a small update to the code at
> http://timgolden.me.uk/python/win32_...rectorychanges
> :
>
> ACTIONS = {
> 1 : "Created",
> 2 : "Deleted",
> 3 : "Updated",
> 4 : "Renamed from something"
> 5 : "Renamed to something",
> }
>
> The correction above is according to entries:
> #define FILE_ACTION_ADDED 0x00000001
> #define FILE_ACTION_REMOVED 0x00000002
> #define FILE_ACTION_MODIFIED 0x00000003
> #define FILE_ACTION_RENAMED_OLD_NAME 0x00000004
> #define FILE_ACTION_RENAMED_NEW_NAME 0x00000005
> in ..\PlatformSDK\Include\WinNT.h
>
> Claudio Grondi

Thanks. I've updated the site.

TJG

[Tim Golden]
>> On the offchance that you haven't seen it, you might
>> look at this:
>>
>> http://timgolden.me.uk/python/win32_...rectorychanges

[Claudio Grondi]
> It seems, that it will be necessary to use some logic based on the
> sequence of events to exactly detect rename and move changes done to
> files/directories, but in principle it is the best approach I know about
> yet.
>
> By the way:
> Is there something similar/same available for Linux?

I've never used them, but I seem to think there are a couple
of similar things for Linux, based on FAM or inotify:

(result of Googling)

http://python-fam.sourceforge.net/
http://www.gnome.org/~veillard/gamin/python.html
http://rudd-o.com/projects/python-inotify/

YMMV
TJG

Tim Golden wrote:
> Claudio Grondi wrote:
>
>> I am aware, that it is maybe the wrong group to ask this question, but
>> as I would like to know the history of past file operations from
>> within a Python script I see a chance, that someone in this group was
>> into it already and is so kind to share here his experience.
>>
>> I can't believe, that using NTFS file system in Microsoft Windows 2000
>> or XP it is not possible to track file events as:
>>
>> - updating/modifying of an existing file/directory
>> - deleting an existing file/directory
>> - creating a new file/directory
>> - _moving_ an existing file/directory (should _NOT_ be covered by the
>> event duo of deleting an existing and creating a new file/directory)
>>
>> Any hints towards enlightenment?
>>
>> Claudio Grondi
>
>
> On the offchance that you haven't seen it, you might
> look at this:
>
> http://timgolden.me.uk/python/win32_...rectorychanges
>
>
> but since it doesn't fulfil your criterion of *not*
> representing renames by a delete and an add, it may
> well not be suitable. Apart from that, I think it does
> what you want.
>
> TJG
Here a small update to the code at
http://timgolden.me.uk/python/win32_...rectorychanges
:

ACTIONS = {
1 : "Created",
2 : "Deleted",
3 : "Updated",
4 : "Renamed from something"
5 : "Renamed to something",
}

The correction above is according to entries:
#define FILE_ACTION_ADDED 0x00000001
#define FILE_ACTION_REMOVED 0x00000002
#define FILE_ACTION_MODIFIED 0x00000003
#define FILE_ACTION_RENAMED_OLD_NAME 0x00000004
#define FILE_ACTION_RENAMED_NEW_NAME 0x00000005
in ..\PlatformSDK\Include\WinNT.h

Claudio Grondi