deleted files

Discussion in 'Computer Security' started by Stuart Miller, Nov 14, 2006.

  1. As I understand NTFS, once a file is deleted then removed from the recycle
    bin it is not recoverable by ordinary means.
    Therer may be utilities out there to do that, and certainly after a defrag
    it will likely be gone. No problem there, but are files recoverable?

    Problem is the other way around, here. I deleted a set of files, emptied
    recycle bin, deleted more the next day and emptied again. Today the first
    set of files were back in the folder that had been deleted from. Only
    explanation I can think of is they there was a brief power outage that day,
    so the system went down and restarted. I remember win98 and 95 used to do
    these automatic registry restores periodically, but I didn't think that
    deleted files and former folder contents were stored in the registry.

    From a security point of view - does this mean that deleted files, with
    recycle bin emptied, are not really deleted?

    Thanks

    Stuart
    Stuart Miller, Nov 14, 2006
    #1
    1. Advertising

  2. Stuart Miller

    Jim Guest

    Stuart Miller came up with this when s/he headbutted the keyboard a moment
    ago in alt.computer.security:

    > As I understand NTFS, once a file is deleted then removed from the recycle
    > bin it is not recoverable by ordinary means.


    Sort of. Read below.

    > Therer may be utilities out there to do that, and certainly after a defrag
    > it will likely be gone. No problem there, but are files recoverable?
    >
    > Problem is the other way around, here. I deleted a set of files, emptied
    > recycle bin, deleted more the next day and emptied again. Today the first
    > set of files were back in the folder that had been deleted from. Only
    > explanation I can think of is they there was a brief power outage that

    day,
    > so the system went down and restarted. I remember win98 and 95 used to do
    > these automatic registry restores periodically, but I didn't think that
    > deleted files and former folder contents were stored in the registry.
    >


    they're not - FAT file systems have two FAT tables. One's a backup; not that
    that does much good, because it's essentially a mirror of the primary.
    NTFS uses journaling and backgrounding to give the illusion of a faster
    filesystem. Now, the backgrounding (which makes heavy use of the large
    caches found on very modern drives) isn't much use to you if there's a
    power cut or if you're running a PVR on your system (you need realtime
    writing to disk - no caching), however the journal is where you become
    unstuck from a security viewpoint.

    > From a security point of view - does this mean that deleted files, with
    > recycle bin emptied, are not really deleted?
    >


    answer: deleting a file on an NTFS filesystem merely removes it from the
    current journal. The file is still physically on the drive. The allocated
    space is flagged for overwriting and bumped to the back of the write queue,
    where it is forgotten about, until it reaches the front of the write queue
    and is overwritten. On an average system, this can take /months/
    considering light usage (browsing, writing documents, etc). On a heavy-use
    system (such as a PVR) this can take a few days. Or even a few hours. Even
    then the chances of that space being entirely overwritten in order are
    fairly remote, so something of the original file will remain - very likely
    enough to use as evidence after a forensic search.

    To expand: a normal format does not erase the contents of a partition.
    Neither does repartitioning. All these do is to rewrite the partition and
    FAT tables. The data area is basically untouched until it comes to actually
    writing data to it with pointers from whatever filesystem resource locator
    you're using (NTFS, FAT, whatever). The only sure way of destroying data
    beyond recoverability (apart from physically destroying the disk) is to
    make multiple passes over hte drive with military-grade hard disk lowlevel
    formatting software*.

    *old Conner AT drives (of which I still have a few) had a notice on them
    which said "WARNING: DO NOT LOW LEVEL FORMAT". This was all to do with the
    fact that if you lowleveled the drive you had to rebuild it using the CHS
    parameters for that specific unit. Get it wrong, you had a brick (or at the
    very least, one which was misconfigured in such a way that you saw a
    dropping sector every couple of seconds - rapidly rendering the drive
    unusable). HD controllers nowadays are smart enough to rebuild themselves
    after a LLF, so it's pretty safe to LLF a drive maybe half a dozen times
    during its lifetime (being a very intensive operation, modern drives get
    bloody hot during a LLF, so definitely not recommended without ample
    cooling!)

    > Thanks
    >
    > Stuart


    --
    -*- Linux Desktops & Clustering Solutions -*- http://dotware.co.uk
    -*- Registered Linux user #426308 -*- http://counter.li.org
    -*- Linux is like a wigwam: no Windows, no Gates, and Apache inside.
    -*- <discl mode="Boilerplate" />
    Jim, Nov 14, 2006
    #2
    1. Advertising

  3. Jim wrote:

    > they're not - FAT file systems have two FAT tables. One's a backup; not that
    > that does much good, because it's essentially a mirror of the primary.
    > NTFS uses journaling and backgrounding to give the illusion of a faster
    > filesystem. Now, the backgrounding (which makes heavy use of the large
    > caches found on very modern drives) isn't much use to you if there's a
    > power cut or if you're running a PVR on your system (you need realtime
    > writing to disk - no caching), however the journal is where you become
    > unstuck from a security viewpoint.


    I'd worry much more about the MFT Mirror (same as FAT table backup). The
    journal is easily cleared by filling it up with writing some bogus data.

    >> From a security point of view - does this mean that deleted files, with
    >> recycle bin emptied, are not really deleted?
    >>

    >
    > answer: deleting a file on an NTFS filesystem merely removes it from the
    > current journal. The file is still physically on the drive. The allocated
    > space is flagged for overwriting and bumped to the back of the write queue,
    > where it is forgotten about, until it reaches the front of the write queue
    > and is overwritten. On an average system, this can take /months/
    > considering light usage (browsing, writing documents, etc). On a heavy-use
    > system (such as a PVR) this can take a few days. Or even a few hours. Even
    > then the chances of that space being entirely overwritten in order are
    > fairly remote, so something of the original file will remain - very likely
    > enough to use as evidence after a forensic search.


    However, with competent tools like SDelete or Eraser you can clear all free
    disk space, all free MFT entries and the journal. Only filenames of deleted
    files with pose a problem, and therefore one should at least rename the
    files before deletion (those and many other tools do that automatically).

    > To expand: a normal format does not erase the contents of a partition.
    > Neither does repartitioning. All these do is to rewrite the partition and
    > FAT tables. The data area is basically untouched until it comes to actually
    > writing data to it with pointers from whatever filesystem resource locator
    > you're using (NTFS, FAT, whatever). The only sure way of destroying data
    > beyond recoverability (apart from physically destroying the disk) is to
    > make multiple passes over hte drive with military-grade hard disk lowlevel
    > formatting software*.


    What about just one pass (because it simply *is* sufficient) with freely
    available tools?

    > HD controllers nowadays are smart enough to rebuild themselves
    > after a LLF, so it's pretty safe to LLF a drive maybe half a dozen times
    > during its lifetime (being a very intensive operation, modern drives get
    > bloody hot during a LLF, so definitely not recommended without ample
    > cooling!)


    Today a low-level format only consists of filling the raw disk with zeros,
    but not rebuilding the internal organization structure. And wenn, you can
    easily do that with a 'dd if=/dev/zero of=/dev/hdX bs=1m' on your own.
    Sebastian Gottschalk, Nov 14, 2006
    #3
  4. Stuart Miller

    erewhon Guest


    > What about just one pass (because it simply *is* sufficient) with freely
    > available tools?


    Do you make this stuff up as you go along?!

    One pass is NOT sufficient to remove the data to a point at which it cannot
    be recovered. It will fool most data reading tools, but certainly not
    systematic, low level magnetic analysis of the platter. For that you need to
    sufficiently break up the magnetic storage to the point there it is
    indistinguishable from random fluctuations.

    One pass might stop your basic data scavenger - it sure as hell won't stop
    the pro's.
    erewhon, Nov 15, 2006
    #4
  5. Stuart Miller

    Jim Guest

    erewhon came up with this when s/he headbutted the keyboard a moment ago in
    alt.computer.security:

    >
    >> What about just one pass (because it simply *is* sufficient) with freely
    >> available tools?

    >
    > Do you make this stuff up as you go along?!
    >
    > One pass is NOT sufficient to remove the data to a point at which it

    cannot
    > be recovered. It will fool most data reading tools, but certainly not
    > systematic, low level magnetic analysis of the platter. For that you need

    to
    > sufficiently break up the magnetic storage to the point there it is
    > indistinguishable from random fluctuations.
    >
    > One pass might stop your basic data scavenger - it sure as hell won't stop
    > the pro's.


    hence my mention of military grade formatting tools - which destroy the data
    to the point where it is practically impossible to recover anything even
    with the cleanest of cleanrooms and all the time in the world - assuming
    you even knew precisely what you were looking for.

    I'm a pro.
    --
    -*- Linux Desktops & Clustering Solutions -*- http://dotware.co.uk
    -*- Registered Linux user #426308 -*- http://counter.li.org
    -*- Linux is like a wigwam: no Windows, no Gates, and Apache inside.
    -*- <discl mode="Boilerplate" />
    Jim, Nov 15, 2006
    #5
  6. "Jim" <> wrote in message
    news:B2s6h.10238$...
    >> Problem is the other way around, here. I deleted a set of files, emptied
    >> recycle bin, deleted more the next day and emptied again. Today the first
    >> set of files were back in the folder that had been deleted from. Only
    >> explanation I can think of is they there was a brief power outage that

    > day,
    >> so the system went down and restarted. I remember win98 and 95 used to do
    >> these automatic registry restores periodically, but I didn't think that
    >> deleted files and former folder contents were stored in the registry.
    >>

    >
    > they're not - FAT file systems have two FAT tables. One's a backup; not
    > that
    > that does much good, because it's essentially a mirror of the primary.
    > NTFS uses journaling and backgrounding to give the illusion of a faster
    > filesystem. Now, the backgrounding (which makes heavy use of the large
    > caches found on very modern drives) isn't much use to you if there's a
    > power cut or if you're running a PVR on your system (you need realtime
    > writing to disk - no caching), however the journal is where you become
    > unstuck from a security viewpoint.
    >
    >> From a security point of view - does this mean that deleted files, with
    >> recycle bin emptied, are not really deleted?
    >>

    >
    > answer: deleting a file on an NTFS filesystem merely removes it from the
    > current journal. The file is still physically on the drive. The allocated
    > space is flagged for overwriting and bumped to the back of the write
    > queue,
    > where it is forgotten about, until it reaches the front of the write queue
    > and is overwritten. On an average system, this can take /months/
    > considering light usage (browsing, writing documents, etc). On a heavy-use
    > system (such as a PVR) this can take a few days. Or even a few hours. Even
    > then the chances of that space being entirely overwritten in order are
    > fairly remote, so something of the original file will remain - very likely
    > enough to use as evidence after a forensic search.


    I'm ok with the forensic matters of total destruction of traces - this is
    more a personal thing.
    Anybody can come in and examine all of my computers all they want - there is
    nothing of great importance.
    But this was a collection of personal and family things for my personal
    journals, and when I was finished I deleted the working copies. Nobody else
    here has the ability or the interest to try to 'undelete' files. I just
    expected that XP would leave them deleted, and not restore them without my
    knowledge or consent.

    Is there an easy way to force writing of the cache? I recall a discussion of
    this in one of the linux newsgroups a few months ago.

    OK, so perhaps it was just bad timing. But now all the important stuff stays
    on the linux machines. Or does ext3 have the same issues?

    I will test this when I have a bit of time - to see if I can duplicate the
    results.

    Thanks for a most useful explanation.

    Stuart
    Stuart Miller, Nov 15, 2006
    #6
  7. Stuart Miller

    kurt wismer Guest

    Jim wrote:
    > erewhon came up with this when s/he headbutted the keyboard a moment ago in
    > alt.computer.security:
    >
    >>> What about just one pass (because it simply *is* sufficient) with freely
    >>> available tools?

    >> Do you make this stuff up as you go along?!
    >>
    >> One pass is NOT sufficient to remove the data to a point at which it

    > cannot
    >> be recovered. It will fool most data reading tools, but certainly not
    >> systematic, low level magnetic analysis of the platter. For that you need

    > to
    >> sufficiently break up the magnetic storage to the point there it is
    >> indistinguishable from random fluctuations.
    >>
    >> One pass might stop your basic data scavenger - it sure as hell won't stop
    >> the pro's.

    >
    > hence my mention of military grade formatting tools - which destroy the data
    > to the point where it is practically impossible to recover anything even
    > with the cleanest of cleanrooms and all the time in the world - assuming
    > you even knew precisely what you were looking for.
    >
    > I'm a pro.


    if you're a pro then i guess you know that the military physically
    destroys any media that has ever contained data above a certain security
    classification because they know that nothing you do with software will
    achieve the data sanitation you allude to above...

    --
    "it's not the right time to be sober
    now the idiots have taken over
    spreading like a social cancer,
    is there an answer?"
    kurt wismer, Nov 15, 2006
    #7
  8. Jim wrote:

    >> It will fool most data reading tools, but certainly not systematic,
    >> low level magnetic analysis of the platter.


    I'm aware of that. But interpolating from the increase of storage capacity
    and the change of encoding methods one can safely conclude that at best two
    overwrites will definitely not leave any traces on the low magnetic layer
    of view.

    > hence my mention of military grade formatting tools - which destroy the data
    > to the point where it is practically impossible to recover anything even
    > with the cleanest of cleanrooms and all the time in the world - assuming
    > you even knew precisely what you were looking for.


    Well, then you have to do two such overwrites. Simple. OK, and for SCSI,
    you have to read the defective sector list, of course.
    Sebastian Gottschalk, Nov 15, 2006
    #8
  9. Stuart Miller wrote:

    > Is there an easy way to force writing of the cache?


    Yes. 'sync' comes with every Unix, and there are various versions of
    sync.exe for Windows.

    But the better way is to overwrite data while bypassing the cache, because
    it doesn't fill the cache with bogus data, which would impact performance.

    > I recall a discussion of this in one of the linux newsgroups a few months ago.


    Well, why don't read the manual of a good tool like SDelete for a detailed
    technical discussion of the issue?

    > OK, so perhaps it was just bad timing. But now all the important stuff stays
    > on the linux machines. Or does ext3 have the same issues?


    Almost all filesystems have that issue. And especially ext3, also it also
    does journaling.
    Sebastian Gottschalk, Nov 15, 2006
    #9
  10. "Sebastian Gottschalk" <> wrote in message
    news:...
    > Stuart Miller wrote:
    >
    >> Is there an easy way to force writing of the cache?

    >
    > Yes. 'sync' comes with every Unix, and there are various versions of
    > sync.exe for Windows.
    >
    > But the better way is to overwrite data while bypassing the cache, because
    > it doesn't fill the cache with bogus data, which would impact performance.
    >


    I see two convenient ways to do this

    1 - open the text file, cut / delete the contents, add some garbage data,
    and save
    repeat cycle to drop the backup files

    2 - delete file as usual, create a new file with same name in same folder,
    write some garbage to it, save, exit wp, delete

    Any significant difference to these approaches?

    >> I recall a discussion of this in one of the linux newsgroups a few months
    >> ago.

    >
    > Well, why don't read the manual of a good tool like SDelete for a detailed
    > technical discussion of the issue?
    >

    Thanks - will do

    It wasn't so much the paranoia of ensuring files to be unreadable, it was
    just the surprise of finding them back

    >> OK, so perhaps it was just bad timing. But now all the important stuff
    >> stays
    >> on the linux machines. Or does ext3 have the same issues?

    >
    > Almost all filesystems have that issue. And especially ext3, also it also
    > does journaling.


    Stuart
    Stuart Miller, Nov 15, 2006
    #10
  11. Stuart Miller wrote:

    >>> Is there an easy way to force writing of the cache?

    >>
    >> Yes. 'sync' comes with every Unix, and there are various versions of
    >> sync.exe for Windows.
    >>
    >> But the better way is to overwrite data while bypassing the cache, because
    >> it doesn't fill the cache with bogus data, which would impact performance.
    >>

    >
    > I see two convenient ways to do this
    >
    > 1 - open the text file, cut / delete the contents, add some garbage data,
    > and save
    > repeat cycle to drop the backup files


    This won't overwrite the data. They might end up in freshly allocated
    clusters while deallocating the previous one. And they'll end up in the
    journal. And in the cache.

    > 2 - delete file as usual, create a new file with same name in same folder,
    > write some garbage to it, save, exit wp, delete


    This will leave the old filename in the MFT/FAT.

    > Any significant difference to these approaches?


    Yes: they don't work.
    Sebastian Gottschalk, Nov 15, 2006
    #11
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. =?Utf-8?B?T3Bwcw==?=

    recovery of MS antispyware files that were deleted?

    =?Utf-8?B?T3Bwcw==?=, May 22, 2005, in forum: Microsoft Certification
    Replies:
    2
    Views:
    380
  2. =?Utf-8?B?QXJuZWw=?=

    deleted files

    =?Utf-8?B?QXJuZWw=?=, Dec 1, 2005, in forum: Microsoft Certification
    Replies:
    2
    Views:
    848
    =?Utf-8?B?Zm1zbWNzZQ==?=
    Dec 8, 2005
  3. John Holmes

    Re: Are deleted pictures REALLY deleted?

    John Holmes, Dec 30, 2008, in forum: Computer Support
    Replies:
    7
    Views:
    645
    §ñühw¤£f
    Dec 31, 2008
  4. Replies:
    3
    Views:
    480
    General Patron
    Jan 1, 2009
  5. Bucky Breeder

    Re: Are deleted pictures REALLY deleted?

    Bucky Breeder, Dec 31, 2008, in forum: Computer Support
    Replies:
    0
    Views:
    610
    Bucky Breeder
    Dec 31, 2008
Loading...

Share This Page