Modern filesystems tend to do journaling, don't overwrite in place, or contain RAID compressed backups or some such, so traditional file shredding tools aren't effective. I want to remove all traces of encrypted files. If people scan the drive for old, unused blocks and discover random-looking data, it doesn't matter. For my purposes the important thing is that once the file is deleted, none of the metadata remains. In simple terms, ideally once deleted it should be impossible to infer that any file ever existed, and certainly any part of the disk which stored its name, size, mtime, etc should be multiply-overwritten as in traditional shred methods. What tools and filesystems should I use?

No offense, encrypting data to prevent theft/espionage is one thing, but there is usually no reason to be paranoid about evidence that there might ever have been an encrypted file on your system, except if it's for an illegal cause.

There is no harm if someone steals your computer that contains a deleted file aptly named "mycompanysecrets.tc" on it. If the encryption is worth as much as 5 cents, they won't be able to get to the contents without knowing the key/password. So, unless you fear being kidnapped by the DoD who will force you to reveal your password so they can discover your plans to explode a nuclear bomb in Washington DC, you're safe.

However, if you STILL insist that all metadata must be overwritten, there is only one way (and luckily it's an easy way). You don't even need a program for that, Windows explorer can do it. Take any arbitrary small file, put it into a folder for convenience, give it a nice long name, and make a copy of the file. Select the file and the copy, and make a copy. Rinse and repeat.
After 20 iterations, you will have created a million files. If you feel like it, do a few more iterations, but 20 should be plenty.
You will now have overwritten the journal and most of the free space in the MFT (or whatever equivalent name your file system uses for metadata store).
Reboot, just to be sure all caches have been synced to disk. Now you can delete the folder containing the million files.

Note that if you use a solid state disk, it is impossible to delete data or metadata in any predictable way because of constant sector remapping ("write levelling"). Having said that, normal hard disks will do remapping in case of hardware failures too, so there is no 100% safe way of deleting something that was written to a device once, other than taking a wrench or a heavy hammer.

Quote: Original post by samoth
No offense, encrypting data to prevent theft/espionage is one thing, but there is usually no reason to be paranoid about evidence that there might ever have been an encrypted file on your system, except if it's for an illegal cause.

right, because the cops are altruistic and care about people. Yeah right, all they care about is getting a conviction, and they will skew whatever circumstantial evidence they can get.

privacy is a right, and people shouldn't need a reason to exercise any right.

No offence, samoth, but that solution isn't exactly practical. It could take a very long time (especially if it's iterated) during which the computer is unusable and what's the guarantee it will actually cover all the possible byte locations?

No, this really needs to be targetted.

FWIW, this problem arose since I'm using the filesystem as a semi-structured non-relational database, the resulting graph has a great deal of meaning in my app.

Quote: Original post by capn_midnight
right, because the cops are altruistic and care about people. Yeah right, all they care about is getting a conviction, and they will skew whatever circumstantial evidence they can get.

privacy is a right, and people shouldn't need a reason to exercise any right.

That's correct, and I'll be the last person saying that privacy is not a right. However, the mere presence of an encrypted file that was deleted does not mean anything.
If you are Joe Normal and the file is not incidentially named "phatwarez" or "mychildporncollection" there is no reason why the mere presence of an encrypted file is any worse than the presence of a letter to your mother.
Also note that a thief stealing your computer and governmental authorities seizing it are two different things. You encrypt your data to limit damages in the former case. If it comes to the latter case, you're in bigger trouble either way, some old data in some deleted file won't matter much then.

Encrypting data is a normal thing, every non-trivial company encrypts anything that could possibly be of any value (usually the entire partition), there's nothing shady about this, and thus you need not hide it. In some companies, any corporate data placed on USB sticks or other removable media must be encrypted too, if you want to keep your job.
Therefore, there's no real reason to erase the metadata, it is entirely legitimate and normal to have encrypted files, and unless you have drawn the attention of The Powers That Are Above The Law upon you, there is little you have to be afraid of (and it that case, you're screwed anyway).

Plus, one usually creates a virtual volume that contains the protected files with a block size that isn't necessarily the same as the outer disk's sector size. This virtual filesystem needs metadata by its own, which is stored as "normal" data on the phsyical disk. If a file is deleted (on the real hard disk), its sectors will eventually be overwritten, more or less randomly. Think of things like log files or Windows file optimisation, if nothing else.
Losing a single sector that way will render many sectors in the encrypted volume unreadable, and with any luck they belong to the MFT.
Thus, unless the CIA incidentially seizes your computer 2 hours after you deleted your plans to murder the president, chances are good that even if they force you to reveal the password, they might not be able to restore much of the data.
And if you're afraid of that happening, start a defrag after deleting the file :-)

Quote: Original post by spraff
FWIW, this problem arose since I'm using the filesystem as a semi-structured non-relational database, the resulting graph has a great deal of meaning in my app.

Solution: Instead of encrypting individual files, put the directory structure including the individual files into the encrypted volume.

All an attacker will be able to see is that there is "some big encrypted file" -- no password means no idea what's inside.
Also, it is probably more efficient CPU-wise to encrypt one volume than to encrypt individual files (key setup etc.).