The Old Joel on Software Forum: Part 4 (of 5) - securing docs & related issues

securing docs & related issues

I want to write/find an app that will lock down (almost) everything using a password of my choice. I am talking about user created files, not OS files or applications themselves.

I've been digging around in cryptography books for a while, and I think I got the encryption part figured out. I wasn't able to find anything on writing secure apps though.

My immediate issues are the following:

1) Once the files are encrypted, I need to be able to access them somehow. The first difficulty is how do I search through them. The "search" thingie that comes with Windows will be useless. I need a grep like utility that can search through encrypted files given the password. Or a Winzip like tool that can show me what's in the encrypted files, and can search in there. I looked for such a thing on sourceforge and freshmeat, but nothing showed up immediately. I am still digging. In the meantime, the best way I can think of writing such a tool is to enrypt the search expression and try to match it to encrypted files so that nothing gets decrypted during the search... Sound reasonable? Not the fastest way, but sure sounds secure to me.

2) Another security concern is when a document needs to be decrypted to be worked on/displayed. Especially if the document is a PDF file or an MS Word file which have special formatting, then I have to decrypt the file and pass it onto the dedicated app to display/edit. This is not the greatest thing since the info is going to be sitting naked in memory somewhere. Are there any ways around this?

3) Lastly, where the heck do I keep my password?! Assuming that I created a console-like app which can access these files, it will need to know about the password or it'll keep asking for it every time it access a file. Leaving the password around (in registry or a file) doesn't sound very secure. Maybe it could keep the password around per session and then toss it out when the user quits the app.
Any ideas?

Any answers to the above issues would be appreciated. I was unable to find any books merging cryptography and practical programming either. If you know of one which will address my problems above and more, please let me know.

Thanks for tuning in!

entell
Monday, April 12, 2004

axcrypt, look on sourceforge.net

Li-fan Chen
Tuesday, April 13, 2004

Quick answer: check out gnu privacy guard: http://www.gnupg.org

More:

In regards to writing secure applications: that's usually (as far as I am aware) a seperate issue from secure storage. There's an addison-wesly book on secure code, but I've not gotten around to it myself. There's also a good primer on http://dwheeler.com

You touch on the big issues with good encryption now. The issue with encryption nowadays is not the security of the algorithm itself, as the keyspaces are plenty big for most purposes. The issue is, like you observed, the people who leave passwords around and other such hacks. Man-in-the-middle is used less than having a secure transport to a fraudulent back end.

First: I think it would be best if you rethink your requirements. You want it to be easy to access, hassle-free, secure, and such. There may be a product that good out there, and if there is I'd love to know about it. But I am not aware of anything that does all that you want.

A solution which I think meets most of your criteria reasonable closely: Put your data in an encrypted filesystem. Search for terms like "encrypted loopback" or "encrypted home directory". This has been implemented on Linux, and may be available on other systems. If you need it on windows, it could be gotten to work through a network share, I imagine.

Keeping your password: This will usually use a key file to decrypt the files. Sometimes (read: nearly always) that key is encrypted with a password. But, you could put the key file on a disk.

You will always have to display the data to the user at some point, which mean's it's not perfect, and there's never going to be anything protecting you once the file's on the screen.

It comes down to this: more paranoia means more hassle. But I think the above is a reasonable solution.

Oh, and don't forget to burn backup copies of your key and encrypted data and store them in a reasonable manner (seperate safety deposit boxes, whatever). Nothing will save you if you lose the key ;)

Mike Swieton
Tuesday, April 13, 2004

I found a few utility programs that encrypt files/folders, and decrypt them based on a password or a keyfile. They are mostly command line utilities though.

AxCrypt is nice but it is once again a command line utility. I'd prefer something that is a DLL so I don't have to access the functionality by calling command line utilities from within the program.

Anyways, I'll dig a little deeper and see where the rabbit hole goes.

Thanks for your input.

entell
Tuesday, April 13, 2004

> I want to write/find an app that will lock down (almost) everything using a password of my choice.

Windows can encrypt files for you, but I don't know how that works.

> In the meantime, the best way I can think of writing such a tool is to enrypt the search expression and try to match it to encrypted files so that nothing gets decrypted during the search... Sound reasonable?

It doesn't sound reasonable to me, but it depends on your encryption algorithm. For example: if I have plaintext "ABCD" that encrypts to "HWIX", if my search term is "BC", I wouldn't expect the encryption algortihm to encrypt "BC" as "WI".

> This is not the greatest thing since the info is going to be sitting naked in memory somewhere. Are there any ways around this?

I don't see a work-around, except to secure access to the RAM while it contains that data.

> Assuming that I created a console-like app which can access these files, it will need to know about the password or it'll keep asking for it every time it access a file.

You could write the app as a command-shell (or, as a service), which asks for the password once (when you run it for the first time after each login), and then stays in memory with your password ... so you only need to enter your password once per login.

> I'd prefer something that is a DLL so I don't have to access the functionality by calling command line utilities from within the program.

You could make the thing more transparent by implementing it as a file system driver (maybe called a "filter driver"), but that would be harder to write.

Christopher Wells
Tuesday, April 13, 2004

>> In the meantime, the best way I can think of writing
>>such a tool is to enrypt the search expression and try to
>> match it to encrypted files so that nothing gets
>> decrypted during the search... Sound reasonable?
>>
> It doesn't sound reasonable to me, but it depends on
> your encryption algorithm. For example: if I have
> plaintext "ABCD" that encrypts to "HWIX", if my search
> term is "BC", I wouldn't expect the encryption algortihm
> to encrypt "BC" as "WI".

Christopher,

I got the idea from Unix. In Unix, when you enter your username and password, the password you enter gets garbled up into a hash and compared to the stored password in the hash format. Therefore your password never gets decrypted. One-way hashes work great for that kind of application, but for what I am trying to accomplish it might not be enough or even applicable.

Having things sit in memory unprotected is probably something I will have to live with, but perhaps at least I could purge things once the user is done. It is probably harder to purge the memory area of another app though. Say you have an encrypted PDF file. When the user wants to view it, you have to decrypt it and call Acrobat reader. Once acrobat reader is done with the file, I have to find the memory area where the file was and purge it somehow. I don't know how easy it is to do the finding although purging would probably be a lot simpler.

entell
Tuesday, April 13, 2004

> One-way hashes work great for that kind of application, but for what I am trying to accomplish it might not be enough or even applicable.

I guess (I don't know encryption) that an encryption algorithm where "encrypt(substring(string)) == (substring(encrypt(string))" would be needed if you're searching for substrings within the file ... and that this would be a "dictionary" encryption algorithm like ROT13, easy to crack.

> It is probably harder to purge the memory area of another app though.

If you're using Windows NT/2K/XP/2003 (not using Windows 95/98/ME) then I think that the O/S should take care of guaranteeing the privacy of RAM: all you would need to do is to unload the Acrobat Reader process which you launched to display the file ... because when the process exits, then the O/S will clean up the process' RAM (ensure that the RAM is reinitialized before it's given to another process).

To me the harder part is that I don't know how you would pass the decrypted document to Acrobat using RAM ... usually Acrobat would expect to find its document on disk ... is there such a thing as a RAM disk? I don't know whether you can do something with memory-mapped files: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/fileio/base/file_mapping.asp

This is the kind of reason why a filter driver would be better (presents a decrypted view of a disk file to an application that reads from disk), but it is harder to implement (it's a device driver).

Christopher Wells
Tuesday, April 13, 2004

The fact that you are asking these questions at all tells me that you should NOT be doing this work. DO NOT ATTEMPT TO ROLL YOUR OWN CRYPTOGRAPHY SYSTEMS. As you've discovered, the math is easy. Key management, etc, etc, etc, is hard. You WILL write an insecure system if you try, so don't try. Find an off-the-shelf solution written by people with years of experience in the crypto space and buy it -- it'll be a lot cheaper and safer.

Or, let me put it this way -- I'm an experienced professional programmer who's written books on security systems and developed cryptography tools for programming languages, and I would not consider myself sufficiently knowledgable to take on what you're attempting as anything other than a fun research project. If I wrote something like that for myself, I would not rely on it for my own use and I CERTAINLY would not suggest that any customer of mine rely upon it.

Eric Lippert
Tuesday, April 13, 2004

Eric,

Thank you for your post.

What I have in my hands is definitely a pet project. Nothing commercial or anything although I want to incorporate it into something else.

As Eric and others mentioned, encrypting files is the easiest part. There are other security holes however. For example, I was thinking I want to encrypt my e-mail files. Then I thought, wait a second... That e-mail is being downloaded insecurily over the internet from my ISP's mail server to my PC. Unless I can plug that hole, encrypting them once they get to my PC is not as effective. There are many other security issues like this one.

I realized that my goal is to provide privacy. A complete security requires many levels of protection some of which I cannot provide by a single application running locally on a PC. All I want to do is provide some basic security measures. For that I could get a commerical product as you suggested, but being the geek I am, I like to know what's under the hood. When it comes to security, most commercial products end up providing nothing more than a false sense of security unfortunately.

That being said, I am looking into using an open-source GPL library. I have read in many articles/books that it is not the algorithm that makes a secuity system good. Therefore, I'd rather use an open source library which is well tested and known to do what it is supposed to do. As the user, I'll (somehow) make sure my keys are not in the wrong hands.

I found the following 2 to be of interest:

* Crypto++™ Library 5.1
(http://www.eskimo.com/~weidai/cryptlib.html)

* cryptlib
(http://www.cs.auckland.ac.nz/~pgut001/cryptlib/)

Do you have any experience with either of these libraries?
Do you know of any other such libraries?

entell
Tuesday, April 13, 2004

Entell,

I would like to second Eric's suggestions and hope you take them to hart. The worst possible thing you could do is try to roll your own setup. It is guaranteed to be bad. Really.

Unlike other "coding stuff", security is unforgiving. Look at it this way: when you are building a system, there will be many things that are suboptimal. This is usually not a major problem. You used XML files where an RDB could have been more performant? You decided on Java where Phyton might have been a better match? OK, but the system runs.
Security is unforgiving. Every single screwup gets you. Anything you didn't know can kill your whole setup.
There is also almost never a need to roll your own. Every OS comes with decent security API's. Don't for a second believe that Open Source or commercial has any simple correlation with quality. Critical implementation flaws are discovered in some of the most high profile Open Source implementations (OpenSSL anyone?), as well as in commercial stuff.

By all means if you are interested dive in and start to learn more. Maybe something like http://safari.oreilly.com/?XmlId=0-596-00242-4 could be a good general introduction to the subject for you.

If it is simply email security you are after, use a mail client that supports S/MIME or PGP. If you are after protecting a lott of files, use an Encrypting File Sytem etc. but before you are going to do all that, first ask yourself the most important questions: What is it you want to protect, what do you want to protect it against and what is the price (not only in $ but also in usability etc.) you are willing to pay.

Just me (Sir to you)
Tuesday, April 13, 2004

I am a flaming douchebag.

Mike Swieton
Tuesday, April 13, 2004

Just me,

I agree not to create my own security scheme. That's a waste of time to begin with when there are many existing, tested schemes out there.

I think I'll go with a crypto library at this point though. I mentioned 2 such libraries in my previous post. Any experience with either one or another library perhaps?

entell
Tuesday, April 13, 2004