Fog Creek Software
Discussion Board




transport mechanism

i have just been tasked with coming up with a solution to allow our clients to pass us files securely.  here are some things i'm considering:

1.  just create an ftp site, and give every client a login.  they upload files to their home directory, and can check back later for a response, that they'd then download.  they can use whatever ftp client they want.  definitely quickest solution for me.  management headache (500+ clients)

2.  make some kind of webpage (HTTPS) that will ask them for their credentials and once logged in give a bit more guidance in uploading their files.  probably use ASP.NET's file upload portion.  I might run into problems with the size limit, possibly have to look into third party software.  still a management headache (this might be unavoidable...)

Any other suggestions?  I'm open to any thoughts and criticisms.  thanks.

nathan
Friday, August 22, 2003

We go the FTP route. If they're sending you data files, they're smart enough to have someone in the IT department figure out how to FTP files to you -- most of the time they're smart enough, but sometimes not.

Be advised, that unless you've got some lower layer protection (VPN, IPSec, whatever) that you're wide open. There is no mechanism in the FTP standard for encryption of the data files, and things like username/logon/password etc. are sent across the wire in unencrypted plain text. For this reason, we've gone with the following:

We've required that users PGP (encryption) the files prior to sending, but sometimes they forget and send us some plain text files (unencrypted). This is AVeryBadThing as these files contain confidential info (medical claims). For this reason, we've blocked access to the FTP server except access through our VPN. So, now we've got 2 layers of encryption -- the packets are encrypted by the underlying protocol (IPSec), and when the individual files are reconstituted into a file, what gets spit out the other side is a PGP encrypted file.

This works well for our data security needs, however it's sometimes a royal pain in the arse when dealing with a new customer -- gotta setup connectivity, VPN, certificates, PGP key management and all that jazz. Most people are fairly clueful about it, but once in a while you get a real bozo and you can eat half the day getting the first file transferred. YMMV

If you want to make things easier, you'll probably want to go the HTTPS route.

Sgt. Sausage
Friday, August 22, 2003

just got more information... looks like the largest files will be 1.5 MB, which is well within the ASP.NET file upload limits... HTTPS is looking better and better... 

Though using PGP is still a good idea...

nathan
Friday, August 22, 2003

What about SSH?

Philo

Philo
Friday, August 22, 2003

Trinity "nuked" ssh in he virtual world...

coo coo cachoo
Friday, August 22, 2003

SSH is a good idea. There are several clients available for both SCP and SFTP. WS_FTP handles it. There is also WinSCP, PuTTY pscp; Transmit on the Mac, and more.

But if you don't have a Unix server it's not an option. On the other hand, it might be worth setting up a dedicated server given the number of users you have.

Nate Silva
Friday, August 22, 2003

FTP is what we use with our vendors. Works great. Everybody knows it. Can be scripted with minimal effort in both Windows and Unix. Why develop a solution when a solution exists in a simple form.

m
Friday, August 22, 2003

"Why develop a solution when a solution exists in a simple form. "

FTP is not secure.
You can make it so with some of the mechanisms described above, but they mostly lose the simplicity/there are zillions of existing clients benefits.

Also you have to poll the server or something to determine files; a POST (or similar) based solution can trigger an event easily. (well, you could write your own FTP server or have filesystem events or... other more-complex-than-nothing solutions). Also a POST style solution doesn't need filenames.

You haven't mentioned what the real requirements are. If it's an occasional file, I'd definately go with a web browser form post over HTTPS--everyone has the client, and the UI is easy to use.

If it's a batch process (every single night), you can use a secured FTP mechanism or secured HTTP mechanism (POST or PUT), both are easy to automate (curl is a good tool).

If it's some interactive frequent process, I'd provide a 'rich' client, with the web browser form post as a backup.

If your customers are unix sysadmins, there's probably some new remote distribution tool. Or FTP.

An email type system is also possible, but the current email infrastructure is so screwed up I'd avoid it.

mb
Friday, August 22, 2003

thanks for the great replies.  the info transferred is highly confidential, and depending on the client will be as often as daily transfers or as infrequently as once a month.

nathan
Friday, August 22, 2003

the biggest problem i see is that we have in-house software that will have to be notified when a file has been uploaded, go get the file, and then create a response, putting it somewhere the client can get it.  this won't necessarily be immediate either.  three days can transpire between a client posting the inquiry and then checking for the reply.  (i'm not sure how long our process takes to create the response file - not more than a few minutes i don't think).

nathan
Friday, August 22, 2003

Surely your software can poll the directory and work out a file has been sent?

It could even send an encrypted mail when it's created the response.

Simon Lucy
Saturday, August 23, 2003

Setting up a dedicated Unix server is a *really* good idea - you don't incur any interface costs (you can either use Samba or your software can SSH in), and you get a bonus layer of security and virus protection.

Philo

Philo
Saturday, August 23, 2003

Anyone has used WebDAV? It looks like a simple enough solution.

All Mighty
Saturday, August 23, 2003

The obvious solution, of course, is to hire me to write a nice custom system that will integrate the whole mess into your information system.

Joking aside, this isn't hideously difficult to do, especially if you use a known protocol like HTTPS that can be scripted.  It sounds exactly like typical EDI data exchanges, save that with EDI you often don't get such nicely standardized protocols as FTP or HTTP.  A simple CGI program with a mechanism for triggering your data processing routine could make this pretty hassle free once it's up and running.

Clay Dowling
Saturday, August 23, 2003

I've done this using HTTPS webdav.
It is built into IIS, and Tomcat. It is easy enough to pop in the apache module.

If the client has one of the later versions of IE, they can just open an "explorer view" in IE, then drag and drop the files into the DAV directory. they can also map a drive. It rules.

rz
Sunday, August 24, 2003

Dove and horseman are transport mechanism too. :-)

Amour Tan
Sunday, August 24, 2003

http://www.faqs.org/rfcs/rfc1149.html

Philo

Philo
Sunday, August 24, 2003

Hey, you can hire me too!
Seriously, it sounds like an HTTP POST with either the accepting software understanding either form encoding or straight data in your data format is a good idea--then occasional users (1x/month) come in via the web interface, and tech savy users (or users w/tech savy admins) use a scripted method. HTTPS handles encryption over the wire; you still need to handle all other security issues (password or client certs and backdoors etc).

The reason I suggest POST vs. PUT is twofold: no need for filenames, and since it's running code you'll get notified and can call your processing system.

"Encrypting transactions on the Internet is the equivalent of arranging an armored car to deliver credit-card information from someone living in a cardboard box to someone living on a park bench."
Eugene Spafford

mb
Monday, August 25, 2003

SSH is an option on Windows.
I'm running OpenSSH under Cygwin under Win2k and it works great.
The Cygwin install isn't too scary and there are some great how-tos about SSH under Cygwin here :

http://tech.erdelynet.com/cygwin-sshd.html

Damian
Tuesday, August 26, 2003

*  Recent Topics

*  Fog Creek Home