Fog Creek Software
Discussion Board




One folder, many machines

I've asked many people this question, but have yet to find a satisfactory answer.

I have 4 machines.

1) My work laptop
2) My work desktop
3) My home desktop
4) My home Media Center PC

There's a bunch of data that I want to keep in synch between my machines.  For instance, I want all my pictures on my home desktop (3) to by in synch with my home Media Center PC (4) as well as my work laptop (1).

I want all my documents on my work computer (2) to be in synch with my work desktop (1).

I want music from my Media Center (4) to be in synch with my work desktop (2).

I've looked at various half-baked solutions such as folder-share, but it just doesn't cut it.  Have you found a solution that will automatically keep stuff in synch between many machines at a very granular level?

Note:  Offline folders isn't flexible enough.

Thanks!

Cybersuraa
Tuesday, March 23, 2004

I'm never put enough faith in bidirectional synchronization to use it.  For a while I did everything on one laptop which I took with me everywhere I went. Now I just keep everything on my work machine and connect to it remotely using Terminal Services from home and from my laptop on the road. It's not the perfect solution.

Joel Spolsky
Fog Creek Software
Wednesday, March 24, 2004

big thumb drive?  (but would limit access to said shared data to one machine at a time, I guess... )

- former car owner in Queens
Wednesday, March 24, 2004

I keep a 256MB USB flash memory card in my pocket at all times.  You could easily fill it up with pictures but for work documents, Quicken database, PDFs I need to read for work, etc, it works great.  I frequently copy the whole thing to a hard drive in case I loose it.

Tim Stills
Wednesday, March 24, 2004

I have been asked to solve a similar problem for my brother. He is a small businessman and doesn't want to pay a lot.  The strategy I was going to use was to set up a cheap linix box for him at the office as a file server running samba and then I was going to try to set up Putty on his home windows machine and try to tunnel through ssh to get to the Samba server.  The idea is then to map a network drive letter on the windows machine to the smaba share.  There is a lot of literature on this available but thus far I haven't been successful. 

Not that I have tried very hard yet.

If that doesn't work I am going to try a similar thing with tunnel plain old ftp over ssh using putty.  I am pretty sure that windows clients can map an ftp site, right?

name withheld out of cowardice
Wednesday, March 24, 2004

Use a source control system!

I think you can get a single-user licence for perforce for free.

SG
Wednesday, March 24, 2004

Most people I know still buy too little mass storage for their needs. If all 4 of your machine had enough storage to 4 identical copies of frequently used files, the sync situation would be painless and easy (you could use a 1 primary, many synching secondaries method of sync in that case)

Unfortunately most people just want to mirror the really important stuff, and use the rest of the storage for unique data they are willing to gamble being stored on only one disk. This maximizes storage potential, but it also maximizes risk and sync complexity.

I fall into the latter camp, so I sync by hand. Lately I have been trying to back up as much as possible so that I can join the first camp.

rsync is something everyone uses in the Linux/UNIX camp, it's been ported to NT/Windows 2K but I don't know if anyone uses it much in production environments.

There are a bunch of expensive shareware ones, try one and see if it does it for you. But before ripping on a piece of duct tape software--try to simplify how you store and use files first (as talked about previously).

Li-fan Chen
Wednesday, March 24, 2004

I haven't tried it myself, but I've heard that a lot of people like Unison.
http://www.cis.upenn.edu/~bcpierce/unison/

Rich
Wednesday, March 24, 2004

The only real solution for your problem is too carry a removeable USB2 hard drive with you all the time, and keep all your data on it. Remember to back it up three or four times a day though.

There are plenty of back-up synch tools; may favourite is Second Copy from www.centered.com which has won many shareware awards.

However sycnronization is still fraught with danger, and if you are dealing with two locations, then the obvious thing is simply to have portable storage as your sure copy.

Stephen Jones
Wednesday, March 24, 2004

I like Unison too. 

It's written in a bizarre language, and the UI is in some wierd cross-platform library that makes it rather horrible, but it seems OK.

It does tell you what it plans to do before it does it, which helps avoid disasters.

Will
Wednesday, March 24, 2004

Has anyone tried SharePoint for something like this (where you're sharing with yourself across multiple systems rather than with others)?

SomeBody
Wednesday, March 24, 2004

Use Laplink, it works great.  http://www.laplink.com/products/llgold/details.asp

Rich
Wednesday, March 24, 2004

Have a look at Beyond Compare.

It is a manual, rather than automatic, sync tool, but when I tried it in the past I found it to be one of those rare pieces of software that made me smile with pleasure when I used it. Definitely worth the $30 if you have a need for what it does.

Ian
Wednesday, March 24, 2004

how about the the ifolder from Novell?

Prakash S
Wednesday, March 24, 2004

What didn't you like about FolderShare?  I use it to keep folders synchronized between my home and office and it works for me. 

Software Guy
Wednesday, March 24, 2004

Try Beyond Compare, it has a scripting language so it can be completely automated.  It will even sync to ftp as well.

JBreffni
Wednesday, March 24, 2004

www.groove.net

Groove is a more expensive solution to this problem, but it works great for me. (It may be a bit of overkill for your needs.)

keithb
Wednesday, March 24, 2004

To original poster- I think you could save yourself a lot of grief simply by keeping everything on one machine and sharing it out to others.

With a fast enough server and network, it won't matter that the files are all coming from one place. (RAID and Gigabit Ethernet are cheap now). Plus it could be easier to back up the central server. (I keep all of my critical data on a 160GB RAID-5, backed up daily to a portable FireWire disk)

Can't really help you once you leave the home network though. (rsync is great if you can find a use for it)

Dan Maas
Wednesday, March 24, 2004

I'd like to ask the original poster just why Offline Folders isn't flexible enough.  Granted it has a few limitations (e.g. you can't delete folders while offline) but I use it to keep My Documents, my Internet bookmarks, and other such stuff synched between my laptop and my desktop and it works great.  I keep the home folders on the desktop, map them as drives to the laptop, and sync them often.

By the way, by default Offline Files doesn't sync things like MDB files but you can configure it so it does (though it's not replication, but merely file replacement).

Kyralessa
Thursday, March 25, 2004

iDisk - mac and XP

http://www.mac.com/1/idiskutility.html

Interaction Architect
Thursday, March 25, 2004

CVS works perfectly for this with very little fuss.  You set up a centralized server that remembers all the versions of all of the files you're syncing; periodically you "commit" to send the changes you've made locally to the centralized server.  (If you're a developer, you probably already have the centralized server running CVS and sshd, and have ssh on all of your other machines, so it's zero hassle.  Otherwise, trust me, installing CVS isn't hard.)  Then you can "update" on the other machines to get the changes; CVS keeps track of what version of each file you currently have, and gets any new versions (which includes file deletions).  Typically I "update" on my work machine when I get to work, "commit" from my work machine and "update" from my laptop when I leave work to go home, etc.  I guess you could run a script to update, then commit, every hour or something, but I think I prefer having to do that explicitly, so I know when the file I'm looking at is going to change.

CVS, of course, provides some other benefits too --- you can use "cvs annotate" to see when you made each change to your address book, for example, and you can undelete files you didn't mean to delete.  (At least until you decide you're out of space on your CVS server, at which point you have to start deleting the old versions of deleted files.)

Joey Hess wrote an article about how he keeps his entire home directory in CVS for these reasons now: http://www.kitenet.net/~joey/cvshome.html

CVS can handle binary files pretty well, but it can't automatically merge conflicting updates in them the way it can with text files.  It's surprisingly fast even for huge files.

Kragen Sitaker
Thursday, March 25, 2004

The main problem I would have with CVS:
* you have to cvs add every file manually. Forgot to do that? The file doesn't get versioned.
* Good luck moving a file.
* If you cvs add a binary file and forget -kb the file gets corrupted. Good luck recovering from that.

Basically telling people to using CVS for folder synchronization is like telling people who want to find a restaurant where to buy a nice fishing rod.

Joel Spolsky
Fog Creek Software
Thursday, March 25, 2004

Thank you to all those who replied.

In my scenario, there's no real _master_ machine... they all have different pieces of data that select other machines require, but there's no UBER copy.  I appreciate that it may simplify things if there was a master, but I have gigs upon gigs of photos, music and video, making it a non-option at the current time.

I also value my network bandwidth a great deal as I stream TV around my house (which takes up most all of the available wireless bandwidth).

My main goal was to come up with an AUTOMATIC solution to the problem, one that wouldn't require manual intervention.

Some remarks in response to some of the comments:

1) Offline Folders: I didn't mention that I backup stuff via FTP from different machines to a server - that's the primary reason why offline folders won't work for me.

2) Sharepoint - Up until about two weeks ago, I was actually on the Sharepoint team, and I felt misplaced loyalty forcing me to give it a try.  It's just not flexible enough, and it's far too manual.

3) Beyond Compare - from the little I know, there are manual steps required.  I'm looking for an "install and go" solution - this is meant as an infrastructure thing, not something I actually want to spend time scripting, etc... (not that it would be too difficult)

4) CVS - you gotta be joking :-)  FAR TOO MANUAL.

I read a recent column today by Walt Mossberg who touted SmartSync: http://download.com.com/3000-2242-10263912.html - I think I'll give it a go.

Laplink also looks promising, thanks for the lead.

Thanks again.

Cybersuraa
Friday, March 26, 2004

Second Copy - http://www.centered.com/

A simple but effective approach.

Phil

Phil Rodgers
Friday, March 26, 2004

I've just had a look at Smart Sycn. At first sight it doesn't do anyting that Second Copy from www.centered.com doesn't, and the latter program does have four or five years of use behind it (I believe IBM use it intennally).

The reason most solutions require manual intervention is that a totally automated one is likely to lhave you overwriting the good copy, or having a load of versions of the same file in cautious mode.

Stephen Jones
Friday, March 26, 2004

You can use an iPod if you have USB 2 or Firewire on all the boxes.  Still manual, though.

The Mac solution for this is iSync (computer<->computer<->web<->phone<->pda<->etc.).  Personally, I keep pretty much everything on my laptop and carry it around with me (as Joel said he used to do).

Jim Rankin
Friday, March 26, 2004

I have been desprately looking and the onyl thing I see that comes close so far is to build a box with ReByte (an embedded linux that turns any crap box into a NAS). It has robust sync capabilities and even internet sync (via WebDAV), but I have yet to find a real review of the device. I believe the system uses Samba as its file-share core, I have yet to figure what they used for the replication (as it does not require a client).

I would use Unison (it is fantastic and sounds like what you need), except for a HUGE problem...it requires SSH client. Hey, Unison team, Samba and CIFS/SMB is enough.

Jason
Friday, April 02, 2004

*  Recent Topics

*  Fog Creek Home