Fog Creek Software
Discussion Board




too many files!

Folks, I got too many files! 240 gigabytes worth! What the heck is all this stuff? You guys must have the same problem, right? What on earth do you do?

Observations:

In the old days I could keep stuff backed up. Just save everything on a CD or two. How the heck am I supposed to back up all this? You guys got DVD writers or what? These home movies are real trouble! Do you keep different sorts of files in different folders for ease in finding backed up stuff? Man that's a lot of trouble!

I think I've given up organizing things. Almost every folder is called junk, cr*p or stuff. I just search for stuff by keyword. That's how it is right? Or do people still try to keep all this junk organized somehow...

Right now I'm just backing up proejct files and email and the rest I've sort of given up on. Someday the hard drive will be no more and I'll lose all these files. That's what it's come down to right? It's now impossible to backup! Or is there something everyone's doing I don't know about?

Dennis Atkins
Sunday, July 20, 2003

I don't know about you, but I download my porn, watch it once and then delete it.

Guy Incognito
Sunday, July 20, 2003

i back things up on CDs. *LOTS* of CDs. As for home movies, I can't really say I have a suggestion.

My usual strategy is to use the fact that there's only a few hundred megs of things of any real importance. Some code, some documents. EVerything of import is *text*.

As for keeping track of stuff, database it. I have my stuff catalogged so I can find out where any particular file resides quickly.

If you really have 240 gigs of things you consider important, I don't know of any cheap solution.

Mike Swieton
Sunday, July 20, 2003

If you've got 240GB of documents, then you're snap happy, 'cos even the mythical monkeys with typewriters can't produce it.

So if you've got a load of home movies, then use a CD/RW or DVD writer. Before you would have kept them on tape. What's the sweat? Being able to keep them on hard drive is an added convenience.

I would say that now's the time to buy a DVD writer for backup. I used to fit all my stuff on a CD but now, no way. Use Second Copy to make the back up profiile for the files and folders that will change, and put the others on CD or DVD (preferably both and twice).

Stephen Jones
Sunday, July 20, 2003

Check out Spacemonger http://www.werkema.com/software/spacemonger.html

It graphically displays your diskusage so you can see at a glance what's taking up all the space. I had a few gigs of temp files and junk on my disk.

Matthew Lock
Sunday, July 20, 2003

And check out system restier if you have XP. It takes up 12% of the partition by default.

Stephen Jones
Sunday, July 20, 2003

http://www.jimmygrewal.com/2003/07/02.html#a92

Prakash S
Sunday, July 20, 2003

"In the old days I could keep stuff backed up. Just save everything on a CD or two. How the heck am I supposed to back up all this?"

It's quite simple. Until you're willing to drop the cash for a monster tape drive and a safe-deposit box, and institute a very regular Backup and Remove Off-Site plan, you won't keep it meaningfully backed up.

I back up the most important stuff only. The data drive is a RAID mirror. That covers me for the most likely problem: drive failure. If someone comes in and steals the PC, or the house burns down, then I lose stuff. *shrug* I'm sure I'll have more to worry about than how hard it was to re-download a bunch of stuff.

Brad Wilson (dotnetguy.techieswithcats.com)
Sunday, July 20, 2003

240 gigabytes??? I can't imagine using nearly that much space (yet). I have about 30 gigabytes of digital camera photos (several years' worth) plus a few gigabytes of code projects, documents, web site stuff, edited photos, email, etc.

I have a RAID (mirrored) system (C:) disk (for protection against disk failures) and I have a second drive (that is, a third physical drive) and I keep up-to-date copies of most of the files (not the applications) on that drive. I have an Onstream 30GB tape drive (only cost about $200) and I do monthly full backups plus daily incremental backups to tape. I create date-based directories for the original photo files and when I take enough new photos to fill up a CD, I make 2 CD copies of that collection of directories and photos, and start a new directory. When I edit photo images, I copy the selected image(s) to a different directory (so the CDs, once created, never become out-of-date). I don't have any offsite backups although once in a while I consider keeping a set of tapes or CDs in a locked cabinet at work.

Another possibility for "backups" is the Martian Wireless NetDrive - see http://www.martian.com

If you don't know what's occupying a large part of your 240GB usage, there are utilities to help sort it out and/or find temp files and clean them up. As Stephen mentions, System Restore can take a large amount of disk space for system snapshots - you can reduce the amount as a setting in System Restore. You can also clean out your browser cache and reduce the space it is allowed to use.

If you have stopped organizing your files and may have multiple copies of the same files or multiple stages of an edit in progress, then you probably just have to take some time and go through and delete the junk and re-organize it.

Philip Dickerson
Sunday, July 20, 2003

http://www.werkema.com/software/spacemonger.html

Just wanted to pass on this excellent piece of freeware.  If you're trying to clear disk space, this is the tool to help you do it fast, instead of wasting all your time sorting through your 30k files, or even 3 meg files which don't really make a difference on today's 100 gig hard drives.

Andy
Sunday, July 20, 2003

Spacemonger ROCKS. I use it all the time.

I propose a 2 stage backup process.

1. Set up a network with a 2nd computer - copy / backup new important files to that computer on a regular basis - weekly, daily whatever you're comfortable with. Find a program to semi-automate this.

2. Create a mirror / incremental backup of that drive to CD-R on a weekly / monthly basis, again whatever you're comfortable with.

I follow a similar process and I've been happy with it. Make sure you clearly label everything. Labelling is an artform.

I don't know that DVD is quite where you want it yet, but if you have no other choice, then DVD might be the way to go. There are competing formats right now, and I don't think the marketplace has settled on one. Also writable DVD media is expensive for the Grade A stuff.

"One backup is almost like having no backup." - I don't know who.

www.marktaw.com
Sunday, July 20, 2003

Oh yeah... as with the previous posters, nearly everything of importance I have is text and takes up practically no space... I haven't made the plunge into digital video and don't plan on it any time in the future.

When my mother sends me that webcam, I may get some streaming video of my nephew to send to her that I may consider precious and keepable, but I'll probably stick them in a folder & burn them whenever it approaches 650mb.

My suggestion to you would be to burn each video to a CD immediately upon transferring it to the computer. If you can the unedited & edited version belong on the same disc.

Partition your drives so that you have a "junk" drive and a "keepers" drive. (drive letters J & K?) Make the "keepers" drive about the size of a CD-RW and when your computer tells you it's full, burn it to CD-RW. Stick a label on it and archive it. If you're really anal, make a database with what's on what CD and start numbering them.

www.marktaw.com
Sunday, July 20, 2003

DVD works, but my preferred solution is external FireWire disks. I now back up all of my projects on two 200GB FireWire disks. I hook them up, do the backup with rsync, then unplug them and take them home. A great poor man's off-site backup...

Dan Maas
Sunday, July 20, 2003

An 8-bay RAID case with 250GB harddisks. Around 5000 USD (not an updated price).

Problem solved.

S.C.
Monday, July 21, 2003

You can buy external RAID setups with 2 drives (use RAID 1 for full mirroring). They're not too expensive, that way you can offload your data onto a separate drive (connected over firewire if you'd like (look at miglia.com) and you're relatively safe if one of the drives fail.

Lou
Monday, July 21, 2003

I think the real problem here is organization - if he can't tell the junk files from the good ones, he has a real problem organizing his files.

And if it's 200gb now, one can imagine that'll double every 18 months along with the rest of the world.

Buy a book about organizing your life and internalize it's lessons so that you have a place for everything and know when it's time to back up, and have everything clearly labelled when you do back it up.

www.marktaw.com
Monday, July 21, 2003

Step 1: Start organizing all the new stuff as best you can.  Don't worry too much about creating the perfect directory hierarchy.  Over time, if you pay attention to where you're putting things, you will find the tree pretty much self-organizes.  For example you realize a directory is starting to get pretty big... so you look over its contents and see that its contents can reasonably be divided into three subdirectories.

Step 2: Organize the older stuff a little bit at a time.  Some of it might go in with newer stuff, some of it might get stuffed into an "archives" type directory.

In terms of overall structure, I am starting to organize my work by project.  I have a projects/ directory, with subdirectories for each project, into which go all of the files for that project.  When it's just a single file (or a couple), I put it into a category directory.  For example correspondence for all letters, and financial for my budget .xls and other related documents.  When it's "read-only" data such as .mp3's or images it gets stored in another tree whose root is simply /data.  (I have /data/media/images, /data/media/music, etc.) 

-Thomas

Thomas
Tuesday, July 22, 2003

Thanks much to all who replied. I'm going to look into the price of DVD media and see if is worth the convenience. Also, I am going to get some CD-RW media in the meantime so I don't feel bad about accumulating all these obsolete CDR disks. The links to programs that help visualize where the big files are were really cool. Regarding organization, I am organized in my email and in my projects directory but the rest of this stuff is just all over the place and I don't really have the time to justify organizing it properly, so I just rely on searching for stuff by keyword when I need to find something and that seems to work fine - the problem is in what and when to back up if at all. Hard drives are cheap but don't really solve the problem since the big film and photo file sets keep growing - hard drives are how I do it now and I have three of them and about to add a fourth. Some of the files I have are backups of other files, which is a problem. Backing all of them to CD is troublesome since some of these files are tens of gigabytes each. I'm going to mpeg compress them and save those files instead of the higher quality originals which is better than nothing.

In thinking about this, there are three things I would like in my dream world:

1. 500 GB Super-DVD media for $5-$10/disk. Probably will never happen. But if it did I could just back up everything all at once.
2. A program that would index my files and then ARRANGE them by topic, subject, people involved, like news.google does. Should put references to files in multiple directories as needed. Idea would be that the file system would automatically create a folder for John Doe Corp. of Plymouth and put our email correspondence and letter correspondence and file attachments and designs done for them all in one spot.
3. A live secretary to handle this and other clerical tasks. it's insane that we engineers do our own filing. It would make far more sense to hire a secretary for a project that is in a time crunch than it would to hire another engineer since half my time is spent doing stuff that someone else could handle.

So in the meantime, there is no easy solution but a dvd might help.

Dennis Atkins
Tuesday, July 22, 2003

Oh and I'm also going to try Thomas' method and see if that's the magic I need to prevent it from getting more out of hand. Thanks.

Dennis Atkins
Tuesday, July 22, 2003

You can get drives that burn CDs at 52x (faster, I'm sure, just going by what's prevalent at the stores). The CD-RW part of the functionality is nowhere near as fast. I would presume with so much data, that speed would be your biggest concern, not the reusability of the media.

Brad Wilson (dotnetguy.techieswithcats.com)
Wednesday, July 23, 2003

*  Recent Topics

*  Fog Creek Home