Fog Creek Software
Discussion Board




Disk space

My home directory was taking up a lot of space. The reason was that only our home directories ever get backed up and I have been keeping stuff there I don't want to lose (since my disk crashed last year and it was a pain to re-install everything).
The system administrator, Dick, sent me emails now and then to delete stuff and each time I did what he asked (I thought). The last time, recently, I deleted tons of big files.
Yesterday Dick sent me an email saying my home directory is too big and if I don't do something about it my account would be locked on Monday.
So I wrote back and explained that I had complied with his requests every time, but he had never said exactly how much to delete. The answer was keep it under 500mb so now I know.
Is this kind of threat typical of system administrators? I felt it was not justified since I had deleted files and since he had not specified exactly what he wanted. And since all I was trying to do was avoid losing programs I had spent a lot of time installing. I felt I was being treated like a disk criminal.

The Real PC
Friday, August 15, 2003

I feel like the threat of "delete stuff or your account will be locked" is not uncommon, but it was pretty poor of him to not tell you what size was your limit

Mike McNertney
Friday, August 15, 2003

This sounds like the old days when I had to delete half my hard drive every 3 months, just to stay below the limit.  After awile, it is diminishing returns, and I was forced to buy a larger drive.

Please beg and plead for more space.  Quit wasting time deleting files, when you can write some code and make the company money. This falls in line with Joel Test Question #9
Do you use the best tools money can buy?

Tell your boss that this system monkey is keeping you from being productive.  I mean 500MB, that is a freaking CD?!

My marketing manager has a 2GB mailbox, and the CEO, at least 5GB... I wouldn't dare ask them to delete squat. This kind of IT red tape is a barrier to productivity.

Signed, A part-time sysadmin

--
ee

eclectic_echidna
Friday, August 15, 2003

Tell them to stuff it and give you more disk space. I bet your job description didn't include 'must spend one day a week using WinZip/tar+bzip2 and del/rm to ensure home directory stays under some ridiculous limit'. And I guarantee you your time could be better spent.

Tom
Friday, August 15, 2003

Tell him that the cost of you cleaning up your files is more than the cost of more disk space.

Flamebait Sr.
Friday, August 15, 2003

this is why :)

http://members.iinet.net.au/~bofh/bofh/bofh1.html

apw
Friday, August 15, 2003

The problem is usually not the diskspace, but the backup system.
If you have enough local diskspace on your machine, you can try to set up a friendly informal P2P backup agreement with a colleague.

Just me (Sir to you)
Friday, August 15, 2003

500mb, at least that's getting reasonable, although I would expect that getting several gigs shouldn't be that terribly difficult for an able IT staff. 

We're stuck at 35 megs for email + files.  Hello? our hard drives are 30+ gigs but partititioned and locked so we only have access to 10gigs (that's before Windows + apps are installed).  And of course we have a corporate policy to keep all necessary files on the lan drives. 

Now how's that work?

Lou
Friday, August 15, 2003

Why don't a few of you look into the cost of backup systems which are capable of backing up huge amounts of data in a reasonable amount of time?  It really doesn't matter how cheap disk space is.

Anonymous
Friday, August 15, 2003

Well, if your Godzilla (tm) backup systems are expensive, then just buy large IDE drives and backup to those!

B.J. Thunderstone
Friday, August 15, 2003

Real PC - how many employees at your company? (ballpark is good enough)

Philo

Philo
Friday, August 15, 2003

Backing up to hard drive is not a viable option.  At most companies it makes sense to have several copies of everything available.  It isn't often that users want a file restored from the last backup.  It tends to be something that was deleted a week ago and they just realized it now.  You also need to have one or more copies offsite in case of a fire or natural disaster.  Hard drives are also sensitive to magnets.  Hard drives have a much higher failure rate than tapes.  Need I go on?

Adding more disk space is not as simple as throwing in another hard drive either.  If the new space is to be added to an existing partition a format and restore of data is often necessary.  If a system currently has one disk and you want to add another and create one partition out of the two you are going to use RAID.  Software RAID tends to be questionable at best, and good hardware RAID costs a lot of money as well.  Not to mention that many servers still use SCSI as the higher priced disks are often more reliable, hot swappable, and SCSI can still provide better performance thanks to it's maximum device count per channel.  Suddenly adding hard drives doesn't look so cheap.

This issue is not as simple as people would like to think it is.  When management won't approve funds to purchase backup equipment policies need to be created.  It certainly is possible that the sys admin is a moron and just makes up policies because it seems like the right thing to do, but that may not be the case.

Anonymous
Friday, August 15, 2003

I meant to type that hard drives tend to be more sensitive to magnets.

Anonymous
Friday, August 15, 2003

> Why don't a few of you look into the cost of backup systems

Say we have 15 programmers... (and assume that they can be modeled by the shape of a sphere...)

Dell PowerVault 122T LTO Autoloader = $6999.99
Uncompressed: 800GB, 800GB / 15 = 53GB per programmer

1 programmer at 50k/year =  50,000/50/5/8/60 = $0.41667 / minute

30 minutes a week for one year = $625
15 programmers = $9375

QED

--
ee

eclectic_echidna
Friday, August 15, 2003

ee, did you include media costs for a full backup system? (that's 9 x a full load of backup media)

Philo

Philo
Friday, August 15, 2003

Media costs, we bought 10 of those 100GB tapes for $70 each, but that was awile ago.

Philo, Are you saying 9 sets of 8 tapes? Now I am confused.

--
ee

eclectic_echidna
Friday, August 15, 2003

Assuming a 5-day week:

Four sets of daily tapes (M,T,W,R)
Four sets of weekly tapes, run on Fridays (Week1, 2, 3, 4). On Monday, following week's tapes come in, current week's tapes go to offsite storage.

One set of "hot spare" tapes.

You could also add twelve sets of monthly tapes, run on the first day of the month, kept offsite.

Monthly tapes are generally only necessary for true archival purposes, of course. You could get away with eight sets. I believe most companies try to make do with 5-7 daily sets (or less!), but IMHO that's a mistake.

So, 8 tapes in a set, $70/tape = $560/set.
Eight sets of tapes + 1 "hot spare" = $5040.

Philo

Philo
Friday, August 15, 2003

Now calculate how long that drive is going to take to backup that much data.

Anonymous
Friday, August 15, 2003

Philo, Yikes! --ee

eclectic_echidna
Friday, August 15, 2003

Depending on how much compression is achieved it will take about 8 to 16 hours to backup that much data with that drive.  That assumes that the drive is attached directly to the server that it is backing up, which isn't very common.  Usually a separate backup server is used that can backup several servers on the network.

Of course the entire calculation is kind of silly for 15 people.  A company with 15 employees is probably not going to spend that much money.  The 112T with two drives at around $2300 is probably more reasonable.  With two drives you are looking at 80 to 160GB depending on compression.  Employee data is not the only thing that needs to be backed up either.  Figure the company has 20GB of other data to be backed up, such as server OSes, company systems with databases, etc.  Now you are looking at 4GB per person on the low end.  I would be pretty surprised if a company this small was even willing to pay $2300 plus media costs though.

And you still need to factor reality into the equation.  Most managers don't look at an employee's time as a cost.  $10K in hardware and media is a real cost.  The sys admin doesn't get to just determine that the drive and media are cheaper and place an order.  Often they must work with what they have.

Anonymous
Friday, August 15, 2003

Additional costs might also include cleaning tapes and software upgrades.  Some software can not handle autoloaders.  Some software can not handle certain tape drives at all.  Hell, you might even want redundancy.  There is no end to how much you might need to spend on this stuff.

Anonymous
Friday, August 15, 2003

On the less expensive side, we just bought an OnTrack dual tape drive, SCSI-160. Cost us $1400, and came with two tapes (120GB each, uncompressed). The tapes are, if I recall, about $100 each. Toss in 6 more tapes, that's an even $2k.

No reason to get unreasonable with auto-loaders and the like. Especially if you limited yourself to backing up to a single uncompressed tape, 120GB / 15 devs is still 8 GB each: 16x as much space. Okay, so the admin needs to swap tapes every 2 days, since you use a tape a day.

Seems like a pointless argument to me. Time of 15 devs is worth so much money comparitively that it's ridiculous.

Brad Wilson (dotnetguy.techieswithcats.com)
Friday, August 15, 2003

And I start sending the 'delete stuff' emails around when their personal folders get over 30Mb....What a nasty Sys. Admin I am..

An Aussie Chick
Friday, August 15, 2003

Also in my experience tape backup systems do tend to need a lot of maintenance and repairs and have fairly limited lifetimes. This is not surprising given that there are quite a bit of moving mechanical parts in such a system.

Backup to disk with backup of the backup to tape for archivation only is a nice alternative to a full tape system.

Just me (Sir to you)
Saturday, August 16, 2003

Hi folks,

Get yourself an account with Netstore (www.Netstore.net). I pay £15 (about $22) per month for backup over the Internet of an *unlimited* amount of data from a single pc.

Data can be restored online or using a CD. Data is strongly-encrypted (using PKI) and only a delta is transmitted. Works like a dream.

HTH,

Mark
----
Author of "Comprehensive VB .NET Debugging"
http://www.apress.com/book/bookDisplay.html?bID=128

Mark Pearce
Saturday, August 16, 2003

Internet connections are generally 3Mbit/sec (~300kB/sec)

You've got 100GB of data to back up.

Do the math.

[I won't get into how promises of 'unlimited' suddenly become 'limited' when you actually put some heavy usage on them]

Philo

Philo
Saturday, August 16, 2003

Hi Philo,

The original question was about 500Mb, not 100Gb - Chinese whispers seems to have inflated that original figure! Even 500Mb of new/changed data is not feasible on a daily basis, so you would initially need to send a CD/DVD of the important data.

From then onwards, only a delta is transmitted, and that delta is compressed before transmission. According to the Netstore statistics, yesterday I generated 5.6 Mb of new/changed data, and that was compressed to 0.7 Mb for transmission.

Mark

Mark Pearce
Sunday, August 17, 2003

Mark, that's 500MB *per employee*. We never found out how many employees we're talking about, but you also have to add in corporate infrastructure stuff if you're talking about a backup solution.

And you should run a full backup at least once a week - living on incremental backups is like standing on a surface that's slowly crumbling under you...

Philo

Philo
Sunday, August 17, 2003

Philo,
There are about 10 programmers. I can have up to 1gb but preferably under 500mb.

The Real PC
Sunday, August 17, 2003

Hi Philo,

Ignoring the initial 500Mb hit, I don't see any problem with 10 developers each having (say) 10MB per day to backup. Assuming most of the data is email, spreadsheets etc, this should compress to around 2Mb, which would transmit comfortably within 15-30 secs per employee.

I don't quite understand your point about incremental backups. Although only the delta is transmitted, this delta is then incorporated into the full backup. In fact, by default, Netstore keeps the previous 20 versions of every file.

I take your point about trying to use this to backup your entire corporate infrastructure. My feeling is that corporate infrastructure stuff probably belongs in-house, for both privacy and size reasons.

Mark

Mark Pearce
Monday, August 18, 2003

Real PC,

any idea on the backup setup used at your place?

Just me (Sir to you)
Monday, August 18, 2003

The disdain for incremental backups is a tape legacy, and shouldn't apply to network backups. It's understandable when using tapes, since you're counting on the fact that every single tape back to (and including) the last full is valid. If you did that for a month with daily backups, you'd be looking at 30 tapes. Statistically speaking, that's not someplace you want to be.

Brad Wilson (dotnetguy.techieswithcats.com)
Tuesday, August 19, 2003

see where the disk space went - www.diskview.com

vjs
Friday, August 20, 2004

*  Recent Topics

*  Fog Creek Home