Fog Creek Software
Discussion Board




Is anyone else completely hosed by MSBlaster?

Where I work, which will remain nameless, we are completely dead - our servers are now in the process of being patched.  We have no email, and I can't get to my source code.  So most of us are reading and socializing this morning.

We are a big - and I mean big - company.  You've heard of us.  Am I wrong in thinking that a patch that's been available for a month should have been applied to all our servers by now?  Did someone screw up, or is it just not possible to get to all the servers in a month?  (This is a serious question - I'm just a code monkey, not not a network admin.)

Grumpy Old-Timer
Thursday, August 14, 2003

It's not that it takes a month to patch, it's that admins feel a need to test patches before applying them. That moves the delta for most companies out to three, six, even nine months sometimes.

The last time I remember a publicly released patch hosing anybody on the Windows NT class OSes was NT 4.0 SP2 (which was a disaster, and caused Microsoft to re-do their testing matrix before release). Since then, I haven't had a patch hose me or anybody I know. I automatically install them all now. Needless to say, no MSBLAST at my house. :)

Security is a "weakest point of entry" problem. The big problem is people who take their work laptops home, and plug them into their always on, never firewalled DSL and cable modems, who then bring the boxes back into work behind the firewall and allow the bug to propagate.

I'd surmise that the end result at some places it that people are simply not allowed to work at home any more because some people are simply not responsible enough. It doesn't take too many people to ruin it.

Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, August 14, 2003

It should be noted that I think ISPs need to take some responsibility here, and firewall off their nodes from each other, as well as from the ISPs network and the public. It's going to be VERY unpopular, but given the current climate, it seems like the right move.

Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, August 14, 2003

Nope, but in theory we could be. As an attack from the outside no, however on a laptop brought back into the office perhaps. Most (I would say all, but thats just asking for trouble) of our desktops are upto date so we'd probably be safe.

Peter Ibbotson
Thursday, August 14, 2003

"It's not that it takes a month to patch, it's that admins feel a need to test patches before applying them."

That's no excuse whatsoever. The problem is complacency by the admins, despite the sql slammer just six months ago.

In a properly available development network, testing a MS patch should take less than 24 hours - slap it on a test box, run through some apps and common services to make sure nothing's obviously broken, then roll it out. Odds are any "lurking" bugs will be found and repatched by MS before you find it.

Show of hands - who here thinks the admins at GOT's company had the patch and were actively testing it for installation on production boxes?

Now who thinks they aren't even subscribed to the MS security alerts? ;-)

Philo

Philo
Thursday, August 14, 2003

"It should be noted that I think ISPs need to take some responsibility here, and firewall off their nodes from each other, as well as from the ISPs network and the public. It's going to be VERY unpopular, but given the current climate, it seems like the right move."

Why should some of us be punished because other people run Windows?

Anonymous
Thursday, August 14, 2003

"Why should some of us be punished because other people run Windows?"

Ooh, trolls. Man, I missed trolls since I stopped reading slashdot.

Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, August 14, 2003

Logically, one should be able to discern from an Microsoft screw up and a sys/network admin screw up. 

1.  As a sysadmin, it isn't hard to find out what new patches are available on a daily basis.  All it takes is a little self-discipline.

2.  The patch has been available since July 16th.  There's no excuse for not patching your servers for 3 weeks.  At the place I used to work for years ago as a computer operator, we rebooted servers on a schedule, updating/patching them as needed. 

3.  You could enable auto-updates, but that is of course at your own risk. 

4.  It almost seems like there's a need for a new position created at most companies: "Security Administrator"

GiorgioG
Thursday, August 14, 2003

Sadly I think philos hands in the air questions are right.
Laptops are the modern equivalent of boot floppy viruses in this case.

I suspect quite a few sysadmins assumed their firewall would stop this. A lot of the more recent MS patches haven't been critical (most of them have been physical access required or browser use required) to apply to servers so they got sloppy.

Peter Ibbotson
Thursday, August 14, 2003

what sort of rinky-dink systems do you guys work with?

where i work, we'll have everything patched without any real testing. But we're a dev shop not a bank or hospital--if something mysteriously dies, we fix it at some point and no one dies or loses significant money.

but a test pass of a real-world system can take a few days per process. do people need to have full-time people on-call to test every random new patch from microsoft/other vendors? and remember, it's the subtle bugs which cause data corruption.

the laptop/vpn problem/email virus/insecure wireless problem is real. the real fix will be a move away from the 'hard shell' security model, because hard shells get small holes drilled in them pretty easily, then the entire inside rots.

mb
Thursday, August 14, 2003

Turns out if we apply the security patch, suddenly all the files we save from 3dstudio get corrupted. The only solution is to uninstall the patch. The worm is blocked at the firewall, but we're fucked if anyone parks an infected laptop near the wireless lan.

Tell me again why I should blindly install patches from microsoft?

xyz
Thursday, August 14, 2003

Firewall your wireless base station, then.  It's a point of access to the outside just like any other.

Alyosha`
Thursday, August 14, 2003

We have had no problems so far with MSBlaster, but that was another story with security patch 823803 - just this month:
http://support.microsoft.com/default.aspx?kbid=825501

Admittedly, it was NT4 and not W2K, but I can understand that admins are reluctant to apply the patches as soon as possible.

GP
Thursday, August 14, 2003

This worm seems to be amazingly agressive.  This morning I stopped in at a small business, whose only connection to the net is one machine that connects via a 56K dial-up once a day to get the email - less than 5 minutes a day connect time. There are no rogue laptops, no other network connections, no one screwing around.

Both of their XP/Win2K were infected.  I was really surprised.

Got no worms cause I smoke cigars
Thursday, August 14, 2003

"There's no excuse for not patching your servers for 3 weeks. "

I am sorry, but that's simply not true.  I run my family's business server unattended for months at a time when I went away to graduate school.  My parent's sure as heck don't know how to check for security updates and patches, and I can't babysit the server when I'm 1500 miles away..

There's no reason our shop has to have a SysAdmin on retainer for this sort of thing - I should be able to set up the server and go.

(Though luckily, I did go home the week after the patch came out and installed it promptly)

Ankur
Thursday, August 14, 2003

In this case there was no excuse. No matter how complex the operation environment, if a shop is not capabe of taking apprpriate action given a full month time, they have a serious procedural problem.
Any shop that was hit by this should do a very very serious operational review. NO excuses apply.

Just me (Sir to you)
Thursday, August 14, 2003

Grumpy, all I can say is MMM

I know who you are
Thursday, August 14, 2003

"Both of their XP/Win2K were infected.  I was really surprised."

Yeah, 1.4 million computers scanning to infect others and your suprised.  Do you work at Microsoft, or just aspire to?
        

Mike
Thursday, August 14, 2003

"Do you work at Microsoft, or just aspire to?"

Slashdot down and you wandered over here to try and talk to the adults today?

Got no worms cause I smoke cigars
Thursday, August 14, 2003

"Any shop that was hit by this should do a very very serious operational review. NO excuses apply. "

So every store which happens to use a Windows machine to check their email once a day needs to do a serious operational review?

Bye-bye Windows TCO. Sorry sir, in order to check your email you must undergo a month of Windows training and spend 3 hours a week updating your machine.

mb
Thursday, August 14, 2003

"So every store which happens to use a Windows machine to check their email once a day needs to do a serious operational review?"

in this situation, they should just use automatic update.  they'd never have to worry about it.

nathan
Thursday, August 14, 2003

Where I work when I plugged my laptop into the network a virus blocker came up and told me about the worm, and 2 minutes later an IT guy came over applied the patch and ran the remover.  All told the thing was off my machine in 10 minutes.

The only reason I even got it is because I take my laptop home and plug it into my DSL connection at home.  No computers within the company firewall were affected.

At previous companies I've worked at, they were never this good.  It does show that it is possible for a vigilant tech crew to keep up to date and keep the network healthy.

Oren Miller
Thursday, August 14, 2003

"Laptops are the modern equivalent of boot floppy viruses in this case."

Peter, I believe this is the fourth time you've brought up laptops on this issue. Why the heartburn for laptops? Or is that just the chink in the "Microsoft can't be blamed for people who don't use firewalls" armor that you've chosen to exploit?

Philo

Philo
Thursday, August 14, 2003

Because people take laptops to networks that the admins can't control. Inadequate firewall is almost always the case with them. How many laptop users (who aren't developers) would put up with ZoneAlarm?

Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, August 14, 2003

Its the chink we have potentially and one thats tricky to solve while allowing the users to carry on working easily.

Peter Ibbotson
Thursday, August 14, 2003

I know a *lot* of neophytes/lay-users that have bought and installed ZoneAlarm or Blackice all by their lonesome. ZA seems to have done an exceptionally good job on their UI, because the people I know using it comment on it, but don't complain.

[They also get a huge kick out of blocking spyware]

Philo

Philo
Thursday, August 14, 2003

Microsoft has released patches with bad bugs since NT4 Sp2.  Security patch Q811493 made many XP computers run at about one-half speed. (MS has since re-released the patch).  If you want to read a bit about it, here is a link to an old, and now out of date article: http://www.microsoft-watch.com/article2/0,,1037305,00.asp

By the way, it took MS about 3 months to fix that patch.

XYZZY
Thursday, August 14, 2003

Regarding slammer, even though the bug was corrected in an early patch, a later patch re-introduced the sql server vulnerability. Are the users that installed the latest patch to blame?

NT4 had a service pack (SP3, I think, but I'm not sure), that rendered all security on Lotus Notes ineffective. Microsoft only test compatibility with themselves (and even that they don't do perfectly) - but they don't, and can't possibly, test for compatibility with all software out there.

If any of you worked in a real production environment, three weeks is NOT a lot of time to properly test deploy patches on critical machines, especially if you consider that these patches and hotfixes arrive on twice-weekly basis.

And Grumpy, and everyone - in my opinion, almost everyone misses the point that the real problem isn't Microsoft or Linux - both will continue to have security problems till kingdom comes. The real problem is monoculture. All your servers are down because all your servers are effectively running the same O/S make and model. If all your cars were the same Ford model, and Ford issued a recall, you'd be without cars for a while.

Yep, a heterogenous environment does take more effort to run (for a start, it really does require clueful administrators). But it pays back on so many fronts, including, but not limited to, the ability to negotiate precises with your suppliers.

Ori Berger
Thursday, August 14, 2003

negotiate prices, of course. Not precises.

Ori Berger
Thursday, August 14, 2003

I didn't see the "You have to have your lips securely attached to Microsoft's ass" disclaimer as I entered this forum, was I not paying close enough attention?

I'd consider myself the original Microsoft fanboy, but that glowing admiration is starting to wane...

Guy Incognito
Thursday, August 14, 2003

IF you as a small shop are not going to do patch testing and approval for whatever reason, just leave the thing on automatic update. This is a reasonable patch process under quite a few scenarios.

Just me (Sir to you)
Friday, August 15, 2003

So let me guess. The slashdot servers are down?

Robert Moir
Friday, August 15, 2003


Hmm... a big company that went belly up due to this worm.

Well, I did hear a rumor that an EDS shop in Atlanta was hosed sometime this week...

non-significant
Friday, August 15, 2003

It's patently clear that many of the self proclaimed experts on this board have never been responsible for more than maintaining their own computer or just a handful of systems at the most.

Anyone who says "Just patch when MS releases. They work just fine" wouldn't last a single day as an IT admin in a large shop. People get shot for less than that.

Last month, MS released 9 hotfixes. In our shop, that's 9 fixes to test on over two dozen servers, all of each running different software.

Oh..And guess what happens when one of the hotfixes royally screws something up? Yeah, that's a lot of fun. And it happens more than some of you think. Time to crank up the restore or spend several hours trying to manually undo all the registry settings. Oh, and do this while supporting 500 users who won't tolerate a minute of downtime.

It's easy to sit there with your single desktop and claim to be an IT expert, but come over to the real world of high availability and hetergenous systems and start spouting off about "just path what MS puts out" and you'd be laughed out of the company.

Get Over Yourself.
Friday, August 15, 2003

Get Over Yourself indeed. 

If they're high availability systems, why don't you firewall them off from the rest of the LAN minus whatever ports necessary?

Simple solution.  Yes, it is doable.  Heck, this way you never need to patch anything ;-)

GiorgioG
Friday, August 15, 2003

"If they're high availability systems, why don't you firewall them off from the rest of the LAN minus whatever ports necessary?"

You don't work in IT, do you? That's ok...

To answer your question, it's because users have laptops. Users take laptops home. Users get infected at home. Users bring laptops to office. Users infect servers.

Some users travel overseas for weeks on end. Some users don't get their laptops updated until two months after we patch everyone else.

Get Over Yourself.
Friday, August 15, 2003

Maybe I didn't make myself clear, or maybe you're just a troll. 

Put all your high-availability servers behind a firewall, separating them from your LAN (i.e. desktops & laptops.)  Only open up the specific ports on those machines that are required.

This would get rid of 99% of your issues.  IMO there's no reason that production servers should be left completely wide open to the rest of the LAN.  There's simply no reason for it.

And yes, I do work in IT.

GiorgioG
Friday, August 15, 2003

Get Over Yourself,

Nobody here is telling you it is easy. It not being easy is no excuse. It is nescessary.
Giorgio is merely suggesting ways that might reduce your need for deploying patches. IP Filtering, zoning etc. can eliminate some of your urgent patching needs and buy you  more time.
It is not because a laptop sneaks by the outer perimitter that there should be no defences after that, and then there should be places nobody sneaks in a rogue laptop.
This time we had one month and some of us could not make it in time. Next time we might not have so much ample prior warning.

Just me (Sir to you)
Friday, August 15, 2003

*  Recent Topics

*  Fog Creek Home