Fog Creek Software
Discussion Board




Nasty virus/worm on my server - what is it?

Does anyone know of free or cheap server-targeted anti-virus software?

Here's my situation: I've just put my new web server (for my new ISV business) at a colo company's data center.  Unfortunately, I didn't have my OS (Win2k) nor my database (MS SQLServer 2k) updated at the time.  Yes, I'm an idiot.   
Surprise, surprise, I now have some kind of a worm or virus on the box.  It causes the box to do a huge amount of outbound data transfer (around 90 megabits per second) until the box is manually rebooted.  I can't restart it using Terminal Services because the box is so overwhelmed by this that it doesn't even respond to ping, let alone to a Terminal Services client.  Unfortunately, I don't know what kind of data transfer (port#, destination IP) is happening because the colo company isn't logging anything that doesn't get to its router.  None of my outbound packets are getting to the router because the colo guys have some kind of filtering that prevents packets from getting through when the demand exceeds some ridiculously high level which my box has been exceeding.

I updated my OS and database.  I already ran McAfee VirusScan and the free edition of AVG on the box.  This turned up nothing.  I fear that these desktop products are not looking for worms and viruses that target servers (ex. SQLServer worms). 
 
I know this is very little information, but it's all I've got.  Can you give me some hints or heuristics about how to find what virus it is?

Here are the basics of my configuration:

-Win 2000 (with the most up-to-date patch, 5.00.2195  Service Pack 3)

-IIS

-MS SQLServer 2000 (with the most up-to-date patch 8.00.763, Service Pack 3)

-Tomcat (4.1.27)

-No SMTP

I humbly thank you in advance for helping me out.

OK programmer, sucky sysadmin
Wednesday, July 21, 2004

If your AV product can't find anything, it's possible some hacker has some customized DDOS or spam software on it. Your best bet is just to wipe the box. Once you've been rooted, this is the only way to really be sure you're clean.

(Yet another reason not to use Windows. Most datacenter operators are well-equipped to determine exactly how a box has been compromised and what's been changed if it's a Linux box).

Neat Chi
Wednesday, July 21, 2004

"Most datacenter operators are well-equipped to determine exactly how a box has been compromised and what's been changed if it's a Linux box"

Cite?

Philo

Philo
Wednesday, July 21, 2004

Time to format the drive and reinstall.  See:

http://weblogs.asp.net/larryosterman/archive/2004/06/18/159482.aspx

example
Wednesday, July 21, 2004

Go to www.sysinternals.com. There are a ton of tools there to allow you to figure out which processes are running (procexp),  which processes have which TCP/UDP endpoints open (TCPView)

If that fails try any of the free network sniffer tools @ http://www.webattack.com/Freeware/network/fwpacketsniffer.shtml

Code Monkey
Wednesday, July 21, 2004

Microsoft has a free antivirus support. Tried it?. Do a Google search on SQLServer viruses.

Database viruses should not be too tough to remove.

Karthik
Thursday, July 22, 2004

Toast the box.

Surely, blatting everything and restoring your last known good backup will be quicker than fiddling about with loads of tools.

I assume you do have a suitable backup. If not, just go back to bare bones (Windows+IIS+SQL, with SPs) and make a couple of backups, for future use. Then put everything else back on.

I find it odd that your co-lo facility can't give you anything from their logs. Even if they filter out high traffic, they must have something, surely.

Nemesis
Thursday, July 22, 2004

Also consider running Microsoft Baseline Security Analyzer before connecting to the network.  It should pick up any patches or security problems you've overlooked after a clean reinstall. You can find it here:

http://www.microsoft.com/technet/security/tools/mbsahome.mspx

And the latest security updates file here (if you don't want to download directly when running the program, ie without a net connection):

http://go.microsoft.com/fwlink/?LinkId=18922

(drop the cab file in the installation folder). Obviously make sure you've got all the latest patches on a CD as well.

el
Thursday, July 22, 2004

I have recently set up a cololocation server --> Windows 2003 Server + SQL Server 200 and all fully patched. Having run MDSA and installed a local packet filter, its fairly solid as long as the logs are checked daily.
Anyway, to the point... I noticed recently my bandwidth usage going up to 1GB a day when the box is still not being used. I used Network montitor (part of the OS) to generate network traffic logs. From here, I found out that the SQL Server on the box was being bombarded with requests from certain IP address ranges. I assume these IP addresses belong to unpatched SQL Server machines with some kind of SQL server virus.
Anyway, I now capture network traffic logs and compare IP addresses to other logs (e.g. IIS) to check for suspicious traffic. This traffic gets blocked in the packet filter.
So, just take a quick check that you do actually have a virus. You could find that you are being bombarded with zombie sql traffic and your SQL server is just responding to that with the 2 finger salut.
HTH
Matt

Matt
Thursday, July 22, 2004

SQL Server blaster will not be picked up by an AV application as it spreads by looking for *running* instances of SQL Server, and infecting that. In general, rebooting the machine will remove the worm. Upgrading SQL Server to sp3 will prevent further infections of the worm, as will closing the appropriate ports.

It is still an extremely prevalent worm - if you have an open port and unpatched SQL Server, you will probably be infected within hours. It's also a spectacularly intense - it does some remarkable stuff to bypass your normal bandwidth limits if you're on a software-controlled bandwidth limit, as you presumably are with a co-lo server (obviously if you're on dialup or ISDN this is not relevant).

Chris Welsh
Thursday, July 22, 2004

Even if you manage to get all the crap removed, the box can never be trusted again.  You *must* reinstall the operating system.  Besides, this is your chance to upgrade to a better one.

Allez-Allez
Thursday, July 22, 2004

Seriously, you *have* to rebuild the box.

Let's say that it appears that you get the bug removed.  Down the road, SQL is acting up to the point you need to call MS support - the DB is hosed, your manager is breathing down your neck - not your best day.

If MS support gets a whiff of the fact that you got hacked/wormed/whatever, they'll straightaway tell you that your configuration is unsupported, and that you need to rebuild the box.

A hacked box is an unknown, period... you're better off rebuilding now than later.

Greg Hurlman
Thursday, July 22, 2004

"Microsoft has a free antivirus support. Tried it?"  Does it work on a server.  A LOT of those tools will not install on the server because said vendor also makes a pricier server package.

Yes you must rebuild - like it or not.

Yes you are an idiot for placing it on the web unpatched.  You understand that so there is hope.  A lot of Windows folk need a lesson like this and then they do much better.  Just reading to do things does not cement them in your mind like actually getting hacked.

"its fairly solid as long as the logs are checked daily."  That's not the kind of baby sitting most people are willing to do.  Perhaps OP should consider Linux/apache.  It's very easy to install I'm told;)  Web is one place where I do give *nix a big nod over Windows.  And no I don't have a cite for you Philo, just going on own experience and what I've seen on this board.  I haven't seen any posts on here about "help my linux webserver got reamed."

Formerly someone else
Thursday, July 22, 2004

Forgive an ignorant question but why would one expose his database directly to the internet anyway?

name withheld out of cowardice
Thursday, July 22, 2004

The default installation of SQL Server makes it available via TCP/IP on port 1433. If you haven't closed this port, are not running sp3, and the computer is visible on the internet, then expect it to get hacked with SQL Slammer.

Chris Welsh
Thursday, July 22, 2004

One more word of advice - it's not paranoia if they really are out to get you.

I've been adminning and developing on top of IIS since v3, and I've never been hacked due to one thing - paranoia.  It was hard to get mgmt to approve things until the summer of Code Red/Blue/Yellow/Nimda/etc - since then, it's been easier.  To admin IIS, you *must* become a Windows platform security expert - for the whole OS.

If you're still learning the admin aspects, see if you can't get your employer to pop for MCSE-Security training materials (http://www.microsoft.com/learning/mcp/mcse/security/windows2000.asp) - it's a good baseline, and should get you started in the right direction.  Just remember - don't stop researching, don't stop reading after you pass the last test.

The server admins need to win every single time - the hackers only need to win once.

Greg Hurlman
Thursday, July 22, 2004

One thing you absolutely desperately need, no matter what OS you're using, is regular OS and system software patching.  The Microsoft tools are big targets, but every time I've been hacked it was on a Linux box, and it was universally because of unpatched software.  Certain packages have security reputations that are for crap, such as IIS, SQL Server and ISC's bind (of these, bind is the worst, and I suspect that the ISC programmers are just bloody dense).

Since it's pretty unlikely that you can do anything to get away from these bad tools, no matter what platform you pick.  Your only solution is to automate the updating of these packages and block off access where possible.  Your SQL Server, for instance, probably doesn't need to be accessible from the outside world.  You might need access from your office, so set up the firewall to allow access only from there and the server.

Buck up.  You're about to become a much better administrator, unless you're rolling over and declaring yourself dead.

Clay Dowling
Thursday, July 22, 2004

"And no I don't have a cite for you Philo, just going on own experience and what I've seen on this board.  I haven't seen any posts on here about "help my linux webserver got reamed." "

Oh, then you should go do some reading:
http://www.google.com/search?sourceid=navclient&ie=UTF-8&oe=UTF-8&q=linux+server+hacked

An admin puts a default unpatched install on the open internet and you're going to blame the OS?

Philo

Philo
Thursday, July 22, 2004

"An admin puts a default unpatched install on the open internet and you're going to blame the OS?"

No.  Although Windows server hacked, did get 5000 more hits.  The OP is at fault.  Although it wasn't apache servers that were putting javascript footers on webpages a couple of weeks back was it?  I do have high hopes for the Winodws juggernaut though.  Like I said they have the ease of install, ease of use, reliability thing about wrapped up.  More work to do on security though.  For the time being I would recommend Apache over IIS if you have the choice.  It can even be apache on Windows.

Formerly someone else
Thursday, July 22, 2004

Worms? Boy, do I know worms!

This is absolutely horrible. You are a menace to society for putting an unpatched Windows server+SQL server on the internet. There is no greater transgression.

Right now, you need to destroy this computer. It is not safe. Not only blat the hard drive, but also destroy the physical hardware. Neither the BIOS, the processor, the memory, nor any other component of this computer may be trusted. In fact the screws on the case are kinda suspicious.

I recommend a 2500 degree kiln for three hours. Then, bury the molten and charred remains in a clay lined pit, with alternating layers of charcoal, sand and topsoil on top to depth of 8'.

In other words, the same treatment that Jimmy Hoffa got. More or less.

Good luck!

And ph'nglui mglw'nafh Cthulhu R'lyeh wgah'nagl fhtagn!

Cthulhu
Thursday, July 22, 2004

People:
This is great stuff, way more expert and specific advice than I hoped for.  I should also say that I got great advice emailed to me from Christopher Hester. 
A few clarifications:

-The colo facility is five hours away (Note to newbie sysadmins: *Don't* choose distant colo facilities.)  Thus, wiping the hard drive is a lot more hassle for me than installing a bunch of tools.

-I'm running my own business here, so I don't have a manager I can pump for funding for training.  Looks like it's school of hard knocks for "sucky"!

Here's what I'm going to do:
-Drive to colo facility. 
-Unplug network connection from box. 
-Reformat and reinstall Win2k with current patches. 
-Install SQL Server. 
-Install SQL Server 2000 Service Pack 3a.
-Disable NetBIOS
-Use Win2k's built-in TCP/IP filtering to filter out all ports except 20, 21, 80, 443 and 3389 (ftp, http, https and Terminal Services)
-Use Win2k's built-in TCP/IP filtering to disallow all UDP ports
-Plug network connection back in.
-Go home.
-Using Terminal Services, install Tomcat, my web app, and all my third party junk I need.         

Here's what I am not going to do:
-Buy a hardware firewall.  I really can't afford it.

What am I missing?  Any corrections, addenda, and thoughts are very welcome

OK programmer, sucky sysadmin
Thursday, July 22, 2004

You can get a used Cisco PIX for around $600 on eBay.  You'll spend about that much on the time lost driving to/from your colo facility with a possible overnight stay.

example
Friday, July 23, 2004

My friend, you have no idea of my cheapskate ways.  For $600 I could visit a server in Timbuktu.

OK programmer, sucky sysadmin
Friday, July 23, 2004

What's wrong with getting a really cheap hardware router/firewall, like you'd find on the shelf at Best Buy?  You're spending as much on gas to drive to your colo as you would on a cheap box, and it might save you another drive.


Friday, July 23, 2004

Well, true, but what would it do for me that I'm not already doing by using Win2k's built-in TCP/IP filtering to shut down every port except those for http, ftp, and Terminal Services?

OK programmer, sucky sysadmin
Friday, July 23, 2004

---"An admin puts a default unpatched install on the open internet and you're going to blame the OS?"-----

Yes. And the word "default" tells you why.

Stephen Jones
Monday, July 26, 2004

*  Recent Topics

*  Fog Creek Home