Fog Creek Software
Discussion Board




Server Babysitting...

I have a strange morning ritual; the first thing I do before I shower or get dressed is I check my email.  I like to know, as soon as possible, what's going on for the day.

A long time ago, I built into our webbased software a system which automatically emails me whenever a error occurs on the site.  It traps most kinds of programming errors and the more mundane server failures.  It's absolutely indespensible for finding and fixing problems.

Today, I got bucketful of emails about connection failures to the SQL server -- the exact error: too many connections.  This is a common error on our server -- it's some kind of resource leak.  Normally I just restart the server once a month and don't concern myself with it.  The load on the server was also about 30 times more than usual for a Monday morning but I didn't think anything of it, probably just that resource leak causing problems.

So, I restarted the SQL and HTTP processes...  and everything was good for about 10 seconds.  After that, the load spiked right back up.  Checked the sites, all running very slow.  Checked all the processes and SQL connections -- there were lots.  "What the heck was going on?"

So now I'm thinking that there is some new worm spreading around the net hammering our server.  I start searching around the server for the logs.  It's just about then that I get an email:  It seems our website is featured front page in the New York times!  We're just being hammered by people checking out the site.  Wonderful, the best opporunity to show off the site and running slower than syrup from a maple tree.  I suppose it could be worse -- it could be slashdot!

I had a bunch of stuff to do today -- but instead I'm baby sitting the server.  Occasionally restarting it.  Occasionally checking the load.  Occasionally posting joel on software.

Mondays...

Almost Anonymous
Monday, July 07, 2003

Here's an idea... Fix your SQL server.

Fred2000
Monday, July 07, 2003

Either use entrprise mgr or qa to look at what is running and what it is doing.

Mike
Monday, July 07, 2003

Mike - never used Profiler?

Philo

Philo
Monday, July 07, 2003

Yes you can find out exactly what is going on with that.  But I thought EM or QA would be a quick way to see what user, app, whatever was problematic.  Then you could trace it with profiler.

Mike
Monday, July 07, 2003

My hosting company rebooted their SQL server because it was busy this morning too. I would never have noticed, but Inktome discovered my site and has been spidering it 24/7 these past few days so any down time generated an e-mail to me.

www.marktaw.com
Monday, July 07, 2003

Did you make sure you have the slammer patch installed on that sql server? Those are some similar symptoms of that worm.

Joel Sundquist
Monday, July 07, 2003

http://isc.incidents.org/port_details.html?port=1434

Here's the poop on the latest activity on the web that directly targets sql server

Mike
Monday, July 07, 2003

So what's the URL of the site?  Now you've got us all curious.

anon
Monday, July 07, 2003

BTW, I didn't mean to imply that we were using MS SQL server.  I guess I should use the even more generic term "database server" rather "SQL server".  My Bad.

Our database is MySQL...

Almost Anonymous
Monday, July 07, 2003

mine too.

www.marktaw.com
Monday, July 07, 2003

Cool, maybe a linux or mysql worm to shut the holier than thou penguinistas up.

Raynor
Tuesday, July 08, 2003

Zone alarm reported a couple fo dozen attempts by SQL destop server to access different IP adresses when I reinstalled it (Zone Alarm), and then suddenly they all went away. Anti-virus found nothing and it doesn't seem to be running anyway. Strange

Stephen Jones
Tuesday, July 08, 2003

Actually something is going on, here are a couple interesting links
http://lists.insecure.org/lists/incidents/2003/Jul/0000.html


http://lists.insecure.org/lists/incidents/2003/Jul/0001.html

Mike
Tuesday, July 08, 2003

Hey, so if there's a MySQL worm then that's reason to abandon MySQL wholesale, right? After all, that's what a lot of the Linux zealots said about SQL Server after slammer... [grin]

Philo

Philo
Tuesday, July 08, 2003

No, I am sure they will find another reason to stick with it.

Mike
Tuesday, July 08, 2003

What I don't understand is why you would ever have your database on an open port on your server! 

We run MySQL but we couldn't get a MySQL worm -- the process is not accessible from outside the box.

Almost Anonymous
Tuesday, July 08, 2003

Actually the New-York Times is worse than slashdotting.

Slashdot's smell is worse though.

Application Specialist
Tuesday, July 08, 2003

*  Recent Topics

*  Fog Creek Home