Fog Creek Software
Discussion Board




FogBUGZ filling up BugEvent table

Hey,

Over the last day or so, the server that has our FogBUGZ stuff on it slowed to a complete crawl.  We think we now know why, and we think it was because of FogBUGZ.

Starting at 6:36:55 AM EST on 9/11/2003, something was filling up the BugEvent table for a single bug, sometimes at about a rate of 5 records per second.  I think we counted upwards of 88,000 records.  They all link to ixBug = 1, with sVerb = Edit.

dllhost.exe on that server went up to over 1.8GB of memory usage.  When we shut down the particular web site that was running this instance of FogBUGZ, everything was fine.

Is this due to Dispatcho or something?  Obviously we don't want this happening again :)

Thanks.

Ryan Duffield
Friday, September 12, 2003

Never seen that before... Sounds more like a runaway dllhost than a dispatcho problem.

If you only shut down your web service and dispatcho was causing the problem, when your site came back up it would continue its carnage.

But then if it wasn't dispatcho, it was either FogBUGZ itself that was adding all those records, or IIS was in crazy land.  I suspect the latter but of course, if its the former, I want to know.

You can enable logging here:
http://www.fogcreek.com/fogbugz/kb/howto/EnabletheFogBUGZLogFile.html

It might generate a lot of data though, so if the problem doesn't happen ever again, set that to 0 and delete the log files.

Michael H. Pryor
Friday, September 12, 2003

Apparently when the web site for that particular FogBUGZ installation is stopped, (we have several for multiple projects), the problem stops.  Start it up again, and the problem also starts again.

Through our own logging, we know the problem started with some event in that particular FogBUGZ site.

It looks like the records being added are one of those "system message undelverable" messages from postmaster@ one of our domains.

We'll turn on logging either this weekend or next week and see where that leads us.

Thanks.

Ryan Duffield
Saturday, September 13, 2003

You may be in a infinite mail loop situation.  You should shut down dispatcho and check the mailbox with outlook express.

Its possible you have set one of your FogBUGZ user accounts to have the same email address as one of your Dispatcho email accounts.  We warn you in the UI not to do this, but there are ways around it.  If this happens and the notifier for the project that Dispatcho is assigning the email to, is itself the mailbox dispatcho is checking and you have mail notifications turned on... then when Dispatcho creates the bug, it will notify the user (itself) and that will generate and email, which will be put in FogBUGZ, which will cause another notification to be sent, and so on and so on.

Michael H. Pryor
Saturday, September 13, 2003

Hi,

Well, the Dispatcho email accounts are definitely separate than any email addresses anyone else personally uses and thus no users within FogBUGZ itself have the same email address.

I guess we'll have to first try downloading the mail via OE to see if that helps.

Ryan Duffield
Saturday, September 13, 2003

Clearing out the mail box seems to have resolved this issue, thanks.  Presumably this happens rarely, but is there any effective way of preventing this from happening again in the future?

Ryan Duffield
Monday, September 15, 2003

Yes, slow down the rate it checks the mail server. There is a setting in the mailbox screen for the number of seconds (maybe milliseconds) between checks.

What is going on is that you are downloading a large message, and because the time between checks is too low, it is attempting to check your mail again before the first check is complete. This cascades and you get the mail loop.

The fix is to give is a good 2 minutes between checks. It doesn't seem to affect Microsoft Exchange (I've never had it happen with it in fact), but SendMail 8 pukes like an infant when this happens.

Marc
Friday, September 19, 2003

*  Recent Topics

*  Fog Creek Home