Fog Creek Software
Discussion Board

4 eng. days to load test...

this is in reference to the latest update on Joel on Software discussing problems with the free trial/demo of FogBUGZ.

Don't shoot me.
I realise that the nature of the problem could be such that only extensive testing (or actual experience) may reveal where the problems were.

I'm not trying to put Joel down here, but wouldn't it have been better to simply 'fix' (optimise) the data server etc. before going online with it?

This would have prevented the potential loss of customers problem as well as being a good investment for the future (ie. in the event there will be more people using the trial/demo in the next release etc.) in the event that the server is ok.

Of course, there would have been no way to know how long it would take to optimise the server before the actual attempt, it would have taken hours to figure out a rough estimate etc.

Feel free to agree/disagree and/or criticise my post, I'm hoping to gain better insight into the issue and correct any faulty reasoning used in the above.

Bob Hu
Saturday, November 09, 2002

I don't think we would have known where the problem was that needed to be fixed if we didn't either do testing or go live.

To be more specific the problem was that the online trial server used Access databases as the backend. We thought that this would be OK; Access is not a great backend for datadriven sites but for the tiny amount of traffic that a single user trying FogBUGZ generates, we thought it could hold up (it has for the past 2 years).

We seem to have discovered that having large numbers of concurrent users, even if they are using separate Access databases, eventually causes some counter somewhere in the Jet libraries to decide it has had enough and it shuts down.

So the mistake we made was assuming that Access could only handle a small number of users *per database* when actually it can only handle a small number of users *per server*.

The fix was to create SQL Server databases for each trial user. That causes another problem (SQL Server can only have 32767 databases, and probably gets wiggy long before that) but we think we have a "scavenging" solution combined with demand-attaching the databases which will let us keep a working set well within that limit. (In fact, one reason we were using Access in the first place is that we thought that we would be able to have as many databases as we needed, limited by available disk space, since each one is a separate file, and we knew about the SQL 32K limit.)

The point I intended to make with my post is that sometimes it's more economically sound to implement capacity improvements only as they are proven to be needed, rather than overbuilding before you know the demand exists.

Joel Spolsky
Saturday, November 09, 2002

Thank you for personally replying to this post.

As I hope to be a manager in a software developement team (once I finish university) I'm always trying to learn more about what is practical, as opposed to what may sound good on paper. In this regard your articles (I've read all of them) have helped me immensely.

Bob Hu
Saturday, November 09, 2002

When I first read Joel's explaination about why they did not load test as well as the justifications for those decisions, I immediately started mumbling things like: "How can he say such a thing", "that is a cop-out excuse" and "the justifications are way off".  The proverbal softball was thrown my way (underhand of course) and I was getting ready to swing and blast it out of the park.

But then I paused and thought, "OK. I've put off some testing before due to time constraints."

Regardless of whether or not I agree with the justifications for his decisions, what I do understand is the conflict between time, resources and tasks. Sometimes you have to make tough deisions and when one (or more) of the variables in the tuple above are fixed (i.e *cant move*) something has to give.

As a final thought, capacity planning never ends and it *is* complex. Especially when your architecture involves more than a simple database and a few webservers. If anyone is interested, there is a good book discussing the theoretical foundations of capacity planning called: Scaling for E-Business by Menasce and Almedia (ISBN 0-13-086328-9)

J Bilger
Monday, November 11, 2002

*  Recent Topics

*  Fog Creek Home