Top Twelve Tips for Running a Beta Test

Joel,

Great article.  We are an ISV and have learned many of the same lessons the hard way.  I especially like your idea of staggering the releases.  I would love to hear from other ISVs about this topic here.  It is one that isn't discussed much (as far as the brass tacks go) and I am intensely interested in it.

Here is my added tip:

* Have an external Alpha test

By this I mean that when your software is at the point where the GUI is more or less frozen and the features are (mostly) there.

For this hand pick a dozen or so of your "hard-core" users, you know, the ones who make a living with your software, give you suggestions all of the time and really seem to care about the future of the product.

Give them early development releases to get their feedback on the general direction of the product and communicate with them often.  On our last product we "adjusted" the direction of several important features and came out with a much superior product.

This type of external Alpha test will give you some feedback from users who know the ins and outs of your product better than most of the development team does because they use it in the real world all the time.  They will point out design and workflow problems that you may have never considered a problem at all.

The Not-So-Philosopher
Tuesday, March 2, 2004

Oh yeah, and reward the Alpha test customers WELL.  We give them multi-user licenses and other good shwag free.  Keep these folks happy, they can save you thousands of dollars!

The Not-So-Philosopher
Tuesday, March 2, 2004

I tell our Alpha testers that you can have input on the color of the house as long as you tell us BEFORE the paint dries.


What I mean is that it's much easier to modify the program (move buttons, change the UI a bit) BEFORE the manual and help file are completed and before final testing takes place.

The real Entrepreneur
Tuesday, March 2, 2004

I'd like to add one point:

For the love of God, please test your applications (and fix the obvious bugs) before you send out your application for a beta test.

I beta tested a product where some of the menu items would crash the application.  Just clicking the item would result in an immediate crash, for everyone.  This is a massive waste of time (both yours and the beta testers) and makes you and your product look bad.

Almost Anonymous
Tuesday, March 2, 2004

Um, what's the difference between tip #7 and tip #10?

Martha
Wednesday, March 3, 2004

> Um, what's the difference between tip #7 and tip #10?

You're the fifth person who noticed that, but the only one who commented. The next revision will replace #7 with the "alpha test" tip above.

Christopher Wells
Wednesday, March 3, 2004

  Recent Topics

  Fog Creek Home