The Old Joel on Software Forum: Part 4 (of 5)

How do you find testers?

Number 9 of the top twelve tips for beta testing says, in part:

"The minimum number of serious testers you need (i.e., people who send you three page summaries of their experience) is probably about 100."

http://www.joelonsoftware.com/articles/BetaTest.html

But where are you supposed to find 100 people who have the time, skill and -- most of all -- interest to do such a thing if:

* you are a small company (i.e. you don't have hundreds of coworkers outside the project)
* your product is not a video game (heck, who wouldn't want to beta-test that?)
* you have a small, or non-existent, user base (e.g. a pre-release beta, or a very specialized product)

Joe
http://www.joegrossberg.com

Joe Grossberg
Thursday, March 4, 2004

That was exactly my question. We don't have 100 customers, let alone people who want to test pre-release software.

We send an email to all of our customers asking for testers, and probably 5 people want to do it.

This is an expensive, specialized application that we only sell to professionals.

Blargo
Thursday, March 4, 2004

Furthermore, as Joel himself says, only a fraction of the volunteers are going to give you useful feedback.

So it's really hundreds, not just 100.

Joe Grossberg
Thursday, March 4, 2004

Joe:

I have read Joel's article and having managed a few beta cycles, I can confirm that it's good advice. I think the 100 tester minimum should have included "(your mileage may vary)". Here's one experience I had many years back that might be helpful.

I inherited beta management duties for a Mac connectivity product that worked with DEC and UNIX servers. They had a list of 50-60 beta testers that they wanted me to use. It became apparent very quickly that 90% were simply on the list to get one of the following:
1. An advance look at the next version
2. Help in making a decision on a large site license
3. Some free software

With my mananger's permission I tossed out roughly half the names and gave strict instructions to Support and Sales not to send me people that just want to kick the tires, but really have an interest in testing it. I also went to an Internet newgroup specific to this type of software and posted an invitation to be part of the next beta cycle.

I ended up finding 12 people that really rocked! These folks easily generated 3 times as much feedback as the other 30-40 on the list and it was top notch data that the developers really appreciated. Just keeping these 12 great testers busy with requests from the developers was a full-time job. Having a message board would have helped, but this was just as people were discovering browsers.

For care and feeding of the testers, we would find extra t-shirts from trade shows and other swag to send them and they *really* appreciated it. Once we shipped the product, they'd get a thank you letter with 1 shrinkwrapped copy.

So my advice would be to start hanging out where some of your users might be and see if you can identify some "thought leaders". Invite them to check it out and slowly develop a roster. As Joel noted, only reward and keep those that offer useful feedback.

QA911.com
Thursday, March 4, 2004

Use a trial period and implant feedback reminders in the app's startup, along with an automatic bug email submitter ?

Joe Hendricks
Thursday, March 4, 2004

In my experience, public beta-testing is only for shrink-wrapped products where customers have a mix of unknown hardware configurations that you can't test in-house. You still need an internal QA testing process of some kind whether shrink-wrapped or not -- public beta is not for catching your bugs.

Its best use is just to make sure it doesn't destroy anybody's system, and/or for a market sanity check if you've been developing in a vacuum. (So when all you get back is "cool program" from a few people, that's a good thing).

But specialized products don't need it, since you're either installing for the customer yourself, or you can specify (or provide) the exact hardware platform you developed and tested it for.

Rick
Thursday, March 4, 2004

What is your goal with the testers?

Are you trying to test for bugs?
Or, are you testing usability?

Here's what we do:

Testing for BUGS
We release the new versions to our paying customers. That way, they get the latest and greatest. And, if they have problems, we hear about it. We fix bugs really fast (usually within a day or two if it's serious) so it's very little inconvenience for the customer.
NOTE: we do pretty through testing in-house before sending this out.

Testing for USEABILITY
I don't think you need 100 testers. Read Don't Make Me Think. He makes an excellent case for useability testing with just a couple of users.

We test with just a few customers who we have close contact with.

Then we release the product. We regularly update the program based on feedback. (Example: very few customers were using our "Deluxe and PRO version features". We suspected this was b/c the trial's button for those features was called "Advanced Features". That scared people (seriously). So we changed it to "More Options...". Many more people started using those features.

The real Entrepreneur
Thursday, March 4, 2004