The Old Joel on Software Forum: Part 4 (of 5)

Beta Testing Pointers

What are the qualities of a good beta tester and how do you ensure you get selected? Should you be a programmer or at least have some programming knowledge? Should you even be experienced in the program field?

What exactly are you expected, as part of your job, to report on? Fist impressions, installation, UI, documentation, ... are all parts of a good product. Do you report everything? The background color is ugly? The toolbar is missing the top border? I can't do <what-ever>. Is all this part of it?

Just about anything you've got I'd like to hear.

Might Just Give It a Whirl
Saturday, May 22, 2004

Well, sending nice gifts never seems to hurt. The bigger the better, IMO.

Anon-y-mous Cow-ard
Saturday, May 22, 2004

We send our beta testers a bottle of wine and some chocolates / biscuits.

Just a nice gesture, we don't tell them they'll get this before hand so we get enthusiastic participants.

Koz
Saturday, May 22, 2004

Probably wouldn't want programmers, unless the product is a tool for programmers. Developers seem, in general, not all of them, to have too much respect for code. They tend to test for what the code says should happen, not what the design says should happen.

Also, a beta tester should have good domain knowledge for the app. You want a user who's more advanced in the app but is otherwise an average Joe in computers, otherwise he'll just work around your problems instead of reporting them.

If you can get people who realize that "Background colour is ugly" and "Feature foo deletes bar component" are both valid issues but should be given different priorities then your much better off as well.

Cheers
John Q

John Q Tester
Sunday, May 23, 2004

John Q

Thanks, I'm of the opinion that you want people who'll report every oddity they find - from people with a strong domain knowledge (who may naturally jump what may be confusing steps) to domain novices (who'll stumble on them). You can prioritise the returns any way you want (I think) getting as much feed-back as you can is the object.

"Excellent feed-back, just what we need." is my objective.

Might Just Give It a Whirl
Sunday, May 23, 2004

Above everything else, I think a good Beta tester has to give good bug reports.

For example, don't report "The background colour is wrong".

Instead, report "The background colour on the password confirmation dialog doesn't match the colour defined by my desktop theme".

Think "How can I direct someone to reproduce this bug" - if the developer acting on the bug report can't reproduce the issue, it won't get fixed.

Of course, sometimes you can't do more than say "I was typing in the password field and the application just disappeared" - but do pass on all the information you have.

To get selected to beta test a product, you generally need to prove your value in advance so that the company running the beta program knows you will provide useful bug reports.

So, start by providing useful (detailed, even) reports on the software you already use. If the company doesn't want to recieve the reports, post them on the web for others to read (that way, a different company can google for your name and see your value).

Just my 2c.

Bevan Arps
Sunday, May 23, 2004

Bevan

Thanks for joining in, way more than 2c in value in there for me. I'm starting to think that what I'm primarily looking for is the definition of a bug. BSOD - that's a bug. Program disappears with the log-in dialogue (that actually happens with this program - under certain circumstances - indeed, a bug.
I don't want to start reporting missing 'features' as bugs or is it if ... it looks like it _should_ do XYZ but it doesn't then that's a bug. Normal function for an ABC type application that's missing ... a bug? Menus/Tool-Bar buttons that are enabled when they shouldn't be ... a bug? I think they are but on the other end all these little things may get annoying.

Thx again

Might Just Give It a Whirl
Sunday, May 23, 2004