The Old Joel on Software Forum: Part 4 (of 5) - Beta Testers

Beta Testers - What to do with them

What do you do with Beta Testers once you have them?

Get them to sign a non-disclosure agreement?
Get them to give you some details (name, age, university etc).

But then what? Just give them a copy of the program and say 'email me if you have any problems/comments'? Or do you give them a more sturucture process?

Aussie Chick
Friday, March 19, 2004

structured*

Aussie Chick
Friday, March 19, 2004

You should probably set up a web forum or something, because that way the testers will collate the bugs themselves by using the same thread. You can then gauge at a glance how many people are suffering from a particular problem.

Insert half smiley here.
Friday, March 19, 2004

I agree with that one, and am organising a discussion board for my site as we speak! thanks.

Aussie Chick
Friday, March 19, 2004

Definitely give them a place to report bugs, and make sure it isn't a black hole (such as just an email address). People like to check up on bugs they've submitted.

What to ask for-- well, some sort of contact info would be good. You probably don't need their full name & address, just a first name and a working email. But you might want to ask about experience level, their preferred operating system, and other things that are relevant to the software being tested.

You'll need to ask the lawyer-types about NDAs and other legalese.

Martha
Friday, March 19, 2004

Collect them and trade for valuable prizes?

Philo

Philo
Friday, March 19, 2004

"What do you do with Beta Testers once you have them?"

Well, Microsoft trys to makes us buy a newer operating system every 3 or so years.

As400
Friday, March 19, 2004

Philo,

this is an interesting concept. Perhaps a website www.betatester.org swap, trade, share etc etc.

Could be a very profitable venture...

----Seriously folks, do I just let the BTs loose with my program, or do I set some guidlines?

Aussie Chick
Friday, March 19, 2004

I say let 'em loose but make it clear they need to describe *exactly* what they did prior to running into any problems.

Then again, you may want to tell 'em to 'spend 50% of your time testing features A, B and C, 30% on features D and E and the rest on the remaining features'.

Make sure they test boundary conditions, i.e. for a group of checkboxes, make sure they 1) leave all checkboxes unchecked, 2) check all checkboxes and 3) check some but not all of the checkboxes. For entering numbers, have them enter lowest and highest values allowed, as well as numbers within and outside of the lowest and highest values allowed.

The Javanator
Saturday, March 20, 2004

Aussie Chick,

What you do depends on what you expect these beta testers to find. If you want to use them as a cheap alternative to a real testing process, give you can try and tell them what to do. I personally wouldn't rate my chance of them staying on board though, as they will spend their time doing the same thing over and over again on different beta versions (This is hard enough if you are getting paid real money).

The way I think about it, a beta test is for a piece of software that is already fully tested (ok, tested to the extent that seems reasonable in the light of resource and time constraints). The point for a beat test is to see what happens if "real users" try and do real things with your software. If you give your users too much guidance, they will only redo tests that you have already done and not really add a lot of value to your product. A beta test is all about finding the guy who wants to doubleclick on something instead of choosing File > Open and makes the software crash.

Florian Heine
Saturday, March 20, 2004

Florian, thanks.

Aussie Chick
Saturday, March 20, 2004

> as they will spend their time doing the same thing over and over again on different beta versions (This is hard enough if you are getting paid real money)

The developer typically doesn't want to do this either: which is why they write automated tests (to do so-called "regression testing").

I'd agree that the reason for beta testers is to give you 'the surprise factor': if you can tell them what to test, then you might as well be testing that yourself.

One thing you might ask beta testers to do is "click this button to run my automated test cases, and send me the results". This is to run your tests on the beta testers' machine (which are running who-knows-what versions of various softwares and hardware).

Christopher Wells
Saturday, March 20, 2004

Aussie Chicken :

What's the URL for your site ?

~ CycleBurner ~
Sunday, March 21, 2004

Javanator -

You describe what your own internal testing should be doing. Needless to say, your software should have passed this simplistic level of testing before asking a beta tester to look at it.

The beta testers should be trying to use the software to do the job you want them to buy the software to help them do.

IYSWIM. :-)

theWeasel
Sunday, March 21, 2004

what does IYSWIM mean?

the url for my site is http://www.calebsoftware.com having said that, the site is not up yet, only a jpg mockup of the site.

Aussie Chick
Sunday, March 21, 2004

IYSWIM = "If You See What I Mean"

Joey the Bellhop
Sunday, March 21, 2004

> What do you do with Beta Testers once you have them

Feed 'em to the fishes.

Monday, March 22, 2004