The Old Joel on Software Forum: Part 4 (of 5)

What do beta testers do?

What is the difference between Beta and Alpha testing?

Is it the job of Beta testers to find code-related bugs, or more to report design related problems?

It would seem to me that code related bugs that are left to surface in the beta testing phase are a result of poor design and poor inbuilt debugging (all of which I am very guilty of...but learning!). The beta testing phase seems to be alot more unstructured then any other part of the design process.

Basically give the program to 100 people, and hope they find bugs?

So what level is my program supposed to be at before I send it out to be beta tested? Bug-free, with the beta testing results expecting to be system or design related?

(Yes I have read Joel's recent article)

Aussie Chick
Thursday, March 11, 2004

I think you'll see a variety of responses to this question, as everyone seems to do it a bit differently.

I've always treated it as a "Get as many people on as many different machines with as many different habits/configs/use-cases to run/interface with the program to turn up any edge-cases and/or bugs that remain".

Lou
Thursday, March 11, 2004

So by the time a product is released for beta-testing, is it expected that the product be basically a production release (except for the fact that edge-cases etc are expected, and user feedback may influence some final changes)?

Aussie Chick
Thursday, March 11, 2004

I was always under the impression that the following conditions existed for alpha/beta/release candidate

Alpha: Not code complete, not necessarily stable
Beta: Adequate code completion, stable
Release Candidate: Code Complete, stable

Personally, I never saw much value to alpha releases other than to get very early feedback as to the design of a product. Input at this phase has the ability to generate large changes in a design. If you've done your research prior to coding, you shouldn't need to do it during coding.

Beta releases have much of the code in it, but not all features are complete. Branding may not be done right. Artwork is rarely final here. The goal is to elliminate bugs in the complete features and start testing the product using real input from real users. Not made up test cases.

Release Candidates are when the code is complete. Here the goal is to find the frequency of occurance of the remaining bugs in real life. Fix the ones that occur the most frequently. Then release.

After that, feedback from the released product goes into the next release phase, and the new alpha starts development.

Rinse and repeat as necessary.

Elephant
Thursday, March 11, 2004

To me, Alpha testing is for useability. It should be done with a small number of users (maybe 2 to 5). The goal is to find where design changes DO need to be made, and find them EARLY.

Elephant wrote : "I never saw much value to alpha ... Input at this phase has the ability to generate large changes in a design. If you've done your research prior to coding, you shouldn't need to do it during coding.
"

One issue here is that you can do all the research you want, but the target customer (end user) may not truly understand what you're creating till they see it.

Now, by "research" if you mean that you showed them mock ups so they could cleary see how it works, then I agree.

However, if they haven't gotten a chance to "play" with at least a mock up, then you need to let them do that at some point, perhaps in Alpha testing.

We do Alpha testing sort of like you'd do useability testing.

The real Entrepreneur
Thursday, March 11, 2004

Good point Entrepreneur .

By research I did mean mock ups. Your typical VB type of thing done up real quick. Further though, I think that for some complex systems, such mock ups don't make sense. When the time taken to create such systems would be equivalent to the time taken to write the real code, I definately see the use of alpha testing.

Thanks for the (IMHO) excellent clarification.

Elephant
Thursday, March 11, 2004

Aren't alpha tests supposed to be internal?

John Topley (www.johntopley.com)
Thursday, March 11, 2004

They don't have to be. It can be very valuable to do alpha testing with some of your hardcore users.

Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, March 11, 2004

These terms tend to be abused. I find prototypes being called Alpha test versions, release candidates betas, etc. Our QA policyt has strict definitions as to what an alpha, beta, pre-release build, release candidate, etc, mean, but the sales and marketing people use whatever term will appease the customer.

Our QA policy calls any build of the software turned over to testing an alpha release. Once a release has gone through a round of testing, the product manager can call it a beta test and send it out to customers.

When I was a product manager I frequently got requests for betas from users who had no intention of providing me feedback. They just wanted an advance look at a release. Instead of involving them with a beta program, i just sent them an uncontrolled pre-release of a recent build.

pdq
Thursday, March 11, 2004

Most public beta releases are really marketing releases, I'd put all of Microsoft's and Oracle's public releases into that category. It isn't really to discover bugs so much as to seed interest and get corporate users switched on.

That's not to say that they don't do beta releases, I'm sure they do, but they won't be publicised.

Simon Lucy
Thursday, March 11, 2004

From a developer's perspective, doesn't it make sense to fix all your known bugs, and then send it out for beta to find new ones? It seems odd to me to send out a program with known bugs for testing.

Rich
Thursday, March 11, 2004