The Old Joel on Software Forum: Part 4 (of 5) - Prioritise features suggested by beta testers.

Prioritise features suggested by beta testers.

Hi Joel,

I just read your article on beta testing. I am developing a code generator product for which there will always be a free version -I will add a professional version at a later date - and was wondering how I could reward beta testers who give me valuable feedback in the mean time.

What do you think about giving priority to features suggested by beta testers who provide valuable feedback?
http://www.dotnetjunkies.com/Weblog/ScottMunro/

Scott Munro
Friday, March 19, 2004

If you do, I hope your beta testers are fairly representative of your user base.

Tapiwa
Friday, March 19, 2004

Shouldn't a beta be already feature complete?
*duck*

I'd say accept all feedback, but do not try to "formulize" your work priorities from them.

Just me (Sir to you)
Friday, March 19, 2004

Is your questions then about how to reward the good testers?

When my company gets big (*grin*) I think I would love to send all the helpful beta testers a shirt. Kind of simple design that reads 'beta tester' or something and has the calebsoftware logo in small letters.

For now, I am thinking about maybe getting a few book vouchers at the university booksotres/or cd vouchers and offering them as prizes. Maybe a couple of random draws for any tester providing feedback. Maybe I'll give a few out to the guys that gave the best feedback(ie hey, its only a $10 voucher, but I really appreciated your feedback, in fact you were one of the most valuable members, so I am going to put your name in the program credits, and hope you will be around next time we need testers).

Aussie Chick
Friday, March 19, 2004

Tapiwa: I think that Joel's article provides some good suggestions for getting quality testers. Whether they are representative of your user base is of course another question.

My product, (a code generator), will be used only by .Net developers who are sold on the idea of code generation. Granted they weren't all made with the same mold but I am hoping that they will have similar goals and use the product the same way.

I just realised that I did not include a link to Joel's article in my fiirst comment.
http://www.joelonsoftware.com/articles/BetaTest.html

Sir : ) Yes a beta should be feature complete. Joel sounds fairly strict on this. He goes so far as to say that adding a new feature means that you need to start a whole new beta test.

I am of course referring to features for the next version - here's hoping that NGenerate is accepted well enough to warrant a next version!

You suggest that I should not "formulize" my priorities based on feedback. I agree that something along the lines of...

"Feature request from number one tester = Number one prioirty feature for the next version."

could lead to some disasterous results. However, is an approach that considers elements of a potential new feature, (such as ease of implementation, availability of other products on the market etc), and weights these with the value of the tester who suggested it necessarily such a bad idea?

I like the concept of a community driven product where new features are suggested and discussed on message boards and blogs etc. Ideally, I could be implementing only new features which the community has already refined for me and seems generally eager to have.

Contrast this with what in some cases could amount to a trial and error approach where new features are added without the benefit of prior exposure to a community of users. The feature may be a welcome addition. It could just as easily receive a 'ho hum' response, (or worse).

There is also a danger that a new feature is considered worthless without a complimentary feature. I am struggling for a relevant example here but am thinking of a car manufacturer who fits out a car with a compartment for a super duper tyre spanner but does not allow space to easily carry a spare tyre.

I put a lot of effort into thinking about the way that a developer will use NGenerate. Being a developer myself, I would like to think that I can understand the motivations that a tester would have for suggesting a feature and be able to make a good decision as to whether or not it should be included. It is entirely possible that I could ignore a feature request because I don't see some hidden benefits.

Is it worth then, taking a leap of faith with a tester, (one you have recognised as providing valuable feedback), who does not have the time or ability to articulate the full motivation for his feature request? The leap would be only a weighting on the new feature but may be enough for a diamond that was going to disappear into the rough to see inclusion in the product.

Aussie Chick: I guess I am fundamentally interested in attracting good testers. Rewarding them with priority for feature requests is the idea I have for doing so.

My concern with T-Shirts and book vouchers is that testers who are not interested in seeing your product improve send in some feedback to score the freeby.

I think your idea of simply telling the tester that their feedback was helpful and that you would be unhappy to see their input end has a lot of merit. This could be an effective means of giving the tester a feeling of ownership for the product. Obviously it would need to be in the form of a very personalised response and make reference to the specific input that was valuable. A mail merge could be very counter productive.

Scott Munro
Saturday, March 20, 2004

If you have a product that your target market will be very interested to see, it seems like it should be fairly easy to get at least a few beta testers.

Do the first beta testers have to know that EVERYONE is getting it free?

Since you're giving it away free (when completed), I assume you're not providing free tech support also, correct?

One benefit for Beta testers: free tech support.

Mr. Analogy
Saturday, March 20, 2004