Fog Creek Software
Discussion Board




measuring quality of testing

I am attempting to measure the quality of the testing my project is undergoing by sowing known bugs into the application before sending it off to testing. By seeing how many of these known bugs the testers pick up I can get an idea for the number of unknown bugs remain in the code. I will likely have to argue this approach to the rest of the team if I want to use my results publicly and am looking for some backup articles on the subject. This is an idea I'm sure I read about on the web before and I'm pretty sure it was on joelonsoftware.com but I can't find the article, does anyone know of the article I'm thinking of?

Thanks

Dave

David Sharp
Friday, January 10, 2003

No, but you could start here:
http://groups.google.com/groups?as_q=defect%20seeding&safe=images&ie=UTF-8&oe=UTF-8&as_ugroup=comp.software.testing&lr=&hl=en

Danil
Friday, January 10, 2003

Or save yourself some time:
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&selm=7asm0f%24d5a%241%40oak.prod.itd.earthlink.net

Danil
Friday, January 10, 2003

Sounds exactly like the Jewish custom of stashing little bits of bread ("chametz") around the house on the night before passover to make sure that your search for bread was truly exhaustive.

http://www.jajz-ed.org.il/hartman/games.html

If you do this, though, I recommend a more binary metric -- recognize that this is just a sanity check to see if QA is really happening at all. My experience with testers is that there are an awful lot of testers that are just not doing any testing whatsoever. It's not clear what they're doing but they wouldn't notice if the application blue screened when you ran it. Hiding known bugs would uncover this. But DON'T try to quantify things or rate testers or treat the result as a numeric value, because if you do you're doing something that's pretty much not scientific and attaching hard numbers to it to give it a scientific air, and therein lies madness. Treat the result as binary: yes, it looks like QA is happenning, or no, it looks like it isn't...

Joel Spolsky
Friday, January 10, 2003

This practice is called "Defect Seeding".  You'll get lots of great articles by searching for this phrase on Google.

Craig
Friday, January 10, 2003

Well said, Joel.

I feel that, if you don't trust your testers, you need to do something about it (through defect seeding, like in this example).  But provided you do trust your testers, then trust them; don't try to quantify their work with particular precision.

Brent P. Newhall
Friday, January 10, 2003

Thanks for all the comments. As Joel mentions, I'm intending to use the results just as a rough guide to whether testing is taking place, not as a hard value as to how good our testers are.

The follow up question I guess would be: how do I demonstrate the results I get to others without revealing the hard numbers which are likely to get abused as described?

Cheers
Dave

David Sharp
Friday, January 10, 2003

What sort of authority do others have, and what authority do you have?

Can you say, simply, "I've come up with numbers, and without getting into specifics, I've found...", and will that be sufficient for those to whom you'd be presenting your concerns?

If you revealed the specifics, how do you fear it would specifically affect your organization?

Brent P. Newhall
Monday, January 13, 2003

As a relatively junior developer on the team, though responsible for the complete front end data-capturing application for the overall system, it is unlikely I can voice my opinion about the results without being asked for some justification and explanation.

So far in the testing half of the ten defects seeded have been found which gives me cause for hope. Unfortunately the half that haven't been found (yet) are the more subtle ones, e.g. obvious values such as the date the information is entered being hardcoded to 7th Jan have been found but others such as numeric values always being multiplied by 100 have not.

My concerns about revealing the raw statistics were originally that if few bugs were found it would be unjustly hard on the testing group. My view has now changed and I'm more concerned that if the majority of obvious bugs are found but few of the subtle ones, it will be used as evidence that testing has gone well when whole aspects of the application (i.e. accuracy of the numbers) may be relatively untested.

David Sharp
Wednesday, January 15, 2003

You appear to be making an assumption with defect seeding that all bugs are coding errors, and ignoring checking the implementation against the requirements and spec., testing how does your application perform as a 'system' etc. You may not care that much about those types of errors, but you need to define your goals first.

The reason I bring it up is my experience has been that testers and developers have opposing (testing) strengths. I've always thought the real benefit of dedicated testers is because they have a different viewpoint and are in a different reward/punishment environment.  Developers normally find most of the bugs.

Somehow defect seeding makes me think of the periodic push towards mathematicly proved correct software. 

Eric Moore
Wednesday, January 15, 2003

Concept in question is called Defect Seeding!

It is described in McConnell - Software Project Survival Guide - pages 227 thru 229 and can be used to estimate defect density. (McConnell claimed to have got idea from early work by Tom Gilb)

Defect seeding is not advocated as a way to measure the quality of the testing team - but rather as a method to estimate number of defects in the software.

Only projects with tons of free time and excellent source control should dabble with this approach (in my opinion) since when you inject defects you can be doing more harm than good.

Note - same approach is used to estimate fish populations in lake by seeding with tagged fish.

Robert Sabourin
Thursday, January 30, 2003

*  Recent Topics

*  Fog Creek Home