Fog Creek Software
Discussion Board

Test Plan Insanity

I'm working on writing a series of test cases for software that is supposed to ship this later this month. As I'm writing the test cases, the sheer number of combinations of various options in the software is making it seem more and more impossible that the test cases will ever be complete.

For example, on one of the tabs in our Preferences dialog, there is PREFERENCE 1, which has three different options A, B, and C. These are represented by checkboxes, any or all of which can be selected at a time (for a total of 8 combinations). On this same tab, there is another option, represented by 3 radio buttons. This gives a total of 24 different combinations of settings that can be input for ONE option. I have to write two test cases for each of these combinations (one case verifies a PASS and the other verifies a FAIL).

So, that's 48 different test cases for one preference option.

This particular product has at least two dozen such preference options, many of which interact with eachother. The number of configuration scenerios increases geometrically with the number of configurable settings added to the software. Mathmatically, I'm seeing thousands of possible configurations. And, honestly, this is not that complex of an application.

I'm a strong believer in the importance of testing, but I'm curious in how test plans have been implemented in the projects that you work on without setting the project late by another 6 weeks.

Benji Smith
Monday, April 22, 2002

This is not that unusual in software testing.  You are going to have to make some intelligent, educated guesses about which preferences interact with which other preferences, and only test the ones that you think are likely to cause trouble with each other.

But if you have a lot of preferences which are likely to interact with each other, that is as good argument as any as to why a feature should not be implemented simply because of testing costs.  I have seen testing departments kill features that seemed simple to implement, powerful, and useful, simply because they were likely to interfere with so many other aspects of the software that they could not possibly be tested in finite time.

Joel Spolsky
Monday, April 22, 2002

Yes, combinatorics will get you every time. I saw a similar situation come up recently where a "simple" set of preferences yielded 14,688 different combinations that should be tested.

When I was faced with this sort of problem in the past, I found it useful to make sure that everyone involved knew early on that not all the possible combinations could be tested.

Here's a way to maximize the benefit of your (necessarily) incomplete testing.

1. Do most of your testing with a "standard" set of preferences.
2. Test each option individually, to ensure that it appears to do what it's supposed to do.
3. Do a little bit of unfocused testing on  "sensible" sets of options - create use cases for "novice" and "expert" users, for instance.
4. Investigate test automation. If you can write a script that iterates through the combinations, that will help increase your test coverage.


Mark Bessey
Monday, April 22, 2002

Write the test-cases and test-automation early in the project... build self-test code into the application.

Do "white-box" testing (source code inspection) to see which options are inter-related (need to test all combinations) and which are mutually independent (should test each value but need not test all combinations thereof).

Christopher Wells
Monday, April 22, 2002

couple of tips:
1. You NEED to work with product managment tell them that if they want to make a release by date X they will need to give you some some "mainstream usage scenarios" which will help you prioritize the top 3 or so combinations of options which are considered most likely

1a. make sure you start testing any additional implmentations only after development has stopped (this is called a code freeze where any additional code can only be checked in to fix bugs, not add features, because it may introduce new bugs) so you need to make sure you have several weeks after product is feature complete to test and fix bugs only.

2. option dialogs, by definition need to be reflected in the api (HMM placebo options dialogs intersting idea), so if you can(have time,know the api well), test the rest of the option buttons using the api, although this is more difficult with api with graphical reflections (such as checking if a font appears correctly).

3. this can only be a stop-gap measure, because once you test these combinations, the developers WILL introduce new bugs through subsequent releases and you WILL need to automate this, if you post what your app is (web,desktop) I can give you more info on this.

Daniel Shchyokin
Monday, April 22, 2002

"...testing any additional implmentations"

sorry meant combinations

Daniel Shchyokin
Monday, April 22, 2002

In the early 90s, "Communications of the ACM" had an article about testing various Unix utilities.  The authors had written software to feed random data to Unix programs (including interactive ones like vi).  They found that about 35% (if I remember) of all programs crashed when fed arbitrary data and some even caused the kernel to panic.  If a combination of data was found to crash a program, enough state (on the testing program side) was kept to repeat the same sequence of data to find out *why* the program being tested crashed.

Now, how does this relate to you?  Have you thought of just randomly setting configuration options and see what happens?  While not as rigorous as testing all possible combinations, you may be able to test a significant portion of the possible states and by doing it randomly, you should get a decent spread of states as well.

Sean Conner
Tuesday, April 23, 2002

Here is your test plan (I'm serious),

"Test all combinations of checkboxes and radio buttons to ensure functionality and output is in line with expectations"
Some examples of expectations are:

(dont list all 48) acknowledge that the tester has a brain.

Tuesday, April 23, 2002

that might well be the test plan, tony, but it is not a test spec. if you want to rely on not only your testers' brains but also their memory then that is up to you, but most of us need repeatability.

Tuesday, April 23, 2002

The test manager at our company actually DOESN'T want repeatability. He insists that test plans be kept vague (e.g., the "expected behavior" states that the software "works as expected.")

He claims this provides better coverage, since different testers will test the product using different sets of inputs, etc.

Of course, he ignores the fact that some of the testers have NO experience with systems like ours, and NO expertise in the field that our products serve, so that "works as expected" is meaningless.

Oh well, at least it ensures job security for developers, since it takes thousands of man-hours just investigating the nonsense bugs that these guys file, since they don't have a clue when the software is broken.

Steve Gilg
Tuesday, April 23, 2002

> that might well be the test plan, tony, but it is not a test spec. if you want to rely on not only your testers' brains but also their memory then that is up to you, but most of us need repeatability.

I've written code to generate a test document where the code automatically iterates through combinations to print the various each-slightly-different test cases plus boilerplate... but more useful IMO is to reduce the number of tests (by decoupling features, e.g. when source inspection tells you that you can), to automate the tests (instead of writing code to make the test plan, write the code to run the tests), etc.

Christopher Wells
Tuesday, April 23, 2002

You either have a very serious design problem or the situation is not nearly as bad as you are making it out to be.

If your large N number of option combinations each produce a specifically different behavior, then your program must be nearly impossible to learn, and is therefore a bad design.

I think you will find that in fact you don't have a*b*c*d*... valid combinations of options, but a relatively sparse subset of the exhaustive matrix.  Moreover, of the valid sets, _behavior_ will be independent of other options settings (again, otherwise your program is hella confusing for a user).  So you may find that you can kill off a huge block of the option matrix with one test case - two valid states which are behavior changing, and all other valid combinations which should not change the behavior.  Automating this can be tricky depending on your language, however.

Something I try to keep in mind as I go along: if a program is hard to test, it is probably also hard to use.


Robert Anderson
Tuesday, April 23, 2002

First, go get a copy of "Testing Computer Software" by Cem Kaner.  You can also look at his site, There was an article about testing undocumented software with not enough time, which might not solve your problem but sure sounds relevant.

Get over the idea of testing every single combination, because you won't have enough time.

1) Prioritize.  Separate the crucial items from the less important ones. Crucial means that this feature can screw up a lot of other features if it's broken, or it's associated with the most risk.  Make sure critical bugs are tested.

2) Find equivalence classes. If you're testing a division function, testing division by 4 and 2 is redundant. Dividing by 2, 0, and the biggest value possible is more informative.

3) Make tests do double duty. If you're testing radio buttons, you're also testing that the dialog comes up. If you have a lot of platforms to cover, and there's a natural division among test cases which aren't dependent on platform, farm them out.  Test the client under windows, the server on solaris, and the db with oracle for one set of tests, linux/nt/postgres for another, etc.

4) Finally, when you've got your plan of attack, show it to  the manager you're accountable to. Make sure your prioritization matches, describe what can be done in the time allotted, and define risks ( like "every new release means x hours of basic testing", or "this section can't be tested until that one is stable").

Look around on the web for the "Testers Network" . It's a repository of articles that came from ST Labs. The company got bought, but the articles are still out there somewhere. It has some articles describing test case design and coverage.

Good luck.

Tuesday, April 23, 2002

Your testing is getting bigger than your project. This is of course, very silly.
It is of no benefit to spend hours and portions of rainforest describing the minutae of checkbox/radio button permutations and combinations, as mentioned above, far better to use an automated testing tool to test the combinations and spend the time developing the test rather that writing about it. At the end of the day your testing must be as efficient as it can be, writing the test plans to cover all options is about as far from efficiency as you can get. Cross all t's and dot all i's is you want too, I have seen test plans like this and my experience was that the tests were either not done because they were too trivial and testers merely ticked the boxes anyway. In the cases where time was wasted testing the minutae sometimes the show-stoppers slipped through the cracks.

Tuesday, April 23, 2002

Take things into account such as release date, budget, staff, complexity, etc.  Don't be dogmatic.  Use judgement, and find a right balance. 

Yes, part of making software is subjectively using good judgement.  (If there wasn't more to it, then it could be automated!) 

Thursday, April 25, 2002

christopher - out of interest, how did you test your code to generate a test document?

Thursday, April 25, 2002

> christopher - out of interest, how did you test your code to generate a test document?

Who tests the testers? I didn't test formally, but 3 ways:
1.  I wrote/debugged the document generation code myself (a programmer moonlighting as a tech writer, I trusted myself to do that)
2.  I inspected the resulting test document
3.  Ultimately, when the test document is used to test the product software, then if ever the test document "disagrees" with the product software then a human inspects to decide whether it's the product or the test document that's in error.

Was it worth the extra complexity? I thought it was the only way that I could do it at all. The test plan was an afterthought on management's part, and they asked for a complete (multi-hundred page) set of idiot-proof test cases (to give to a small army of QA interns) to be written in a couple of months. Automating it was the only way I knew of to develope/maintain a document of that size in that timeframe (copy-and-paste, and global search-and-replace, only go so far ... they're insufficient where e.g. the structure of all test cases change).

To help with step 2 (inspecting the document) I built variable information into the name and sequence number of each test case. For example if you have three variables each taking a value 0 through 9, then the corresponding test case titles have names beginning with 0.0.1, 0.0.1, 0.0.2, ... and looking at the auto-generated table of contents at the begining of the document, it's easy to see whether you have 1000 test cases titles which follow the above pattern.

My one feeling of guilt about it is that I wrote the document generator using Script/VS which few people know, so when I left (I was a contractor, and management had no interest in how I did something as long as I did it) they would have had a difficult time maintaining it (but, management can't have everything). The next time I did anything at all similar (I was told to convert an SGML document to HTML, while the SGML document was still being written by someone else) I did that my writing a Rexx program, which the permanent people there would have been more familiar with.

Christopher Wells
Thursday, April 25, 2002


The test plan is finished now, with 200 specific test cases (some test cases require the same task to be performed multiple times with different settings).

When I made my original post, I overreacted a bit regarding the number of configuration permutations. There are some settings that produce different functionality when combined with other settings, but the number of these types of settings is somewhat rare, and most options are tested on their own rather than in combination with other settings.

Thanks for the input.

Benji Smith
Friday, April 26, 2002

*  Recent Topics

*  Fog Creek Home