Who handles writing test documents?

Any opinions on how this should be done?

Crimson
Thursday, April 29, 2004

In our shop it's : "He who writes the spec, writes the test."

Anybody see any problems with this? Assuming that the above mentioned "He" is technical enough to write the test cases as well as the docs -- which "He" is in our shop, no matter which "He" writes the spec.

Sgt. Sausage
Thursday, April 29, 2004

Yep,
The problem is, is writing tests the best use of his time?

The Artist Formerly Known as Prince
Thursday, April 29, 2004

This depends lots on the product and the dev environment.
Like most startups, we use - out of necessity - a
basically "agile methods" approach with "light" specs,
lots of refactoring and constant testing.  (But we don't do
pair programming as we only have three developers who
mostly work from home.) 

Anyway, the developers write their own tests.  We
created an elaborate test infrastructure which makes
it easy to add tests to the test universe, so that they
can be run all at once in a "lights-out" fashion.

x
Thursday, April 29, 2004

Also, we don't write test DOCUMENTS, if by documents
you mean elaborate descriptions of the tests.  We do
write comments in the test scripts explaining what
they're meant to do.  (We have about 3,000 test
scripts, so a "document" for each one would be a huge
PITA - and more importantly, would never get read anyway.)

x
Thursday, April 29, 2004

"He who wants to write the code, must write the test. First." :)

Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, April 29, 2004

"He who writes the spec, writes the test".

Un, no.

The spec writer will be biased, and will have more understanding of the DUT than anyone else.

Someone else should read the spec, then write the test based on his interpretation of the spec.  (Different interpretations are common in an office having diverse ethnic backgrounds with varying degrees of ability in English.) In all likelihood, you will find errors in the spec, which at this stage in the project, is the right time to find them.

The verification engineer should write the test plan, consulting with the designers as required.  This works for software just as well as it does for ASIC design.

David Jones
Thursday, April 29, 2004

In our shop, if you want to write code, you write tests. Tests come first. No tests, no code check in.

anon
Thursday, April 29, 2004

my only concern is that writing the test 1st then the code will prevent many scores of bugs from coming in.

As anon stated though, no test, no check-in.  So I write the code 1st.  Now I need to check it in, so I write the test.  Woops!  I just let hundreds of possible bugs in.  Since the code was written first, my mental bias is going to be to develop a test that fits the code and proves that it works.  You can't avoid that.

I think the best solution is to have your coders read the spec and code.  Then have testers read the spec and write tests.  Then have other testers read the code and write tests.  Now you've got the code being checked and tested as code (white box), you've got the spec being checked by both the testers and the devs for problems, you've got the testers using the spec to develop tests for the code (black box), and if you want to have your devs write unit tests for good measure, all the better.

Moral of the story, why does everyone allocate 10% of the product's lifecylce for testing when everyone knows that testing takes 50-60% of the product's lifecycle?  So skip the lie and do it right from the start.

Elephant
Thursday, April 29, 2004

The initial question is a little vague, so the answer is it depends on what level of test documentation you are talking about.  Depending on the scale of the system you are working on, you will end up with test documents at the system, component, and unit levels.

If you have a properly written spec, it will specify the behaviours you need to verify at the system level.  You do not need to have the spec writer authoring the system test plan, but they should verify that the test plan covers all specified requirements.  Tests at this level should be black box and closely emulate the intended deployment environment.

Component level testing will verify the interaction of multiple units.  As such, the test documents at this level should be written by a technical or team lead.  The testing will verify the functional aspect of a subsystem as a whole so testing needs to be authored by someone who understands all of the desired interactions and functions.  Testing will probably be black to grey box. 

Unit level testing and documentation should be written by a developer.  Ideally, the test software should be written by someone other than the application developer.  While having the application developer write the tests is an option, the use of a second developer adds a second set of eyes and interpretations to the problem which may help detect discrepancies between the implementation and the requirements.  Testing will be white box.

!
Thursday, April 29, 2004

In our shop, developer writes unit tests, customer writes acceptance tests.

"Integration Tests" are unit tests, only bigger units.  "System tests" are just unit tests, only one really big unit.

regards,

Matt H.
Thursday, April 29, 2004

Typically, the developer writes unit tests and some integration tests; a verification engineer should write integration and system tests.

Ham Fisted
Thursday, April 29, 2004

"The spec writer will be biased, and will have more understanding of the DUT than anyone else."

1. What is DUT?
2. In what way would the spec writer be biased?
3. Wouldn't you want someone writing the test to understand the desired OUTPUT, but not be invested in the actual code?

Mr. Analogy
Thursday, April 29, 2004

DUT == "Device Under Test".  Could be a circuit or module, or subroutine, or...

Designers tend not to see their own bugs.  As well, if the designer has a concept in mind, he may spec it, but omit a detail.  Upon re-reading, this omission is not necessarily evident.  In effect, the designer is "blind".  A designer may also overlook a required operating mode when doing the spec.

If a third party, having good domain knowledge, but zero knowledge of the DUT, reads the spec, these omissions become apparent, and often guide the development of test cases.

It's basically peer-review, but at the initial specification stage.

David Jones
Thursday, April 29, 2004

  Recent Topics

  Fog Creek Home