The Old Joel on Software Forum: Part 3 (of 5) - Auto tests and an autotest method

Auto tests and an autotest method

Hi,
Probably like most software development companies we are constantly battling with the issue of having the team produce good quality software with a very low rate of errors. One of the ideas we are currently consider is to make auto tests a part of the actual software components developed.
So considering component oriented development - our standard is that the programmers develop auto test code which tests the public API of their component and includes all the main test cases.
The idea is instead of making this an external script it because an autotest method of a class or an autotest class in a component. That way we can call object.Autotest() on any component to test it. The idea of this is that the autotest is maintained and extended along with the code and can be extended to include new test cases when errors are discoved with the component.
The autotest methods would return a string with details of the results of the test, or generate some type of error of course...
Ideally we want this to be used in conjunction with automated testing tools to provide the highest level of QA.

What do you think? Has anyone tried this approach?

Pete
Tuesday, October 28, 2003

Why not just use one of the many unit testing frameworks out there instead of reinventing the wheel?

Not that doing that is always a bad thing, I'm just curious as to what lead you to your current idea.

Joe Blandy
Tuesday, October 28, 2003

pete there are still a ton of things that can go wrong:

1. tests wont be atomized- if there are multiple tests for a class, how will you know which one succedded and which one failed will you be sure?

2. test results will still have to be manually read, if it produces a string someone will have to read and worse INTERPRET the string

3. It is rare that a test class will not need other classes,for example in the J2EE mythos, how would you give your "tested" servlet a request object? youd have to do some pretty heavy digging in tomcat to get an instance of it (known as hard integration tests). What about session, what about EJB or Corba objects?

4. Soft integration, if you need to print up a report how do you simulate a connection string you will use, or how will you simulate protocol error or network simulation errors(soft integration)

the artist formerly known as prince
Tuesday, October 28, 2003

Some further information - we are developing in ASP/COM and .NET (all new development in C# .NET and ASPX).

Joe,

"Not that doing that is always a bad thing, I'm just curious as to what lead you to your current idea. " -

To be honest we haven't really used the unit testing frameworks, so maybe they are the best approach - can you recommend any?
(I have seen some links from http://www.extremeprogramming.org but we haven't really check them out)

However another thought is that having the autotest as part of the software that we can use the full power of the language to test, and the autotest code becomes part of the component and can be run on remote servers without need to install 3rd party tools, etc.

The idea is to make it a central part of a programmers development and something they develop as they code a component.

the artist ...,

You mentioned "1. tests wont be atomized - how will you know which one succedded and which one failed will you be sure?"

Maybe I misunderstood the point but to clairfy - The idea is that the output of the test class or method returns a string saying which tests ran and what their return values are, and displays any error messages. This output would usually be displayed in an ASPX page showing the outcome of the tests.

If there are any exceptions then the ASPX page which calls the autotest could e-mail the person managing the build...

"4. Soft integration...how do you simulate a connection string you will use"
- the autotest method or class (this is a guideline maybe there are multiple methods) could take parameters for values like this.

"or how will you simulate protocol error or network simulation errors" - the autotests would not be designed to trap this but to contain test cases to ensure that the public API is functioning correctly as per the specification - so when a developer says "I am finished" it is a step they must perform to ensure that someone else can take there component and reduce the likelihood of there being any obvious bugs in the API.

Pete
Wednesday, October 29, 2003

In J2EE, you can use Cactus to test servlets. It's an extension of JUnit.

See http://jakarta.apache.org/cactus/

John Topley (www.johntopley.com)
Wednesday, October 29, 2003