Fog Creek Software
Discussion Board

Automating software testing, tools/howto?

Hi all!

I've been having a look around the archives, and though I've found quite a lot of info, I still haven't got a clear answer to my doubts.

In my company we're developing software that is both extremely low level (I/O and network filter drivers, customized windows logon dlls and such like) and client / server (integrated into Active Directory and with GPOs). Up to now we've used a "manual" testing metodology, that is, install both server and client components, and manually exercise them to see if they break.

As a side-product of the testing we've done, we are in the process of developing a fairly comprehensive "test tree", ie, a set of features to test and interactions between them (such as "if A fails, then B and C can't be tested, but D can"). We'd like to try make this into something a little more systematic and, specially, automated process. I am quite confident that there must be some way / testing framework / tool that might help in the process.

The first thing we'd like to do is to be able of generating a comprehensive list of "test to make", ie. from the requisites and interactions tree mentioned before, to be able to "parse the tree" - there must be some better term, I'm sure - to get a list of the specific tests we have to make to be sure we've covered a certain set of features to test / failures to verify.

Ideally, we should be able to load that list onto some testing software so that the testing is made automatically (in that way we're sure we don't miss a test).

_ANY_ help and/or pointers to possible resources will be more than welcome. From searching the web I've found quite a lot of "academic" studies on software testability, and theory of individual component testing, but I haven't been able to find a comprehensive testing solution. Maybe there is none, but....

Thanks a lot...

Javier Jarava
Friday, March 7, 2003

Here's a Wiki page on the subject; scroll to the bottom for a whole bunch of links:

And here is a huge list of Testing Frameworks in various languages:

Brent P. Newhall
Friday, March 7, 2003

One useful idea: Mock Objects [1]. Most frameworks at the link are for Java, but the concept isn't language-specific. It's particularly useful for I/O tasks or actions that run in some kind of container -- they take the place of the framework objects and allow you to set information (like writing to an I/O stream) and query the object afterward (like counting the number of bytes written to the stream) to see if the proper information was set.


Chris Winters
Monday, March 10, 2003

*  Recent Topics

*  Fog Creek Home