The Old Joel on Software Forum: Part 4 (of 5)

Test frameworks

The recent thread on test plans spawned this. I'm currently defining, soon to be implementing, a framework for running regression tests. Most are automatic, some will be manual.

Any recommendations on existing frameworks I could leverage off of? I was thinkign Python, but just saw junit on sourceforge cuortesy of the aforementioned thread. I'm sure there are others out there.

What I need to generate is:
Each test has a pass/fail status
A report on how many tests passed, how many failed, how many ran.
A report on how many failed that passed last time, how many passed that failed last time.
Some fancy test metrics to justify my job to management.

I've done this before, usually rolling my own. I've done it in sh, perl, Python, and C. Mebbe it's time I use a pre-existing framework for this.

It needs to run in either a DOS/Cygwin, or a Unix environment.

Snotnose
Wednesday, May 19, 2004

I don't get your question.
If you do want to create unit tests, you'll certainly want to use the test framework available for your implementation language (JUnit if your program is in Java, PyUnit for Python, etc.).
If you want to create acceptance tests, then a unit test framework is not really the right tool for the job. Or is it ?

Pakter
Wednesday, May 19, 2004

There are 3 kinds of tests: auto, manual, and unit. Auto and manual are acceptance tests, about half of them started life as a unit test. The manual tests do something, then ask the user "does the screen look like this?", the user says yay or nay. I need to specify when I start the test which subset of auto/manual/unit I want to run.

We have all sorts of features, with different teams working on each feature. My job is to pull everything out of version control overnight, build it, and run the regression tests on it. The goal is that, once this is up and running, I spend maybe 1 hour a week on it.

As the tests themselves are written in javascript I was thinking to just use javascript for the framework. But the more I learn Javascript the more I want to use Python.

Good thing I spell checked this one, my spelling really bites today. Sorry about that first post.

Snotnose
Wednesday, May 19, 2004

So what's the difference between an "auto test" and a "unit test"?

Ryan
Wednesday, May 19, 2004

A unit test is one type of auto test. A unit is basically a requirement in the code. For example, when I pass certain values into this method, an exception is thrown - so the aim is to write a piece of code that passes those values into the method, and fails unless an exception is thrown.

Snotnose - it is a good idea to also automate the acceptance tests. The fit framework (in particular fitnesse) is good at this, although you don't want to test too much, since there is a certain point where the value drops off. It also takes a few days to get up to speed, but it has easily paid off for us. (It takes about 30 seconds to do a complete application test, which used to take half an hour).

Rhys Keepence
Wednesday, May 19, 2004

In my mind, a unit test is what the developer runs and can be either auto or manual. Auto and manual tests are what I run.

I'm looking very hard at STAF, developed by IBM and open sourced. See http://www.research.ibm.com/journal/sj/411/rankin.html and http://staf.sourceforge.net/index.php for more.

Snotnose
Thursday, May 20, 2004