Fog Creek Software
Discussion Board

Testing *Unit tests

I was reading an article on refactoring the other day where two pair programmers were doing some test first development. During their session they found that their test suite was getting awkward, and happily went away and refactored it.

I suddenly realised that they were refactoring their tests, but there were no test cases to prove the tests were okay!

This sounded like quite a dangerous operation to me. I might be taking things a little too far, but I might have left the test cases alone.


David Kalnins
Wednesday, April 7, 2004

Ugh. I hope they made a business case for refactoring the tests.

From this short anecdote it's hard to tell, but it sounds to me like a typical inmates-taking-over-the-asylum situation. Don't they have real work to do over there?

Joel Spolsky
Fog Creek Software
Wednesday, April 7, 2004

Umm, the likely reason that they decided to refactor the test cases is because they were adding a real piece of functionality and were finding it awkward and time consuming.

Oren Miller
Wednesday, April 7, 2004

The greatest temptation would be to "refactor" the unit tests right after you "refactor" the code, so they match up again.  Which defeats the purpose of unit tests.

The right solution is to put the new "test functionality" (whatever that means...) in a new test suite, and then run both suites every time.

Thinking about it more, the only way to verify that a unit test was refactored correctly would be to run it successfully against every version of the software since the beginning of time.  That is the stupidest thing I have heard this week...

Manuel M. Garcia
Wednesday, April 7, 2004

Oh yeah, and you would have to verify full coverage, in case the "refactoring" was mainly removing tricky corner conditions...

A great deal of work for exactly zero benefit, because nothing is stopping you from creating additional unit test suites, and putting the new functionality there.

Manuel M. Garcia
Wednesday, April 7, 2004

The point isn't to write perfect code, it's to:

1. Specify how the code should work with an example
2. Exercise the code in an isolated fashion
3. Add more of these 'exercises' in the future as you catch additional bugs

Wednesday, April 7, 2004

If you write your unit test *first* ala XP, then you'll have ample experience with your test failing.  That is one way to demonstrate you have a valid test.

So... test fails, fails, then passes.

Mr. Analogy
Thursday, April 8, 2004

I think it's a % of value thing.

If your code has a 10% defect rate, this is unacceptable.

If your code has 10% defect rate, and you have automatic tests with a 10% defect rate, your net defect rate should be around 2% (0.1*0.1*2 for bad luck).  This may be acceptable.

So you may be willing to tolerate 10% errors in tests.  Now, to have tests for tests, to get the overall defect rate even further down ... well, most people think that's just silly.  Like Joel said, most people have work to do. 

Of course, for life-critical systems, you might want to do something like that.  I think most of those people program in C with matLab, and use symbolic requirements.


Matt H.
Thursday, April 8, 2004

David Kalnins:

"I suddenly realised that they were refactoring their tests, but there were no test cases to prove the tests were okay!"

Speaking for us, there is a different flow when refactoring or altering the behaviour of a unit test, as opposed to doing work to product code under test.

The safety net afforded by unit testing has gone, so any changes have to be *reviewed*. Slowly, carefully and not at 11:00pm by tired eyes! As to how we review, it varies from person to person and from case to case - informal code reviews, pair-programming or buddy programming.

"This sounded like quite a dangerous operation to me. I might be taking things a little too far, but I might have left the test cases alone."

I don't know what this article described, but I presume thay were 'refactoring' in the strictest sense - ie. they made no changes to what the unit tests actually did with the code under examination, rather they changed the structure of the test code.

So I suppose they would have ended up with exactly the same test cases as before.

<Rhetoric>Why on Earth would anyone want to do this? This takes me on to...</Rhetoric>

Joel Spolsky:
"Ugh. I hope they made a business case for refactoring the tests."

Some strawmen to burn first:-

1. It might be an open-source project. It's art for art's sake.

2. It's a pedagogical example of refactoring - it just happens to be unit test code getting the treatment.

3. They are successfully relieving a gullible client of excess money, so why should they worry? The asylum turns out to be a five-star hotel. :-)

Your standpoint appears to be that refactoring unit test code is not connected with adding value to a commercial software product.

I'd say that depends - if you do unit testing already, presumably you are doing it for commercial reasons, eg:-

1. The customer will not accept an obviously defective product.

2. The consequences of defects in the product may have serious legal repercussions on your own company.

3. You are aiming for a more predictable pace of software development, avoiding firefighting. This helps keep project costs down overall on big projects - although you will certainly go more slowly in a day-to-day sense with all those tests to worry about!

4. Your project is large enough that the entire design can't be carried around in somebody's head - so as new people join the team (or when existing developers forget the details!), you need something that acts as a specification for all the detail in these myriad classes and functions.

If this is the case, then your tests are indirectly adding value - so you might want to refactor the tests for much the same reasons as for code that is obviously part of the product.

My own experience is that single unit tests can sometimes span several screenfulls of text - if I don't apply decomposition refactorings, I rapidly get to the point where I (and more importantly, the reviewer) can't understand what's going on.

Another common case is that several unit tests share a certain overall approach in common - it might be the test cases, a setup of a series of interconnected objects or a sequence of operations that are repeated in separate tests but with slight variations.

In such cases, test refactoring is definitely a plus.

I have to add that we don't automatically refactor tests as we go - it tends to happen when we see the next unit test is turning out to have far too much in common with a previous one, or when halfway through the intent of a test to a reviewer, the reviewee admits that their code is less comprehensible than it appeared to be the previous evening!


Thursday, April 8, 2004

A unit test is also a documentation of the code. Read the unit tests to see how the code is supposed to be used.

In that retrospect I see two business cases to refactor the test code:

1. You want clear test code so it can actually work as documentation.

2. You want to simplify the test code so it is easier/faster to add new tests. Then you can produce more for the money.

I usually refactor common test cases and test setup. The tests themselved will always keep the same set of assertions.

Thomas Eyde
Thursday, April 8, 2004

I forgot: There is no refactoring in the strictest sense. There is the correct sense and the wrong one. Refactoring is about simplifying the code while the behaviour is unchanged.

So if you refactor your test, run it and it fails where it previously didn't, it is then obvious that you also changed the test behaviour. Go fix the test. The production code is untouched, so it is still correct.

Thomas Eyde
Thursday, April 8, 2004

If you refactor your test code and something fails that didn't before, how can you be sure it's the test code you broke?  What if you fixed the test code to find a new problem in the production code?


Thursday, April 8, 2004

I think refactoring tests is every bit as important as refactoring code. Why? Because a unit test is a client of an API just like the rest of the production code that uses it, and thus you have to change the test when you change the thing being tested.

If your unit test is a complete mess (and I think it's much easier to write messy tests than messy code), changing the actual production code to keep it maintainable and comprehensible is that much harder. If your unit test is factored well, changing your production code is easy.

I've seen unit tests that are giant copy-and-paste messes. They inevitably are the single most painful things to change when you decide to modify the code being tested. If you let them get out of control, your system becomes brittle just as rapidly as if you let production code get out of control.

(And as for 'testing the tests' -- this has come up in my head quite a few times, too. In the end, I've settled that "you can't have infinite regress" and stopped at one level of testing, combined with test-first development -- which does indeed provide substantial evidence that your tests actually work.)

Andrew Geweke
Thursday, April 8, 2004

Eponymous, as long as we are talking about refactoring as a behavioural preserving activity, then there are no such thing as fixing a test during a refactoring. Ergo, if the test is broken after a refactoring, then the refactoring is not done correctly.

Besides, with a normal potion of intelligense, it should not be too hard to figure out where the error was introduced, in the test or the production code.

So there is no chicken/egg.

Thomas Eyde
Friday, April 9, 2004

As a side note, I always enjoy XP advocates doing gymnastics trying to explain their philosophy to us pagans :-)

Monday, April 12, 2004

I've been playing with "code test, then code" a bit and I find that half of my test failures are due to incorrectly coded tests.

I end up changing the tests when it's clear that I made a mistake setting up a test case, and I change the code when it seems like the test is correct.

I don't think there's a way to write correct tests on the first pass, any more than there is a way to write correct code on the first pass, so I guess this must not be what unit testing is intended to address.

Eventually I wind up with a tests and code that are *consistent* with each other, and the tests express the intended behavior of code.

Ham Fisted
Tuesday, April 13, 2004

Just other day, I've subclassed a repository class to have it store the data in a sql database instead of a file. It had some unit tests that I didn't like, each one tested too many things, a missing functionality made all of them fail. So I implemented my new sql class using the tests to verify it, and refactored the tests till everything worked.

The answer to your question is that the tests and the code are coupled. You don't to need an "test of the tests" (which sould also be tested), because your code that verify your tests.

Paulo Eduardo Neves
Thursday, April 15, 2004

The orig XP book talks about unit tests in terms of "confidence."  I'd never take them as guarantees; that's what proofs are for.

The one thing I occasionally do though is make sure a test run fails, so I know things are working properly.  Which reminds me, I need to modify my test system so there's a type of test that's meant to fail and reassuringly spit out, "Good, a bad test failed!"

Tayssir John Gabbour
Wednesday, April 21, 2004

*  Recent Topics

*  Fog Creek Home