Fog Creek Software
Discussion Board




What on earth does a unit test do?

I've only recently looked at XP in any depth and I feel that I must be incredibly stupid - I simply don't understand it! I studied electronics and computer science in the late seventies, since when I've mainly worked as a mathematician/statistician in the telecoms world. I write a lot of code (these days largely VB, VBA and the occasional bout of C) and I have a fair number of regular users but I've never followed any particular methodology. As in, I very rarely plan anything, just get stuck into the code. I've downloaded the test framework for VB and - well, I'm buggered if I can understand what the hell is going on. I mean what exactly_is_a unit test?? I've spent (wasted?) hours of work time reading the XP sites and trying to get my head round it. Am I simply too old? Or am I in fact just very stupid?

Graeme Henderson
Tuesday, May 21, 2002

A unit test is a way to ensure that your code does what you want it to do.

If you have a function that calculates insterest, you write a unit test that provides the input for the funcrtion, and checks that the interest calculated is correct for a known value.

You do this informally when you setp through code or when you run your program and check that it does the right thing on the screen

The advantage here is that the thing you checked for last week gets checked when you run your unit test this week.  You don't have to remeber to do it.

adam
Tuesday, May 21, 2002

Assumption:  A developer tests code against a preceived set of criteria before declaring the work to be "done".

So, the code written around the developed code constitues the unit test.  Sometimes (often) the code cannot be tested alone and may be dependent on an external environment:  properly initialized hardware, a running server supporting the proper interfaces, or other software within the system being up and running.

A developer will often check in code to source control which does not pass all the unit tests, especially in the early stages of a project, because other developers will be satisfied with working with a partially working interface rather than no interface at all.  (Checking into source control does not constitute being "done", but let's not get tied up with that right now).

OK, so that seems pretty rational...  Now make the assumption that the proper test criteria are pre-established and accurate.  In other words, at some design meeting everyone agreed what the interface looked like, how it behaved (threading model, blocking calls, exceptions, etc.) and that the problem was sufficiently bounded that a unit test could be written to validate the code for all conditions.  Obviously, this is not always possible, but assume for this instance it is.

So, what is frustrating is when a developer says something is "done" when it hasn't been run through a reasonable unit test.  How many times have you heard the phrase "it works over here" as though that is supposed to validate bug free code?  AARGH.

So what unit testing is intended to guarantee is that it will run under all circumstances in all anticipated environments. The only mystery then is anticipating all possible inputs or variations in the system (and they are legion). 

One more reason why minimalist code is always best.  For every conditional statement you introduce a necessary test case.  I could rant for days on this...

Nat Ersoz
Tuesday, May 21, 2002

Nat,

I take it you've had problems with this before? ;)

Marc LaFleur
Tuesday, May 21, 2002

I would highly recommend using Unit Tests in any programming language... Languages with introspection capabilities such as Java are easier to manipulate with Unit tests but even language like C++ have them.

Unit tests and unit test frameworks like JUnit (or XUnit in general) allow you to develop code "test first".
In a nutshell the idea is to get the functionality of the code working first and then worry about how it looks.
This means:

1. Write tests
2. Write the code to get the tests to work.
3. Repeat 1-2 ad nauseum.
(at this point the code is, well, disguting)
4. Refactor the code - i.e. make it look nice.

But wait, if you make it look nice, move pieces of code around and various other beautification processes, you are SURE to introduce bugs, right?

Yes... But guess what, you already have the FULL test suite to notify you of the bugs... You wrote them first, remember? So, it's an easy job to get the test to run AGAIN...

Now, you have the tests, the nice *working* code, and now all you need is to... Eh, nothing actually...

Another important use for unit test, which is often overlooked is as a way of formalizaing invariants in interfaces (or pure virtual class a-la c++). When you specifiy the interface you usually only specify the methods' signatures. If you also provide a "Test Suite" (the XUnit name for a bunch of "Test Case"s (The XUnit name for a test testing one aspect of something))... Anyway, if you also provide a test suite, you can formalize other aspects of your interface's invariants.

In this case, an implementor would not only need to adhere to the interface's signature, but would also need to pass the tests... This means that he would adhere to a lot of the semantics of the interface, as captured by the test suite.

The test suite also constitute a very concise form of documentation of the stuff tested. This could, for example, resolve question like "Does 'empty' mean Null or does it mean 'containing the empty element'?" Well.... Just look at the test...

One last comment, it is often the case that the test cases themselves are buggy. Moreover, during the course of devleoping a class, component or function which is tested by a suite of test cases, the test cases themselves may become obsolete, test useless things (and hence be removed) or even be refactored to be more efficient...

Hence, both sides of the testing equation - the tests and what they test are subject to change as time goes on...

Amir Kolsky
Tuesday, May 21, 2002

http:\\www.junit.org

Amir Kolsky
Tuesday, May 21, 2002

Marc,

I have issues.  Lots of issues...

Nat Ersoz
Wednesday, May 22, 2002

Graeme,

From your email address I guess your from the UK, so you've probably been to IKEA.  For the rest, IKEA is a furniture store.

At IKEA they have demonstrations of their testing.  For example, their will be a sofa, and a mechanical device that basically pushes a bottom shaped piece of wood into the sofa over and over again.

They are demonstrating their quality control.  The mechanical device is simulating the effect of people parking their bottoms onto the seat over and over again, day in day out.

A unit test is like one of those machines.  Is is basically a contrived piece of code that can very quickly test your piece of code (or unit) to make sure it is up to scratch.

The test unit framework is just to help you develop the unit tests in a standard way.

Ged Byrne
Wednesday, May 22, 2002

FWIW, unit tests don't have to be complicated.  They can be implemented informally on projects of any size.

I recently begun implementing unit tests on a small development project of my own.  I have a checklist of features to test.  Each feature has a set of tests to perform, which looks something like this:

*** Settings ***
1. Open Settings window.
2. Change a text-based setting.
3. Click "Save & Close" to close the window.
4. Open the Settings window again.
5. Verify that the text-based setting was changed.
6. Click the "Show console window" checkbox, making it checked.
7. Save changes and close the window.
8. Close the application.
9. Run the application again.
10. Verify that the console window is displayed.

Once I feel I'm ready to release my application to the world, I run through all these tests, verifying that the basic functionality works.

This could be done in any environment, even at work when you're just updating a feature.  Write a quick list of steps that will verify that the feature will work, and run through them before checking in your code.

Brent P. Newhall
Wednesday, May 22, 2002

I wrote a chapter on unit testing in Python: http://diveintopython.org/roman_divein.html .  Actually, very little of it is Python-specific, and the Python code involved should be easy to read.  The chapter steps through the process of designing and implementing a library to convert integers to and from Roman numerals.  Start by defining your requirements, then write your test cases, then write your code, then refactor your code with various optimizations and make sure all your unit tests still pass.  I have found this last part (refactoring with confidence) especially useful in the real world.

As others have said already, test suites can define your interface at a semantic level.  Not only should a test suite test a variety of known good inputs, but it should test as many bad inputs as possible.  There is no way to express negative numbers as Roman numerals; if I try to convert a negative number to the toRoman() function, what happens?  Also, internal consistency: toRoman() and fromRoman() are mirrors of each other, so fromRoman(toRoman(n)) should equal n for all values of n in the acceptable range.  And so forth.

One last note: several "Extreme Programming" gurus have contacted me to inform me that the chapter has a fatal XP flaw (BigDesignUpFront -- I write too many test cases before writing any code).  To which I say: you can use unit testing without accepting every other rule of XP.  You can write unit tests after the code is written (for instance, in response to a bug report -- the most likely scenario).  You can write lots of unit tests up front.  You can write unit tests in lieu of design specifications altogether.  Do whatever works for you.

Mark Pilgrim
Wednesday, May 22, 2002

Mark... if I remember my reading of XP correctly, then it encourages you to do exactly what you advocate - meaing adjust the specific portions of the methodology intellegently to meet "your" specific situation.

I take it the term "guru" should be interpreted as someone that still has faith in the one true and proper path (e.g. there is one and only one correct answer to all questions), and NOT as someone that really has a clue!

Joe AA.
Wednesday, May 22, 2002

Hear hear! Marc!

Adopt what you need, when you need it.

Design has several "levels"... High, Medium and Low.

High level design is something that has to be made, otherwise you have chaos.

Medium level design is a good thing to think of, it usually has to do with which components you select, which languages etc.

Low level design should be done the XP way, i.e., done by coding. Nonetheless, I advocate writing as many tests when you can, this is because if you think of a test, you don't want to lose the thought.

AMIR KOLSKY
Wednesday, May 22, 2002

OK, I was too abstruse in my last comment, and got some questions regarding what the heck I mean:

If you are coding a test, and you think of another test (or a bunch of 'em) just code them. You don't have to do 'em one test at a time if you can think of more than one.

If you have the semantic definition of an interface and you want to code all tests at once, do so. If you really want to see the green after every test, comment out the test bodies and uncomment 'em one at a time...

Amir Kolsky
Thursday, May 23, 2002

Thanks to all of you, especially Mark Pilgrim. When I read through Mark's tutorial it all clicked into place; its the first time I've seen a really clear explanation of the philosophy and _practice_ of XP. I'd developed a mental block somewhere along the way - I think I'd made up my mind that XP was really difficult and ended up not seeing the wood for the trees. (Although in hindsight quite a few of the things I've read make it much more complicated than it needs to be). So, I've written my first test-code-first class and it seems like a natural way to work (although it was a very simple class!). For the first time it made me think properly about the inputs and outputs of my methods rather than just diving in and concentrating on the functionality. Given that I work as an individual rather than as part of a team I don't think the full XP shebang will work for me, but nonetheless I am cautiously converted..... they're only rules, right?
As a side-effect, I've never worked with Python before, but such is the excellence of Mark's tutorial I will be doing in future. Thanks again.

Graeme
Thursday, May 23, 2002

I'm assumnig that the unit tests are actually code so how do you decide where a problem exists? Is the bug in the code or in the test? Should we write a test case to prove that our case test works? I looks like this could go on ad infinitum so where does it end?

I'm not trying to be facetious. I agree with testing your code but where is the line drawn?

Marshall Harrison
Thursday, May 23, 2002

I do agree with Marshall - one of the reasons I ended up banging my head on the desk over XP was looking at the test code that tests the test code in VBUnit. Talk about circular referencing my brain! I've gone for a simple, single test suite for the class I'm writing and that seems far enough to go for me. Its only useful as long as its useful........

Graeme Henderson
Thursday, May 23, 2002

Marshall, you draw the line at a reasonable point where you feel you have tested as much as possible within reason. Say we have method a that returns a string containing "hello world". We simply test that the method returns that string, nothing more.

If the test is written correctly then your code should pass when it is finished. It's very simple and it doesn't take a brain surgeon to write unit tests. I tend to write unit tests for my core classes and every method/property I can test within reason. For instance, I am writing a mail application. Now I can test that an attachment is saved properly to the hdd by testing it's size vs. what the size was when it was attached. But to write a test that made sure the base64 encoding of the attachment is working properly might be too much because I would have to write a test that decoded the attachment and tested the output, making sure it exactly matched the original in every aspect. This might be a little overboard because if a binary file is encoded improperly it simply won't open/run properly. Some things have to be left to manual testing. I say automate as much as you possibly can within reason.

I write suites of tests so I can run them all in one build. I don't write tests for my tests. The idea is to eliminate the chance of a bug in an automated way.

Ian Stallings
Thursday, May 23, 2002

Thanks Ian,

When writing the tests, are they written in the same language as the program you are testing or is there a tool with it's own testing script for testing the code?

I'm all for the idea of unit testing but I'm just not sure how to do it cheaply and easily. I don't want to end up having to write a major program to test every program I write. There must be some way to simplfy things.

Also any links would be appreciated.

Marshall Harrison
Thursday, May 23, 2002

The other point worth mentionng about *unit* tests is that by definition they test individual *units* ie small parts of the system; a single subroutine, function, interface or whatever.  This is in distinction to testing your entire application; ie the thing you get when you glue all those units together.

Matthew Webber
Friday, May 24, 2002

Marshall,
Yes the unit tests are usually written in the same language. For instance when I code in java I use Junit as a framework to write and run my tests. When I code in C# I use the NUnit testing framework (the two are virtually the same). The unti tests are small classes testing what will happen in both good and bad situations. IE what will happen when we pass in a param that is expected vs. what will happen when we pass in a param that is null, etc. When the test is passed it's no guarantee that you won't encounter bugs, it just lowers the chance of a bug appearing, saving you time in the long run.

I cannot stress how useful it can be when you are refactoring or adding additional features. Knowing you haven't broken something gives value to these types of tests. The learning curve for unit testing is pretty low and the payoff is big.

Ian Stallings
Friday, May 24, 2002

Thanks Ian.

Sory to sound dense on this but I do have one last question. From the examples I've seen on the Internt I'm wondering how it is actually implmented:

1. is the test written right into the code of your program?
or
2. Are the routines to be tested cut and pasted into a new code in the test unit?

Either way it looks like it could cause problems - the first could make your executable bigger (for languages that cant ifdef the code out) and the second could create cut and paste issues that hide or cause other bugs.

Thanks.

Marshall Harrison
Friday, May 24, 2002

On my current project our unit tests are placed into a seperate package/namespace that is not distributed with the release build. For instance, when using C#, I create a .test namespace that I compile to a debug build. When we release a build to the public we do a seperate build without the unit tests. That's just the method I use now developing shrinkwrapped software components. I have done Java projects in the past where we also gave the client the unit tests for future developers to reference. It depends on your circumstances and target consumer of the software.

Ian Stallings
Saturday, May 25, 2002

Ian,

Thanks for all of your help.

Marshall Harrison
Saturday, May 25, 2002

*  Recent Topics

*  Fog Creek Home