Fog Creek Software
Discussion Board




Experiences with XP "Automated Testing" ?

Hi,

I'm investing using Automated testing on our next programs.  Automated testing ala "Xtreme Programming"'s approach of writing a Unit test first (e.g., using Dunit, Junit, Nunit).


What sorts of experiences have people had with that?

What is a good first step to see if this will work?

(I"m inclined to simply write a test procedure for the next few classes I create).

DUNIT, for exmple, seems to launch right into a whole complicated framework. I'm looking for a more gradual way to try it "and see if it works".

Mr. Analogy
Monday, June 28, 2004

I've used it on a couple projects now, and have had good success, including:
  x less risk in making significant changes to code base
  x less time spent debugging (in general)
  x less time determining causes because I can generally not worry about code under automated testing being wrong.

Suggestions:
  x Get a good refactoring tool.  When code that tests are coupled to changes, you'll regret manually fixing test code.
  x Keep tests focused on one thing.  Don't test methods or classes.  Instead, test one facet of a method.  It makes it simpler to determine cause of test failure, and allows more expressive test names.
  x If something is hard to test, don't give up on it.  Usually, there's a way to make it easier testable, and this leads to more loosely coupled design.
  x Treat bugs as proof of a missing test case.  When a bug is found,  figure out a way to implement a test case

Some things I've found hard to test are UI (due to incomplete APIs), DBs (due to persisted state), and race conditions.  If your code will largely deal with these, good luck.  Maybe Delphi is better with these than others...

Read
TDD by Kent Beck
  http://www.amazon.com/exec/obidos/ASIN/0321146530/
Pragmatic Unit Testing
  http://pragmaticprogrammer.com/developers/starter_kit/ut/index.html

If you don't want to drop the cash, at least read these bits:

http://www.xprogramming.com/xpmag/acsBowling.htm
  http://www.xprogramming.com/xpmag/acsBowlingProcedural.htm

http://redsquirrel.com/dave/work/babySteps/

Also check out mock objects:
  http://www.thedelphimagazine.com/samples/1677/1677.htm

Jeremy Dunck
Monday, June 28, 2004

Honestly? Automated unit testing has changed my life.

I'm not kidding about this. Automated testing may be the single biggest productivity enhancer that I've ever encountered. I won't claim that will happen for everybody, but for the kind of coding I do, it's been invaluable. A few reasons why:

- It's always an eye-opener to write extensive test cases and discover that the simple function you were certain was correct had all kinds of weird behavior in corner cases that you otherwise wouldn't have caught.

- It's absolutely invaluable when you need to refactor, because you can make changes and be confident you didn't break anything.

- It tends (at least for me) to encourage better up-front design. I've learned that if a class is difficult to test, it's probably because there is a problem with the interfaces. (Note that I don't do much GUI programming, which undoubtedly has different testability challenges.)

- If you're disciplined about putting test cases in when you discover bugs, it also provides regression testing. If you make sure the examples you cite in your documentation are also part of your test suite, you can make sure that your code actually performs the way you claim it performs.

My suggestion: Don't spend a bunch of time just yet worrying about the big fancy frameworks. Just write a couple of test cases that instantiate some objects and call a few methods and compare the actual results with expected results. To quote another XP maxim, "Do the simplest thing that could possibly work." Convince yourself that testing has value. Then look for ways to make the test management more efficient.

John C.
Tuesday, June 29, 2004

Just saw Jeremy's post and would second his points. Debugging time is often reduced in my experience as well, I think because comprehensive unit tests help isolate problems close to the source instead of letting them propagate to a point where they cause some totally unexpected behavior.

Testing against a db can be tricky, but I've found a couple of solutions that work for me. One is to have a test schema with the same tables etc. as the real db but populated with known data that I can manipulate and can expect specific results from. This is nice because I can again test weird cases that might not occur in typical data. A second approach is to build mock db objects that take the db out of the equation entirely and just fake responses in ways that I specify.

John C.
Tuesday, June 29, 2004

Interesting thread.

How do you guys organize your tests?  Do you put them in the same source file as the code it is testing?  Or does that get messy?

What about assets related to testing?  i.e. binary files... some of the stuff I write is very computationally intensive.  Thus, testing it is a real pain.  So I would like to associate some test assets that will be fast to crunch with it... but managing all those test assets could be a pain, I would imagine.  Our source tree is already quite messy.  : )

Or should I just use a framework for testing... I haven't really looked into any.

Interested in any C/C++ or Python specific answers, particularly about any testing frameworks that add value.  Are they easy to pick up or just impose unnecessary restrictions?

Roose
Tuesday, June 29, 2004

I've been unit testing all my non gui code for about a year now, it has defintely improved the quality of my code.

It does get difficult when your relying on system services which aren't always consistent but its worth persevering.

For C++ development I've been using CxxUnit which does some code generation for you, it's ideal for small projects but I don't think I'd want to use it in a large team.

The next stage for me is to figure out how to do the GUI testing.

Tony Edgecombe
Tuesday, June 29, 2004

I've also been doing varying levels of unit testing with C++ for over a year now. I agree with all the 'life changing for the better' comments. It's the best and most productive new technique I've tried for a long time and as a nice bonus it improves the design ;)

I don't use a framework; I wanted to get the simplest thing working so I just started writing tests. I now have some test support code that I regularly use.

I've blogged about my testing experiences here:

http://www.lenholgate.com/archives/cat_testing.html

Len Holgate
Tuesday, June 29, 2004


Ditto the Automated Unit testing has changed my life.

If you aren't doing OO code, then "xUnit" type frameworks may feel a bit heavy and clunky.  Same deal for C++ code - cppUnit and CPlusPlusUnit just feel ... clunky.  Hard to work with.  Easy to get wrong.

You might want to look into Perl's Testing Features:


http://www.wgz.com/chromatic/perl/Test-Tutorial.pdf


Good luck!

Matt H.
Tuesday, June 29, 2004

Ditto the ditto. I won't ever go back to non-automated unit testing if I have my way.


Tuesday, June 29, 2004

As a historical clearification, unit testing and automated
testing existed long before XP.

son of parnas
Tuesday, June 29, 2004

As far as mechanics, I've always kept my tests in a separate file(s), but that may simply be a matter of personal preference or habit.

When I need lots of external data for something, I just create a /testdata directory, dump data files in there, and read them in the test code.

It goes without saying that all this stuff gets checked in to source control.

John C.
Tuesday, June 29, 2004

==>Honestly? Automated unit testing has changed my life.

I have to agree whole heartedly about that one! It's changed my life.

We use NUnit for .NET dev, and a custom framework (we do a lot of VBA/Access and I haven't yet found VBAUnit <grin>)

The biggest benefit for me is it got rid of that sick feeling every time you make a minor change -- I know I'm making this change, it's small, it's simple, but it *shouldn't* break anything. It shouln't... It can't ...

And then you get the call. It may be a week later, a month later, whatever -- you get the call that something's now broken. After a little investigation, you find that the small change that *couldn't* break anything -- well ... it did. It broke something BigTime! (I'm sure nobody knows what I'm talking about <grin>)

Anyway, with a library of tests built up for the application, now I can be pretty darned well sure I've not broken anything, as one or more of the tests will fail -- before I release the changes to production.

It's been a lifesaver!

Sgt. Sausage
Tuesday, June 29, 2004

Where can I dowload this "CxxUnit"?  I have googled for it, and don't see anything.  Nothing on the newsgroups either.

?
Tuesday, June 29, 2004

perhaps cppunit?

mb
Tuesday, June 29, 2004

"""How do you guys organize your tests?  Do you put them in the same source file as the code it is testing?  Or does that get messy?"""

I use separate files, usually in a 'tests' subpackage within a package.  So, tests for package X go in an X.tests package.  In this way, test code doesn't get loaded during normal operation of the library.


"""What about assets related to testing?  i.e. binary files... some of the stuff I write is very computationally intensive.  Thus, testing it is a real pain.  So I would like to associate some test assets that will be fast to crunch with it... but managing all those test assets could be a pain, I would imagine.  Our source tree is already quite messy.  : )"""

Keep in mind that you needn't have only one test suite.  Indeed, XP recommends you have at least two distinct kinds of tests: unit tests and acceptance tests.  It sounds like your computationally intense tests with lots of data are probably acceptance tests, and it's fine to create a separate package structure for that, to run periodically.


"""Or should I just use a framework for testing... I haven't really looked into any."""

I use Python's "unittest" module.  In each test module, I create a 'test_suite' function that constructs a test suite, and in the '__init__' module of the tests package, I create a 'test_suite' function that imports the other test modules and combines their test suites.  So, running 'python unittest.py somepackage.tests.test_suite' will run all of the package's tests, or I can specify an individual module's worth of tests to run.  (I usually mainly run tests for the specific stuff I'm working on, but I always run a systemwide unit test before checking code in to CVS.)


"""Interested in any C/C++ or Python specific answers, particularly about any testing frameworks that add value.  Are they easy to pick up or just impose unnecessary restrictions?"""

I recommend Python's unittest framework highly, although there are also those who speak well of the Python doctest module.  I don't know about tools for C and C++, but since all the C I write these days is Python extensions, I use unittest to test those too.  If you're using Boost::Python or one of the "inline C++ in Python" tools, you may want to use those, since you'll only need to learn one testing tool, and you won't have to take the extra effort of writing tests in C or C++.

Phillip J. Eby
Tuesday, June 29, 2004

*  Recent Topics

*  Fog Creek Home