The Old Joel on Software Forum: Part 3 (of 5)

automated testing

My team doesn't use automated testing, but we plan to start using it next year.

We'll be doing mostly web apps in Java and .NET. They are internal enterprise applications.

Mostly we must do functional and regression tests. We must see if the forms are inserting, updating and deleting records correctly, and if the business transactions are updating the tables in the way they're supposed to.

Any ideas about a good tool to create and run these automated tests?

Fernando Correia
Wednesday, November 26, 2003

Try:

http://www.xprogramming.com/software.htm
http://www.junit.org

For Java, there's JUnit. If you're doing web development, there's HttpUnit and Cactus.

I'm not overly familiar with .Net but I know there are some unit test frameworks for .Net as well.

For a fully automated acceptance test tool try FitNesse.

You're in luck. There's such a wealth of unit test frameworks for the languages you're using that there's really no good excuse not to use them.

Good luck.

anon
Wednesday, November 26, 2003

One other thing. Since you're working with databases you will most likely also want to look at mock objects.

Google for more info on mock objects.

anon
Wednesday, November 26, 2003

NUnit is .NET's version of junit. http://www.nunit.org/

Ian Olsen
Wednesday, November 26, 2003

We've started to use NUnit, and have really been impressed with it. It takes advantages of .NET syntax so you can specify what's a test and which classes have tests.

Walt
Wednesday, November 26, 2003

I've used both JUnit and NUnit, and I'm really really fond of the fact that NUnit uses metadata (attributes) instead of class inheritance. Java should considering adding something like attributes to the language... it's just amazingly useful in so many places.

Brad Wilson (dotnetguy.techieswithcats.com)
Wednesday, November 26, 2003

I know I can use JUnit (and maybe NUnit) do automate tests by creating objects and calling their methods.

But can I still use them if I must simulate a user interacting with the web site?

I mean, not calling a class' InsertRecord or ChargeCreditCard method, but simulating the input on the web form, and then looking at the database or viewing a result page to see if the operation was correctly done.

Fernando Correia
Wednesday, November 26, 2003

> Java should considering adding something like attributes to the language... it's just amazingly useful in so many places.

Metadata is slated for java 1.5 -- the ".Net-envy" release.

Portabella
Wednesday, November 26, 2003

UI interaction is not the appropriate place for unit testing.

In my experience, automated UI testing is: (1) hard, (2) frustrated, (3) hard, (4) time consuming, (5) did I mention hard? It's generally not worth it automating UI testing. The tools are all either awful or impossible to learn, or more commonly, both. Change the UI and ruin your tests.

Your customer (representative) should be using the product and telling you what UI issues there are.

Brad Wilson (dotnetguy.techieswithcats.com)
Wednesday, November 26, 2003

We do automated testing on the UI objects (as well as nearly all the java objects down below). While this is not true UI testing, it has shown itself to be very valuable.

If you're running client/server type apps, setting up tests is so error prone that its almost useless (not for developers' unit tests, but to hire testers to do manual tests of this nature).

Anyhow, our test guys have done a superb job at automating nearly all tests on all the client objects - including the UI objects. So yes, if the UI changes the tests break - but the test code is maintained in the same source control system as the client code, so they are in sync. Spend the time and money, its well worth it.

hoser
Wednesday, November 26, 2003

There is ASP Unit which lets you do web page testing with Nunit. It's still in a pre version 1.0 stage, and while it looks like it may turn out to be a good product, it's more trouble than it's worth at the moment.

http://aspunit.sourceforge.net/

Bill
Wednesday, November 26, 2003

Hi Bill,

Since I am now semi retired, I will share with you the secret of automated UI testing:
here it is (Drum Roll.....) "Input automated, verification manual", have a tool like Auto-IT go through the app and simply have it do printscreens of the ui, when you want to verify something. If your app has keyboard shortcuts use those as much as possible.

Then if the verification passed, save the printscreen, and assuming you use identical data (and don't have timestamps anywhere on the screen), you can then simply go back and do binary compares. Note this is not 100 percent automated, but it gets you through the most labor intensive part of ui testing quite well, the only thing this method lacks is a recovery method, i.e. its kind of hard to program around dialog boxes saying fatal error DB not found etc ... but it is still better than going through everyting in the ui by hand

Feel free to send

the artist formerly known as prince
Wednesday, November 26, 2003

meant to say, feel free to send me royalties

the artist formerly known as prince
Wednesday, November 26, 2003

Automated testing from the UI is too damn hard, and becomes error prone. It's also largely useless - it doesn't pick up the cause of problems.

The way to go is to come up with a good object orientated design which allows you to mock inputs. This isn't easy, but once you have it, your test cases become easy. As an added bonus, code quality improves.

Rhys Keepence
Wednesday, November 26, 2003

If you use the M.V.C design pattern to implement your GUI, you can use the *Unit frameworks to test your UI.

A few words about MVC are here:

http://www.cs.indiana.edu/~cbaray/projects/mvc.html

An explanation on how to write tests for GUI designed in this manner is here (for .NET):

http://www.codeproject.com/dotnet/tdd_in_dotnet.asp?target=nunit

(Go to "Testint the User Interface", but note that the contents links are broken - just find that in the page).

Eli Golovinsky
Thursday, November 27, 2003

Note that the original post did not talk about unit testing, but about functional and regression testing. While I'm a big fan of unit testing I believe it is also extremely useful to have some sort of automated regression test of the system *as a whole*. To do this you need a tool which gets as close to possible to simulating real users using the system.

Projects I have been involved with have used Rational Robot to perform this sort of testing. It drives the GUI and checks for results on the GUI and/or in the database.

While I'm not a big Rational fan, in fact I've tried some of their other tools and not been impressed, I am very impressed with what the testing guys achieved with Robot.

Didn't actually use it myself, but saw the automated tests running and it was pretty cool. The guys would let regression tests run overnight and check out he results in the morning. To cut a long story short, it allowed our testers to catch bugs with a minimum of effort.

Robot is not just a simple 'screen scraper'. Its quite well integrated with IE and is HTML aware so the test scripts won't break if you make minor changes to page layout.

cheers,
Peter

Peter McKenzie
Sunday, November 30, 2003

Hi,
If you are thinking for long term solution then you can
think of evaluating tool independent framework. One
such framework is available in open source.

http://safsdev.sourceforge.net

If you have any queries, do let me know.

[Niteen]

Niteen Yemul
Sunday, November 30, 2003

Thank you all for the comments. Peter is right: I want functional and regression testing.

Fernando Correia
Monday, December 1, 2003