The Old Joel on Software Forum: Part 4 (of 5) - Keith Ray's opinion on GUI testing...

Keith Ray's opinion on GUI testing...

Check it out here (short blog entry): http://homepage.mac.com/keithray/blog/2004/04/27/#AccpetanceTests

I think it's too simplistic and ignores a host of other things that jump up with GUIs (drawing, refreshing, key click combos, etc).

What do you think?

Crimson
Thursday, April 29, 2004

I think anyone who expects to be able to test GUIs hasn't written many. I used to be very much into XP and GUI testing, but I found the following problems when actually doing it:

- Because GUI tests do not involve simple input/output checks, they are usually very fragile and difficult to write.

- GUI testing discourages you from making improvements to your GUI. When you have dozens of tests that depend on your GUI functioning a certain way you will not be thrilled to rewrite them.

- The 'correctness' of a GUI is subjective. Tests can't take into account feedback, performance, appearance, and many other qualities of a GUI. In the end, a good designer will always take their GUI for a drive before release.

Matt
Thursday, April 29, 2004

I think this is why you try to stablize the GUI as early on in the development as possible (stubbing out the implementation as necessary). Not only are you able to write automated tests for it, but you also get the benefit of being forced to think about use cases and what your app actually DOES. You're forced not to churn if you get the UI right the first time. Additionally, you have the illusion of rapid progress early on in the project, although that may be more of a liability than a benefit.

Alyosha`
Thursday, April 29, 2004

I agree with Matt. The nature of GUI's just aren't conducive to automated testing. There's some functionality you can automate, but it only tells you part of the story. In my opinion, you're almost always better off having a human sitting down and manually going through everything.

Alyosha, I don't think I've ever been in situation where I've gotten the UI situation right the first time. :) Usually the customer sees the GUI and thinks of 8 billion other cool things to add and are already living in a future where the GUI can do the new things they want done.

Yeah, this happens with any piece of software, but it's particularly prevelent with GUIs.

Crimson
Friday, April 30, 2004

It's true that the best way to find bugs in your GUI is to have a real live human being sit down and test out the UI. Almost all GUIs of any complexity are very buggy at first, especially compared to the back-end (where behavior is more predictable and less prone to subtle usability issues and so forth.)

However, the problem is in regression testing. It is extremely expensive to thoroughly regression test a GUI manually, especially if your software goes through a lot of iterations and (as a result) test cycles. If you want to have confidence that you haven't broken what used to work, it's often more than worth the initial pain to write automated UI tests, preferably ones that actually exercise the UI through some sort of UI automation tools.

I owned testing for a product that had a very rich, very complex UI and my biggest regret throughout the project was that we didn't do more UI automation up front, since thorough regression test passes took 3 calendar weeks (12 man weeks.)

As with most testing, regression tests are where the automation really matters...not finding the initial bugs.

Mike Treit
Friday, April 30, 2004

I used to do GUI development, and now do "back end" stuff. There is no contest, the UI stuff is _much_ harder to get right.

B*ggered if I ever want to do _that_ again
Friday, April 30, 2004