Fog Creek Software
Discussion Board




Testing is a FUNDAMENTAL skill for programming


The more I think about it, the more it seems that TESTING is one of (if not THE) fundamental skills for programming.

1.  Microsoft rates this skill very highly.  Most of the little "microsoft questions" I've seen are about how to test : a salt shaker, whatever.

2. A test is really an OBJECTIVE set of goals.
Program will read SAMPLEFILE.txt and product the following results.
So, the user might say "it has to display HTML" (web browser spec. A test might say : it has to display all of these elements " <A> bla sdfkjssd </A>  <table>" etc. etc.

As the Mythical Man Month puts it: Razor Sharp tests: I.e., it's clear whether you pass or fail.

3. An iterative "test-code-test" approach to programming seems much more likely to result in an actual working product. It keeps you focused on the GOAL.  Without realizing it, I've always written software in the XP "test first" (I just hadn't learned to AUTOMATE the damned tests yet).

4. The test is like a compass sighting when you're hiking: it is a constant reminder of where you're going.


5.

This probably seems terribly OBVIOUS to most the JoS folks, but  it wasn't to me until recently. I was DOING all of the above things because they seemed to be the right thing to do. Now I think I was lucky to stumble on THE important skill.

(BTW, I've always written software this way merely because it as commercial software and I was a newbie and VERY concerned about not catching errors, etc.).

Mr. Analogy
Wednesday, July 07, 2004

Pretend I deleted that 5. above (with not text because I felt it would be redundant).


I guess I should have written a test for the posting, first. <g>.

Mr. Analogy
Wednesday, July 07, 2004

yep :)  another unfortunate fact is that most programmers suck at it.

Seriously though, a good tester is worth at least as much as a good programmer IMO.

I was very lucky in finding the one we currently lose...unfortunate I was unable to dissuade her from getting a boyfriend, becoming pregnant, and moving back to the states.

So we will be losing her again in 5 more months ;(  she will be hugely missed and very hard indeed to replace.

FullNameRequired
Wednesday, July 07, 2004

I have no doubt, having done it both ways, that the only sane way to write code that will grow and live a long life is test first.

Brad Wilson (dotnetguy.techieswithcats.com)
Wednesday, July 07, 2004

But how do you automate unit testing for GUI modules? For concreteness sake, suppose we have a dialog with a list view in it and a couple of buttons. How do you write a test first for such a thing?

Floridian
Wednesday, July 07, 2004

"How do you write a test first for such a thing?"

You call SendMessage() to emulate user input.

Ron
Wednesday, July 07, 2004

"I have no doubt, having done it both ways, that the only sane way to write code that will grow and live a long life is test first."

heh, unfortunately (good though that method is) that is not in itself sufficient.
Software _must_ be tested by people other than the programmer, and preferably by actual users or people able to duplicate the bahavior of actual users as closely as possible.

There is no way around that....writing tests first is good, and catches certain kinds of bugs, but its not even close to being 'enough'

FullNameRequired
Wednesday, July 07, 2004

SendMessage won't emulate all kinds of user input correctly (plus you've got to worry about a million different ordering details).

Anyway, people don't write tests to make sure that console I/O works correctly either.  Generally the most value with testing comes from separating/abstracting the concerns of your application (a very valuable activity in and of itself) and testing those abstractions.  Once you're satisfied that the thing you wrote will do what you wanted, you can hook it up to everything else in your application.  If you don't do this kind of verification, then you'll have to search through the mess left behind by all of your interacting components to figure out which ones are faulty.

This is one of those things that people who run "interactive interpreters" are always going on about.

Kalani
Wednesday, July 07, 2004

One of the fundamental things, i learned in my short 3.5 years career as programmer is the way Testing approached is really important.

Currently, I am alone in the project and there is no independent tester(s).  Whenever the code is changed, I test the changes with the assumption that everything changed will go wrong and keep a target for the no of bugs. Surprisingly, many times I could find more than targeted no of bugs.
In the very early days as programmer, I used to unit test the code with wishful thinking that code will just work fine.  Now I understand the simple truth
“If you don’t find the bugs, somebody else will find them and it may going to be embarrassing.”

>> But how do you automate unit testing for GUI modules?
It is always better to separate the application functionality code from the GUI code. With this layered approach, functionality code can be conveniently tested in non-interactive mode.

anonymous
Thursday, July 08, 2004

I agree entirely with your article. However, how many of us have worked for companies that do not think the same way ? Usually small sweat shop type places where testing, bug tracking and other essentials are regarded as expensive luxuries that detract from the bottom line. After all, what are customers for ?

I've worked at too many of these places, and that is one of the reasons I am very wary of working for small companies, it seems to be more prevalent there.

WhatTimeIsItEccles
Thursday, July 08, 2004

I hope we can all learn something from the original posting: read it.  Use your eyeballs and your brain.  I have poured through hundreds of pages of documentation from big companies, and I have seen about 6 major grammatical errors per page.  For a 500-page manual, that means about 3,000 glaring mistakes that any fluent English reader would pick up.  These are mistakes that help to make users confused.  If you have a typo in code, what happens?  Same thing with people and writing text, albeit not as severe.

sir_flexalot
Thursday, July 08, 2004

Personally I think there are multiple sides to this discussion. 

Yes, testing is a fundamental skill for programming.  But it can also be a crutch--i.e. one can always take on the attitude that "I'll just write this code, and then find the problems in testing."  I think that was the foundation of the "infinite defects" methodology.

Testing should be a process of trust, but verify. 

By the time you reach your first unit test, you should have very carefully considered the code's impact and its adhererence to spec.  And you should never assume that testing will take the place of those activities.

There's an undercurrent to the theme that testing is important.  It's not testing, per se, that is a valuable skill, but the willingness and ability to critically look at one's own work.  That's an end-to-end mentality; it begins with design and follows through to good, critical testing.

no_defects
Thursday, July 08, 2004



Thanks for all the responses above.


1. Programmer should do the unit testing. 
Otherwise, it's very easy for the tester and programmer to become adversarial, even with the best of intentions.

2. GUI testing should be done with a real person. You can't automated USEABILITY testing, which IMHO, is 80% of what GUI testing is for.  And, while you're doing useability testing, you can test for that other 20% (GUI bugs).

3.  We do the above and then send out a limited run of the program, burning our own CDs.  After 9 years now, we don't actually have many bugs turn up in this final stage, but it's still there as a comfort level.
How many times have you heard about throwing a batch of 1,000+ CDs in the gargage b/c testing (or lack thereof) missed a bug) ?

Mr. Analogy
Thursday, July 08, 2004

The thing about most developers is that they tend to test the code to see if it does what the code was written to do. This is wrong.
There are advantages to the developers testing the code/compile level of bugs, like typos in the code or the wrong objrct being called. But you need proffessional testers to do the code/spec testing, to see if the code does what the design intended, as opposed to what the developer intended. Not to mention iteratively improving the design as the use of the app becomes clearer.

-John Q.

John Q Tester
Thursday, July 08, 2004

"The thing about most developers is that they tend to test the code to see if it does what the code was written to do. This is wrong."

That's why you must write the tests first, not the code first.

Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, July 08, 2004

"There is no way around that....writing tests first is good, and catches certain kinds of bugs, but its not even close to being 'enough'"

It's astonishing to me that you read this into my statement. As a developer, the only way for me to write code quickly that is pliable and relatively (logic) bug free is by writing tests first.

I made no statement that talked about acceptance testing (i.e., does it flow well for the user), or that I felt that my testing was "enough".

Sheesh. Trolls.

Brad Wilson (dotnetguy.techieswithcats.com)
Thursday, July 08, 2004

I think the fundamental challenge in software development is requirements analysis.  Almost invariably in my experience the big problems for a product (or company) are rooted in writing the wrong code, not so much in writing faulty code.  If I may oversimplify:  once the system is implemented, you can fix the bugs but you can't change the requirements.

I'm not saying testing isn't important, of course.  Actually, I suspect that a powerful effect of TDD is that it focuses so much fresh attention on the big question:  just what are we trying to build here?

How many times have you been writing some unit tests and realized that your assumptions about what the module needs to do were wrong or incomplete?

secutus est
Thursday, July 08, 2004

Most programmers test for that the ordinary cases work, without too much thought to the extraordinary ones. A good tester tries the extraodinary cases too. An exeptional tester tries the apparently extraordinary case that turns out to be commonly done by the user.

MilesArcher
Thursday, July 08, 2004

Hi Brad,

"It's astonishing to me that you read this into my statement. As a developer, the only way for me to write code quickly that is pliable and relatively (logic) bug free is by writing tests first."

good for you :)

"I made no statement that talked about acceptance testing (i.e., does it flow well for the user), or that I felt that my testing was "enough"."

??? and I made no statement that said you did.

"Sheesh. Trolls."

wot?  Im not a troll, my post in no way attacked you or impugned on your skills.....whats ya problem?

FullNameRequired
Thursday, July 08, 2004

There are at least 6 (probably more) types of testing, all essential;

1.  Unit testing - where the programmer tests his own code to make sure it does what he/she (yes, there are some of us out there) thinks it should do;

2.  Destruction testing - where a tester (not the programmer) tries to (and usually succeeds in) breaking the program by clicking the wrong buttons, typing text into numeric fields, etc etc, without any idea of what the thing's supposed to be doing;

3.  Functional testing - where someone (not the programmer) who understands what the thing is supposed to do checks that it does it;

4.  Volume testing - where someone/something generates huge amounts of data to check that the thing doesn't collapse in a heap with al lot of data;

5.  User acceptance testing - where the key users check that it does what they wanted it to do;

6.  Reality testing - where the real users start to use it and soon get to know the software better than the developers.

I agree entirely with the person above who said it's better to catch the bugs before you deliver the software than afterwards. Also agree that testing can unfortunately be the first thing that gets dropped when timescales get tight.

Does anyone know any really good automated testing tools?

Miranda (UK)
Monday, July 12, 2004

*  Recent Topics

*  Fog Creek Home