Fog Creek Software
Discussion Board




Frequency of different types of defect

Let's define three kinds of defect as follows:

a requirements defect is when the requirements are wrong; the cure is to change the requirements.

a mismatch defect)is when the code does not do what the requirements tell it to do; the cure is to change the code until it does.

a misuse defectis when the language is just plain misused; e.g. leaking memory or using a null pointer.

Question: what is the relative frequency of these three defect types in your work place?

When I asked this at a conference recently  I received lots of different answers.  What does this group think?

Keith Paton
Thursday, January 03, 2002

Almost all defects here are "misuse", and almost always in unanticipated situations. I interpret this to mean that we have a good sense of the requirements before we write the code, a good ability to write code to meet the requirements, and a testing department with a sense for pathological cases. It probably also means that we need to do a better job of simulating pathologic input in our unit tests.

Jeff Paulsen
Thursday, January 03, 2002

I am interested in your reply.  In my work I regard mismatch defects as those that can be detected only by people whereas I regard misuse defects as detectable by programs (code auditing tools).

Can you give us an example of a typical mis-use defect?

Keith Paton
Friday, January 04, 2002

A common C/C++ example would be

if (x = 1)  etc, etc.....

Simon Lucy
Friday, January 04, 2002

Yes, this is trapped by LINT as informational message 720.
(Boolean test of assignment).

I am anxious to gather a collection of misuses that people have found in practice.

I shall send a copy of the gathered collection to each contributor.

Contributions and pointers are welcome.

Keith Paton
Friday, January 04, 2002

x / y and x % y where y wasn't checked to be different from zero.

Evgeny Goldin
Friday, January 04, 2002

I think I misunderstood you the first time. It might be better to say that our defects are a mixture of all three types you distinguish; here is a made-up example. We have a complex sort of transaction in which items on orders can be moved, shared, and changed. The requirements are clear about all of these, but there might be a plain misuse of the language as a side effect of all three occuring at once.

Would you consider that a misuse defect or a mismatch defect?

Jeff Paulsen
Friday, January 04, 2002

A nasty "misuse" defect I was guilty of many years ago:

Have a function to look up an array index. It returns -1 to indicate that the thing it was searching for was not found.

Now use it in the following way:

infoArray[ FindMyThing() ] = 0;

You know, in C, -1 is a perfectly legitimate array index. And we spent about a week wondering why some data was being corrupted. Whoops.

Chris Tavares
Friday, January 04, 2002

Chris,

What about an automatically applied  test for
return value of FindMyThing not checked?

This requires that you are going to be disciplined about
which fcns should have their return values checked and which not.

Keith Paton
Friday, January 04, 2002

Jeff

was that a mis-use of the natural language in the requirements or a mis-use of the programming language?

I would define a mis-use of the PL as a construction that would be "wrong" whatever the requirements.

So memory leaks would be a mis-use; adding when you should have subtracted would be a mismatch

Keith Paton
Friday, January 04, 2002

<quote>
You know, in C, -1 is a perfectly legitimate array index. And we spent about a week wondering why some data was being corrupted.
</quote>

Excuse my ignorance, because it's a long time since I did C/C++, but isn't there a range check option in every compiler?

Jan Derk
Friday, January 04, 2002

I believe there is another significant type of defect.
I would call it an 'interface' defect, e.g something is possible even though the requirements do not say that it should be or should'nt be, say for example the user can hit a button, but it requires another step to be performed before the user should be able to hit the button. The user hits the button (without completing the required steps) and something weird happens, maybe causing an error, maybe not. Maybe they've just formatted the hard drive, but they don't know. This could be a requirements defect, but the requirements would need to be as thick as a few phone books to define all UI expectations.

Tony
Friday, January 04, 2002

60/35/5

Working in Java, I've noticed there's a lot less weird  "stare at the code until you realize there's a typo" errors.  (But we don't use case statements, so that helps.)

OTOH, the occasional stuff I do in perl is more like 20/30/50.

Scott
Friday, January 04, 2002

Replying to some above comments:

First off, in my own defense let me repeat that this was quite a while ago - about 6-7 years. I've definately learned some things since then, and I wouldn't use this sort of interface anymore. I'm much better at this stuff now, honest!

Automatically checking for FindMyThing's return value being checked? How would you add the checking? In code, or via some tool? Adding it in code is rather difficult; the need for yet another variable, etc. And how would you express the constraints for an external checker? This check needed to be at runtime, after all, not compile time. An assertion was inappropriate, since "Not found" was a perfectly reasonable return value. Just not in this specific case...

As far as range checking in the compiler - not in 1994, on an embedded system, using an old for the time version of GCC (1.x series, I believe). We were lucky we had function prototypes. I honestly can't say I've seen range checking in any compiler I've used, but I don't switch around very much and I generally don't use plain arrays much in my code anyway, so it hasn't been missed.

Chris Tavares
Saturday, January 05, 2002

Chris Tavares wrote:
"As far as range checking in the compiler - not in 1994"

Reminds me of the days when I was still inexperienced and innocent and they made me write Fortran using a line compiler and vi without decent debugging tools. I once searched 3 days for a bug which was caused by a line of code being more than 72 characters long. The bloody compiler just cut off that line without issuing so much as a warning. Thinking back I can't believe how inefficient I was.

Chris Tavares wrote:
"I honestly can't say I've seen range checking in any compiler I've used"

Maybe I'm just spoiled. First thing I do when starting a new project in Delphi is switch on Range, I/O and Overflow checking. No more fuzzy errors, ever. If the small performance penalty is important I switch the warnings off in the final code, but most of the time I just leave it on.

Jan Derk
Saturday, January 05, 2002

I'd say in my experience Mismatch is the one that happens most often.  Usually because projects I've worked on have been evolutionary in scope and the requirements change thus causing the mismatch with the code.

If I was asked this two years ago I would of said Mis-use, but over the years I've adopted a multitude of techniques to prevent obnoxious mis-use errors like if( x = 0 ) ( if ( 0 = x ) throws a compiler error and is quickly solved and thus one of those techniques ).

-Hosed

Lucas Goodwin
Saturday, January 05, 2002


if "findPlace" always returns a valid quantity we can use it without testing.
if it sometimes returns an invalid quantity then we should test before using.

The trick is to tell an automatic tool which functions are which.

Then the tool checks that you tested when you should have.

I do this all the time in my code auditing systems.

Keith Paton
Saturday, January 05, 2002

In reply to Simon Lucy's post there, to finding time and time again this if(var = 1) in the code.

There's a commonly known workaround to this, it works wonders and works on many languages. :)

Get used to put the constant first, and you'll never have that problem again, because the compiler will let you know about the violation.

if(1 == var)

Beka Pantone
Sunday, January 06, 2002

I think I'd rather be free to write the way I think
if(x==1)
and have LINT warn me when I say
if(x=1)
by mistake

Why should we work around defects in the language when our tools can allow us the best of both worlds?

Keith Paton
Monday, January 07, 2002

I used to write "if (1 == x)" code. Like the previous poster mentioned, that is NOT how I think and it grew tiresome. People will spend more time reading my code than I will spend writing it, so I now think the code should flow with the reader's natural train of thought: "if (x == 1)".

My employer paid a lot of money for the compiler I use; let's get our money's worth. ;-)

btw, bugs like "if (x = 1)" are the least of my team's problems..

cop
Monday, January 07, 2002

In any case, the trick does not work for the case of
if(x=y) used mistakenly for if(x==y)

Keith Paton
Monday, January 07, 2002

Actually, I'd add a couple more defect types.

There's an "Unclear API" defect, when you call someone else's APIs incorrectly (either third-party software or a layer than another team owns). For example, since I don't have much experience with LDAP, my LDAP code may occasionally fail. [This might be a misuse defect.]

There's also a "special case" bug, in which there's some unusual situation that breaks your code. [This might be a mismatch defect.]

These categories, along with unclear requirements, are the ones that I find most often in my own code.

Jared Levy
Tuesday, January 08, 2002

Here's a couple odd defects that you don't hear about, but are the worst in the world.

- Painful integration DEFECT
It works fine, handles all conceivable cases.  And when you need to hook code up to it or extend it, you feel like going postal because there are so many unpredictable subsystems.

Caused by both Spaghetti Code and lack of docs.  Cured by refactoring.  While it's "invisible" to the end user, it's like buying an electric drill that gives unyielding grief if you don't use certain preset drill bits.

Happens all the damn time.


- Impossible-to-solve DEFECT
"What do you mean, Java leaks memory everytime you bump the computer!?"

No amount of care, proofs or Tech Support can save you.  The platform's just broke.  Caused the complexity of the world.  Cured by voodoo and the art of Workarounds.


- Security DEFECT
You may think this is a requirements defect.  But it happens so perversely often, that it gets its own category.  Sometimes an emergent property of the system makes it possible to have a secure design doc, while still being exploitable.

Sammy
Tuesday, January 08, 2002

*  Recent Topics

*  Fog Creek Home