Fog Creek Software
Discussion Board




When is a bug a bug?

We're trying to convert our IT department from fire-fighting mode to proactive. One task is finally creating procedures. We're using a huge complex custom software application and the whole IT team is expected to report bugs. I'm writing the procedure, and I'd like to be explicit.

So, when is a bug a bug? Is it any time the app doesn't do what the user expected (except, of course, if the user entered/selected erroneous data)?

Is it the above, but only when it conflicts with the User's Guide? (Doing the task in a non-optimal manner and getting unexpected results isn't a bug.)

Is it any time the app doesn't perform the way the designers expect, regardless of the user's expectations?

joel d canfield
Tuesday, September 23, 2003

A bug is when the actual output does not match the expected output, given the inputs. You can't leave out the "given the inputs" part, because it changes the expected output. (For f(x,y)=x/y, the expected output is usually x divided by y. But for y=0, the expected output is some sort of error message.)

And I've now written 'output' so many times that it no longer sounds like a word.

Martha
Tuesday, September 23, 2003

A bug is any violation of the Law of Least Astonishment.

Alyosha`
Tuesday, September 23, 2003

I would make sure to distinguish beteween features
and bugs. If you have features that people want
in your bug database then your bug count will
look out of whack.

compile error
Tuesday, September 23, 2003

"Doing the task in a non-optimal manner and getting unexpected results isn't a bug." - depends on the unexpected result. Is it a system crash? That's always a bug, no matter what. Is it an error message? That's not a bug. In between those two extremes, it's probably a bug, but of varying severities. It just gets very hard to define in a procedure.

That's why I'd stick to a nice, generic definition. It's easy to mark a bug "invalid"; it's much harder to fix all the little issues which you didn't enter because you weren't sure if they qualified as bugs.

Martha
Tuesday, September 23, 2003

In the end a bug is anything your user would label as such. You can't talk sense into people - if your customer believes some behavior is a bug, then it should be in your bug database. If not and if your customer finds out that you tracked his bug differently than other bugs (like on a piece of paper), then he'll think you're not taking his issues seriously.

Johnny Bravo
Tuesday, September 23, 2003

"A bug is when the actual output does not match the expected output"

Expected by whom?  The code author?  The user?  The QA people?

Robert
Tuesday, September 23, 2003

"depends on the unexpected result. Is it a system crash? That's always a bug, no matter what."

Baloney.  Maybe this code is a system crash recovery test driver.

NEVER overgeneralize. :)

Robert
Tuesday, September 23, 2003

What Johnny Bravo said.

And, this is also why "bug" is a dumb word to use for this.  I like "issue", instead.  You may or many not have a bug, but your customer definitely has an issue, and that issue needs to be resolved in some way.

Another creative term: "incident".  It may not be a bug, but *something* happened, so it's an incident.

A case-tracking system I created at my current employer didn't have a "bug" classification.  There were only Questions, Problems, and Requests.  (With lots of subcategories like "Problem - Service Interruption", and "Request - Feature Enhancement".

The problem with "bug" is that nobody wants to admit that there are bugs, in part because of management stupidity.  My wife uses a POS program at her store, Cash Register Express, which the support staff claims has no bugs, as a matter of company policy.  They absolutely refuse to ever admit that anything in their program is a bug.  It doesn't matter if you're sending them a screenshot of a program crash, you're making a "feature request".  That's a pretty stupid attitude, but I'm guessing there must be some kind of incentive system in place there to punish bugs.

One part of the problem, is that using the word "bug" makes it open to question in the first place!  This whole thread exists because it's not even 100% clear what *is* a bug, so of course people are going to argue about it, and try to argue their way out of whatever incentives exist.

Now, if customers have "issues" or "problems" with software, this is an indisputable fact, since it is the customer who decides whether something is an "issue" or a "problem" for them, and they will always be right.  :)

Now, that doesn't fix improper incentivizing(?) around problem metrics, but it *does* encourage a focus on say, a metric of "problems per customer", rather than a "total bug count".  But, if customer-contact people are the ones who enter the issues, and there is no way for developers to take revenge on them, then the developers can't manipulate the metric and it's a bit less dysfunctional.

Phillip J. Eby
Tuesday, September 23, 2003

When I've worked in QA, I've always used the criterion that "anything that doesn't work the way I'd expect is a bug". Of course, that pre-supposes that I know what I'm doing with that particular product.

I've rarely seen a situation where too many bugs are reported against a product. As someone else mentioned, it's easy enough to categorize defects as "feature requests" after they've been entered. On the other hand, if somebody doesn't report a problem, you might miss a really serious bug.

You want to encourage everyone to put as much information into the bug tracking database as possible - feature requests, "only saw this once" reports, everything. Use the bug reporting system's features to cut down on the clutter, but make sure you have a good total picture of the state of the software.

-Mark

Mark Bessey
Tuesday, September 23, 2003

as the software always works as the code describes it, the code itself cannot be buggy. crash is a normal behaviour, the result of the code.

so then bug becames what is not expected, or doesn't work as the user wants.

this leads that everything the user ask will be a bug. it doesn't work as i expect.

sad but true. we created a software for a layer. image who had better skills to determine what is a bug.

na
Tuesday, September 23, 2003

A bug is when the program doesn't do what the design documents say it will do in a given circumstance.  Everything else is something else: a feature, a suggestion, a note, whatever. 

In my experience you need an iron clad definition of a bug or everything will be lumped in there.  If not, you will get into pissing matches with the authors as they will think their problems are the most important thing in the world. 

An important thing to stress is issue has to be repeatable.  I have had waste a significant amount of time squashing bug notes that were one time failures induced by external factors; like the OS crashing. 

D
Tuesday, September 23, 2003

I am with Philip on using "issue". We have issues reported and the customer, product owner and development teams used to waste a tremendous amount of time making their case on bug versus feature. When we got a little smarter, we realized the important thing is to prioritize the issues with the customer and get them happy. Figuring out what is and isn't a bug usually leaves one side unhappy.

m
Tuesday, September 23, 2003

I think that it's not worth distinguishing between bugs and features.

Call them "issues" and define them as any case where you would like a code change to be made.  Some percentage of the issues are bugs, some are features, and some of them are people complaining about the milk in the cafeteria.

This also has the benefit of making it harder to count out how many bugs there are, which is, as Joel says, a very bad idea.

Flamebait Sr.
Tuesday, September 23, 2003

You need to think a bit more widely than 'bug'. There are several things here that fall into that space:

Incidents - product does not appear to be working as expected

Problems - Definitely a problem the root cause of which has not been determined

Errors - The root cause of one or more problems. The failing components have been identified.

A problem may manifest itself in one or more incidents.

An error may be the root cause of one or more problems.

Any system that attempts to cover this all by just classifying things as 'bugs' is not going to cut it (sorry Joel).

This is all commonsense brought to you under the guise of ITIL. Check it out.

Gwyn
Tuesday, September 23, 2003

Everything your end-users perceive as a bug is a bug.

For some it might seem extreme, but even when the user entered erroneous data I call it a bug. That one gets classified as a GUI bug. If it appears, additional data input checking is added to prevent it from happening again. If the user guide conflicts with the application then it is a documentation bug.

The beauty of being this obsessed about bugs is that you have to provide much much less support.

It is also one of the reasons why developers should, at least part of their time, provide customer support. Every support request is actually a bug report that did not make it to the bug database. With each customer they help out, only one thought should be on the developer's mind: What can I do to prevent this from happening again? Improve the GUI? Improve the documentation? Improve the online FAQ? Fix the installer?

About the bug/issue/feature naming: Bad developers get defensive about bugs. Good ones know they happen and fix them with the highest priority.

jan Derk
Tuesday, September 23, 2003

If you're looking for criteria, what about using a

What did you do, What happened, What was expected, how do we repeat it? approach. 

Everything else requires investigation, these are bugs.  That way you can direct energy at bugs and allow QA or another group to investigate the issues. 

Lou
Tuesday, September 23, 2003

You really need three different types of item - bug reports, new features or upgrades, and planned work. These can be relationally linked of course.

As well as making planning easier, it protects you from the very real danger that someone will come along one day, count the "bugs," and report to someone higher up that there are lots of bugs.


Tuesday, September 23, 2003

> ...even when the user entered erroneous data I call it a bug, ... a GUI bug.

This is bad practice. That's not a bug. It's an opportunity to make the application more robust.

My definition of a bug is that a piece of functionality doesn't do what it's meant to do, at the code level. Problems arising from users not understanding or using the application are opportunities for improvement, not bugs.


Tuesday, September 23, 2003

This is an interesting topic, and one that comes up all the time between testers and developers.

I'm a software tester by profession. My job, distilled to the basic essence, is to find bugs in software. Obviously the question of "what is the definition of a bug?" is one that needs to be answered.

What you will find is that programmers have a very different idea of what constitutes a 'bug' than your average tester - just take a look at some of the definitions given in this thread - typically the programmers idea is that a bug is very narrowly defined.

A tester will invariably have a much broader definition - personally I use a definition like "Anything that is wrong with the software is a bug." 

I don't care if it's "by design" according to the developer. I don't care if it matches the spec exactly. If I see a problem with the software, it's going in the bug database - and we can argue about it in the bug triage meetings.

Developers can be wrong. Specs can be misguided. Customer requirements can be written by people who don't understand the core problem.

A good test process encourages this sort of critical thinking, and does not get hung up on whether something is a "bug" or a "feature" or an "issue"; if it's going to result in work by the dev team, it belongs in the bug database.

Mike Treit
Tuesday, September 23, 2003

Sometimes the flowers in my flowerbeds spread into the driveway. When they do, I call them weeds and pull them.

zen gardener
Wednesday, September 24, 2003

> A tester will invariably have a much broader definition - personally I use a definition like "Anything that is wrong with the software is a bug." 

It's legitimate for developers to be cautious on this issue. The tester's job is to find ways the software can be improved, and his definition of bug seems to encapsulate this.

But in many settings, this subtle difference in meaning is never articulated or preserved, and the list of bugs is used as a mark of what's "wrong" with particular products, or even software in general, and by corollary with the programmers who produced the software. There's nothing wrong with programmers being careful over the definition.


Wednesday, September 24, 2003

> A good test process encourages this sort of critical thinking, and does not get hung up on whether something is a "bug" or a "feature" or an "issue"; if it's going to result in work by the dev team, it belongs in the bug database.

<rant strong=true>
Sorry Mike but that's patent crap. It may satisfy your need from the testing silo but it doesn't satisfy the overall development process's needs.

You have incidents, problems, errors and changes. Some changes are as a result of errors, other are as a result of requirements change.

You cannot manage all these as a list of bugs. They each have different attributes and lifecycles.

I get really frustrated when people only look at the part of the development architecture they can see from their own corner. It leads to shoddy solutions and incompatible processes.
</rant>

gwyn
Wednesday, September 24, 2003

And another thing...

The reason we differentiate between problems ("bugs") and requirements changes is because the developers pay to fix their work but the customer must pay for requirements' changes.

I found a project in ICL where an internal customer had agreed to pay £50,000 for a solution but the customer kept changing the requirements (and he was a bit of a bully so people wouldn't stand up to him) and eventually it had cost ICL the best part of £1,000,000 (yes that's right, 20 times as much) to produce the solution. Poor (well, no) change management was to blame.

gwyn
Wednesday, September 24, 2003

Joel,

>> So, when is a bug a bug? <<

Are you trying to track bugs only, or track anything that might involve a change to the system?

If you're trying to track bugs, and only bugs, you have a problem. Given N people, you'll see N+1 definitions of "bug". This sets end-user against designer, designer against tester, and developers against everybody.

A better idea is to have a "change management" database rather than a bug database. Anything that might potentially result in a change to the system goes into this database. In this way, there's no argument about categories and no definitions war.

Of course, each entry into this database still needs to be prioritised (to allocate work) and categorised (to manage lifecycle). This unenviable job is given to a change management person / team / committee. The job will certainly need input from end-users, testers, developers etc, but the final decision about prioritisation and categorisation belongs to the change management people.

Mark
----
Author of "Comprehensive VB .NET Debugging"
http://www.apress.com/book/bookDisplay.html?bID=128

Mark Pearce
Wednesday, September 24, 2003

gwyn:

I have a feeling you must use an ineffective bug tracking system / process.

Your statement that different types of issues have different life-cycles, and that (in your case) developers pay for fixing errors while someone else pays for requirements changes, doesn't imply to me any good reason to try and track these in different systems, or not treat them as 'bugs'. If you have a simple process in place for reviewing every bug, and a good bug tracking system, it's a trivial matter to keep track of who owns what.

In fact, having everything in a central place is infinitely more efficient for everyone on the project than trying to segregate everything based on definitions nobody will ever agree upon.

Mike Treit
Wednesday, September 24, 2003

Quite obviously, anything that "ought to be changed" needs to be tracked, prioritized, and hopefully acted upon. Call them all incidents if you like.

As pointed out, if the software performs as designed but doesn't satisfy the requirements, it is not delivering value. You need to act on that.

But there's an issue that the development team needs to track closely: software that does not perform as designed. If you can't consistently deliver code that passes aceptance tests and/or unit tests, you don't have a sustainable process: you're rolling the dice and hoping to get lucky one day and ship.

So by all means track all kinds of issues. But you have to have a way to identify and track cases where the code does not meet spec, regardless of whether the spec describes a system that delivers value.

Reginald Braithwaite-Lee
Wednesday, September 24, 2003

I take issue with the comment that "anything the customer thinks is a bug is a bug". We get tech support issues raised by customers that they think are bugs in the software, but turn out to be their unreliable network.There is not a lot an application can do if the OS can't get to the network drive that has the data on it.

Or, the application is for putting square pegs in square holes and the user customizes it to try to use round pegs and it the pegs either don't fit or are wobbly. Enhancement request, issue, whatever, but it's not a bug.

I've been down this road with QA people all the time, it's always debateable and somewhat arbitrary. Therefore, you need to track both enhancement requests and bugs and then prioritize them in a reasonable way.

pdq
Wednesday, September 24, 2003

I'm a developer, but one who very strongly believes that I want to hear about all problems of any kind that might relate to my software in any way.  If I don't get told, it'll just happen again, after all, and that's not going to increase perceived quality of my work any.

On the other hand, even I sometimes don't want to hear it.  The one developer who puts trivial comments such as "This log message should have two spaces not one" in the bug database does annoy me all out of proportion to the developer who sends me a polite instant message asking if I object to having the spacing changed.

If there is no reasonable possibility it could reoccur later, AND it takes no more than five minutes at zero effort to fix AND the cause of the problem is obvious from the description...I won't report it myself, and I'd rather other developers mention it to me casually.  (Although I do use the bug tracking tool to report something trivial in my own code, if there's a reason I can't immediately spend that five minutes fixing it.)  On the other hand, I know that testers can't necessarily measure this stuff -- if the bug has reached them and they have the time and patience to report it, it needs to be tracked no matter what it is.

On to the other side of things...bugs I like:
A) Reproducible description of how to cause the problem -- this is ideal.
B) System does not match docs.  Obviously at least one needs to be changed, and these are often a hint at design issues.
C) System does something unexpected.  Either the system should change, the docs should [exist, and] explain why that's the expected behavior, or it's a 'known issue' we don't intend to fix right away -- in which case it should still be documented somewhere, like the bug system.
D) A long, detailed as possible story about something that happened in testing once, but can't be reproduced.  (Along with the usual version numbers / dates / platform info, provide the exact text of errors, copies of log files, copies of input, etc.)  It's a *lot* easier to mark this as unverifiable and ignore it, than it is to try and remember "what happened that one time" when a seemingly related problem pops up later.

People who bug count don't want to see C and D bugs in the tracking system, and often want B bugs relegated to documentation error (whether or not it'd be better to change the code).  But bug counting is a waste of effort anyway -- what's important is what the unresolved issues *are*, not how many there appear to be.

Mikayla
Wednesday, September 24, 2003

I'm a manager and I agree with Mikayla.

(Perhaps Mikayla is a 99.9%er? ;-) )
I however, do count bugs, not to determine some arbitrary level of quality but to predict release dates. You can watch the trend line of all of the schedule bugs for a project and get another datapoint on when the ship date will be from what the development lead is estimating.

pdq
Wednesday, September 24, 2003

*  Recent Topics

*  Fog Creek Home