The Old Joel on Software Forum: Part 3 (of 5)

The anatomy of a bug

With URL linking turned off, you'll have to do more work than usuall (copy paste a url, dude ...), but it is worth it.
At http://headblender.com/joe/blog/archives/microsoft/001280.html Joe Bork gives us an interesting in depth account to show why some bugs are left in released products ...

"Of course, Microsoft ships software with bugs. You might think that, as a software tester, I get very angry about this. You would be right. Mostly, I get angry about bugs that aren't discovered until after the product has been released, since this means that the test team missed something, and we tend to take things like that rather personally. However, there are a few cases where we (and by "we," I mean "me and some other people on my team") find a bug, and perhaps the developer has even produced a fix for the bug, and yet we ship the software without fixing the bug.

I can hear people screaming "You do WHAT?" at the top of their indignant lungs, so please -- calm down. In the interest of lesser opacity, I'll take one (of a very few) of these kinds of bugs as an example, and talk in excruciating detail what the bug is, and why we decided not to fix it before RTM (release to manufacturing)."

Just me (Sir to you)
Monday, October 27, 2003

If we do a run of 1000 test iterations and we have a
data access error on the 900th iteration, should this
prevent us from shipping?

If a tester files a bug because they don't like the font
spacing on a line should we not ship?

If a seldom used feature does not work and it
would take 2 months to fix, should we
not ship?

If a feature is not ready should we not ship?

You have to assign a priority to each and every
feature and bug and then make judgements. It's no
different than any other part of life.

Priorities are based on:
1. revenue
2. customer commitments
3. quality level
4. quality level relative to important customers
5. desires for what we want the product to be

No where in this list will be "bug free software."

son of parnas
Monday, October 27, 2003

Why are the keys in the ActiveSync section of the registry? It seems to me that all problems would disappear if they would be in VS's section, and that it's also The Right Place(tm) to put them, because they're installed by VS and used by VS, not by ActiveSync.

Roel Schroeven
Monday, October 27, 2003

Take the number of vehicles in the field, A, multiply by the probable rate of failure, B, multiply by the average out-of-court settlement, C. A times B times C equals X. If X is less than the cost of a recall, we don't do one.

Danil
Monday, October 27, 2003

Good quote... More at http://www.imdb.com/title/tt0137523/quotes :-)

Frederik Slijkerman
Monday, October 27, 2003

Lets turn the flaming pinto argument around.

We have 100 customers who are dying to get the latest release because it's going to save them tons of money. There is a bug that might affect one of them in a minor way. Should we hold off for another week or so just to fix that one bug.

Another couple of weeks, you say, it's just a couple of minutes for the developer? Yes, a couple of minutes for the programmer, a day to build and check. The balance of the week of another round of QA just to make sure nothing important got screwed up.

That assumes that we either don't find another bug in the mean time and we didn't break anything with the correction.

You can't have zero defects in software. If you think you can, you haven't thought hard enough about what a defect is.

pdq
Monday, October 27, 2003

"Another couple of weeks, you say, it's just a couple of minutes for the developer? Yes, a couple of minutes for the programmer, a day to build and check. The balance of the week of another round of QA just to make sure nothing important got screwed up."

...AND time for the product documentation people to change the user manual and change the help files. If the bug fix is big enough, the marketing people might have to change their blurbs about how it works.

A one-minute change for a programmer can equate into weeks and weeks of delay once the change ripples down the entire chain of people involved in releasing a product.

Norrick
Monday, October 27, 2003

Actually, my read of the article says those keys ARE used by ActiveSync -- VS uses ActiveSync as part of its installation and debugging process, and those keys modify ActiveSync's behavior to support this activity. Having those keys in the VS section would require modifying ActiveSync to look for Visual Studio, which would seem to be a bad idea, especially if any other companies wanted to also support debugging of Windows CE devices.

Ben Combee
Monday, October 27, 2003

"...AND time for the product documentation people to change the user manual and change the help files. If the bug fix is big enough, the marketing people might have to change their blurbs about how it works."

Err...how many bugs that would take a developer a couple of minutes to fix will require manual and marketing changes? I'm completely in agreement with the linked article that there is a point when you ship, and no software is perfect, but I think your example is a tad rhetorically ridiculous.

Dennis Forbes
Monday, October 27, 2003

"(As an aside, I always picture "eXtreme" programming practitioners using phrases like "Dude, gnarly unit test!" but maybe that's just me. I'm not a big fan. But I digress...)"

I found that the most remarkable quote of the article.

Leonardo Herrera
Monday, October 27, 2003

Well, we don't seem to be able to cycle in under a week no matter how small the bug is. I am assumming no marketing or significant documentation changes.

Let's say the tab order isn't correct in a dialog. This still takes a week to cycle. I agree with the article that it's a tough call to decide to ship.

I take exception to the comment that implies it's greedy corporations that cause bugs in software.

pdq
Monday, October 27, 2003

Dennis,

"Err...how many bugs that would take a developer a couple of minutes to fix will require manual and marketing changes?"

Well the change documented in that article would need an update to the manual to explain to the user how to do the manual registry repair option, and also explain what that function was doing.

Probably not in a product like VS.Net where they might assume the instructions "We cant repair this automatically. Please run the reg file to fix it." is enough for the average user, but if a similar bug was found in MSWord then there would be at least 2 (*no of languages) pages of help to update / create (KB, F1 Help) so you can see for a company that has as much documentation and development going on in so many languages that this simple change could easily become many weeks of work.

ChrisO
Monday, October 27, 2003

But that's a change to the documentation because the bug _wasn't_ fixed. That doesn't explain why you'd change the documentation if a bug _were_ fixed.

Kyralessa
Tuesday, October 28, 2003

"That doesn't explain why you'd change the documentation if a bug _were_ fixed"

Two cases I can think of OTTOMH:
1) There is a version out there whose manual describes the bug
2) Fixing the bug causes other described behaviour to alter. Sometimes a developer gets behaviour from a method which he believes is correct and so he works with that. Then it turns out that the behaviour wasn't correct so the client code works differently after the bug fix.

Tuesday, October 28, 2003

Well the fix as described *did* change the behaviour of the two products, as it changed their interaction - given that the bug was an interaction bug, that much is obvious.

As such, its quite likely that it would require a documentation change in at least one of them (VS2003 I'd say) to note whats going on.

Robert Moir
Tuesday, October 28, 2003

While I agree with the overall points fo this article, I'm glad that this guy is a tester not a coder...

I can understand the his concern about adding this check to the main code path. I think this is a valid concern. But it begs the question of why they are adding it as a pre-deployment check as opposed to a post failure check. Once the deployment fails you could check if the keys were missing and then do whatever recovering is necessary. Personally I would have put up a message box telling the user that registry keys are misssing and to import the ProxyPorts.reg in the VS.Net directory. The users are assumed to be developers -- they should understand what this means and how to correct it. It is certainly much more preferable than getting a generic 'Deployment Failed' error.

Billy Boy
Tuesday, October 28, 2003

If you assume the users are developers, you assume incorrectly. What about students in a college learning programming skills?

One of the main points of the article was that they didn't want to assume things about the users (e.g. most devs have admin access but they didn't want to assume this would be the case).

Robert Moir
Tuesday, October 28, 2003

If you are using a development tool, what are you? Hopefully a developer.

I may agree with you that students wouldn't understand, but at least the admins that they complain to will have a fighting chance of not having to reinstall the entire shooting match -- at the university I went to this would make the difference between the lab being down indefinitely (no time to rebuild all that -- just use the emulator) or having things up and running again in a short while.

I wouldn't assume that the butt in the chair is the one fixing problem.

Hang So
Tuesday, October 28, 2003

Just out curiosity what is Microsoft's policy on personal blogs discussing company process. I've seen quite a few blogs recently by Microsoft employees.

christopher baus (www.summitsage.com)
Tuesday, October 28, 2003

They appear to encourage it.

pdq
Tuesday, October 28, 2003

> Just out curiosity what is Microsoft's policy on personal
> blogs discussing company process. I've seen quite a few
> blogs recently by Microsoft employees.

The policy is pretty straightforward. Do not discuss confidential stuff (like schedules for unannounced products), be polite and friendly, stick to technical topics (more or less), and focus on solutions, not problems.

We think that our software dev process is quite excellent, so the more people who know about it, the better. In fact, I discussed it today in my blog!

Eric Lippert
Tuesday, October 28, 2003

So are you saying that you can't have a blog about your reptile hobby. Why does msoft care about that? Or is that just on their servers?

christopher baus (www.summitsage.com)
Wednesday, October 29, 2003

We're encouraged to keep it _primarily_ technical. Raymond Chen, for example, pretty much has a technical topic every day, but also talks about curling, learning to speak Swedish, and other non-technical topics.

And of course, these are guidelines for blogging on blogs.gotdotnet.com -- if you want to run your own server, then you go right ahead and say anything you want. (Modulo nondisclosure agreements, etc.)

Eric Lippert
Wednesday, October 29, 2003