Fog Creek Software
Discussion Board




Internal vs Shrinkwrap software bugs

Firstly, I suspect Kent Beck of being glib. If XP doesn't have a bug database, it's because you're encouraged to write bugs on index-cards and put them into the estimating process. At this point they magically become features. This is a good practice because it forces you to estimate and trade off bug fixes in the same way you do new features. It strips away the illusion that bug fixing is something that magically happens at some point perpendicular to the schedule.

However, it doesn't mean you don't have a bug database, it just means your bug database is a pile of index cards. Compare the Mozilla project's use of bugzilla as a database for feature requests.

Secondly, Joel states that shrinkwrap software has "higher bars for ease of use and lower bars for bugs" than internal software. While the former is, in fact the case, the latter is not, it's just a case of different priorities.

Take stability, for example. If Citydesk had a crashing bug that gave it an MTBF of four or five days continuous use, only a tiny percentage of the users would notice. Even if the bug were reported, the effort required to find and fix something that rare and obscure might not be worth the time that might be used adding a new feature to make the vast majority of people who only keep the program open a few hours happy.

On the other hand, if one of your company's internal systems goes down that often, it's going to end up costing you a lot of money. Someone I was talking to recently who works at a big investment bank was desperately trying to replace his CORBA ORB because it was crashing about once a month, and every time it happened their whole call-center was going dead, and costing the company hundreds of thousands, even millions of dollars.

There are other classes of bug that are more important in internal software than in shrinkwrapped software, simply because of the scale of the application. The original XP project, C3, was responsible for printing the pay-cheques for (I think all) Chrysler employees, so obviously it was going to be held to a much higher standard of reliability and accuracy than your copy of Microsoft Money.

On the other hand, Joel's conclusion is right, even if his premise (that the bar for bugs on internal systems is lower) was wrong. Many bugs that must be fixed in shrinkwrapped software can be ignored in internal software, because in internal software if someone says "Doctor! Doctor! It hurts when I do this!", you can often say "Well, don't do it."

This disposes of a lot of the classes of bug that are really hard to write automated tests for. Take the "Polish Windows" problem. Most localisation problems occur in the UI, and the UI is generally the hardest part of an application to write tests for even /one/ version of.

If things like your deployment environment are fixed, you have far fewer variables that you can't control, and thus you can write better tests that properly exercise how the application is going to be used when it's deployed. That does make life easier when you're hunting bugs.

Charles Miller
Wednesday, May 08, 2002

In XP bug is not listed as new User Story (feature card), unless it is a big change in functionality.

XP said - when bug is found, it should be demonstrated in the automated functional tests. Then developer should demonstrate it on code leve - in UnitTest. Then fix it.
Benefits are - this is more direct way to cimmunicate problem, and this is a way to ensure it is fixed forever.

Kent is too extreme. He is right - in XP you dont need BUG TO BE FIXED IMMEDIATELY database, but you still need support issues database. You have a question, you log it, you have to decide if it is a bug or change request or user is stupid. If it is a bug, it can be minor and you may want to fix it later when you have a time to polish your app.

Roman Eremin
Wednesday, May 08, 2002

BTW, you know - it is possible to deal with Polish Windows bugs in XP way.
In our system we did not have much problems with execution under different OSes, but we had OS-related problems with installation process.
So currently we have an automated way to test installation under Win95, Win98, Win98SE, WinME, WinNT and W2K (we can add more if we need). This is done on one machine, unattended and whole process took about 2 hours. This is done with VMWare profiles.
Test script is almost OS-independent, so if a problem is reported, we can demonstrate it in the script and immediately see if exists in other systems.

Joel's case is more complicated - he will need about 20-30 VMWare profiles to be sure he cjvered everything. But adding a new system to the test suite should not be that hard - you just create one more VMWare etalon profile and run same test script on it.

Roman Eremin
Wednesday, May 08, 2002

*  Recent Topics

*  Fog Creek Home