Fog Creek Software
Discussion Board




Paul Graham says we don't need bug tracking softwa

In his collection of excellent essays, specifically the one entitled "The other road ahead" (http://www.paulgraham.com/road.html) Paul Graham states -

"We never had enough bugs at any one time to bother with a formal bug-tracking system."

This somewhat shook my world view since I am of the camp that any successful commercial sofware project is as good as its bug tracking system.  Currently my small project (using Fogz bugtracking system) has logged around 200 bugs in the last two months. While these range from feature requests to incompatibility issues or even just musings, I attribute our attention to detail and fast turn around time to this system.

Is Paul Graham on to somthing or is he wrong in this instance? Would his Yahoo shopping software been improved with an official bug tracking system?

Jon Kenoyer
Saturday, July 03, 2004

Paul Graham is on something. :-p

The system is not about how many bugs you have. It's about having a single, authoritative list of bugs. You can get by with an Excel spreadsheet if you have a few bugs, but I'd claim that's still a "bug database".

Brad Wilson (dotnetguy.techieswithcats.com)
Saturday, July 03, 2004


I haven't read the article, so I'm not sure of the context in which he said that. However, I can't imagine not having a bug tracking system for any project that is larger than a couple of files.

Trying to build even an average sized application without a bug-tracking tool is foolhardy. There is simply no way that you can keep track of all of the bug, tweaks, feature requests, etc in your head.

Of course, bug tracking software is only a requirement if you have bugs. If you write "perfect code" (as a former co-worker of mine bragged about) then I guess you don't need bug tracking systems...

Mark Hoffman
Saturday, July 03, 2004

"With Web-based applications, everyone uses the same version, and bugs can be fixed as soon as they're discovered. So Web-based software should have far fewer bugs than desktop software. At Viaweb, I doubt we ever had ten known bugs at any one time. That's orders of magnitude better than desktop software."

(from "The Win for Users" on the same page)

www.MarkTAW.com
Saturday, July 03, 2004

Paul Graham is right. Just don't create any bus in the first place and you won't need any bug tracking software. it's as simple as that.

Dennis Atkins
Saturday, July 03, 2004

As he says:
> Bugs turn up quickly.

And devotes a lot of text to bugs, so it seems unlikely
they didn't have bugs. Just because you don't
track bugs doesn't mean you don't have them. And
once you go to an actual release cycle, customer support,
new feature schedule, have a qa team etc, you need
a tracking system.

son of parnas
Saturday, July 03, 2004

I've never once written a bug into any of my code, as I am much smarter than all of you or anyone any of you has ever worked with.

So I don't really understand all of this bug tracking nonsense.

Mr Fancypants
Saturday, July 03, 2004

I'm just waiting on Joel's "Paul Graham is going to die soon" article.

Philo

Philo
Saturday, July 03, 2004

Is that bitterness I smell Philo?

Eric Debois
Saturday, July 03, 2004


The question is whether you need a bug database SYSTEM.
I think you always need a LIST of bug, it's just how formal you need it.

If you'll have a lot of bugs or more than one person needs access, then you need a Database (or "system")

MY EXPERIENCE
We have 18 comercial programs. We've never needed a bug DATABASE, but we track bugs. It's probably on the order of 10 per program open at any time

I've always done consisten unit testing on our pograms though.  Without realizing it, I've always use the XP approach: code and test. (Well, OK, XP says I should test, fail, code, test, pass. Will be doing that in the future!)

I am no genius programmer, I was just always very paranoid about bugs when I first started programming. (My first program was a comercial program). So, I always tested a lot. 

Mr. Analogy
Saturday, July 03, 2004

It's always fun to watch people get up in arms at the suggestion that everyone isn't as screwed-up as they are.

: "We built our trillion megawatt hoobafunken software with three programmers, straight out our heads and into code, and it has none of the problems but every feature of Acme Corp's HoobaMaster, which they have 300 'engineers' and 700 testers working on.  Oh, and it performs and scales better."

:: "Well that's fine for some amateur product, but I've been in this business since Eniac, and I *KNOW* you must have at least 100 head and a specially certified process to do serious work"

(Umm, no.  Just because you don't have the talent to do it without so much wheel-spinning and cattle doesn't mean that's true for everyone.)

: "We didn't have many bugs, and fixed them all within minutes of finding them anyway, so we didn't need a grand scheme to track them."

:: "What??  He's lying.  He just doesn't *know* that he has the same thousands of bugs I have!"

For some insight into the cause of this malady, read "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Lead to Inflated Self-Assessments" by Justin Kruger and David Dunning of Cornell University.

Tally Ho
Saturday, July 03, 2004

Eric - nope, was tongue-in-cheek. [big grin]

Philo

Philo
Saturday, July 03, 2004

> . It's probably on the order of 10 per program open
> at any time

I have had systems with over 1000 features scheduled
for various releases planned 5 years out.  I want a
bug system.

And if a product has over 100 components and
there's a bug per component and if you have 10 products
that's 1000 bugs.

And you have all the bugs that QA finds that aren' t
really bugs but must be resolved.
And you have the wonderful 1-pixel alignment type
bugs. And you have customers who need to see all these
issues, as well as numerous internal departments.
It seems few here have a system where the
customer actually gets access to your internal systems
and wants to see every twitch.

If you have your freeware poker program you developed
all by yourself  then write bugs on toilet paper.

Otherwise the bug needs to be assigned to a person
and a release, it has to be prioritized, scheduled,
resources made available for the testing and development.
For example, the lunar lander simulator can only be
used by so many projects. People need to plan the fix
and related fixes, people need to know what build
it is in, have QA write tests to verify that's fixed, then
know what release it is in, and be able to tell customers
all the bugs and the number of lines of code that changed
and the test procedures and perform the root
cause analysis as to why it existed and perform a FMEA to
show that you've handled similar situations and it
will never happen again.

Not everything is yahoo whatever where nobody
really gives a shit if it works or not.

son of parnas
Saturday, July 03, 2004

^^^^^^^^  right here

See what I mean?

Tally Ho
Sunday, July 04, 2004

I've worked on projects with hundreds of open bugs, all carefully filed in a bugs database, and I've worked on projects where the bugs list was so small that it was "maintained" by Emacs in a text file.  The key difference isn't the size of the project, but the use (or not) of the Zero Known Bugs methodology.  Fixing bugs as soon as they're found means that the bugs list will never swell too much, and the bugs will be easier to fix, too.

You still need a bug tracking system, however, for a variety of reasons: to pass bug information from QA to development; to track feature requests; etc.  But with Zero Known Bugs, the bugs list will transform from a sabre-tooth tiger to a kitten.

Oren
Sunday, July 04, 2004

Don't confuse the size of the product with the size of the team.  They don't correlate very strongly in this age of big teams of mouth-breathers.  Big team almost always means bad team, and that requires a bugs database.

Tally Ho
Sunday, July 04, 2004

Ho, do you even know what FMEA means?
Have you ever been in the sort of project
that cares what it means?

Have you ever been on project with millions
of users that all get to call up and make bug
reports that need to be tracked and handled?

Have you ever done work that is more complex
than fixing spelling or an index problem? Let's
say MS has a bug called "Fix Security." Do you
think that bug can be fixed  immediately? It
would take an immense amount of time to
figure out where the current problems are and
the potential fixes. It might spawn thousands
of bugs over multiple releases over multiple
years over multiple teams.

Big teams don't mean bad teams. Small teams
don't mean good teams. Good people mean
good work gets done.

How many people do you think work on a big cisco router?
Do have an idea of the number of separate
technologies and skills that make such things
work? The teams are big. The problems are complex.
The problems are often not cost effective to fix
immediately and are scheduled for later releases.

Yet your response to all this real world fact will
be something insightful like "see what i mean"
or "big is bad" or "fix bugs fast".

son of parnas
Sunday, July 04, 2004

Son of parnas: I'm not sure *you* know what a FMEA is. The entire FMEA process is really meant for manufacturing, not software construction. FMEA's just don't fit the software construction process very well.

I'd love to see you try to build a process FMEA and a set of control plans for programmers. Building an instruction sheet for a CNC operator ain't exactly the same as providing a set of specs to a programmer. Its an entirely different ball of wax.

Heck, I doubt you know the difference between a design FMEA and a process FMEA. I do; I'm a programmer for a tiny company whose sole offering is a FMEA tool.

Dave
Sunday, July 04, 2004

> FMEA's just don't fit the software construction process
> very well.

If you mean it is hard because programmers don't
want to do it and it takes a lot of work then i would
agree.

That FMEA doesn't fit for software is not true at all.
If you build a class 5 switch, for example, you will do
an FMEA for the entire system, that includes software.
For hardware it is much easier than software.  Having
been through the process several times, unless
i was dreaming, i can state it definitely applies to software.

son of parnas
Sunday, July 04, 2004

I would love to see a few pages of one of your FMEAs. That's assuming you know the difference between a design FMEA and a process FMEA. And your earlier comment shows that you've probably confused a Design Validation Plan with Control Plans with design & process FMEAs.

And your crack about the difference between software and hardware -- that's a doozy. If your hardware FMEA's are relatively short, then you simply haven't asked enough questions about your product/process.

I'll bet you build these in Excel, don't you?

Dave
Sunday, July 04, 2004

> I would love to see a few pages of one of your FMEAs.

They go on for ever and ever.

The hardware is easier because it is fixed. I never said
they were short or easy. Quite the contrary. And
hardware has very detailed specs. For a capacitor we
know how it could fail, we can tell what happens, how we
detect it, isolate it,  notify about it, compensate for it, etc.

On a complex software system it is daunting.

son of parnas
Sunday, July 04, 2004

Define forever.

I'd still like to know how you build a process FMEA for software. And you still haven't demonstrated you know the difference between process and design.

You build these in Excel, right?

Dave
Sunday, July 04, 2004

> Define forever.

1. For everlasting time; eternally: No one can live forever.
2. At all times; incessantly: was forever complaining about the job.

> I'd still like to know how you build a process
> FMEA for software.

Doesn't software describe a machine? So how
is it different?

>And you still haven't demonstrated you know
> the difference between process and design.

I can touch my nose with the tip of my finger
while walking. Is that a sufficient performance?

> You build these in Excel, right?

Why don't you just pitch your tool.

son of parnas
Sunday, July 04, 2004


> Define forever.

That's a pretty simple request -- how many lines are on one of your FMEAs? Just pick a representative product.

I ask about process FMEAs because if you manufacture(not just design) software and claim to use FMEAs then you *must* have a process FMEA and a Control Plan. My guess is that you didn't even know that there was such a thing as a "process FMEA" as opposed to a "design FMEA" before I brought it up. Otherwise you would have some sort of a reasonable answer, even if it was wrong.

>Why don't you just pitch your tool.

I'm not interested in pitching our tool. I'm interested in finding out if you use Excel for your FMEA's. Because Excel does not scale well at all for even mildly complex products. You simply cannot claim that your enterprise effectively utilizes FMEAs if Excel is your tool of choice. Unless maybe you are in the pet rock business. And every purveyor of a decent FMEA tool knows this.

And since you haven't answered the question, its not hard to assume that your department uses Excel.

Dave
Sunday, July 04, 2004

Here is a good question to see what you really know about FMEAs: Do you use RPN's and if so why? Or why not?

Dave
Sunday, July 04, 2004

Let's start at an even more basic level: what is an RPN?

Dave
Sunday, July 04, 2004

FMEA? Excel? RPN?

I use email and so should the two of you ;-)

.
Sunday, July 04, 2004

> Otherwise you would have some sort of a
> reasonable answer, even if it was wrong.

Actually, i am not answering because i don't
appreciate your tude which is more like
"have you quit beating your wife?"

My intent was to communicate
that different worlds get very complex
and you can't go without a bug database. Ho's
attitude is one that says his experience allows him the
priviledge of declairing absolutes about how things
should be, when i am sure he has never been
in particular product spaces that are much different
than the consumer software only market place.

Cisco making a router, for example, is manufacturing,
and software is part of what is being manufactured.

son of parnas
Sunday, July 04, 2004

Translation: SOP uses Excel and doesn't know what a RPN is.

me
Sunday, July 04, 2004

Wow, there's a misleading title if I ever saw one.

What he actually said was: since his program had fewer than 10 bugs at any given time, he (and the one other guy in his company) didn't need any bug tracking software.

How'd you get from that to "PG says we don't need bug tracking software"?  (Do you get paid for click-throughs?)

That's slashdot-worthy spin.  Whoa.

anonymous coward
Sunday, July 04, 2004

Getting back to the actual topic of this thread...

I would vote that all software development can benefit from using a bug tracking system (as opposed to "needs").  And this isn't just because we offer one for sale (actually, we started selling it because folks liked our internal project tracking program and wanted to buy it)...

Sure, if you only have a couple of products, a small number of developers and strive for zero defects, you will likely be able to keep your bug list in your head.  No tracking required - something that makes developers dance for joy!  If you have a bad day or a customer calling in with a "small" list of defects/wishes, you might have to jot down your list on a piece of paper (or even in Excel), but once you have dealt with the list, you can throw it away.

Except that you then have absolutely no way of determining the difference between one version or another (in terms of bugs/features).  Assuming that your customers all use the same version (web application), that they have no control over upgrades, and that you don't care whether your customers will notice if you have to revert to a previous version this isn't necessarily a big deal. 

But:
1) You will have no idea how long a particular feature took to implement (human memory is notoriously unreliable) - so you aren't learning from your experiences.

2) If a bug, or variation of a bug has come up several times before, you won't know the precise details, if you even know or remember that it has occurred before.  Maybe that bug is actually a symptom of something else (sloppy development cycle...)

3) If a client (or several clients) has asked for a particular feature more than once, but you have declined to provide it, in some cases it may be important to know when said requests took place.  Or just handy to have a way of prioritizing features (with only a handful of items on the agenda at once, future desirables tend to get lost without a formal system, and are very difficult to search through)

4) If someone wants to know which features/bugs occurred with which version, you can't tell them. (Eg  Investors often want to see what you have accomplished, if you live in Canada and qualify for research and development credits this is information you need to know)

5) An accurate assessment of the risk of implementing new features or bug fixes isn't really possible unless you implement said fixes one or two at a time.  In which case, unless your software is very straightforward, you are either burning a ton of unnecessary time on duplicate testing, or you are not testing adequately.  Given the general opinion of web apps - perhaps the second is more common.

Unfortunately, I am hopelessly biased.  But then, so is Paul Graham. 

Phibian
Monday, July 05, 2004

Holy Hijacked Threads batman!

Dave and son of parnas, you guys should really get a room or something.

The title does seem a bit hyped, but if someone says "we don't need bug tracking software since our list of bugs is so small" then its not much of a jump to "we are doing something right, and other people are doing something wrong".

Not to sound too lofty, but I think it is a good idea to use the Socractic method to learn new things. In this case we have two opposite camps (one must always use bug tracking software versus using a certian language/approach one doesn't need bug tracking software). Hopefully, by comparing and contrasting the two we can glean new ideas to help us.

There was mention of a Zero Bug Methodology. Are there any links to more on this topic? I did a google search but there was too much background noise to find a specific approach by that name.

In the end, I think it is so easy to set up/use software like Fogz bugtracking system that anything but the most trivial program would benifit from it.

Jon Kenoyer
Monday, July 05, 2004

Jon - for the Zero bug thingy, try:
"The Joel Test"- see no. 5
http://www.joelonsoftware.com/articles/fog0000000043.html 
or
from Dexterity Software "Zero Defect Software Development" http://www.dexterity.com/articles/zero-defect-software-development.htm)

a cynic writes...
Monday, July 05, 2004

Good counterpoint Phibian.

the only thing Phil graham ever did was write a behemouth in lisp that was tossed out as being unmaintainable the first time anyone went to make changes to it. I hardly think he is someone any one should be listening to and his attitude about bug tracking merely confirms that he is no developer.

Stan Dangrass
Monday, July 05, 2004

Stan, what do you know about Paul Graham's product?

He produced something with a number of unconventional technical choices -- a team of only three people, web release vs. desktop release, languages like lisp, perl, etc chosen for power rather than political reasons.

(Viaweb eventually hired something like 60 people in order to get bought, but put them on different projects, to reduce communication overhead.)

Google's cofounder revealed here:
http://murl.microsoft.com/LectureDetails.asp?895
that other search engine companies refused to buy/license Google's technology! Because it "search quality wasn't competitively important." So this company has a track record of not knowing what to do with holy grails and geese laying golden eggs.

Tayssir John Gabbour
Monday, July 05, 2004

"So this company has a track record of not knowing what to do with holy grails and geese laying golden eggs."

I'm referring to Yahoo here.

Tayssir John Gabbour
Monday, July 05, 2004

"the only thing Phil graham ever did was write a behemouth in lisp that was tossed out as being unmaintainable the first time anyone went to make changes to it."

(Assuming Stan's not writing fiction here to "prove" his opinion, a la an Ayn Rand novel.  ((Does Stan have references?))  Stan, that's and awful lot of personal vilification from somebody whose name doesn't turn up in Citeseer or on the list of Turning Award nominees.  Bravo on the bravado.)

A reasonable interpretation of such events could be that the people who tried to maintain it didn't know LISP and didn't want to learn it.  Often it's easier to rewrite something than to bend your mind to fit the pattern of another programmer, and even when not finally easier, it's the route many people choose.  Of course it might have just been shit too.  *We'll* never know.

Or perhaps the Yahoo inheritors simply weren't *smart* enough to grok it.  Shakespeare and any web application alike can be produced randomly by an infinite number of monkeys over an infinite period of time.  Using Java or PHP apparently reduces either the time period or the number of monkeys.  Slightly.

Tally Ho
Monday, July 05, 2004

Perhaps if they'd had a bug list, even if those bugs were fixed in 5 minutes each, they'd have been able to learn from it and not have to throw the whole thing out.

Of course, since the program was always perfect, it probably didn't need source control either.

mb
Monday, July 05, 2004

(first 'they' in sentence above = Paul Graham et al; second 'they' = Yahoo)

mb
Monday, July 05, 2004

Yeah, this 1-line or so comment of Paul's shouldn't be taken that seriously... it's not like he wrote a long screed about it.

The problem comes in when someone says you ALWAYS need bugtracking, or NEVER need bugtracking. Uh-oh.

Tayssir John Gabbour
Monday, July 05, 2004

exactly.

Tally Ho
Monday, July 05, 2004

Wow, I haven't seen anyone who can even spell FMEA in several years, let alone derail a thread with it. FMEA = Failure Modes and Effect Analysis. Its a tool to help you analyse risk, mostly used in manufacturing. RPN = risk priority number  ( = severity * probablity of occurance * likelihood of detection).

Reasonably good when you are evaluating the potential bugs when making a door assembly, but not so easy to use when deciding the potentiality of some software. Especially when the PHB pulls the severity and probability out of his fanny perpendicular (and makes you use them). Like any good pareto chart user, you tackle the $1,000,000 bugs before you tackle the $100 bugs. Although every PHB wanted the $100 bugs fixed first because those were the ones that showed up on his desk.

In past companies, the organized ones I have worked for, we usually use an excel spreadsheet for keeping track of bugs, feature requests and what has been deferred (like, the hardware hasn't arrived yet). It ended up being a punchlist when you sit down with the customer to hammer out what are the priorities in the next week or 2.

PS. to derail the derail, I would like to state that Paul's best essays are:
What You Can't Say. and
Why Nerds are Unpopular.

Peter
Tuesday, July 06, 2004

*  Recent Topics

*  Fog Creek Home