Fog Creek Software
Discussion Board




Automatic Error Reporting

We have implemented a form of automatic error reporting into the latest version of our program (using the BugzScout.dll for FogBugz - but that is not important).

We hope it will help us to find bugs in our system that our testing doesn't catch (we know users love to do things completely differently to even the most creative tester) and so I was just wondering how have other companies who have implemented this sort of functionality found it.

Have you found it useful to your development/support teams? Did it help you to track down any bugs that you would never have thought to test for? Did it annoy the hell out of your users? (Ours is VERY unintrusive, it basically pops up the error screen and says "Send" or "Don't Send" with room for a "'What were you doing?' edit box")

It may seem silly to ask this after I have implemented it, but after hearing Joel say how it improved CityDesk and how Microsoft says it even helps Windows we decided it would be useful for us. But I just thought I would like to get some more comments on this sort of thing.

Chris
Monday, June 30, 2003

Keep an eye out for my feature article about this in the next issue of STQE magazine!

Joel Spolsky
Monday, June 30, 2003

Any links for that Joel? Will it be a free article (please)?

Chris
Monday, June 30, 2003

It is very useful because you get to know about errors you wouldn't otherwise hear about. And if you include enough information (stack trace etc) about the error, it might just be enough to fix it.

One thing is that many users are loathe to enter any information about what they were doing at the time the error occurred, let alone write a reproduceable bug report!

Frederik Slijkerman
Tuesday, July 01, 2003

I have been rolling my own error reporting in ASP code. My error handling code emails me a copy of the error and adds to it a log file.

The real advantage is that you can see exactly what caused the bug and you know how often it happens.

Matthew Lock
Tuesday, July 01, 2003

I'll second Matthew's method.  I inherited several hundred thousand lines of ASP code, and adding a custom error page that emails me the error details has made it quite easy to catch existing bugs in code.

I use Outlook to organize the errors as they come in, so I've got a live task list, of sorts, that I can fix on an as-needed/as-available basis.  It's worked out nicely for me at review time, too:  "I fixed 17,000 bugs this year," etc.

If I was more motivated, I'd script something to integrate that into the Tasks feature of Outlook, but that would really cut into my slacking time.....

Jeff MacDonald
Tuesday, July 01, 2003

We've been using automated error reporting for a while and find it helpful in keeping up with the bugs.

What I'd like to know very often that our form doesn't report right now is how severe the error is. I've been thinking about adding a drop down along the lines of:

How often have you experienced this error?
- Every time i try this operation
- Very often
- Occasionally
- Never seen it before

Having that info would help us prioritize our bugfixing.

Big B
Tuesday, July 01, 2003

Yes we do have the "What were you doing" field as optional.

And Big B. we are hoping the "How many times this has been reported" feature of FogBugz will help us do what you are saying without having to get the user to decide - because to a user every error "happens all the time" even if it is the first time they have had it.

We also like the fact in FogBugz that it is easy to provide an automatic answer to the user if we have already fixed it - or a tip on how they can avoid the error until we have fixed it.

We use foxpro so it is quite limited in what can be captured (basically a stack trace and error number / message) but so far from internal testing that has been enough to track down or at least narrow down where a bug is.

Chris
Tuesday, July 01, 2003

I think our users would answer such a question honestly. They're usually pretty good about saying it's not a big deal when they can try again and it works.

The reason I want to ask is not that I can't look through the log and see how many times an error has been reported, but that I don't know how many times it has happened but NOT been reported. Some users don't report an error until it has happened 5 times or is stopping them from doing their work - others report every single one they find.

Big B
Tuesday, July 01, 2003

Thats a good point you make. Maybe something for us to think about in the future.

Chris
Tuesday, July 01, 2003

*  Recent Topics

*  Fog Creek Home