Fog Creek Software
Discussion Board

how do you do QA?

Yes, I know how. But, are you using an automated testing tool? how much work is having an automated testing tool? do you have dedicated resources to do qa?

this is probably the biggest weakness at my the company that I presently work for.  We essentially unit test, there maybe some system test/uat/qa, but overall i feel like we do a monthly release (just bug fixes, no new enhancements) that is not fully tested.

the best scenario that i had was a group where every 1.5 weeks was a build (no, not every day). at the end of it, we would spend easily .5 - 2.5 days testing. and if necessary, have to work saturday. We made it a point so that it didn't happen. Even though it wasn't automated, we would all have test scripts for different areas of the system.
When finally we were ready for a release, we would spend 1 week doing this "System integration test" before going to UAT.

Any thoughts? why are so many short cuts taken? and how do you convince mgmt that more should be done?

Friday, July 16, 2004

One thing that I've had some success with...

Prior to me coming aboard (Dec), we had very few formal requirements docs, few meetings with clients, no automated testing procedures, and little collaboration.

On one of the *many* active projects we have, I was assigned to test a few modules.  It was annoying, time intensive (4+ hours each time), and I was prone to missing something and/or being lazy and skipping something.

So I went through and spent 3 days building testing scripts for my portion.  Now, the next time the lead developer told me to test, I started the scripts while he were still at my desk, stalled the him for a 2-3 minutes, and then told and showed him the results.

His expression was priceless....  I thought he had wet himself.

*THIS* is how you convince people to create automated testing procedures.

Friday, July 16, 2004

The whole modern 'quality' movement is about prevention over cure. 

The problem is management (and american culture) that is addicted to adrenaline and fire fighting.  We are too busy fire fighting (cure) to ever have time to do fire prevention.

What's ironic is that fire fighting without building support infrastructure just generates more fires.  But Heroics is big and recognized.  (Notice on the project:  The guys who have to work super massive overtime to make the project date are heros - the ones who did responsible scheduling and get to leave at 5:00 PM every night are slackers.)

I'm a prevention guy:

Testing can be prevention - in that it prevents the customers from finding your bugs, and it prevents all sorts of debugging pains. 

Of course, the best time to do testing is -early- in the lifecycle, so start with unit tests. 

Basically, a function is a combination of inputs, outputs, and transformations.  An awkward combination of functions, or one that is vaguely specified, is going to be a problem.  So you need to think about the API, it's use, and in I/O/Transforms early.  So I suggest test-driven development as a design activity.

That this gives you a regession test suite is just a real cool side benefit. 

I would add a big of a system test phase (can still be done by coders), and then finally have a really good customer acceptance test phase and you start to be in great shape.  (If you are writing shrink-wrapped software, I would make testing team that is a simulant of a customer. Probably a trainer, a customer service rep, the PM, and a full-time tester.  This group needs to define acceptance tests for specs before you begin coding.)

Just a handful of thoughts about testing.  To get back to your first question: I'm very careful about calling testing QA, because crappy specs, skipped design, and code-like-hell coding is a recipe for disadter, and calling testing 'QA' sort of implies that testing can fix this problem.  It won't.  At that point, the best testing can do is to tell you how bad the software is.  You can then test/fix/re-test until you decide the software is good enough to ship.  It will still be bad, just not -as- bad.

On the other hand, if you are doing things well - iterative development, good specs, good design, responsible coding, etc, then testing can be really cool in that it can give visibility ("How far are we from done?") and confidence ("Yes, I think we really are done.")

Does that help answer your question?

Regards, (Formerly Matt H.)
Friday, July 16, 2004

>I would add a big of a system test phase

Should be "Bit of a system test phase", not "big"

ugh.  I made a few other typos, but I hope that's the only one that can be mis-read to change the intent of the post. :-( (Formerly Matt H.)
Friday, July 16, 2004

What's QA?

Oh, you mean where you send your beta version out to customers and listen for screams?

Friday, July 16, 2004

i wouldn't call them beta versions. they are all the bug fixes for the month. my problem is that even though i might fix one thing, what's not to say i broke something else?

it's a large and complicated app, so it is possible.

yes, in a way we wait for screams, hence having 4 resources devoted to the help desk. in the same token, we only have 7 developers and 1 business analyst.

Friday, July 16, 2004

qa is short for quality assurance

Friday, July 16, 2004

What's a beta version... we just deliver on the promised date.  (Which is usually today.)

Friday, July 16, 2004

QA is "Quota Appropriation".

Consultant to Management: Oh! You are a <insert whatever> outfit? Do you do UT? Do you conduct RA? What about QA? Give me money.

Management to Finance: We have to implement the English alphabet. Give me more money.

Finance to Marketing: The English language needs funding. Get me more money.

Marketing to Production: Linguistics is expensive. Make sure the product cures the common-cold, is as easy to use as Pantene Pro-V shampoo, and ensure that it costs only as much. Finance wants more money.

Production to Consultant: We do not have money. What do we do?

Consultant to Management: ~


Friday, July 16, 2004

I've done it KC's way. I created automated scripts a few years  ago. Their care and maintenance is fairly minimal requiring just a few changes with each new release. I run them in minutes and can ferret out many bugs before shipment.

But y'know what? It's just not recognized. It's simply taken for granted.

On the other hand as Formerly Matt H. says, management is addicted to fire fighting. There was one project last year that was done in a panic and someone shipped a special out to a customer without doing any QA. Bypassed it completely. You know what happened.

In order to put out the resultant fire we worked a few hours on a weekend and sent a good (meaning tested) version to the customer before he got back in Monday morning. A couple of us got mega-Kudos for that from top management as well as the customer.

We were heros, which is a wonderful thing, however nobody learned the lesson that sending out untested crap is what caused the fire in the first place. Nobody. It'll happen again and I'll be the hero again simply for doing what should have been done prior to shipment.

Friday, July 16, 2004

Our QA creates a matrix from the contractual requirements (all the "shall's") and testing just consists of checking that each are satisfied. If they happen to come across a different bug, they'll put it in the database, but they don't check for non-requirements stuff specifically.

Friday, July 16, 2004

>I wouldn't call them beta versions. they are all
>the bug fixes for the month. my problem is that
>even though i might fix one thing, what's not to
>say i broke something else?
>it's a large and complicated app, so it is possible.

You want a regression test suite that you can just run again and again everytime you make a change, that goes through all your code and re-tests that you didn't break anything accidentally, and you didn't break the way other units expect your unit to behave (integration bugs.)

You want automated unit tests.  We should talk ... (Formerly Matt H.)
Friday, July 16, 2004

yes, but more. i want functional testing and regression testing.

now to add the kicker ...

it's an MS Access app! yes, it's true. i'm also trying to push a conversion to .NET. But, the managers are scared. Part of the problem is that we spend so much time bug fixing/support calls that we don't have the time to move forward.

so anyone know of automated testing tools for MS Access apps? my thought is that even if it was manual test scripts that at least it would get done. we would then cut down on the number of bugs, hence have more time for .NET.

Friday, July 16, 2004

To amplify what Matt H says, to some people QA is documenting the process that you use to create software, as crappy as that may be.

And possibly, add a testing phase where the goal is to document (but not necessarily fix) everything that might possibly be a bug. At worst it's a CYA activity.

Friday, July 16, 2004

Arrange your automated unit tests in a suite and you have a full regression test suite!

You say it's access.  Is that VBA or just a "true" database?  Or VB?

I'm pretty sure there are tools to help you.  If not, rolling your own is not that hard.  Tell us more about your project ... (Formerly Matt H.)
Friday, July 16, 2004

To anyone that's interested:

I'm vaguely working on an article for a web-zine about "QA, falsely so called."  If you are interested in reviewing, giving input, etc, drop me a line. (Formerly Matt H.)
Friday, July 16, 2004

AutomatedQA ( is a good tool for planned, incremental, and sustainable automation testing.

I spent a year building a library for WinRunner that would make it more robust (HA!), flexible, and give it a high-level interface. Don't go there. Mercury has not taken that tool past static HTML days. It does not even have a DOM.

Not only did management's lip service to automated testing dry up quickly, but the existing QA department viewed it as a job threat.

As long as an organization's goal does not include knowing what works and does not work within a few hours of a build, QA is a hard row to hoe.


Richard C Haven
Friday, July 16, 2004

thanks for the input richard.

anyways, in all reality it's a VBA app. MS Access is the front-end and used for the reports. The business logic is really VBA functions. It's SQL server on the backend.

Friday, July 16, 2004

VBA Unit:

I haven't tried it, but xUnit-type testing frameworks are usually very good. (Formerly Matt H.)
Friday, July 16, 2004

thanks! I'm going to test it out. If it works, then i will do what is suggested above and just write some scripts and then show it to management.

it's a step in the right direction!

Friday, July 16, 2004


"Our QA creates a matrix from the contractual requirements (all the "shall's") and testing just consists of checking that each are satisfied."

I don't know about anyone else, but this set the alarm bells ringing in my head. Are you making a big assumption that the stuff in the contract is correct? What about the code you have had to generate to plug the inevitable gaps in the contract req. spec?

Ian H.
Friday, July 16, 2004

I've been reading Steve McConnell's new "Code Complete 2", and he has some very interesting hard data concerning QA activities.  I was fascinated to learn that code reviews (and especially formal inspections) catch substantially more bugs than testing.  I don't know much more about the original data than I read in Code Complete, but I thought it was rather eye-opening.

Of course, how do I do code reviews when I'm the only developer?  (Short of leaving the code alone for a week and then reading through it later.)

Jesse Smith
Monday, July 19, 2004

*  Recent Topics

*  Fog Creek Home