Fog Creek Software
Discussion Board




Delivery to QA

My company's current policy is that no product goes to QA until a fully packaged install is available, the tought being that we need to QA the install as well as the product (QA is involved in product specification, but then doesn't see anything until the end of development)

I have several problems with this method:
1)  It delays the start of QA until the very end of development which increases the cost of finding/fixing bugs.
2)  You then only have the development team's appraisal of the state of development.
3)  The urge to push a product out the door increases once someone (mangement) believes that a packaged install is available.
4)  Time overruns SEVERLY impact the QA schedule.  If QA has 3 weeks to test a product and development runs over by 1 week, you have either shortened your QA time, or have pushed back other products to get QA the full time they need.  The cost of this could be amortized over the development cycle if intermediate builds were available.

Are these valid judgements on my part, or do most QA departments not accept anything but a packaged install?

wyoming johnson
Thursday, February 21, 2002


If you use the staged delivery technique, then you can release a Full, working product over a series of builds.  Each Build has a certain (growing) set of functionality that can be tested by QA.  Each build is ship-worthy (allthough the first few builds are severaly crippled.)

Then, if you are running late, you can just ship the n minus 1th build.  That way, you can effectively cut features yet ship on time without increasing resources.  (Trying to go for all 3 out of 3 is well, honestly, probably not going to happen.)

Then again, staged delivery is cool in theory, but in practice, I've found it requires a disciplined approach that is often rejected for "code like hell" as the code complete date approaches.

Just my $0.02,

Matthew Heusser
Thursday, February 21, 2002

We have a "daily setup" that goes along with the daily build, so when there are added features to be tested by QA (actually most of the testing is done by the documentation group because for writing the documentation they go through all the software again and again anyway) they install the daily setup and start from there. Normally they will take the current beta version, though. We have a fixed release schedule of four weeks, where we branch a freezed version from our code base. A setup is created directly after the freeze for that, too. Of course, keeping the setup up to date all the time involves one developer (out of 20) working on nothing else.

Jutta Jordans
Friday, February 22, 2002

I've worked on several huge projects where it would have been simply impossible to wait to the end to send things to QA. In these cases, the setup team is working at the same time as the main dev team, and their first product is a "setup" that is just a list of instructions: copy these files from the dev server to this folder, then run this reg file, then use this executable. The point is to get *something* that can be used by the QA folks (who, we presume, are at least more experienced with computers than the general public) even though it's not the final setup.

The thought of waiting to the end to do QA strikes me as insane. If you're not doing QA as you go along, you're not really doing daily builds, because part of daily builds is that they're supposed to work, and how would you know if they didn't?

Mike Gunderloy
Friday, February 22, 2002

> The thought of waiting to the end to do QA strikes me as insane.

We have different kinds of QA, for different kinds of build and release.

Developer QA (unit test) is what you do to your new stuff.

After that, it goes to the ITG (Internal Test Group) who work with and support developers. They'll test anything you ask them to: you tell 'em how to test (including how to install, if necessary) and they will test it, no problem.

After ITG, a build might take any of several routes: 1) nothing (it's not a releasable build); 2) promotion to the VCS mainstream (used by other developers); 3) a beta release (sent to a beta customer for beta testing); 4) an engineering release (a version that is accepted by an existing customer, but which hasn't been through our 'STG' and which isn't sold by the salepeople); 5) a release candidate (which is sent to STG, the System Test Group, who are much more arm's-length than the ITG and who will only test a [installable] release candidate; 6) a release (a version which passed STG, and is available for purchase from the corporate salespeople).

In brief, STG would want something installable, and ITG won't insist on that; they're two different kinds of QA, with different purposes.

Christopher Wells
Friday, February 22, 2002

*  Recent Topics

*  Fog Creek Home