Fog Creek Software
Discussion Board




They Just Fired the QA Guy

I work for a small company that does natural language processing for commercial and government projects. We've got around sixty employees. Roughly 30% of the employees are software developers. Another 30% of the employees (like me) are in the professional services department, doing custom implementations and configurations for clients. The other 40% of the employees are in management and HR.

We had one QA guy. Yesterday he was laid off. There are (to my knowledge) no plans to replace him. The rumor I've heard from someone in the engineering department is that programmers will now be required to do all of their own testing.

I'm worried.

Our software is very young. And, since we're involved in a a very science-y field (NLP), our software is constantly going through R&D to improve its performace and capabilities. How can we possibly do R&D without QA? Are they crazy?

And there are, of course, the normal bugs in software that crop up and might not be noticed by a single developer. Many of these bugs won't be noticed until they wind up in the hands of the users. Essentially, this decision will turn my department into testers (since we're the primary users of the software).

But even more than the problem of having to fix the natural bugs that occur in software or the problem of having to manage unpredictable R&D output or even the problem of turning my team into software testers, I'm especially worried about what this might mean for the financial status quo of the company. What kids of software company gets rid of QA?

Yikes!!
Friday, August 01, 2003

Running a development team without QA specialists is not *always* as crazy as it sounds.

Here at SourceGear, we recently made a switch wherein QA/testing is now the responsibility of the developers.  But this doesn't mean we're taking testing less seriously.  In fact, we made this change because we consider testing our code to be just as important as writing our code.  Our programmers don't necessarily *like* the idea of doing their own QA, but that doesn't mean that we're taking QA less seriously.

Does your company seem to have a plan for how QA is going to be taken seriously once it has been absorbed into the developers' job description?  Or did they simply lay off the QA guy on the assumption that testing will somehow happen anyway?  The difference is huge.

Eric W. Sink
Friday, August 01, 2003

The idea of independent testing is that developers make assumptions and subconsciously avoid bugs they know about; some things may also simply not occur to them. Testers worth paying for are *professionals* who know how to break things. A good QA person will bang on every button, menu item, shut down the app in the middle of processing, etc, etc, etc. They will almost have a sadistic glee in making software fail.

The only way I can imagine credibly using devs as QA is if you have more than one development team, and each team tests the other teams' software.

Actually, this *might* work - give an award for the team that finds the most bugs each month. Then you get double-dipping: the devs will work harder to find and fix their own bugs (so the QA dev team doesn't get them), and in QA mode they'll work harder to find bugs. Only downside is then you might get into arguments over whether something is a bug or not... :-/

Philo

Philo
Friday, August 01, 2003

My experience:  developers test code to make sure it works, QA tests code to make it NOT work.  The two are mutually exclusive, and I never trust developers to test code, period.

GML
Friday, August 01, 2003

While I believe that all programmers should be unit testing the code they write, I don't believe programmers should be doing the QA testing.

In theory, I like the compromise Philo suggested. However, in order for this to work both teams have to be ready for testing at the same time.

One Programmer's Opinion
Friday, August 01, 2003

I concede that there are very good reasons for having testers be distinct from programmers.

And yet I claim that it *is* possible to find programmers who have the ability to be good testers.  It is much more comfortable not to even consider that possibility, since we don't really want it to be true.  The notion that programmers are fundamentally defective people, incapable of good testing, is a nice place to hide from a body of work we simply are not interested in doing.

The choice is often one which is made according to context.  Small companies usually have fewer specialists, choosing instead to fill their rosters with multi-talented people who can wear multiple hats.  Larger companies tend to employ people with tighter, more specialized job descriptions.

Eric W. Sink
Friday, August 01, 2003

Anybody thinking about costs here???
A QA person would be a lot cheaper than a programmer, thus making programmers do QA themselves is a waste of resources.

Mr Curiousity
Friday, August 01, 2003

Although, if the software is really complex then both a programmer and a QA person would need to have a lot of training and may be MS or PhD. Then there are no cost savings when a dedicated QA person is hired since the salary differential is insignificant.

Mr Curiousity
Friday, August 01, 2003

Let me answer the actual questions asked...


"How can we possibly do R&D without QA?"

You can still to R&D, but less of it - since the R&D folks are now spending time doing QA.


"Are they crazy?"

Probably not.  Deluded perhaps, but not likely crazy.


"What kids of software company gets rid of QA?"

One that doesn't care as much about quality as something else.  In this case, I'd guess it was a cost-savings measure.  IMHO, a short-sighted cost-savings measure that will come back to haunt them and wind-up costing more than was saved.

bpd
Friday, August 01, 2003

Costs do matter, but there again, context is critical.  If you can hire testers a lot cheaper than programmers, you should seriously consider having them be distinct. 

In our situation, that doesn't work.  Our major product is a version control system, and as much as possible, our testing is done by creating automated test programs.  Our testers need the same skills as our programmers.

There are still good arguments for separation, but in our case, salary differential is not one of them.  YMMV.

Eric W. Sink
Friday, August 01, 2003

BTW, my (completely uninformed) guess on the company that did this:
1) The QA guy was not fired as a cost-cutting measure. There were other reasons. (for cause; qa guy did a "pay more or else", something like that)
2) The decision not to rehire is a penny smart, pound foolish move - "why do we need to replace him? We'll just get the coders to test?"

IOW, it's not some well-thought-out strategic move, it just happened in an elevator.

Philo

Philo
Friday, August 01, 2003

Yikes asks, "What kind of software company gets rid of QA?"

The question I am asking myself is "What kind of 60 person software company has 40% managers/HR?" That's 24 managers and HR people to keep track of 36 developers/consultants. That is a  heck of a lot of management overhead. Granted, I have never worked in a 60 person company, but it still seems like a lot of managers & HR people.

IMHO, separate QA groups are good for some kinds of software development, but not all. I have been involved in scientific/engineering software most of my career. Most of the defects I've seen dedicated QA people find are of the "I did X, Y, Z and it crashed" (critical) or "The form X doesn't conform to standard Y" (not critical) kind. They almost never find problems like "When I tried to do Z, I got the wrong result" (critical) since they don't know what the right result would look like. (Sometimes the software developers don't know this either).

[Trying to find a non-pointy-haired-boss explanation for why Yikes' company got rid of the QA guy.] Perhaps one of the many managers took a look at the types of defects that the QA guy was finding and decided that the vast majority of critical defects were found using other methods.

BTW, all the studies I've seen (or had quoted to me) indicate that a ratio of 1:2 or 1:3 QA people to developers is "industry norm". Therefore 1 QA person for approx. 18 developers was way below industry norms. Perhaps the developers in Yikes' company were already doing most of the QA.

In any case, if it were my decision, I would have laid off one of the managers :).

A. Manager
Friday, August 01, 2003

If I was the only QA guy testing 18 developers' code, I would want to be paid more too. :-)

runtime
Friday, August 01, 2003

"They almost never find problems like "When I tried to do Z, I got the wrong result" (critical) since they don't know what the right result would look like. (Sometimes the software developers don't know this either). "

A.Manager is right on the money here. I was in a similar position, and sometimes it took us weeks to figure out if particular behaviour/result was a bug, or a feature. No QA person would ever be of any help, not that we had any ... Most such discussions went  together with a heavy dose of denial when bugs were claimed to be features :-)

I even once wrote a chapter in a paper describing behaviour that was due to a bug, yet I thought I was braking new ground :-))) Luckily I caught it before the paper got submitted. 

Mr Curiousity
Friday, August 01, 2003

Eric,

>> Our major product is a version control system, and as much as possible, our testing is done by creating automated test programs.  Our testers need the same skills as our programmers. <<

Well, first of all, you know the internal logistics of your company a lot better than I do, and incidentally you're a lot more successful than I am, so take what I say with a pinch of salt. But...

IMX, there are several types of testing.

Developers tend to focus on code, working from the inside outwards. They use assertions to verify data flow and structures, use the debugger to verify code flow and data, use unit testing to verify each function, use integration testing to verify sub-systems, and do some system testing to verify functionality. As you say, many of these testing tasks can be automated.

QA, on the other hand, tend to focus on features, working from the outside inwards. They use scenarios to test real-world features, use their previous experience of the customers to test unlikely but real scenarios, use global tests to verify real-world feasible inputs, use regression tests to verify that defects stay fixed, do compatibility testing with previous releases, and generally look for quirks and rough edges. Many of these testing tasks *cannot* be automated.

Whilst I knew all of this academically, it was really brought home to me emotionally when I worked at Enron. We were a team that built energy trading software, and we worked with a separate QA team that *really* knew how to do their job. I was astonished at the number of issues caught by the QA team, but not the developers.

For example, the QA team knew that trading exhange XYZ always had a wobble between 3 and 4 pm, so they specifically tested the link between our software and that exchange at this precise time. QA knew that trader Jeff liked to bunch all of his prompt products at the top of the screen, so they explicitly tested that scenario. And so on and so forth...

Mark
---
Author of "Comprehensive VB .NET Debugging"
http://www.apress.com/book/bookDisplay.html?bID=128

Mark Pearce
Saturday, August 02, 2003

Good QA testers also have some programming ability, with which they can use to write test scripts to directly call code or run tests in an automated testing tool like WinRunner.

Some PHBs think of QA people and technical writers as "overhead", and have programmers doing QA and writing long documents and manuals.  They are too stupid to realize that it is the *activity* of QA and documentation that is the overhead, not the people who do it.  Somebody has to do it, so it's better to have dedicated professionals doing it instead of eating up programmers' time and distracting them from their primary activity of developing the product.

With such a high overhead of management in a small company, and bone-headed decisions like this, this company will be filing for bankruptcy within the next 2-3 years.  If as much.

T. Norman
Saturday, August 02, 2003

Testers like looking for mistakes.

It's a different mindset.

Whenever I see a piece of software that needs testing I think; how can I break it.

When I write a little thing of my own, I think, how cool!

That's why the jobs should be separate.

Stephen Jones
Saturday, August 02, 2003

Stephen,

Yes, I think you have it right. When coding, my assumptions tend to become fixed early, and therefore are difficult to change. A QA person usually tests my code using a different set of assumptions. As Philo says, a developer can do effective QA, but not on her own code.

One study showed that 50% of defects that were clearly shown on report printouts were not detected by the developer, whereas a separate tester missed only 10% of these defects.

Mark

Mark Pearce
Saturday, August 02, 2003

To answer if programmers can do their own testing:

Yes, if you're happy with it being done badly


I think it should be obvious

(a) A person who specializes in an activity (in this case QA) is likely to be better (and getting increasingly better over time) than a person who doesn't

(b) A person who is measured on, responsible for, an activity (QA) is likely to do better than a person who is measured on and responsible for something else (like producing code etc)

(c) An 2nd set of independent eye looking over something, will always spot things the initial programmer didn't spot.

(d) Programmers, of necessity, and correctly, understand things in programming terms, which usually even a different type of understanding to a broader overview, you might get from a QA person.

If you follow the no QA argument to it's logical extreme, why have anybody who checks other people's work

- Hey let's get rid of company auditors. CFOs can just add the figures up twice to double check them

- Let's get rid of nuclear safety inspectors.  The guy running the plant can just double check the readings on the dials.

- Don't ever ask for a second medical opinion. Just ask one doctor, wait 5 minutes, and ask him again.

- etc etc.

S. Tanna
Saturday, August 02, 2003

Mark Pearce does a nice job pointing us some more of the differences between QA and coding.

Truth be told, our current structure here at SourceGear would have to be regarded as somewhat of an experiment.  I assume the day will come when our company grows and we once again separate QA and coding into distinct groups. 

But when that day comes, I believe our current structure will have been a good step toward that eventual goal.  Our developers today will become managers of newly formed teams, as they will have gained invaluable experience by taking full responsibility for the shipping quality of the code, not just the initial creation of same.

And when we do grow and hire QA specialists, I will expect them to have the same skills and background as if I were seeking programmers, and I'll expect to pay them comparable salaries.

Eric W. Sink
Saturday, August 02, 2003

Have sales replace the QA for UI integration testing , do your own API testing

Daniel Shchyokin
Saturday, August 02, 2003

Eric your 2nd and 3rd paragraphs seems to contradict each other about organization.

FWIW, I think testers should be in the same group as developers. I also think having testers doesn't absolve developers of responsibility to test their work and produce as good quality as possible.  Lastly I agree that testers should be comparable in salary, responsibility, skills etc., to developers - it's just they have different responsibilities and skills.

S. Tanna
Saturday, August 02, 2003

>"And when we do grow and hire QA specialists, I will expect them to have the same skills and background as if I were seeking programmers, and I'll expect to pay them comparable salaries."

Very wishful thinking.  Your company won't be able to grow to that point if large chunks of programmer time is being spent on QA and there is no dedicated QA person in the company to uphold the quality of the software. Big companies with established market share can afford to make mistakes like that, but not small growing companies.

Your one chance of producing a decent product without any QA staff is to release alpha or beta versions to large numbers of potential users and provide a mechanism for them to report bugs.

T. Norman
Saturday, August 02, 2003

Wishful thinking?  Yeah, maybe so.  :-)

I admit I'm crazy, but I also claim that I'm not as crazy as I look.

Sometimes I just get tired of facing the same old problems.  Sometimes what I really want are *new* problems.

Eric W. Sink
Saturday, August 02, 2003

Since Eric's product is a version control system, by which I assume he means a software version control system, like Perforce, it's kind of a special case. A development environment, like MSVC or Metrowerks, would be similar. Your target end users ARE programmers, so using non-programmers as QA would mean that your testers would be less likely to recognize buggy behavior.

That kind of product is indeed best tested by programmers, but it would be better if they were different programmers than those developing that particular product. The idea proposed earlier in this thread of having multiple product teams that each tested the other's product would be more likely to uncover hidden assumptions, etc.

Teri Pettit
Sunday, August 03, 2003

Yes, it is best to have programmers testing a product for programmers. But *good* QA people are also programmers, even if they haven't done something nearly as complex as enterprise level application development or writing compilers.

T. Norman
Sunday, August 03, 2003

As long as only one big engineering change is being made at a time, so it's easy to see where any problems come from, it shouldn't be particularly fatal.

If I lived anywhere near the midwest, I'd like to work there.  I also have certain ideas concerning programs and defects.

rohan j.s.
Sunday, August 03, 2003

The main problem I'd have with QA by devs is that it is very hard for the devs to get rid of the "implicits" in their software model when they put on their QA hat. This is the main reason why that software you thought was bugfree fails at the first instance your sales person lays his hands on the mouse.
As for the original poster, I think R&D programmers would be the worst possible kind to do QA. The nature of their mindset is to take higly risky excursions into new stuff with a high chance of failiure bit with a good payoff in the few cases it succeeds. Decent QA is not exactly going to be their idea of fun, and you will just get camouflage and compliance behaviour out of them.

Just me (Sir to you)
Monday, August 04, 2003

This sure isn't how Microsoft runs things.....

Jason McCullough
Monday, August 04, 2003

Metrowerks has its own SQA organization that's under different management from engineering.  They are involved in reviews from marketing requirements forward, and the SQA manager has sign-off authority on the final products that we ship.  Sometimes it has been difficult managing the feedback from the test organization, but it's been invaluable for improving the quality of our products.

Ben Combee
Monday, August 04, 2003

*  Recent Topics

*  Fog Creek Home