|
What is the value of testing?
Please tell me what value there is in testing or having "system testers" or Quality Assurance Analysts? What value do they really bring?
JKeys
Wednesday, March 31, 2004
Cheaper to fix a problem when it exists on one CD than it is once it's been copied to 10,000 CDs.
If software engineers had to support thier own products, there would be very few bugs.
I do and we don't.
Mr. Analogy
Wednesday, March 31, 2004
Um...finding bugs is the primary value of testing.
Do you want to know how buggy your software is? Or would you rather not know?
If you want to know the former, hire a good test team. If you'd rather be blissfully unaware of your flaws, or prefer to wait for your customers to tell you, don't.
Having a dedicated test team is the first step toward producing really high quality software. I would guess most teams that develop software without giving it to some dedicated testers to gnaw on believe they have decent quality level, but are instead simply ignorant of all the problems they haven't found.
Now, I'm a professional software tester, so I'm sure this perspective isn't shared by all (or probably even most) developers, but it's true.
As a side note, at Microsoft we don't use the term Quality Assurance because it's really an inaccurate description of what the role of a test team is. Assuring quality is the role of everyone on the team (dev, PM, test), not just the role of the test team.
Mike Treit
Wednesday, March 31, 2004
Good answers. I'm a little dissapointed that a professional tester responded before more developers. The reason being, I am professional tester myself. I was a bit insulted when one of the development managers I support asked me to attend one of their development staff meetings to explain to the developers what the value is in having a tester on every project. Isn't it obvious? I've been in testing for over 7 years. I guess I just simply don't understand why developers don't want testers. I'm trying to find out where you developers are coming from so that I may address your ill conceived notions. : )
JKeys
Wednesday, March 31, 2004
You need to test in order to ship a high-quality product to the customer. And a business needs happy customers to succeed.
I'm a developer at a company without any dedicated testers. Consequently, I spent lots of my time testing stuff. Writing code is more fun, but I'd much rather uncover problems earlier rather than later.
Julian
Thursday, April 1, 2004
Well, here is an answer from a developer:
We don't use testers because the only people capable of testing our product properly are developers themselves, and developers get bored very quickly if you ask them to just test and test ...
Now, our product is perhaps unusual -- it is a specialized kind of compiler. We do run it through a set of 5000 or so regression tests every night. Unfortunately, that's no guarantee of quality: there is always some weird code construct that we don't compile correctly. But no sane person/tester will ever write code that way.
So the tester could just pull random stuff off the net, try to compile it, and report that it "didn't work", which is useless. For a useful bug report, the tester must pick that random stuff apart, understand it, understand why it didn't work, and create an automated regression test out of it.
By which time he pretty much became a developer, is bored to death with testing, and we give him something more interesting to do.
Employed Russian
Thursday, April 1, 2004
I am often surprised at how, having faced a day of hard slogging rearranging some fancy dialog or something, testers return the next day to point out the blatantly obvious stuff like "this field is misnamed" etc.
The *worst* person to test code is the person who wrote it imho.
i like i
Thursday, April 1, 2004
> Good answers. I'm a little dissapointed that a professional tester responded before more developers. The reason being, I am professional tester myself.
JKeys, if you wanted to promote your own role, why did you pretend you were asking an unbiased question at the start? Sounds a bit Machiavelian to me. Maybe that's why development groups aren't keen on working with you.
On the general question, the presence of distinct testing groups has its place in professional development, but it can also be part of a management culture that sees development as a commodity.
In that culture, cheap inexperienced and talentless developers will be chosen, then given poorly planned schedules. To handle the resulting mess, your average Dumb Manager will turn to Testing, sometimes to the extent of giving them higher authority, which causes conflict in the development team.
This of course sends all the good developers away, so the system stabilises with poor developers and a dependency on extensive testing.
JM
Thursday, April 1, 2004
It's actually not obvious. Do you write code? Some companies turn testing into a code-writing position. Do you know how to automate tasks so you're not a bottleneck?
While it could be a timesuck meeting, it's also your chance to tell them what you offer. Maybe you could explain how you can save them from painful redesign because catching bugs earlier cuts them off before more multiply on that spot. Less stress for them than when an important customer finds bugs. In this meeting, your role is about making developers happy. If you were meeting execs, your role becomes about making them happy.
Offtopic, it raises a red flag that you envisioned a scenario where bumbling developers would make fools of themselves as you publicly chastised them... There are myriad reasons why testers are necessary. One is that proving software correct is not perfectly effective. Another is that performance is often not captured by static analysis.
Tayssir John Gabbour
Thursday, April 1, 2004
I'm a developer and I like testers because I hate to:
1) Draft up complex test procedures
2) Continuously run said test procedures
3) Maintain different operating systems/configurations and check my code against all of them
4) Make sure my code change didn't break one out of a million little things which were working before in unrelated modules
5) Test more than what it takes to say "It compiles; ship it!"
That's why we pay the testers.
MR
Thursday, April 1, 2004
" I'm a little dissapointed that a professional tester responded before more developers"
Ummmm... I was the first to respond and I'm a developer. I develop, test and (most importantly) support my software.
That last task is a feedback loop. It provides a very strong incentive for good testing, which provides an incentive for better development.
BTW, if you're looking for JUSTIFICATION for testing, check out some of Steve McConnel's books. I think in Code Complete he talks about cost of a bug increasing by an order of magnitude in each phase:
1. Development
2. Beta Testing
3. CD Duplication
4. Customer
So, it's roughly 100x as expensive to fix the bug after a customer gets it (i think this is convervative, btw).
In general, I think programmers are NOT the best people to test thier own software. IT's like proofing your own writing.
However, by having the programmer ALSO support the program, you encourage better testing.
Mr. Analogy
Thursday, April 1, 2004
Had another idea: have the testers provide tech support.
Then you can hire someone who may be less expensive (needs much less expertise) than a software developer, but you still get superior testing.
Tester has a strong incentive to test well: makes support much easier.
Mr. Analogy
Thursday, April 1, 2004
Since no one's said it yet, I'll recommend "Testing Computer Software" by Cem Kaner et al. It's a bit dated, but the core of it hits the nail firmly on the head. Take the examples w/ a grain of salt, and you can get a ton out of the book.
Elephant
Thursday, April 1, 2004
Not doing QA on software is like living in tornado alley and not carrying homeowner's insurance! You feel incredibly smart at how much money you've saved until you're looking at a pile of matchsticks.
The cost of catching an error in development is far less than having your good customers find it and decide to be someone else's customer.
Ned Ryerson
Thursday, April 1, 2004
One of the roles of the QA group in agile organizations is to write the (hopefully automated) acceptance tests for the system under development, to give form to the user stories that the developers are working on.
Their role becomes much less about verification that the system performs as specified and much more about specifying the system in a way that can be checked automatically. They also serve as a bridge between the developers and the Customer, able to describe as acceptance tests what the Customer is asking for in terms of user stories.
Robert C. "Uncle Bob" Martin gives a great presentation on this titled "Testing Is About Specification, Not Verification" that goes into the details of how this works.
Chris Hanson
Thursday, April 1, 2004
Here's another anecdote I just realized was publicized so I can't get in trouble for sharing it...
A company built some trading software for a major Wall St. player. When it went live, thousands of buy and sell order pulsed through the system. Shortly thereafter someone noticed a "glitch". From what I am told, when a glitch happens and tons of money is flying around, you better be ready to cover the losses if you do anything to mess it up.
They finally pulled the plugged after about 1-2 hours of scrambling to fix it on the fly. This glitch ended up costing them $700,000.
Ned Ryerson
Thursday, April 1, 2004
A testing group can devote more time and resources
to testing in all the different possible configurations.
They can spend time developing more complicated
tests that take longer to set up.
QA should have entrance criteria that must be met
before they test a load. This is usually a short form of the larger test suite. Developers are responsible
for making the test pass. Developers will run the long
form if they can.
This may make sense for your product or not.
Some tests take weeks to run and take many
millions of dollars of equipment that must be
shared smartly. I don't think it is one size fits all.
son of parnas
Thursday, April 1, 2004
The value they bring is "making sure the product matches the business requirements." *Theoretically* developers could do this, in the same way developers could balance the books at the company and design the advertising, but they don't, and you don't want them to, for various cultural reasons too complicated to go into.
Ideally, they're involved at every stage of the process - early on, they're kind of like a devil's advocate product manager, analyzing and challenging implicit assumptions, costing, mismatches between business goals and what everyone wants to do, making sure the specification actually includes all planned functionality, the functionality doesn't include anything not in the spec, and so on.
As time progresses the role morphs into doing the same thing at a more technical level. People rarely state it formally, but functions that behave incorrectly, performance that isn't cost-effective, and bad security aren't really not technical problems - they're unfulfilled business requirements.
Unfortunately most places either force testers away from their most effective role by either:
1) Hiring people incapable of doing anything more than whacking through dialog boxes.
2) Trapping themselves in the code/fix loop through bad management.
Myself, I find it a lot more interesting than development. It's a hell of a lot harder, for one thing.
Jason McCullough
Thursday, April 1, 2004
Wow, thanks for the replies!
<<We don't use testers because the only people capable of testing our product properly are developers themselves...>>
UH! What an untrue statement. I hope you don't mean the developers that wrote the code! They are the least capable of testing the product!
Re: Developers being bored with testing. They probably never received formal test training. Write code to test code. Write tests to design code. What' s boring about that?
I hire recently hired a developer to test, he codes test harnesses and tools to test our products as well as system tests using white box methodologies. I must mention, he has formal software testing and security testing training.
And thanks for the tip on the Cem Kaner reference. I actually know Cem Kaner and solicit interns from his training programs at Florida Tech, what a great professor!
As far as the comment that I can't sell QA and that's why I'm here.. that's a fair statement, but I don't have to sell QA to the developers, they don't have a choice in the matter. Like it or leave it. I'm just trying to get a perspective on the matter that I just can't understand.
In my opinion, QA is step up from Dev. I think you'll see this trend continue as companies face litigation from inadequate software. QA resources will soon be the best of the best of their developers, practicing QA and testing.
Has anyone seen a change in their company's QA process since the sarbanes - oxley bill of 2002??
JKeys
Friday, April 2, 2004
JKeys, right at the start I accused you of being Machiavelian, and attributed your communication problems to perfectly justified responses from the developers.
I see you have now confirmed this: "I don't have to sell QA to the developers, they don't have a choice in the matter."
I also pointed out testing can be a tool in toxic management environments, and thus a symptom of them, and you confirm that as well with your comments: "In my opinion, QA is step up from Dev. I think you'll see this trend continue as companies face litigation from inadequate software."
In other words, these types of organisations can only attract very poor developers. Organisations that go this way are not sustainable, because you must have your best developers creating the new software.
JM
Friday, April 2, 2004
Perhaps if you can locate people who have the hacker spirit and enjoy questioning peoples' assumptions, you could build yourself a nice Black Team. Creating is not always exciting; reacting could be a rush.
Tayssir John Gabbour
Friday, April 2, 2004
The more I think of it, the more exciting it would be to have organizations where I fight the testers. Alternately, I switch and go face-to-face with developers.
Current developers may have political problems what that idea, since you're naturally in an antagonistic role, but boy would that be an interesting future.
Tayssir John Gabbour
Friday, April 2, 2004
"Organisations that go this way are not sustainable, because you must have your best developers creating the new software."
As an amusing analogy, why is it that the best athletes are assigned to playing defense in football?
Jason McCullough
Friday, April 2, 2004
 Recent Topics
Fog Creek Home
| 
