The Old Joel on Software Forum: Part 3 (of 5) - Small ISV's

Small ISV's - Do you have testers?

We all know testers are good, but as a small shop it can be hard to afford a full time tester when you only have a team of about 3 developers.

So what techniques do you use? Do you get the team break out of their developer role and do a few days of solid testing here and there on a project? What about University students looking for cash, anyone had good experience using them? What about a small group of customers looking for a significant discount?

Ben R
Sunday, January 11, 2004

We do a bit of it all really.

One thing we do is have the training and customer service guys bang on it. They are not formal QA, but that are pretty good at it.

Marc
Sunday, January 11, 2004

"Do you get the team break out of their developer role and do a few days of solid testing here and there on a project?"

Yup.

"What about University students looking for cash, anyone had good experience using them?"

It's cheaper (ie free) to use relatives, girlfriends, wives, friends, etc. ;)

"What about a small group of customers looking for a significant discount?"

Oh yeah. Them too.

Almost Anonymous
Monday, January 12, 2004

Interesting question, (I am almost an ISV I think!) just coming up to the testing stage.

Of course you can’t test it yourself, I wrote a little function at work, the function pops up, it says ‘client code’ and ‘filename’, you enter both and the little macro saves the for you. Nice. Simple. I used it for months, never ever ever realised that it didn’t say ‘filename’, it actually said ‘Happy1’.

I am trying to nab some university students, but it is hard, I have no money to pay them. The best I can offer is a reference. Which, truth be told, if I was offered this at university I would have taken it, I mean a few hours a week, and a little experience plus a referee in the right field, all done from their own home.
(I mean I am so willing to do the same thing now, I would love to find a small ISV that would let me do some beta testing, document writing, anything, give them a few hours a week, learn a bit and get a reference, it would help me a lot)

Still, I can’t find anyone (yet) who feels the same. Still looking though, and the local uni is just about to go back.

Aussie Chick
Monday, January 12, 2004

Maybe you can hire a part time, or even worst, a part-time who is part-time tester, part-time staff of something else. I am sure you'll find something, as long as you can train properly.

Li-fan Chen
Monday, January 12, 2004

We are are small software company in Melbourne but don't have testers. Yes sometimes it sucks because I can't test all of my programmer's work, and they often screw things up, mainly due to inexperience. But I would rather have my programmers learn to test their own code properly rather than let another person think for them.

At uni (and it shows in the workplace) young programmers often don't care much about the consequences of bugs. After all, its not treated as a big deal if you screw up on a small assignment. Plus testing isn't tought very well (or at all) in my experience. But we live with that and try to make them better programmers.

I hire a lot of students... sometimes our job ads attract huge numbers of applicants, sometimes not. It depends on the time of year. Worst times are of start of exam periods (June/November) and out of semester dates (Dec, Jan, Feb, and July). Best times are start of new school semesters and just after exam periods, when students have both time to look for work and many need to earn money.

We like students because (a) they are cheaper, (b) they don't expect too much and (c) they haven't been corrupted by other big business organisations <grin>

PDF
Monday, January 12, 2004

PDF, what kinda of operation are you running where you can't afford the proper programming to get the results your customer need? Is it not billable?

Li-fan Chen
Monday, January 12, 2004

Basically we are ending our R&D mode of operation. In the past few years we have been selling minimal number of systems to stay afloat, and to cut down on support. We have very few local sites that are happy to pay for our first class software and support when they need it.

Well thats the spin I put on it :-)

Yes our software isn't of the highest quality, but the types of customers using our software right now are not in need of shrink-wrap tight software. We are slowly improving and rely very much on the latitude that rookie programmers give us re: working conditions and pay.

PDF
Monday, January 12, 2004

Well PDF, maybe your competitors won't have software with quite so many bugs and can use it to leverage your customers away to them. Unless you're in an extreme vertical market that doesn't have any competitors - yet.

Believe me - code that's written by people who can "get things done" but who aren't "smart" can hang around for years frustrating the hell out of anyone who has to sell it or support it, simply because the business cost of refactoring it all to be more maintainable and adaptable to new requirements is too high.

Yes, I agree that developers should take more responsibility for unit testing their code. But a lot of developers are not particularly good at testing the functionality and usability of the entire product. And by having testers close to hand, you're likely to find more bugs before your customers do, and the testers won't hesitate to feedback to the developers about it. So after a while your developers will learn to test their code otherwise the testers will keep throwing them bugs to fix until the cows come home...

Personally if you've only got the capital for 3 developers, I'd rather have 2 developers and a tester.

Better Than Being Unemployed...
Monday, January 12, 2004

We're a small shop with a 6-person development team. We have two types of testing. Smoke testing and QA testing.

Smoke Testing
This is performed after a programmer thinks he or she has coded to spec, but before the code is submitted to code review. This is the "I think I coded what you wrote up in your spec, but can you check to be sure?"

This step is performed by the usability expert on the team (who happens to know exactly how the software should work from the end-user's perspective.)

Not working through a detailed script, the smoke tester simply exercises the functionality that was supposed to be coded, and typically finds 90% of the "easy" bugs that would tie up most of a QA tester's time.

We consider smoke testing part of the construction portion of each 1- to 2-week phase.

QA Testing
This is performed after smoke testing is complete and all bugs fixed, and code is reviewed and updated from code review.

On our team, this is performed by the triple hat-wearing Training Manager and Documentation Lead. He is in charge of creating detailed testing scripts with actions and expected results. The goal of QA testing for us is to test every flow of every use case to the letter. If it's in the requirements, it's tested during QA.

(This is in contrast to smoke testing, which is not intended to provide absolute, 100% coverage.)

The common theme for us is that testing is ALWAYS performed by those who are closest as possible to the end-user. The only time we have programmers performing testing is when we are close to a significant milestone and we want as many people banging on the system as possible, and no new code being written.

For us, this has the side benefit of not only testing the code to ensure it meets the spec, but it exposes the functionality to an "end-user ambassador" as quickly as possible. This helps us catch (*gasp!*) requirements mistakes or missing requirements.

As much as we try to capture all necessary behavior before any code is written, sometimes it's not until you sit down and use the product that you realize you missed one small part of what's supposed to happen, or you realize something is more confusing than it sounded on paper.

This process seems to work well for us. We've had two
"beta" releases of portions of the application since implementing this process and so far, the defect rate seems pretty low. (I believe one or two crash bugs were reported and perhaps 4 or 5 other small bugs.)

That's my $0.02.

Dave

Dave
Monday, January 12, 2004

If you put the programmer in charge of tech support, you'll find them VERY motivated to test the heck out of the software.

I do programming, tech support, and testing. I find it very efficient.

I think it's probably difficult to find somone who's really committed to testing the software. It's a thankless job ("thanks for finding that error in my code, d**khead").

But the guy doing tech support sure cares if there are bugs.

The real Entrepreneur
Tuesday, January 13, 2004

In my opinion you won't last long as an ISV with out QA.

christopher baus (www.baus.net)
Tuesday, January 13, 2004