Fog Creek Software
Discussion Board

Joel's Test and Productivity

Over the last six months, we (i.e. my two-person project team) established many items from Joel's 12 Step Test. Starting at, uhm, one single point, our current score is about 8. Life did not get easier, though. All the productivity we expected to gain is eaten up by writing specs, maintaining the build, (and, ahem, fixing bugs) etc. I'm a little disappointed. For me, there are three possible reasons:

(a) I'm too impatient; the productivity boost caused by following Joel's Test will soon kick in, and sky-rocket our productivity to a level never known before.

(b) My team is too small. Joel's Test works well for large development teams (say, about the size of MS), but not for two lonely guys.

(c) I completely misunderstood the purpose of Joel's Test - it is *not* about productivity. Then again, what *is* it about? Quality? But this should be linked to productivity!?

Bottom line: I must admit that 'productivity' is a fuzzy notion, and that I, by no means, can measure it precisely. If I say that it seems  that not much has changed, this is a feeling rather than a measurement.

Does anybody have experiences with introducing Joel's Test? And what are the results?

- Roland
Monday, January 20, 2003

I don't think it's any surprise that it takes longer to release something if you're scoring well on the Joel Test. But stop and think - is what you're releasing now of better quality?
I think it is, because you admit you're now spending time fixing bugs!

Also, is your two man team two developers or two people in the company overall? Bear in mind that marketing, support, testing, training, manufacturing etc will all be interested in what's in your spec and what gets shipped.

You might just be going a bit overboard. For my (currently) two man team, I keep bugs in an Excel spreadsheet checked into SourceSafe. Took about two minutes to set up and works fine.

Doing the build took about a day to setup, for a two man development team just a batch file will do. The important point here is that once you've set the automatic build up, the time you lose doing that is easily offset by the vastly quicker time to produce a working build, and you've got a better guarantee of it working.

I don't have an individual spec, but instead I have lots of "mini specs", one for each bit of functionality for the product. This has the advantage that each spec takes no more than a few days to write, we can write a full schedule out of it, there's not much maintenance needed, and it never goes above 6 - 7 pages; and most of that is screenshots. This is good because our management would fall asleep trying to read it if it was longer....

I adpoted a lot of the processed outlined in the Joel Test before I'd ever read the Joel Test, and they worked great for me. You might not find yourself releasing things more quickly, but you'll definitely find you _do_ save time in the long run because you don't have to waste time with .xxa, .xxb, .xxc releases every time the customer spots a silly bug....

Better than being unemployed...
Monday, January 20, 2003

Classic problem : we already have a very hard time handling quantity (cf. creative accounting over @ Enron et al.), but we have even less visibility as far as _quality_ is concerned... which is as much if not _more_ important when it comes to abstract products like software.

In other words... the fact that you deliver high-quality software while your competitors sells crappy stuff _is_ an important asset.

Talk to your customers about whether they understand that buying cheaper software (cheaper because you will have spent less time thinking it through, building high-quality code instead of spending time + money fixing bugs and sending patches) or more expensive but of higher quality (hence less headaches to IT people keeping up to date.)

In short, productivity is a very hard concept....

Frederic Faure
Monday, January 20, 2003

It is rather difficult to tell from what you've written why you are not seeing productivity gains.  The fact that you aren't measuring your productivity is an indication that you may not be sure of what to expect.  Don't expect the productivity gains to "soon kick-in".

There is not much special about the Joel test.  Most of what is there are standard software engineering practices condensed in to a quick easy to understand check off list.  If you are writing scripts that take a few days each to produce, these factors won't help much.  Once you get to anything larger: two or more developers or a few weeks to write the software, then these practices become essential for maintaining software quality. They don't help much with just producing lines of code. And you don't have to be MS size to use them.

DeMarco and Lister in "Peopleware" named their chapter on team building "Teamicide" because they realized that they could tell us how to kill a team, but couldn't give any formula for building one.  The factors on Joel's list are similar in that following them won't make you productive, instead they eliminate distractions to productivity.  You still have to design and write the code, not just expect following the list to make you productive.

Monday, January 20, 2003

According to Joel, the Joel Test is supposed to measure the quality of your software team. ("to measure how good your software team is").

Looking at the article more carefully, he doesn't actually  elaborate on the benefit of having a "good software team".  You could interpret the benefit as improved quality and improved ability to work together as opposed to an actual productivity increase, - because there is no question that doing things like maintaining a bug database, using source control, writing specs etc etc adds overhead which sometimes can sometimes seem like terrible overkill, especially in a small group. ("That will take so little time to fix that entering the fact that it is fixed into the bug database will take more time..." or "Updating my bugs knocks me out of the zone" - guess where we have our main problems :/ )

But I think that the perceived loss of productivity due to the "overhead" is really mainly perception, because things like "fixing bugs before adding new features" not only improve the quality of your product; they also reduce the time you spend fixing the bugs 6 months from now when you don't remember anything about the code. Don't forget that fixing bugs before shipping is much less expensive than fixing them after the fact.

Also, some of the individual points in the Joel Test have obvious benefits (which he does explain) which are not necessarily upfront productivity gains - eg source control makes it easier for two programmers to work together and reduces the possibility of code loss in the event of problems such as disk failure.

After all, you are not going to see the productivity benefit from having a smaller risk of data loss unless you actually have a disk failure and/or backup failure.

In my opinion, when you are evaluating the changes you've made to your process based on the Joel Test, it would probably help to look at some of the non-productivity related benefits because most of the time I think you'll find that the productivity "gain" is difficult to measure. 

Monday, January 20, 2003

The other thing to remember is that the benefits of a good software process increase as the size of the team gets bigger. Some things are pretty much non-negotiable; you should be using source code control unless you can recreate every line of your program from memory. But the other points should be implemented to an _appropriate_ level.

Take requirement specs. The purpose of a requirement spec is to ensure that every person on the team is working to implement the same thing, that they know the goals and trade-offs, and how the part of the system they are writing interacts with the other parts. A one person team that needs to be enough detail to remind you what Component B of the system was going to do while you are writing Component A. With a ten person team it needs to be a lot more detailed, because A and B are being written at the same time, and you need to be sure that Al and Bert (who are writing them) understand the finished product in the same way.

In short, you probably won't see big benefits on a two person team. But I'm also pretty sure that, provided you keep your level of documentation appropriate, you will see some benefits. And when you become a four person team, or six person, the experience will be worth it.

David Clayworth
Monday, January 20, 2003

If you are killing more bugs now then you might not see a gain at once, because it takes longer to ship code, but you'll see a gain at the end of the year when you haven't had to patch, rebuild, repackage and re-ship the same code over and over. Which takes less time than getting it right.

Have you lost time, or just become a lot more honest about where time is being used up?

As mentioned elsewhere, have you gone a little overboard? I always think that its too easy to go after some new methodolgy like its a religion whereas it should only be a guide.

Robert Moir
Monday, January 20, 2003

If you are killing more bugs now then you might not see a gain at once, because it takes longer to ship code, but you'll see a gain at the end of the year when you haven't had to patch, rebuild, repackage and re-ship the same code over and over. Which takes less time than getting it right.

That should be "Which takes MORE time than getting it right."

Mondays. Tch.

Robert Moir
Monday, January 20, 2003

*  Recent Topics

*  Fog Creek Home