Fog Creek Software
Discussion Board




Programmer Performance

The business folks at my company want to start tracking performance in all departments.  Ideally, they'd like concrete, non-subjective, metrics to "compute" this.  I.E., actual time of completion vs. estimate; actual time of delivery vs. estimate; number of billable hours, etc.
I'm looking for others, and maybe some good/bad reasons for using them.
Are there "industry best practices" for programmer performance?

Ken
Monday, July 19, 2004




Hahaha.  You're screwed.

LoC (Lines of code) are normally a terrible idea.  ROI (Return on Investiment) is a nightmare to establish.


Your best bet is to get your users/clients to vouch for you.  You want your users/clients to ask for you on their projects.  I have 3 groups within this company that have specificly asked for me.  It's hard for a boss to dispute anything when the client knows that you'll communicate with them, work to clarify things, and deliver a solid system.

KC
Monday, July 19, 2004


Sorry, I didn't mean that to come out quite that way... it was initial reflex.

KC
Monday, July 19, 2004

> Ideally, they'd like concrete, non-subjective, metrics to "compute" this.  I.E., actual time of completion vs. estimate

Is this to this measure the output of the programmer, or the accuracy of the estimate?

Christopher Wells
Monday, July 19, 2004

Start looking for a new job.

sgf
Monday, July 19, 2004

Did those estimates come from the developer or from the PHB? Most of my grief has come from supervisors pulling numbers out of the backside of their underpants. The person controlling the estimates has life or death control over you.

An example of why estimate vs delivery is a real bad metric: The Denver Airport baggage handling system. The development managers said it would take 4 years to build. The sales team said "the airport would be finished in 2 years, and that's when the handling system is being promised for completion: you have 2 years to do it in."

The handling system was finished 4 years after it started. Was the handling system finished on time, or 2 years late? Most managers think it was 2 years late, because some sales idiot promised it in half the time it could be finished in, and that is the number that was widely published and what they all remember.

What ever you decide to measure is what you are going to get. If you measure billable hours, you will see folks acting like lawyers: billing for time in meetings, billing for lunch, etc.

What do you really want to measure? What does it mean when you say "a programmer is productive?"

Peter
Monday, July 19, 2004

"What ever you decide to measure is what you are going to get."

And you'll get NOTHING else.

These sorts of extrinsic measurements (and rewards based thereon) cause you to be less focused on your work and more on the extrinsic measurements. You'll be thinking "how many hours did I bill today" instead of "what button name is going to be clearest to the customer -resulting in fewer tech support calls, happier customers, and higher net revenue".

Mr. Analogy
Monday, July 19, 2004

"Industry Best Practices for Programmer Performance" -- unfortunately, the 'Industry' is evolving so fast, and it has so many different niches (web development, C++ Dev, Java Dev, Windows Dev, Unix Dev, Linux Dev, Database Dev, DOD2167 vs DOD 498 vs IEEE 12207 vs CMM, DFD vs RUP vs AM vs XP) that there are so many 'Practices' that there is very little consensus about which ones are 'Best'.

The SLOC (SourceLinesOfCode) metric is concrete, but means little across different platforms and languages, and probably measures 'the wrong thing'.  In other words, is it better to crank out 200 lines of grubby code, or spend the same amount of time to do the same function in 20 lines of code?  And if today I 'clean-up' and remove 180 lines of grubby code, is my productivity negative?

The first hurdle is trying to measure ANYTHING.  The next hurdle is how to use those metrics.  From what I've seen of "Business Types", they need to use metrics in a very simplistic way. 

The best use of metrics is 'predicted versus actual', so we can predict how much more time is needed to complete the project.  These would be predicted versus actual for:  Number of objects created, Number of objects needed, 'Function Points' created and needed, etc.  The worst use is 'he is better than the other guy', as this creates and rewards metric creep.

And, of course, if you are going to USE 'predicted versus actual', the sales guys and managers MUST keep their hands off the 'predicted'!  It is hard enough to try to come up with some realistic 'blue-sky' prediction on code size and functionality.  To then have a manager or sales guy tell you that it should take half that time is  very demeaning and demotivating.

Also, be aware, in software the 'Actual' rate of completion of code is very non-linear.  Lots of time up-front in design (no code).  Lots of code in the middle (we're Coding!).  Lots of time at the end integrating, testing and fixing (not much code, in fact we may be removing code).

AllanL5
Monday, July 19, 2004

As much as possible, use metrics that balance each other out, such as time-to-market and customer satisfaction.

Derek
Monday, July 19, 2004

After using SLOC, POPs and all sort of adhoc metrics I found out that ... hmmm ... software metrics alone are not that useful.  "How long does it take to implement X feature?" does not have a simple answer, no matter how much the upper management wants one.

Building up a historical database of software development records won't help to much either. The amount of information that needs to be collected and analyzed is huge (and creates a proportional overhead).

Dino
Monday, July 19, 2004

Ken, I implore you to take some of these comments to the decision maker of this new policy and present it to them in a polite but forceful manner.

I've done this a few times with threads that were about issues we have at work and the outcome has been good. The smart people on this forum can usually express their point of view better than I can.

Personally, I think that measuring estimate vs. actual time and customer satisfaction are two good ways of doing it.

Wayne
Monday, July 19, 2004

Ken's neither a programmer not someone about to be affected by this type of performance assessment. He seems to know zilch.

He's probably an analyst from some IT industry "research" outfit trying to learn the basics for a report they will eventually sell for $4,000 a pop.

Management material
Monday, July 19, 2004

Keep in mind, Ken, that any metrics only make sense when the tasks that compared programmers are working on are also well comparable. That is, you can't hope to objectively compare performance of two guys while one is working on UI and another on DB interaction.

Egor
Tuesday, July 20, 2004

Don't use KLOCs to compare one coder's performance with another. Better coders may well write fewer KLOCs. But given that each coder' style is fairly constant, measuring the output of the whole group of coders over a longish time frame (a month, or a quarter) may give you an idea of how productive the group is being.

It's probably more important to measure defects in the code produced, however. Check out Tom Gilb, who has plenty to say on the subject,

Dave Hallett
Tuesday, July 20, 2004

Guys, guys, Ken doesn't give a stuff what you think.


Tuesday, July 20, 2004

Measure bugs.  Empower your customers to fill in bug reports.

Typically, empowered users aren't actually going to fill in bug reports.  Hence low bug counts.

i like i
Tuesday, July 20, 2004

...Your best bet is to get your users/clients to vouch for you.  You want your users/clients to ask for you on their projects.
    I'm trying to implement a situation where we have chief programmers that will choose from the staff who will work on their projects.  I figure this will at least give me the opportunity to weed out the dead wood.

...Did those estimates come from the developer or from the PHB?...
  Typically, the programmer doing the coding will provide the estimate.  In the case of new, less experienced programmers, the senior programmer on the team will estimate, and then multiply by the "newbie factor."

..."What ever you decide to measure is what you are going to get."
And you'll get NOTHING else.
    Agreed.  The metrics we've had up to now have put the sloppiest, least elegant programmer in the department above all others.  That's why I want to change the way we do things.

...Ken, I implore you to take some of these comments to the decision maker of this new policy and present it to them in a polite but forceful manner.
  I'm the decision maker of the department, but the guy pushing for the measures is the CEO.  He wants to be able compare the "productivity" of one department (say SD), to another (CS).

...Ken's neither a programmer not someone about to be affected by this type of performance assessment. He seems to know zilch.
  Programming for 12 years, managing for 5.  My staff, who I actually care about, will be affected, and through them, so will I.  I'd love to speak more to you about your apparent clairvoyance, but since I know nothing, I guess I'd better get back off to kindergarten.

...Guys, guys, Ken doesn't give a stuff what you think.
  Not all managers are like you.


The way I typically judge performance is based on the elegance of the code, and the logical way in which the various pieces, or even a single algorithm, is put together.  Unfortunately, these tend to be more subjective than objective.  One of the desires for the objective measures is for when a charge of favoritism is made.  Concrete measures can't be argued with.

Anyway, thanks for all the feedback.

Ken
Tuesday, July 20, 2004

Ken - That's cool, it's nice when someone tries improve the performance measuring process.

But..."but the guy pushing for the measures is the CEO.  He wants to be able compare the "productivity" of one department (say SD), to another (CS)."

Good luck with that part. Comparing different programmers is difficult, but you're at least you're comparing apples to apples. Comparing the "productivity" of software development to customer service is like asking whether your car or your house is performing better.

I liked the other things you said, it's just this particular one that's

Will Gayther
Tuesday, July 20, 2004

The only way to compare objectively is to give multiple programmers the same task, and then you can compare them based on the time taken, bugs, LOC (fewer=better).

Anything else will result in distorting behavior to optimize whatever is being measured.

NoName
Tuesday, July 20, 2004

Of course, I'm sure your CEO didn't reach his position as a result of performing well to objective metrics.

NoName
Tuesday, July 20, 2004

I disagree with the "only objective way" proposed by NoName. Shorter code is not better if no-one but the author can make sense of it. Completing a task rapidly is only of value if the code is readable and works. If you want some code that doesn't work, I can do any job instantly.

I would say, give all the programmers the same task to complete within a reasonable time limit, and score them on the number of defects in the code they write. Hopefully you have a QA dept that could do this. What else really matters?

Dave Hallett
Tuesday, July 20, 2004

In Tom DeMarco's book "Why Does Software Cost So Much" there's an excellent essay, "Mad about measurement", exactly on this topic.

Kobi
Tuesday, July 20, 2004

Rereading more carefully, I see that NoName did actually mention bugs. My apologies. I still think that's the most important measure, though.

Dave Hallett
Tuesday, July 20, 2004

>I would say, give all the programmers the same task to
>complete within a reasonable time limit, and score them
>on the number of defects in the code they write. Hopefully
>you have a QA dept that could do this. What else really
>matters?

The issue here is that you've got to create a set of tasks that test all knowledge/skills that are important to the organization's goals.  If you give your programmers the task of implementing a programming language then the ones with prior experience doing this will destroy the others.  But if you sell video games and you need language designers, graphics/geometry programmers, physics programmers, "AI" programmers, and so on, then the language test will not be particularly useful.

Making a test for each type of activity is a good step (or a 'better' step anyway).  But comparing test scores between departments can be extremely problematic.  If you have two students and one gets high marks in sciences and the other gets high marks in language arts, which is the "better" student?  It's not a meaningful question.

But what is it that you really want with this testing?  Is the purpose to identify your top performers and reward them, to provide them with even more incentive to do excellent work?  If so, there might be other methods of doing this, appropriate to your business.  For example, if small teams develop products then give each team some percentage of the profits on the product.  Mix up the team compositions over the course of a few projects to make sure that hangers-on have some chance to fall off.  After you've done that a while (and everybody has had a chance to get a feel for everybody else), let the programmers vote on who they'd like on their teams.  This might make the importance of particular people (and particular pairings of people) very clear.

Kalani
Tuesday, July 20, 2004

I should have also added Performance to the list of objective criteria.

Still, no matter how much you try to obtain objective metrics there will be subjective things that cannot be quantified, such as readability. Just as with an athlete although there are objective measurements such as points scored, there are still subjective elements involved with evaluating their ability.

As long as the LOC is properly defined to minimize artificial manipulation (e.g. chained method calls such as x.y().z().q() should be counted as multiple lines), less LOC will almost always be better in the subjective measures such as readability.

NoName
Tuesday, July 20, 2004

Ken the Original Poster asked:
> Are there "industry best practices" for programmer performance?


Go read the book "Peopleware" by Lister and DeMarco.

http://www.amazon.com/exec/obidos/tg/detail/-/0932633439/qid=1090337268/sr=8-1/ref=pd_ka_1/104-6268678-1326361?v=glance&s=books&n=507846

Mark S
Tuesday, July 20, 2004

Ken, what you want to create is a productive, fun-to-be-in environment. Yes, people work, better and harder when they are happy and work is fun. One depressed guy can take down his whole team in no time.

Don't create a bureaucracy; better off, look at agile development methodologies. The overall development process will be more efficient and you'll get what you want without alienating all your programmers. Remember, your people are the ones getting the work done and when things hit the fan, they are the ones that will get you out of trouble – one way or the other :-)

Dino
Tuesday, July 20, 2004

...Don't create a bureaucracy; better off, look at agile development methodologies.

That's one of the things I'm trying to do too:
http://mkennethclark.blogspot.com

My position has always been that good programmers on an energetic team will produce good software.  The measurement of their success has always been subjective, based on my "opinion" of their skills.

Anyway - thanks for all the good feedback.

Ken
Tuesday, July 20, 2004

mmorpg leveling efficiency?

Ken
Tuesday, July 20, 2004

Google for "heisenberg"

mike
Tuesday, July 20, 2004

<quote>I'm trying to implement a situation where we have chief programmers that will choose from the staff who will work on their projects.  I figure this will at least give me the opportunity to weed out the dead wood.</quote>

This is fine, if your objective is simply to axe programmers who don't make the grade.  (Kinda reminds me of the demon Crowley in _Good Omens_, who heard that talking to plants would make them grow better, so every few weeks he'd pick up a plant, walk it around to all the other plants saying, "this one couldn't hack it; say goodbye," and then toss it in the dumpster.)

Now, there may very well be people on your team who'll never be good developers (using Eric Sink's distinction).  If they're really never going to improve, sure -- give them an invitation to the world.  But for everybody else, what about something like the XP process of pair programming with frequent pair switching?  Speaking as a solo developer, the one thing I miss most about working on a team is the opportunity to work (and swap ideas and skills) with other bright people who are good at things I'm not as good at.

Try that for a little while (say, 3-6 months) before moving to your "lead developers pick their staff."  Then you'll at least have people making their team selections based on actual one-on-one knowledge of how the other team members work.  Add in summary reviews of coding sessions and you might not even want to move to your other plan, since with everyone working with everyone else, your "dead wood" won't get a positive review from anybody.

<quote>..."What ever you decide to measure is what you are going to get."  And you'll get NOTHING else.
    Agreed.  The metrics we've had up to now have put the sloppiest, least elegant programmer in the department above all others.  That's why I want to change the way we do things.</quote>

...and you think using different metrics is the way to make things *better*?  See:  http://www.joelonsoftware.com/news/20030115.html and more importantly: http://www.joelonsoftware.com/articles/fog0000000070.html

(Oh, wait, you didn't say you wanted things better, did you?  You just want them to be different.  Carry on, then.  ;> )

Sam Livingston-Gray
Wednesday, July 21, 2004

When trying to find things to measure, don't forget 'has a good or at least non-negative influence on the work of the other programmers.' :-) 

Suppose that:

Developer A helps fellow developers to be more efficient/effective and to improve their own skills.  Developer A happily spends a few minutes of time to save someone else many hours or days of frustration.  Developer A's help to others produces significant beneficial effects on the whole group's aggregate work.

Developer B helps no one, but plays loud, obnoxious music, and distracts fellow developers with porn spams.  At all developer-only meetings, Developer B wipes nose-pickings onto every document within arm's reach, and continually contributes stupid, insulting suggestions, so that other people hardly have a chance to be heard.  Many of Developer B's co-workers are so distracted with frustration and rage that it impairs their performance on most every metric measured.

Developer B has the highest *individual* productivity in Developer B's whole group! 

Developer A is rather good, but did not measure up to Developer B in individual productivity, being 'worse' by 0.0001 units.

I wonder which one should be promoted to a managerial position?

Chris Niswander, http://bitboost.com
Sunday, July 25, 2004

*  Recent Topics

*  Fog Creek Home