The Old Joel on Software Forum: Part 3 (of 5) - Example of Optimizing to a Performance Metric

Example of Optimizing to a Performance Metric

Just found this article that talks about the timers at a Burger King drive through and how in order to ensure the staff that was working would 'measure up' the writer as a customer was asked to do something that didn't make sense to him...

Just another example for the argument against performance metrics. Onces there is a measuring stick in place, people will optimize their behaviours to it in any industry.

Ray
Friday, September 26, 2003

Sorry, should have included the article. D'oh!
http://www.fool.com/news/commentary/2003/commentary030926ram.htm?source=mppromo

Ray
Friday, September 26, 2003

Great article, thanks.

Passater
Friday, September 26, 2003

We had one of these at a Taco John's (regional Taco Bell competitor) when I worked there in high school. We *never* asked a customer to pull ahead on an order, unless the customer behind them had their food ready.

That being said, perhaps ours was less sophisticated. It just kept an average of all times on an LED, and then at the end of the shift, the manager recorded them. So, before we left, we drove on and off it a few times. A bunch of 1-second times drops that average like a stone! ;)

Dignified
Friday, September 26, 2003

I don't think that providing one example of an abused performance metric means that all performance metrics are bad. And believe me, I've seen plenty of examples of misapplied and abused performance metrics. But I have seen many more examples of well applied performance metrics that are tamper resistant.

Performance metrics are best applied when the process that is being measured can be standardized. I've worked in manufacturing and my wife works in mortgage underwriting. Both are environments where standard metrics can be applied with good results.

Software development, on the other hand, is much less standardized, and therefore performance metrics can be difficult to apply. But that doesn't mean they're bad in all cases.

Nick
Friday, September 26, 2003

"I've worked in manufacturing and my wife works in mortgage underwriting. Both are environments where standard metrics can be applied with good results."

Yeah - witness the growing trend of bankruptcies filed, to the point that the banks lobbied to make bankruptcies harder to get. Yep, those loan worksheets work like a champ!

Philo

Philo
Friday, September 26, 2003

True, not all metrics lead to this kind of behavior, but most of them do when the particular metric's importance is allowed to dominate over the rest or can be faked.

Like using lines of code to measure productivity, while ignoring bug rates and maintenance effort. Or using budget compliance as the primary measurement of a manager's success, resulting in decisions like laying off the $85K expert instead of the $60K average guy because that makes this year's budget numbers look better.

T. Norman
Friday, September 26, 2003

Performance metrics are often counterproductive.

For example, focusing on bug counts leads developers to argue whether something is a bug, leading to unnecessary hostility and discouraging people from filing bugs. As another example, increased standardized tests in public school lead teachers to drill students for months about how to do well on those particular standardized tests.

In both cases, aiming for overall quality, without objective metrics, would generate better results.

Julian
Saturday, September 27, 2003

"Yeah - witness the growing trend of bankruptcies filed, to the point that the banks lobbied to make bankruptcies harder to get. Yep, those loan worksheets work like a champ"

Actually, this underscores my point. My wife works as an underwriter in the non-conforming (higher credit risk) mortgage business. Many of the companies that have entered this market in the last few years are flight by night operations sweeping in to capitalize in the low interest rates. Most also don't have standardized processes for ensuring the quality of their loans. Sure, they have underwriters that underwrite to guidelines, but the quality of the underwriting is not as important as the turn time and volume to most of them.

But, back to my main point - these companies don't bother with performance metrics, whereas my wife's company has been very solid on this front. Her company has used the loan default metrics to go back and adjust their rates, guidelines, and training where necessary. Looking at the past two year's worth of data, their monthly deliquency rates have gone steadily down from 0.8% to 0.6%.

I am a firm believer in performance metrics. If you want to improve something, measure it. It may not apply to all areas of performance or all types of businesses, but it does to a lot of them.

I understand why many people have a bad taste in their mouth when it comes to performance metrics. I've had my share of piss-poor metrics mandated upon teams I've been on. But overall, the main problem with performance metrics is not whether they should be used, it's with how they are used.

Nick
Saturday, September 27, 2003

The problem with metrics is that we're not robots. So you will always fuck people who don't conform to the metrics. So long as you don't have a problem with screwing random strangers, this is not a problem for your business.

But I've met a *lot* of people who swear by the system until they get screwed by it personally (DA's that become defense attorneys, credit underwriters who become credit repair counselors, IRS auditors who become tax attorneys).

I'll bet your wife budgets her credit in such a way to maximize her credit score - not because it's the best way to manage credit, but to game the system (it seems like a *lot* of loan officers keep 2-3 credit cards and make purchases on them regularly, then pay them off in full - why not simply keep one card and pay in cash when possible? Because that lowers your credit rating). In other words, she optimizes to a performance metric.

Philo

Philo
Sunday, September 28, 2003

I have to say that I agree with Nick. The metrics aren't the problem, it is how they are used.

In the Burger King/Taco Bell/<insert drive-thru place here> example it is obvious that the "time per customer" metric is being used to either punish the staff of a particular shift or as a way of ranking staff or something like that. Since the measurement of the metric has a direct impact on some aspect of the staff's lives like compensation or employee of the month or whatever, the staff has incentive to optimize for the metric. If on the other hand, the metric is a short-term thing used to gather information on customer service times and then look for ways to optimize the overall proceedure that the staff follows or to detect trends in what time of the day certian foods need to be closer to being 'customer ready' then the staff wouldn't have a need to optimize to that metric.

I think the bottom line is if the metric is going to be used in any way that affects the employee, it will be optimized for and an info you gather from the metric will be meaningless.

Ray
Sunday, September 28, 2003