The Old Joel on Software Forum: Part 3 (of 5)

Why people can't know.

Joel's comment about not being able to tell people it's a joke and they just don't get it reminded me of this article:

http://www.apa.org/journals/psp/psp7761121.html

It's both fascinating and rather scary.

Mr Jack
Wednesday, January 28, 2004

<g> explains why the majority of programmers believe they are above average in ability...

FullNameRequired
Wednesday, January 28, 2004

I got into business with a guy who seems to fit the bill with this exactly. Once I had worked with him for a while I discovered he was highly incompetent and totally unable to see it. He went from one disaster to another leaving a path of destruction behind him. Investigation of his past business dealings revealed this to be norm with all people who delt with him. Unfortunately, he also had plenty of money in the family, so when the business started to die he decided to take it out on the rest of the people involved via his lawyer, totally unable to see that he was the cause.

Craig
Wednesday, January 28, 2004

So humility is not uniquely the province of religion.

Tayssir John Gabbour
Wednesday, January 28, 2004

You don't need a study to see the truth of this phenomenon. Just watch American Idol.

Oren Miller
Wednesday, January 28, 2004

This study tells us what we already believe. I read the first quarter or so of the article and my opinion already is that the study is poorly designed.

I do agree that everyone considers himself to be above average in general but there are always the highly competent individuals with low self-esteem to muck things up.

name withheld out of cowardice
Wednesday, January 28, 2004

I suggest you read the whole study before judging its merit - I suspect you'll find those elements that you identify as 'poor design' are accounted for in the later experiments.

Mr Jack
Wednesday, January 28, 2004

I doubt it, though I might continue to see if you are correct.

Example: Using psychology students at a single college and giving them standardized test questions introduces a significant selection error. How would future experiments make up for this?

name withheld out of cowardice
Wednesday, January 28, 2004

I pondered this for some time when I taught software engineering.

I wondered:
Why is it that the students that make the most mistakes are the the least concerned about those mistakes, and the least concerned (worried) about future performance.

I finally realized it was quite simple:

These people completely ignored the results of thier work. They were never concerned about mistakes because they completely forgot them as soon as they made them.

So increased thier self-view, but prevented them from learning from thier mistakes.

Ironically, IMHO, the less skill someone has (the more poorly they perform) the more likely they are to adopt denial as a self defense.

Ironically, this leads to more poor performance.

It's hard to be honest about our shortcommings, but it's the only way to deal with them.

The real Entrepreneur
Wednesday, January 28, 2004

An odd side effect or perhaps corollary. An old Pathologist once told me that in dealing with surgeons he found that the ones that were very good and sure of themselves just tended too assume that everyone they worked with, the pathologists in this case, were likewise competent (until demonstrated otherwise). Thus these surgeons just accepted what he said while the less confident ones would incessantly question his methods and conclusions.

Of course this could all be observer error but I thought it was interesting

name withheld out of cowardice
Wednesday, January 28, 2004

Read Study #3 for the answer to your question. Particularly the "false-consensus effect" discussion.

Yes, this paper is not all-conclusive. It does provide a very strong suggestion. However, to fully validate it, it is normal within science to repeat the experiements with as many different variables as you can use. The more consistent the results are, the more likely the hypothesis is true.

-T.J.

T.J.
Wednesday, January 28, 2004

I had the same problem with my own perspective of how other people's ability to develop codes. I always have assumed for a long time that most people are at least as good as I am or better. I keep looking at the gurus (which in hindsight are actually rare), and their productivity capability, and wonders how I could keep pace with it. But I always assumed that others are as decent as I am.

But that was before my previous boss introduced the concept of 'code review'.

My perspective of the majority of other people's performance with coding in relationship to myself was re-calibrated practically overnight.

I am at a level where I know I would not be able to haul ass for Joel's company (I probably would be one of the worst coders there), but that is a symbol of how high of a quality level Joel demands.

But I am also at a level where very few people could code rings around me (the gurus). Don't you guys hate knowing that you're damn good, but definitely not good enough?

-T.J.

T.J.
Wednesday, January 28, 2004

Experiment three doesn't answer my question. It seems as poorly designed and the one about logic.

name withheld out of cowardice
Wednesday, January 28, 2004

See also this article, from the New York Times:

"Among the Inept, Researchers Discover, Ignorance Is Bliss"

http://tinyurl.com/33vc

Alex Chernavsky
Wednesday, January 28, 2004

Real entrepeneur, I think it can be seen as a process of natural selection.

Good b-shit artists will be able to bluff their way through even if they're poor programmers - say - whereas those without those social skills won't.

The corollary is that the nicer a guy or girl is, the less capable he or she probably is.

I like beaches
Wednesday, January 28, 2004

"Example: Using psychology students at a single college and giving them standardized test questions introduces a significant selection error. How would future experiments make up for this?"

Well, yes - it's a relatively small study and more research is needed. But I see no reason to believe it's findings are flawed simply because it concentrates on college students. And quite frankly even if the result did turn out to be particular to college students it would still be interesting.

Mr Jack
Thursday, January 29, 2004

Mr. Jack-

The point isn't even "college students", it's students at the same college. The college itself has selection criteria based on, among other things, the ability to reason logically. One would expect that Cornell students in general would be in the top half of the population as far as the ability to solve problems in logic. It may even be the top quarter. A study that proposes to learn about incompetence but fails to include the bottom 3/4 of the population is invalid and teaches us absolutely nothing. I don't mena to be derogatory, but I will. In a hard science this sort of study simply wouldn't pass muster.

Another point. The people at the bottom seem to think they are at about the 60th percentile while those at the top seem to think they are in the 70th percentile (on the things studied...approx.) Instead of concluding that the better people were also better at estimating their performances, wouldn't it be simpler to conclude that most people think they are slightly above average? Thus people who are slightly above average would be the best estimators? Did the study even say how the slightly above average people were at this? My recollection is not but maybe I just missed it.

Also, in experiment 3 in which each participant judged how well others did. I'm not entirely clear on the method here but if we are dealing with a set of college students who are above average for the population and the competent and incompetent each think they have the right answers, wouldn't the people who actually have the right answers always be better at estimating other people's scores?

How would the "incompetent" cornell students do against the non-college attending population at identifying English sentences? My guess is, significantly better. I also dare guess that the people who truly don't know good grammar might actually know this. THey might know from years of failing high school classes that they ain't no good at book learning.

The study purports to demonstarte that which you believe and thus you assume it is good. It is not.-

name withheld out of cowardice
Thursday, January 29, 2004

Coward,

I refer you firstly to footnote 1 discussing what they mean by incompetence.

Your hypothesis that everyone judges themselves a bit above average is, of course, an obvious one. However the studies demonstrate that it is not all that is going on. The incompotent not only overjudge their comparitive performance, but their actual score. In a way that the competent do not. The incompetent were unable to recalibrate their ability relative to the group by viewing the test papers of others; the competent were. This is not to say that people don't typically judge themselves just above average just that there is a more interesting phenomena going on.

Maybe the really, truly thick know they are really, truly thick. This study does not tell us. The study does say "Finally, in order for the incompetent to overestimate themselves, they must satisfy a minimal threshold of knowledge, theory, or experience that suggests to themselves that they can generate correct answers. In some domains, there are clear and unavoidable reality constraints that prohibits this notion. For example, most people have no trouble identifying their inability to translate Slovenian proverbs, reconstruct an 8-cylinder engine, or diagnose acute disseminated encephalomyelitis. In these domains, without even an intuition of how to respond, people do not overestimate their ability. Instead, if people show any bias at all, it is to rate themselves as worse than their peers ( Kruger, 1999 )."

It is just one study, its results will therefor always be tentative, and cannot be assumed to be wildly extrapolatable. I maintain it presents an interesting and highly plausible hypothesis none-the-less.

I have no idea why you assume this report supports my existing prejudices.

Mr Jack
Thursday, January 29, 2004

Name Withheld: Your criticisms are all very well and good, but the study is entirely consistent with my intuition and with decades of personal experience dealing with idiots. Therefore, I believe the study.

Another coward
Thursday, January 29, 2004

Mr. Jack:

I dunno, does it? Are you telling me that this study came as a complete revelation to you, destroying illusions you had been living under for years? If so I am sorry and I reccommend you not make up your mind yet. The idea that this is a start and further study is required is just plain stupid. This is not a start. If someone wants to investigate this idea he should design and execute a good study. In science (and pseudo-science such as this) we don't just dismiss the fact that the study design is poor. Poor design invalidates the results.

Your response I find very post-modern.

name withheld out of cowardice
Friday, January 30, 2004