The Old Joel on Software Forum: Part 4 (of 5) - Nielsen is getting on my nerves

Nielsen is getting on my nerves

http://www.useit.com/alertbox/20040301.html

"For example, studies show that intelligence declines by birth order. In other words, a person who was a first-born child will on average have a higher IQ than someone who was born second. Third-, fourth-, fifth-born children and so on have progressively lower average IQs. This data seems to present a clear warning to prospective parents: Don't have too many kids, or they'll come out increasingly stupid. Not so. "

From this, he goes on to dismiss statistics in usability completely.
<<Yes, experts get better results than beginners from qualitative studies. But for quantitative studies, only the best experts get any valid results at all, and only then if they're extremely careful. >>

One does not know where to start to refute this piece of junk. In fact the experiment he is talking about- measuring IQ;s is itself completely erroneous. It is impossible, i repeat IMPOSSIBLE IMPOSSIBLE IMPOSSIBLE to conduct such a statistical experiment on IQ's RELIABLY. The number of VARIABLES are so many, so many that the entire experiment itself can be called to question. For example, in a poor African family, the first child can have good education. When the second one comes, the family may discover that they dont have money to send him to school too. This may cause the IQ difference. As another case, for the first child, the parents may have enough time to devote to his education. for the second child, they may not have the same amount of time - because there are two kids. This may cause the IQ difference.

My dear Nielsen . Have you not heard of the term "ALL OTHER THINGS BEING EQUAL"?. For the experiment to be STATISTICALLY correct, the experimenter MUST have demonstrated that there was NO OTHER CAUSE that was interfering with the childrens education.

I remember carrying out a simple statistical experiment. Tossing bread into the air and determining whether there was a better chance that bread landed butter side down. One guy just took the bread and tossed it into the air several times and wrote the results. The statisticians rejected his experiment as FALSE and gave him a POOR GRADE. Why?. Because he may have unconsciously tossed the bread with DIFFERENT FORCE each time he conducted the experiment. The rest of us had to build mechanical devices to toss the bread. This is so that we can reliably demostrate that we had nothing to do with the force with which the bread was thrown and it was the same each time. This should give an idea about how rigorous statistics is. You simply cant draw a conclusion without demonstrating you took every care that ALL OTHER THINGS WERE EQUAL. To prove that every other thing was equal in the case Nielsen talks about is impossible. He should have been sitting with each of the parents and making sure everything was equal.

JUST TALKING TO 20 PARENTS ADN THEIR KIDS AND MEASURING THEIR IQ'S IS NOT, I REPEAT IS NOT NOT NOT NOT A STATISTICAL EXPERIMENT

Karthik
Monday, March 8, 2004

Do you read the following paragraph? He basically explained the IQ study was worthless. So what is your point again?

DJ
Monday, March 8, 2004

I'm betting Karthik was kid #12.

Kyralessa
Monday, March 8, 2004

<<He basically explained the IQ study was worthless>>

Yes. He is using a worthless study to damn statistical usability. He is extrapolating the IQ study to imply that statistical usability may not give the right results.
That is erroneous. I am not saying that Nielsen wrote that the usability study was right.

Karthik
Monday, March 8, 2004

My point is: Nielsen is trying to imply that the IQ study is a statistical one and it is not.

Karthik
Monday, March 8, 2004

Actually, Nielson says that the IQ studies are wrong, by excluding information about parent IQs; Which seems to indicate that he thinks the studies are not statistically valid.

If Nielson bothers you; then don't read his columns.

Bleh
Monday, March 8, 2004

That's my feeling, too. Nobody makes you read him. Like those people who bitch about Howard Stern. I don't like him, so I don't listen to him. That's good enough. Removing him from the air because you don't like him just makes you an asshole who can't play well with others.

Brad Wilson (dotnetguy.techieswithcats.com)
Monday, March 8, 2004

If Nielsen doesn't trip your trigger. Pick some other self appointed web guru that does.

Mike
Monday, March 8, 2004

Karthik,

Are you the youngest in your family?

Cletus
Monday, March 8, 2004

<<Grin>> I seem to getting a lot of fire for this. Let me explain.

No i am actually kid #1. My main complaint is that Nielsen is in a position of influence. There are a lot of hard working, conscentious people in the usability area/statistics. By simply denouncing it as an inexact science using examples that are totally wrong, he is giving it a bad name.

Nielsen is right when he says that the IQ tests are wrong. Where he errs badly is in assuming that it is a correct statistical test. No statistician worth his salt will ever consider it as a statistical test.

Its something like this . We all know Quackery is wrong. Nielsen uses the example of a quack study to say that statistics is itself an inexact science.

Its not. I have heard of dozens of companies who use and profit from such studies.

Karthik
Tuesday, March 9, 2004

> <<Grin>> I seem to getting a lot of fire for this. Let me explain.

Of course. This is the JoS forum that you're talking about, people here are very cult personality oriented and Nielsen belongs to their particular pantheon of deities: Spolsky, Greenspun, DeMarco, Fowler, Don Box, etc...

If you bitch about a guru: prepare for the Gulag.

your address is never revealed
Tuesday, March 9, 2004

No. Neilsen is one of the ones who's a bit doubtful. Same with Fowler.

Tuesday, March 9, 2004

Nielsen is an ignorant.

Everybody with a bit of common sense and some experience using Windows apps and doing a bit of tech support knows as much as Nielsen.

Jaxxxon
Tuesday, March 9, 2004

I think Nielsen is right on.

The reason IQ declines by birth order is the older kids beat the brains out of the younger ones. QED.

anon
Tuesday, March 9, 2004

I would have thought the more probable explanation would be that the first born interacts solely with adults, whereas the second one is interacting with his elder sibling.

Stephen Jones
Tuesday, March 9, 2004

I find it interesting that although Nielsen gives six justifications for his conclusion that most usability studies should not focus on quantitative results at the expense of qualitative results, you mention only the third in coming to your conclusion: "from this, he goes on to dismiss statistics in usability completely".

So you ignore his other five points and in refuting the only point that raises your ire you don't even address his comment.

Personally, I think he could have added a seventh justification, the justification that most usability studies cover phenomena that have so many variables that it is impossible to conduct statistical experiments reliably (or your point, I believe...).

Phibian
Tuesday, March 9, 2004

<<So you ignore his other five points and in refuting the only point that raises your ire you don't even address his comment.>>

Not really. I did want to refute some of the other. But unless he supplies the data, it’s difficult. The example I chose was the silliest, but was sufficient to expose Nielsen. He was using an absurd study (he himself says its absurd) to prove that a legitimate branch of usability was difficult. He confuses collecting a bunch of numbers from a bunch of people with statistics !.

Difficult to understand why people want to defend what cannot be defended. There are reasons against quantitative analysis, but "difficult" (Nielsens phrase) is not one of them. Its like saying theoretical physics is difficult. So please avoid it. Theoretical physics is difficult only if you have not bothered to study it. For a practitioner of physics, it may not be so difficult. For a properly trained usability consultant, statistical usability may not be difficult.

Come to think of it. Statistical Usability is just like C,C++,C#,Visual basic. You need to study it, understand it and learn how to apply it. Will not a C++ professional laugh at a Visual basic programmer when he (VB guy) says that C++ is "difficult"?.

Judging from the example Nielsen said about usability, I doubt he knows much. What has he studied?. Statistical usability requires formal study. If you interview 100 people
and gather a bunch of data, its as much statistics as sphagetti.

A statistician would never cite the example he cited to prove that statistical usability is difficult.

I had a professor once. Absolutely top notch at statistical usability and worked under him for some time. While personally i am a novice, my professor used to teach us how these "Gurus"- often fool laymen who have little experience in statistics. He used to give the example of Taguchi- who borrowed an example already existing and rebranding it as a novel thought. You cant make out Taguchi is rehashing old stuff if you dont knw statistics.

So having seen valid practitioners, its difficult to see a statistical usability quack like Nielsen use a totally erroneous example to damn statistical usability.

Karthik
Tuesday, March 9, 2004

For all your ranting I think you are the one that is being illogical. It did not seem to me that he dismissed quant statistics completely. He merely says most people will get more bang for your buck from qualitative studies. This makes sense to me.

His example of the butterfly ballot was very good. Testing with 100 users would not have revealed a statistically siginificant problem, but people using common sense might have picked up on the problem with those ballots.

NathanJ
Tuesday, March 9, 2004