Does commissioning a study really indicate bias?

Obviously a topic near and dear to my heart. [grin]

Trying to stay strictly objective, what is your opinion on the idea that when a company *commissions* a study from a reliable independent source it will affect the outcome of the study?

I'm not questioning the potential for bias when a company *conducts* the study, or when the study is conducted by a traditionally biased source (for example, "Slashdot Study Finds Linux used in 99% of Households" or ".Net Weekly finds no evidence of Java in business"); but let's say there's a think tank that has historically been even-handed. If they release a study commissioned by MS that shows some positive result of MS tech, is it really reasonable to think that a company that builds its business on objectivity would throw it away on one study?

The big caveat is that it's possible the company that commissioned the study retained release rights, which means if the study comes out negative it won't see the light of day; but that doesn't affect the reliability of the released study.

Just looking for some opinions. Or a flamewar. Your call.

Philo

Philo
Saturday, May 8, 2004

"Trying to stay strictly objective, what is your opinion on the idea that when a company *commissions* a study from a reliable independent source it will affect the outcome of the study?"

The way I see it is this:

The reliable "independent" source is a business.  In the case of software developers (lets say our favourite target: Microsoft) they are looking for a study they can use in advertising.  Now, If the study isn't positive, the client CAN'T use it.  I submit that it's not good business to produce a product that is useless to your clients!

For an independent source to produce a positive report doesn't require a lot of corruption.  Slightly changing the methodology of the study and the variables investigated can alter the results substantially.  And it's easy to know, ahead of time, what methodology and variables might be favourable to your client.  And it's obviously good business to do just that.

"The big caveat is that it's possible the company that commissioned the study retained release rights, which means if the study comes out negative it won't see the light of day; but that doesn't affect the reliability of the released study."

You can commission 5 different studies on the same subject (say TCO) and they'll all likely have 5 different results.  Now lets assume that 4 are unfavourable and 1 is favourable...  which one will be released?  Is that study reliable?  Standing alone, it probably is reliable.  However, Compared to the 4 that didn't get released -- you could question that study's methodology.

Almost Anonymous
Saturday, May 8, 2004

Yes.  Rampant cynicism and the polarization of the vocal minorities on either side of an issue guarantee that any commissioned study would be widely viewed as biased.

Doug
Saturday, May 8, 2004

Any company that commissioned an unbiased study would surely be open to a lawsuit by shareholders :)

As you have pointed out, the  companies normally get round the problem by not releasing studies that are unfavourable.

The basic problem though, is that a company that takes on a commission is only doing so because it needs the money, and it knows full well that it is more likely to get repeat custom if the first report is favourable. Rmember that years back auditors such as Arthur Anderson were considered to be impartial.

My favourite story regarding this comes from more innocent times. There was an English Lit academic who had devised a theory that claimed that you could identify an author by the average number of syllables he used in a word (or maybe the average number of words in a sentence - I can't remember how).

He was given a very large grant by the Bacon society to go through all of Shakespeare's plays together with the works of the various candidates for having written them, including Marlowe, Bacon the Earl of Essex, Elizabeth I, Ben Jonson. After a couple of years research (for this was done in the sixties when computing power was not what it is now) he concluded, as the Bacon society hoped, that Shakespeare's plays weren't written by Shakespeare; the problem for them was that he came to the conclusion they were all written by Christopher Marlowe!

(As Christopher Marlowe had died some twenty years before Shakespeare's last known play was written this did present one or two niggling doubts - but the guy had been so generously reimbursed by the Bacon society, that he could spend the next three years writing his book to prove that Marlow hadn't really died at all but instead had hidden in Ann Hathaway's second best bed. Admitedly, he never got round to explaining how the author of "Dr. Faustus" could have followed it up with such absolute drivel as "Titus Andronicus" or "Henry V!", but that is another matter).

Stephen Jones
Saturday, May 8, 2004

There is definite bias.  When somebody pays your company $5 million, are you going to tell them that their product sucks compared to the competition?

They also don't have to worry about losing their reputation for objectivity, because intelligent folks will assume bias anyway, and the rest will soak up the study results as if they're 100% true.  No loss either way.

Those that take objectivity seriously are those that refuse to run studies on companies that are paying them.  Like Consumer Reports magazine which has no advertising.

NoName
Saturday, May 8, 2004

There is also the issue with the what and how of what is being studied. For example, small changes in frasing can alter the outcome of a poll pretty drasticly.
So, the company ordering the study can very well imprint its bias on the study through the requirements.
This is something that even the natural sciences have to struggle with, and they dont have anywhere near as much room for opinion or nebulous concepts like TCO.

The best way, IMHO to find out if a technology suit you or not is to find others who have done simmilar things and talk to them yourself.

Eric Debois
Saturday, May 8, 2004

> somebody pays your company $5 million

If objectivity is desired, I wonder whether it's possible to conceal the client: the client uses a lawyer to hire a research agancy, and the lawyer doesn't disclose who's paying for the study: like the "blindness" people are supposed to use in clinical trials.

Christopher Wells
Saturday, May 8, 2004

Clinical experiments depend on having a control group for comparison. I think the studies philo is refering to are more in the realm of social studies or something, where you cant really create two separate groups and just test your theory on one of them.
Blind study means that the subjects are unaware of which group they belong to.
Double blind means that the researchers or people leading the experiment are not aware of this either.

It depends an awful lot on what exactly you are investigating, but its generally hard to design blind studys out side of a lab.

Eric Debois
Saturday, May 8, 2004

What is the purpose of the study?  Is it to convince people that Company X's product is superior to the competiton?  If so, and people find out that Company X commissioned the study, does it really matter if it is objective?  The damage is already done.

Scot
Saturday, May 8, 2004

What damage? 

Microsoft has commissioned lots of studies like this in the past and they still do it, don't they?

I think they're just aware that initially there will be a more or less loud backlash of public opinion when it comes out that the study was commissioned, but that the MS-favorable-results will be out there forever and the backlash over the commissioning will blow over in a relatively short time.

Herbert Sitz
Saturday, May 8, 2004

I think the PUBLIC doesn't necessarily see the study as biased.  I think that you really need to lead them by the nose and say "look, study XYZ was COMMISSIONED by ABC and it just HAPPENS to show that ABC is good. Seems biased, eh?"


I work with medical professionals and even THEY don't see a study as biased  (necessarily) if it was comissioned by the company the study benefits.

HOWEVER, *I* personally would think there's a strong incentive to be biased,f or the reasons stated by other posters above.

Mr. Analogy
Saturday, May 8, 2004

right, this is a common trick done by software companies (microsoft is an example), drug companies (to the point where they get people to do 'presentations' at conferences, where the entire content of the presentation except the presenter's name is provided by the drug company), other science-related grant issues (no research is done into the dangers of cell phone radiation because noone will pay for it) (the current administration is also apparently pulling funds for papers which disagree with the govt's desired results).

so yes, a commisioned study is almost certainly biased, especially if it's released. but people still believe these studies are independent, so the process will continue.

otoh, a study for internal use only may not be biased.

mb
Saturday, May 8, 2004

The answer is that of course commissioning a study will involve bias. Mainly it will be in selecting the research firm.

This is not wrong. It's just democracy. Microsoft will be careful as to which firms it selects, and so will IBM and Red Hat. The research firms know they have to steer a delicate balance between avoiding reports that are obviously biased, and losing future work.

The accounting firms are the worst examples of this. They will do anything for a few million. Look the other way while management robs the company then issue an audit certificate, and so on.

Saturday, May 8, 2004

Philo, are you Bill Gates?

Li-fan Chen
Saturday, May 8, 2004

Could you give an example of the kind of study we're discussing? I think that would help make things a little more concrete.

Without seeing an example, I might believe that there are always 100s of little details to get wrong... and it may be that more errors are made which favor the client.

Tayssir John Gabbour
Saturday, May 8, 2004

Re: Bill Gates as Philo,

It turns out that Bill Gates is indeed Philo, and also responsible for the posts of half of the regular posters here.  The other half are aliases for Joel Spolsky, including all the 'employees' of his 'company'.  The sporadic technical questions are created by an advanced AI project--in its goal of passing the Turing test, it has attained a milestone of the project, to manage to masquerade as a non-native English speaking teenager.  So far, the AI has fooled everyone. 

Deleted posts are not deleted manually.  Instead, they are submitted to the FBI's Carnivore program, and are deleted if they receive a sufficient ranking.  All threads are immediately deleted if they use the phrase 'invoke Godwin's law' TWICE.

Questions submitted to 'Ask Joel' take some time to answer because they have all been outsourced to India.  As the cost of outsourcing increases, and the pendulum of public opinion sways, Joel is considering moving Q&A back in-house, especially after the incident.  You remember, right?  The one where, due to a small paperwork mixup, one of his contractors answered a question about Linux by posting his tax return forms.

Joel's signature, which states that he is a Google shareholder, is appended to the end of every twentieth post.

While we're here: it turns out that Robert Scoble is inhuman.  I don't know exactly what he/(it?) is, but no human being is able to post forty entries a day, every day of the week, for as long as he has.


Have a nice evening.


Pete

pds
Saturday, May 8, 2004

Doesn't necessarily indicate it, but it is sure implied.  It makes it look like the company funding the study bought the results.

No one I know takes the Microsoft "Get the Fact's" campaign seriously.  Get the FUD would be a better name.

Mike
Saturday, May 8, 2004

Actually outsourcing provides the classic proofs of this.

McKinsey's been doing work for Nasscom for years (and would like to continue doing this.) Nasscom asks them to do a study on offshoring. Guess what? Sending jobs to Indian creates jobs in America and makes everyone richer.

Gartner gets lots of business for Indian outsourcers and has a growing office in Mumbai. They do a study on offshoring. Guess what? Offshoring benefits everyone.

.
Sunday, May 9, 2004

Actually the process of creating a non-biased study (within reasonable limits of course) is very well known and pretty much understood. (<g> I only wish I could remember it a little better)

put very roughly its:

(0) Decide as exactly as possible what the study is going to investigate

(0.5) Decide on a set of questions that are as 'neutral' as possible and do as little as possible beyond extracting the pertinent info.

(1) Select the participants of each group as randomly as possible from within the groups of interest (in the case of a MS TCO stdy it would presumably be experienced Linux Admin people vs experienced Windows Admin people)

(2) Put the questions to the selected groups and faithfully record every answer

(3) Record, look at, view and play with the gathered data until some conclusions about its meaning can be found

(4) publish the conclusion, the raw data, the questionaire and the process by which the people were found.

Thats pretty much it, anyone who doubrs the conclusions can check out the actual data and if necessary point out the possibility of a different conclusion.

If no one can do this persuasively then the conclusions are accepted.

The chances of (to pick a company at random) Microsoft using anything even vaguely approaching this method are nil...they have nothing to gain and everything to lose.

FullNameReqired
Sunday, May 9, 2004

A survey or study commissioned by any body, governmental or corporate, might result in a biased result.  More likely is that those findings which support whichever bias is wanted are emphasised by the sponsoring organisation.

For instance, a margarine manufacturer may well emphasise results in an independant study that support the consumption of margarine as a factor in reduction in hair loss, whilst at the same time ignore or downplay parallel findings that overconsumption of margarine contributes to impotency.

Whichever, the consumer of such reports should always treat subsequent spinning of them by other organisations as exactly that, spinning gold from straw.

Simon Lucy
Sunday, May 9, 2004

"Philo, are you Bill Gates?"

?

Where does that come from on *this* thread? IMHO it's a perfectly objective, neutral question. (May I point out that IBM, Redhat, and Oracle all commission studies as well)

Philo

Philo
Sunday, May 9, 2004

FullName, your description is for a *survey*, not a study. Different things.

.
Sunday, May 9, 2004

"*commissions* a study from a reliable independent source "

The word "independent" is absolutely innapropriate to that place in the sentence.  Reliable?  OK, maybe.  Less-overtly-biased?  Fine.  Independent?  That's a joke.

What's funny is how many folks in the IT world are completely oblivious to these things.  I love hearing things like "so-and-so at VendorX tells me that the VendorX-a-ma-bob is the fastest and most reliable thing-a-ma-bob available for our situation, so I think we should choose that."  [Overheard (surreptitiously) from an actual tool-selection meeting consisting of a dozen managers.  No irony sensed.  No animals were harmed, nor intelligent people involved, in the above mentioned activity.]

raga
Sunday, May 9, 2004

Hope you are not commissioning your thing with Directions on Microsoft folks.

Jason
Sunday, May 9, 2004

Let's put it this way -- if you could get a bunch of whistleblowers like Feynman, Nader, and maybe Chomsky, and they all could write their thoughts on how the study was conducted, as well as what they could do better next time, then I'll agree that's a pretty damn good "independent" report.

(Yes I know Feynman's dead, substitute someone who's willing to piss people off in pursuit of what they think is the truth.. but with a scientfic background.)

Tayssir John Gabbour
Sunday, May 9, 2004

"your description is for a *survey*, not a study. Different things."

of course they are different things.  that is still a good basic blueprint for a study. or, if not, perhaps you would like to provide your own?

FullNameRequired
Sunday, May 9, 2004

It's easier to do, and certainly to describe, an unbiased survey than an unbiased study.

A survey is like a scientific experiment, although of course the choice of questions can sway results. But a "study" is an interpretation of facts, views or whatever, and interpretation by definition will be subject to the interests, expertise and bias of the researchers.

.
Sunday, May 9, 2004

"But a "study" is an interpretation of facts, views or whatever, and interpretation by definition will be subject to the interests, expertise and bias of the researchers."

which is why its so important to publish the raw data along with the conclusions and to ensure that the process was as provably neutral as possible.

The idea isn't to get total agreement on the results, nor to stop people disagreeing with your results.  The idea is to get agreement that your conclusions are reasonably based, and to ensure that only the most maddened fanatics are accusing you of hiding the real facts.

I really dont see what you are blathering on about?

FullNameRequired
Sunday, May 9, 2004

FullName, the selection of which "facts" to investigate, let alone report on, can decide the outcome of a study, even if subsequent interpretation is as straight as a die.

This happens all the time. Governments have favourite research firms that they know will deliver the outcome they want, because of shared interest. Businesses do too. And for that matter, so do social activists.

Sunday, May 9, 2004

" the selection of which "facts" to investigate, let alone report on, can decide the outcome of a study, even if subsequent interpretation is as straight as a die.
"

well, duh.

thats why you make all the decisions public.....

ah, forget it, its not like it matters.

FullNameRequired
Sunday, May 9, 2004

We often look for a reason not to trust something
as reasonable heuristic in a complex world.
A quick and easy rule is not to trust  a
favorable commissioned study. Assuming trust is
clearly a bad policy. After that trust is very difficult
to establish, so it's easier to move on. Tribal affiliations
change everything though.

son of parnas
Monday, May 10, 2004

Unbiased study? Do these things exist? Surely at least in the case of a commisioned study the presumed direction of the bias vector is clearly stated, and as a consequence the results might actually be less biased because one would have to be more carefull.
The most biased studies are probably going to be those done by "volunteers" or "independants". Since they do not have to cover for the obvious bias vector, they are free to manipulate at will.


Naaahh. I'm rambeling. No?

Just me (Sir to you)
Monday, May 10, 2004

It is of course possible to have an unbiased study.  However I'm not sure how you can show a lack of bias if the study is favourable to the organisation who commissioned it.  Catch 22 methinks.

a cynic writes...
Monday, May 10, 2004

  Recent Topics

  Fog Creek Home