The Old Joel on Software Forum: Part 4 (of 5)

Trusting any reviews

In an earlier thread the poster expressed dissatisfaction with the state of reviews on Amazon. I agree and I don't think this phenomenon is limited to Amazon. I used to hope the web would help people decide what to buy based on the free flow of information but once people caught on to this idea the guerrilla marketers ruined it. For stuff one has a continuing interest in, such as programming, he can resort to a forum such as this and take opinions of regular posters.

For one time purchases though, this doesn't work. I see a problem waiting for a solution. Surely the combined brain power of this board can come up with a solution.

How does one design a website that allows people to review products they use while minimizing attempts to manipulate the system?

name withheld out of cowardice
Monday, February 9, 2004

For one-time purchases, I tend to use cnet.com. I do note that Amazon has a quasi-karma system in place which lets you vote on whether a review is useful or not, but it's still not all *that* useful. The primary problems I see are (a) gathering a critical mass, (b) making it trivially easy both to write your own review and to rate reviews others have written and (c) overcoming the natural human tendency to not bother reviewing something unless you thought it sucked. ;>

Sam Livingston-Gray
Monday, February 9, 2004

Isn't CNET though just the professional reviewer concept that we have had in magazines for years? I'm also not just talking computer stuff. When Iwant to buy a car Iread car magazines. When you want a stereo I read Stereophile or Absolute sound. If I want a new kitchen gadget what do I read?

My hope was that there would be a web site. There are such sites like ePinions but don't they fall victim to Guerilla marketing? Is there a process that could reliably eliminate this?

name withheld out of cowardice
Monday, February 9, 2004

You aren't going to find a useful kitchen appliance review site, because intelligent people don't take the time to write reviews of their toaster.

Monday, February 9, 2004

I have found reviews to be very helpful on various websites. For example, when I want to buy a consumer electronics, I will read reviews, if available, on CircuitCity's site (though would never buy it there). I have found very honest feedback and it has directly resulted in my decision to buy or not buy the product. Maybe we are just not looking in the right places or expecting too much from online reviews.

Amazon reviews are interesting in that they can often lead to other books. Humans have opinions, they share them, they are flawed.

m
Monday, February 9, 2004

"If I want a new kitchen gadget what do I read?"

I usually look at Cook's Illustrated (http://www.cooksillustrated.com/) or Fine Cooking 9http://www.cooksillustrated.com/). Hope this helps. :)

Rich
Monday, February 9, 2004

or www.egullet.com

mb
Monday, February 9, 2004

Google and others are trying to push through full-text search of books. Unfortunately publishers seem to resist this with their technical books, even though it's really the way to break free from untrustworthy reviews.

(There are counterexamples to their fears. Paul Graham's _On Lisp_ goes for ~$230 used, despite being absolutely free online and a reprint in the works. The market speaks. Also, Safari may have counter data too. I can't access their sales info, but it looks like a lot of publishers are signing up despite the IP theft potential.)

In the earlier thread, Gareth mentioned finding reviewers who share tastes. It worked for me too; authors aren't building accounts with long histories and character (yet ;) so it's immune to the throwaway account problem. Though if Amazon's actually editing stuff heavyhandedly, many bets are off.

There's something I did to ascertain the trustworthiness of a used book seller... I'd ask a question like what edition/printing they carried, and all I'd care about was the manner of their response. (Which was great for them because I'd do this with newbies, and my glowing recommendations immediately led to a bunch of sales and recommendations.) You can probably convince an author in a similar way to make excerpts available; most good authors have nice websites full of writing and excerpts anyway. It's always in their interest.

Also, check out Apress's forums; this is probably a big issue with them and you could have a great dialogue.

Tayssir John Gabbour
Monday, February 9, 2004

I think knowing the reviewers is key, so what a site would most need to do is to make it easy to get to know, or at least know about, the reviewers. Provide photos (of course, they needn't be real), easily-accessible links to lists of all their reviews, some way to communicate with them.

Add something like Amazon's "how useful was this review to you?" feature -- but make it possible to tell who voted which way for each review. (100 guerilla marketers all voting for one another? Boring.)

Every user has (perhaps not explicitly) some measure of trust in every other. You can say explicitly that you do or don't trust someone else's reviews, or the system can deduce the fact from (1) your "how useful?" judgements on their reviews, (2) how well their ratings and yours -- of products and of reviews! -- agree, and maybe (3) whether they are vouched for by other people whom you trust.

Some way of telling the Powers That Be that you suspect so-and-so of being a marketer in disguise. Enough complaints about someone, and they get investigated by a real human being, and kicked off the system if they're thought to be abusing it. (There's clear danger of meta-abuse here.) People making bogus complaints about others are of course noted for future reference... :-)

An annotation facility, so that you can comment on specific parts of someone else's review. Annotations get rated too, and you don't see ones that the system thinks you find unhelpful. (Or maybe you see them, with greatly lowered probability, so that eventually you have the chance to pass judgement directly.)

Cross-references. "If you liked this, look at that". Lists. "Here are some books I think are particularly good." All, of course, identified by their creators, and therefore all trusted by you to a variable extent.

Many good reviews will not be on this site. So, define a mechanism for people to export reviews they post elsewhere, lobby to get that mechanism used (e.g., by Amazon, newspapers' review columns, etc), and make the site act as an aggregator for sources of reviews that you (i.e., any particular user) like. Some care is needed to prevent abuse of this feature; trust in external reviews probably shouldn't be transitive at all.

But, frankly, I don't think any of this will work. Reviewing something requires too much effort for many people to want to do it; the barrier can be lower when the reviewing happens at the same place as the buying (As on Amazon), but that's exactly what we want to avoid. Oh, and who's going to pay for all this? Is there any possible answer that doesn't immediately make it much harder to trust the system?

Gareth McCaughan
Monday, February 9, 2004

I no longer trust Amazon's top 100 reviewers, I think most of them are paid to write their reviews, and I'm sure there are plenty of reviews for products that they have no personal experience with.

I agree that the "People who bought __ also bought" is good. Typically since it involves money (buying two things) most people won't do it. The problem is, obscure items sometimes don't have much in the way of correlated purchases, or if they do it's The Lord of the Rings and The Pirates of the Carribean because some items are purchased by just about everyone and can throw the recomendation system off.

I'm going to guess that any system can be subverted by someone determined enough.

One great thing about Amazon is that each reviewer has a homepage, and you can add them to your list of reviewers to keep track of. This way, when you find someone who writes a review you trust, you can keep track of them.

All the lists, "Listmania," and "So You'd Like To..." have obviously be subverted. My favorite is to find a listmania list that's all books by 1 author, or all 1 brand of product, with maybe 1 other product thrown in so they can cross advertise.

And on Amazon you can see who your favorite reviewers have added as their favorite reviewers and so on, so there you go it's Amazorkit. Sure this system has been subverted too, but you have to use your judgement.

www.MarkTAW.com
Monday, February 9, 2004

Speaking of trusting reviews, check out this IMDB forum thread about Dirty Dancing 2 (don't ask, let's just say my being there is an incident that involved a lot of tequila, oh and my girlfriend loved the first movie)

http://www.imdb.com/title/tt0338096/board/nest/3772691

Anyway, you'll find 3 posts - the only 3 posts that aren't negative - and they ALL say "Go to Apple.com to view one of the three sizes of trailers (In Quicktime TM Copyring Patent Pending Only)"

Talk about your geurilla marketing.

www.MarkTAW.com
Tuesday, February 10, 2004

> Copyring

Copyright... Grrr, why can't I type lately?

www.MarkTAW.com
Tuesday, February 10, 2004

Blame the geurillas

Firefox
Tuesday, February 10, 2004

Consumer Reports magazine ( http://www.consumerreports.org ) was set up long ago (at least 30 years) to do unbiased testing and reviews of consumer products.

Cubist
Tuesday, February 10, 2004