More Amazon Shenanigans?

We have posted in the past about Amazon.com reviewers — their motives, their celebrity, and even some reviewers who seem to game Amazon’s commenting system.

Much more recently, I blogged about a strange shakeup in the Amazon best-seller rankings. From the comments that followed, it appears that the Amazon algorithm wasn’t re-jiggered, and that the change had nothing to do with Amazon Prime and changes in shipping costs. Rather, the most plausible explanation seems to be, as several commenters noted, that Amazon was holding a big sale on inexpensive children’s books and that word of the sale caught fire on the Web, driving a ton of traffic (and sales) to Amazon.

One comment on the thread had nothing to do with the ranking shakeup, but instead concerned the familiar topic of reader reviews. It was written by a fellow who’d prefer to be known only as Richard, and it is really interesting:

I once did an experiment on Amazon by registering a dozen different accounts under different names and email addresses. Then I used each name to click on those “I liked this review” and “I didn’t like this review” buttons. I discovered a few interesting things.

I used a couple of the phony names to go to the top ten Amazon reviewers and systematically go through their reviews clicking mostly “I didn’t like this review” [buttons]. I didn’t realize before, but the people in the top ten are just fanatics about their reviews. They spend all their time on Amazon posting reviews and trying to cultivate positive votes. When it came to their attention that someone was giving them negative clicks each day, they grew dismayed. I got several emails to the email account set up in those names, asking what was going on and justifying the quality of their reviews. I responded that I didn’t like their reviews and had every right on a public Web site to give them negative votes. A few days later, ALL of my negative votes were removed from all the reviews of all the top ten reviewers, indicating to me that they complained to Amazon, which simply deleted all of my negative clicks.

With amusement, I noted that all of my positive clicks under that name remained visible. Apparently if I systematically dislike a group of reviewers and reviews, my votes get deleted, probably for some sort of abuse or something. But my positive votes continue to be highly valued.

Separately, I used another phony name to give lots [of] negative votes to another reviewer, just because I found his reviews pedantic and self-absorbed, and took a virtual dislike to him. After a few days, he too complained to Amazon, which promptly deleted my negative reviews (but why? my votes were cast in good faith, I really really hated his reviews, so I had every right to give them negative votes, that’s why Amazon gives us those little vote buttons in the first place). This time, I sent a note to Amazon customer service, asking why they kept deleting my negative votes but not my positive ones, and whether there is some numerical ceiling to the amount of negative votes we’re allowed to cast each day. A few days later, I got a saccharine note back advising me that Amazon values my input and my opinions and would never delete my votes, but sometimes they take a few days to tally up and appear on the Web site. If I would check back in a few days, surely my votes would register. I did so, but of course those votes never reappeared.

Anyway, after that little experiment in how the positive and negative review votes are rigged, it doesn’t surprise me at all that Amazon rigs their bestseller list.

If Richard is right, maybe the folks at Amazon read this paper by Judy Chevalier and Dina Mayzlin.


Robert

Amazon's reply to Richard's email may be a bit strange, but the fact that his votes were removed is not.

A bunch of fresh accounts breaking the general voting pattern (lots of negative votes on posts that had generally been voted positive) and all posting from the same IP and. That Amazon caught this fraud is (in my eyes) a sign of good health and shouldn't be labelled as shenanigans.

Dave Rivers

As a moderator in a forum on the web it comes as no surprise to me that web administrators would delete the negative reviews.

Too often, recent members on the web get involved in flame-wars in forums and blogs... they tend to be juvenile in character, and their content gets stomped out based on complaints by people who are regulars in the system.

What would be most interesting would be if the author would keep an account a long time, write REAL reviews himself (become a "valued contributor") and THEN cast negative votes.

Would his votes be rescinded then? Or would his contributions (even if negative) be valued?

Jason

@ #1: If that's the case, why wouldn't they take action against the reverse (fresh accounts dishing out mass positive reviews at will on a single reviewer?) What's to stop the top reviewers from creating fake accounts of their own and giving their real account tons of positive reviews? Sounds like nothing. I wouldn't be surprised if it's already happening.

Doug Sunshine

Sounds like Richard needs a girlfriend

Michael Henke

It is certainly reasonable to remove ALL "single-click" (like/dislike, without any commentary) reviews posted by multiple new accounts arising from the same IP address. Of course, such removal should immediately precipitate an email from amazon.com customer service elucidating the reasons for removal. This email should also include amazon.com's "customer review" policy. Transparency provides credibility.

Of course, in this case, only the negative reviews were removed, and I infer that Richard's commentary-laden review(s) were also removed. This action certainly casts reasonable aspersion on the credibility of amazon.com's reviews.

While those willing to offer their time and effort reviewing for amazon.com may be merely "enthusiastic", their incentives merit some attention.

Perhaps Dr. Levitt might be interested in analyzing those incentives?

Mary

How did the reviewers know the identity of the user who was giving them negative votes? Negative COMMENTS are marked with the identity of the person who left it, but Yes/No votes cannot be identified, unless I am mistaken. So I am not sure how the reviewers knew to contact Richard to complain.

Christine

Of course they delete negative reviews ... they are trying to sell the books, not discourage you from buying them!

Michael Henke

Responding to Mary, I imagine they can trace each click to an IP address, regardless of the account's username. I'm certainly not positive about this, but it seems logical.

I agree with Christine that positive reviews help amazon sell books (as it is interested in selling all authors carried), but consumers obviously value the reviews, else why would amazon have provided such infrastructure? These consumers might be less responsive to reviews that are "incentivized".

For instance, if collusion allows amazon to use the review system to shore up sales of certain products (notably, books it has too many of) in a "quid pro quo" manner, their reviews might not be as valuable an asset. While I haven't undertaken to examine how such collusion might occur, given its immediate benefits, I'm sure it is soluble.

Josh Millard

After a few days, he too complained to Amazon, which promptly deleted my negative reviews (but why?

Why? I'm going to assume that Richard wasn't thorough enough to run each of these fake accounts from a distinct IP address. If Amazon had, from his previous experiment, identified his IP (or IPs) as belonging to a problem user, they might well see the later experiment as just more grief from the same source. Certainly it's something we run into over at Metafilter, and that's just a few tens of thousands of users, which has to be peanuts to Amazon's reviewer-base.

If you're going to build a rating system as big as Amazon's, you're probably going to pay a lot of attention to the problems that come with users trying to game the system. Not all those problems are created equally, of course, and not all of them will strike Amazon as equally important to address, but by now I'd imagine their administrative playbook is pretty thick, and "single user running multiple accounts maliciously" has got to be in there.

Read more...

mat

Is "jiggered" pc?

Tom

Amazon desperately needs a system, or second review group, where only those whoe purchased the item through Amazon may review it. The HD DVD vs. Blu Ray is getting out of hand with reivewers giving 5 start to every release in one camp and 1 star to any release in the other, even if it is the same movie!

Political books run into the same problem. If you could filter reviews by those who had purchased, I would turn it on and never look back.
Tom

Mary

Michael,

It seemed that, according to Richard, the _reviewers_ contacted him after receive the barrage of negative clicks. I certainly can see how Amazon.com could IP trace the clicks, but the reviewers are just Amazon.com users and cannot do that.

Juan

I had a negative review of an MP3 player removed from Amazon. After only 2 months the player died, as I am an engineer I opened it and found the sweat had corroded the contacts. I wrote this in my comments and after a week it was deleted.
They do delete negative feedback.

Joe

Stephen,
Seems like the comments to this post refute it, once again.

Your blog post essentially says: Oops, I guess last time was my mistake! Commenters straightened me out. But what about *this* one, eh?

And again your commenters are straightening you out.

First, there's no way that Richard could have performed this experiment with the chronology he gives:
- The reviewers that he voted on can't tell what customers voted Yes or No on the "Was this review helpful to you?" buttons

- Even if the reviewers could tell, they couldn't have discovered his email address to complain directly to him

- I'm not sure how he could *tell* definitively that *his* votes, positive or negative, on any review were deleted or not deleted! When I vote on a review, go to another product, and then go back to the first product, the voting buttons look the same. There's nothing that says "You voted Yes on this" or "We have no record of your vote." Just the same "Yes" or "No" choices.

It doesn't seem like it's really an "experiment" when there's no real data collected. Richard can make no conclusion about his hypothesis based on observation.

Second, Richard's behavior with a bunch of new accounts had the pattern of fraud, so it makes sense that his votes would be treated as malicious. Again, not that he could tell one way or another what happened to his votes, anyway.

This is a pretend experiment that highlights nothing other than expected behavior on Amazon's part.

So, Stephen, back to you. You now have two recent blog posts with sensationalistic titles ("Earthquake!" "Shenanigans!") that, on further examination, turn out to be "Oh, never mind." Maybe a little more due diligence, or modesty in headlines, is in order?

Read more...

JC

I'm also curious how these reviewers would get the email from someone clicking those buttons. That's shenanigans!

coolrepublica

I never buy a book with too many positive reviews. I am always seeking out the negative ones. If the negatives reviews make no sense or the reviewers reasons for hating the books seem not founded in logic, I buy the book. If the negative review make sense and their reason for hating the book rings true to me, even if the book has a bunch of good reviews, I don't buy the book.

This techniques has yet to fail me. You can try it if you want.

Richard

Thanks for the comments, all. I'm the fellow who did the little experiment on Amazon. Just to clarify, the reviewers who commented back to me did so when I posted short comments on their reviews, which may have allowed them to identify me, although I actually had the (completely unprovable) feeling that they contacted Amazon with their concerns and Amazon identified me to them. I had that feeling because I received three or four comments from them over a day or two, and then no more. The comment fo me from Amazon came directly in response to my email to their customer service address.

As for the notion raised by one comment (Jason's), many reviewers certainly do create their own phony identities in Amazon so they can give their own reviews many positive clicks. I decided to run this experiment when I stumbled on the Top-100 Reviewers and, like any bored person who took some statistics in college, became fascinated by the dynamics of that list.

I discovered by observation that if two people have the same number of posted reviews, that the one who has more positive votes on her reviews is higher on the list than the other. In other words, for reviewers striving to climb painfully to the top of the Top Reviewers list, positive reviews give you an edge over the competition.

This phenomenon is easy enough to observe. You'll see somebody post a review of shaving cream or life savers or some other silliness, and then within three days you'll see it has eleven positive votes, and no negatives. Then, if you look at that same reviewer's other reviews, you'll see they all have an identical eleven positives and no negatives.

Of course, the most famous example where such "vote pumping" DOESN'T occur is the #1 reviewer, Harriet Klausner, who has been the subject of newspaper articles over the years amid suspicion that she is actually a consortium of readers, given her strangely eclectic tastes in books and her ability to read and review ten or more books each day. To my knowledge, Amazon has never released more info on Harriet, nor has she been interviewed or otherwise gone public. She's been the #1 reviewer, by a mile, for years and years.

I may be a naif, but all this made me a bit suspicious, so I set up my own phony identities to see what would happen in the reverse case, when someone gives a dozen negative reviews. My results as noted, are that serial negative votes are deleted whereas serial positive ones are not. I don't complain about Amazon's right to delete my serial reviews, I suppose I expected that to happen, it's their website. But I was amused that a) they didn't contact me before doing so, b) they didn't provide me a rational explanation afterwards, c) they only deleted negative reviews, not positive ones, and d) they continue to turn a blind eye to the pattern of positive fraud by top reviewers.

As Dave Rivers noted in his comment, it may have been more useful from a scientific standpoint had I kept careful records with dates and numbers of votes, so that I could quantify my findings. Alas, I didn't. It was more of a whimsical time-killer than a rigorous experiment. If someone wishes to replicate, and improve upon, my methodology, I would be delighted to read their results.

Read more...

Drew

This is an interesting phenomena in online businesses. Although I'm less savvy about the particulars of Amazon.com's reviews, I did notice something peculiar about reviews on Sears.com that relates to this discussion.

I purchased a Kenmore 70pt Dehumidifier that quit working on me nine months after purchase. After reading the reviews, I couldn't believe the number of negative reviews. (Why did I purchase it in the first place!) When I posted mine, a relatively benign negative review, it never appeared. This made me wonder, "maybe there are TOO many negative reviews to post?"

What was also interesting is the trend in the reviews over time. The highest ratings were delivered by those who had only recently purchased the machines, while the worst ratings by those who had owned their machines six months or more. The inluence of American consumerism is unavoidable -flocks of people writing reviews about objects they recently purchased, as if they were extensions of their being.

Kudos to Sears and their Kenmore branded dehumidifiers, truly the greatest disposable appliance put onto the market to date. According to the reviewers on Sears.com, many of them have owned and used dehumidifiers that lasted for 15 years (or more) until they purchased this most recent model. And despite the negative reviews across an entire year, Sears continues to sell this model at the same price with the same complaints. Brilliant.

Interesting ethical questions here.

Read more...

James

I hope he was going through a proxy or using some other mechanism to cover his tracks, otherwise his votes would surely look like outliers.

If amazon are anything other than completely useless I'm sure the first thing they did was fire up a little script that showed them how long he'd been a registered user for (probably not long), how many negative reviews he'd given out, and how his votes compared to the rest of the voting population. His experiment would be a lot more convincing if he shared his methodology

paulwesterberg

A friend of mine also noticed that amazon quickly deleted his negative review of a product. Amazon is in the business of selling things, they need to push the product they have and bad reviews drive down sales. You can't trust them because their business depends on sales and advertising.

Just like you cant trus't the freakonomics blog to post negative comments about the freakonomics book
such as the preface and introduction pretty much suck - like filler tracks on a CD, and the core of the book - 5 Whole Ideas! - chapters 2-6 are only 174pages which is way too short. Its like they ran out of good ideas but they still had pages to fill so they also included an Epiloge, Notes, Acknowledgments, an Index, and new for 2006 an Expanded Bonus Section.

I wouldn't be surprised if top positive "reviewers" at Amazon got "rewarded" with gift certificates, discounts free shipping etc.