Further Evidence That Wine Tasting Is Wildly Subjective

(Photo: Tony Alter)

A few years ago, we did a podcast on whether expensive wine tastes better. There is now further evidence that the answer to that question is no — even for elite wine critics. Winemaker Robert Hodgson recently collaborated with the California State Fair wine competition on a little wine-tasting experiment:

Each panel of four judges would be presented with their usual “flight” of samples to sniff, sip and slurp. But some wines would be presented to the panel three times, poured from the same bottle each time. The results would be compiled and analysed to see whether wine testing really is scientific.

The first experiment took place in 2005. The last was in Sacramento earlier this month. Hodgson’s findings have stunned the wine industry. Over the years he has shown again and again that even trained, professional palates are terrible at judging wine.

“The results are disturbing,” says Hodgson from the Fieldbrook Winery in Humboldt County, described by its owner as a rural paradise. “Only about 10% of judges are consistent and those judges who were consistent one year were ordinary the next year.

“Chance has a great deal to do with the awards that wines win.”

(HT: The Daily Dish)

Enter your name...

Presumably it would be more accurate to say that chance has a lot to do with which wines win, assuming that the wines aren't awful. A flawed wine (e.g., corked or with green bell pepper flavors) probably doesn't get entered into these competitions. Winning a medal might accurately indicate that the wine is not truly horrible.

Enter your name...

I wonder if they ever try things like adjusting the rankings based on the order in which wines are tasted. If, like election ballots, the first or last has a higher or lower value in general, then you could statistically correct for that.


Amazing! Who would have thought that humans swishing liquid around their tongues can't be considered scientific instruments?

The influence of music is interesting too.


Agreed. I'd file this paper under "obvious", but I know too many people who will probably refuse to believe it.


maybe we should turn the issue on its head and ask if the judges just distinguish the badly tasted wines and then ceteris paribus the winning wine award is a subjective blend of network, luck, subjectivity (taste) etc

D. Raymer

A few years back I went to a wine tasting party where you each bring your favorite bottle. I'd heard that the manager of the local Trader Joes would be there, so I brought a bottle of their "Two Buck Chuck" (Charles Shaw, $1.99).

It didn't win, but scored right in the middle along side of $40 bottles, and the manager didn't recognize his own rotgut!


To really determine if wine is priced according to quality don't you have to do a take home test, give several bottles of each wine to drink over the course of weeks in typical wine drinking scenarios? What these studies seem to indicate is the way wine tastings are set up are poor.


Interesting phenomenon.

I'm curious to know if anyone has done a similar study with whiskey. Specifically, I think this would be interesting with regard to single malt scotches where people are willing to pay more for certain bottles that they say are 'better'.


Ha! Did anyone read the whole article - they've got a nice reference in the last sentence:

...Sadly, he didn't explore how long it would take a monkey to type a wine review.


Interesting and entertaining, but ultimately not very convincing. If you were taking graduate level Research Methodology, you would have a lot criticism from the department. First be very specific about your subject. If it is the accuracy of wine tasting, be very careful not to stray. Beer tasting is not part of wine tasting, no reason to even mention one with the other. Next step, who are your experts? Chefs? Grocery Managers? A local wine connoisseur? Do you really think anyone considers them experts? I could give you my medical opinion, but if it were wrong would that discredit the opinions of trained medical doctors?

Also, you need to match up their claims with your counter claims much more accurately. Find out just what it is they claim to be able to do that you and I can not. Write to your nearest Master Sommelier and see if they can point you to better resources for this. I can assure you they make pretty clear claims and you might be surprised how much they agree with you about nonsense included in judging wines.

Also, you seem to be making a connection that isn't there between wine prices and wine ratings. Bare Foot has many bottles of wines that are highly rated, have won medals, and sell for less then $5 a bottle. That is one of many that my local Master Somm points out each year that are priced low and rated very high. It is simply that large quantities are available, so demand does not raise the price very much. He also points out bombs, that include very expensive wines that are not very good. He is quite critical of very expensive wines.

Finally, drop the smarmy comedy act. You are not John Stewart, and unlike John your comedy routine comes off as intellectual vanity. I can assure you that many in the wine industry are very well educated and very intelligent. You won't be laughing if you have to debate one of them in public. Their knowledge of history and chemistry will quickly humble you. Remember, we are the skeptics so and we are the ones who are proponents of science, what good are we if we do not hold ourselves to the same standards as we do those we criticize?



This is from a Master Sommelier. A challenge? Set it up and test with a new wine pro group. This time with the true, masters of wine, either Master Sommeliers or Masters of Wine.


I'd be interested to see those results.


That’s actually surprising that you can fool so easily even experts. I wonder if it’s the same for beer? You can test it with current beer revolution in Europe


I have heard about this experiments that demonstrate that expensive wine is no different from cheap wine. My issue with these experiments is do they really simulate the "real world." When is the last time you drank wine and didn't know the brand...in other words, who in the "real world" does blind taste test.

I also heard about an real study done on brandy in a book I read. Brand A was losing market share to brand B and the executives didn't know why when brand A bet brand B in a blind taste test. In taste test where the brand A and brand B were tasted without being blind, people preferred brand B. Now when they put brand A in the bottle of brand B and vice versa people preferred brand A in the bottle of brand B. In other words, it was the bottle that made brand B tasted better. When the company made their own fancy bottle, they started to gain market share and bet Brand B.

So it is possible when people think they are drinking fancy wine, it taste better. And it maybe is not completely in their mind, maybe their taste buds get more sensitive. If this is true, is there any way to test it?