Further Evidence That Wine Tasting Is Wildly Subjective

(Photo: Tony Alter)

A few years ago, we did a podcast on whether expensive wine tastes better. There is now further evidence that the answer to that question is no — even for elite wine critics. Winemaker Robert Hodgson recently collaborated with the California State Fair wine competition on a little wine-tasting experiment:

Each panel of four judges would be presented with their usual “flight” of samples to sniff, sip and slurp. But some wines would be presented to the panel three times, poured from the same bottle each time. The results would be compiled and analysed to see whether wine testing really is scientific.

The first experiment took place in 2005. The last was in Sacramento earlier this month. Hodgson’s findings have stunned the wine industry. Over the years he has shown again and again that even trained, professional palates are terrible at judging wine.

“The results are disturbing,” says Hodgson from the Fieldbrook Winery in Humboldt County, described by its owner as a rural paradise. “Only about 10% of judges are consistent and those judges who were consistent one year were ordinary the next year.

“Chance has a great deal to do with the awards that wines win.”

(HT: The Daily Dish)

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 11


  1. Enter your name... says:

    Presumably it would be more accurate to say that chance has a lot to do with which wines win, assuming that the wines aren’t awful. A flawed wine (e.g., corked or with green bell pepper flavors) probably doesn’t get entered into these competitions. Winning a medal might accurately indicate that the wine is not truly horrible.

    Well-loved. Like or Dislike: Thumb up 11 Thumb down 3
  2. Enter your name... says:

    I wonder if they ever try things like adjusting the rankings based on the order in which wines are tasted. If, like election ballots, the first or last has a higher or lower value in general, then you could statistically correct for that.

    Thumb up 5 Thumb down 4
  3. Skeptic says:

    Amazing! Who would have thought that humans swishing liquid around their tongues can’t be considered scientific instruments?

    The influence of music is interesting too.

    Well-loved. Like or Dislike: Thumb up 9 Thumb down 0
    • Bob says:

      Agreed. I’d file this paper under “obvious”, but I know too many people who will probably refuse to believe it.

      Well-loved. Like or Dislike: Thumb up 7 Thumb down 0
  4. chris says:

    maybe we should turn the issue on its head and ask if the judges just distinguish the badly tasted wines and then ceteris paribus the winning wine award is a subjective blend of network, luck, subjectivity (taste) etc

    Thumb up 1 Thumb down 0
  5. D. Raymer says:

    A few years back I went to a wine tasting party where you each bring your favorite bottle. I’d heard that the manager of the local Trader Joes would be there, so I brought a bottle of their “Two Buck Chuck” (Charles Shaw, $1.99).

    It didn’t win, but scored right in the middle along side of $40 bottles, and the manager didn’t recognize his own rotgut!

    Thumb up 1 Thumb down 0
    • Mike says:

      To really determine if wine is priced according to quality don’t you have to do a take home test, give several bottles of each wine to drink over the course of weeks in typical wine drinking scenarios? What these studies seem to indicate is the way wine tastings are set up are poor.

      Thumb up 1 Thumb down 0
    • Enter your name... says:

      I’ve heard from a chef that each “wine” in the Two Buck Chuck line is actually several different wines, so that some bottles are just fine and others are not very good. It’s impossible to be consistent with that kind of volume. Unfortunately, there’s no way to know which batch is one of the better ones.

      Thumb up 0 Thumb down 0
  6. Sol says:

    Interesting phenomenon.

    I’m curious to know if anyone has done a similar study with whiskey. Specifically, I think this would be interesting with regard to single malt scotches where people are willing to pay more for certain bottles that they say are ‘better’.

    Thumb up 2 Thumb down 0
  7. MTT says:

    Ha! Did anyone read the whole article – they’ve got a nice reference in the last sentence:

    …Sadly, he didn’t explore how long it would take a monkey to type a wine review.

    Thumb up 1 Thumb down 0
  8. DST says:

    Interesting and entertaining, but ultimately not very convincing. If you were taking graduate level Research Methodology, you would have a lot criticism from the department. First be very specific about your subject. If it is the accuracy of wine tasting, be very careful not to stray. Beer tasting is not part of wine tasting, no reason to even mention one with the other. Next step, who are your experts? Chefs? Grocery Managers? A local wine connoisseur? Do you really think anyone considers them experts? I could give you my medical opinion, but if it were wrong would that discredit the opinions of trained medical doctors?

    Also, you need to match up their claims with your counter claims much more accurately. Find out just what it is they claim to be able to do that you and I can not. Write to your nearest Master Sommelier and see if they can point you to better resources for this. I can assure you they make pretty clear claims and you might be surprised how much they agree with you about nonsense included in judging wines.

    Also, you seem to be making a connection that isn’t there between wine prices and wine ratings. Bare Foot has many bottles of wines that are highly rated, have won medals, and sell for less then $5 a bottle. That is one of many that my local Master Somm points out each year that are priced low and rated very high. It is simply that large quantities are available, so demand does not raise the price very much. He also points out bombs, that include very expensive wines that are not very good. He is quite critical of very expensive wines.

    Finally, drop the smarmy comedy act. You are not John Stewart, and unlike John your comedy routine comes off as intellectual vanity. I can assure you that many in the wine industry are very well educated and very intelligent. You won’t be laughing if you have to debate one of them in public. Their knowledge of history and chemistry will quickly humble you. Remember, we are the skeptics so and we are the ones who are proponents of science, what good are we if we do not hold ourselves to the same standards as we do those we criticize?

    Thumb up 0 Thumb down 2