Am I Good Enough to Compete In a Prediction Tournament?


Last spring, we posted on Phil Tetlock’s massive prediction tournament: Good Judgment.  You might remember Tetlock from our latest Freakonomics Radio podcast, “The Folly of Prediction.”  (You can download/subscribe at iTunes, get the RSS feed, or read the transcript here.)

Tetlock is a psychologist at the University of Pennsylvania, well-known for his book Expert Political Judgment, in which he tracked 80,000 predictions over the course of 20 years. Turns out that humans are not great at predicting the future, and experts do just a bit better than a random guessing strategy.

Good Judgment is Tetlock’s latest project, an ambitious plan to put 2,500 volunteers to the test in a forecasting tournament sponsored by the U.S. government. The Good Judgment  research team includes Barb Mellers and Don Moore, with an advisory board of Daniel Kahneman, Robert Jervis, Scott Armstrong, Michael Mauboussin, Carl Spetzler and Justin Wolfers. The criteria for selection is for those “who have a serious interest in and knowledge about world affairs, politics, and global economic matters and are interested in testing their own forecasting and reasoning skills.”

Considering myself someone who fits this description, I signed up to represent Team Freakonomics in the tournament. I have an econ degree from the University of Chicago (and consider myself a decent tarot card reader) so, why not? Plus it pays $150 a year to answer some questions online, and at worst I could use a “random guessing strategy” that would give me pretty good odds in this game.

The entry process started innocently enough, with a survey designed to gauge one’s interest in forecasting. Then came a test of world knowledge, which was hard for a couple of reasons. First, Google searching isn’t allowed; additionally, the test is timed, so you’d barely have time for Google in any case. Second, the test questions asked for a range of how true a given statement might be. For instance: a certain country’s GDP is $X in a given year; how true is that claim? This made the exercise incredibly difficult, as I have enough knowledge to give an extreme answer – true or false – but not enough to give a more subtle one.

I also knew about “anchoring” (which Richard Thaler speaks about in our “Mouse in the Salad” podcast). My mind was probably playing tricks on me with answers that seemed right, but probably weren’t.

This whole process did a number on my self-esteem: if I didn’t know about the world here and now, how could I possibly predict the future? I ended up completing the survey in segments; luckily there were some LSAT-type logic questions at the end and a fun IQ-shapes game.

I know what I was thinking by the end: boy, this should definitely pay more than $150! And I’m not even in the tournament yet!

Thankfully, I got notice yesterday that I’ve been accepted into the tournament (phew!). So let the games begin! Now all that’s left is for me to decide on a strategy: random guessing or actually trying to predict the future. Given what Tetlock’s research shows, that’s a tough call.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.



View All Comments »
  1. Brendan says:

    It was a bit different for me. The timing on the world knowledge test was not a time deadline, instead it just tracked how long it took you to answer, I assume I could have taken as long as I wanted answering without repercussion. I still ended up rushing through it.

    …I think the directions said that there would be variation in the experience for many players. Good luck Bourree!

    Thumb up 2 Thumb down 0
    • Joshua Northey says:

      I was disappointed I never got a survey email. I am a bit of a polymath and extremely knowledgeable about the world and its history which are two of my stronger subjects.

      Good luck to the volunteers. It will be interesting to see the results.

      Thumb up 1 Thumb down 0
  2. James says:

    Why are you people so stuck on this “humans are not great at predicting the future” nonsense? In fact, humans have worked out a lot of very effective ways of predicting the future. Indeed, pretty much all of science and engineering are about making predictions of the future. When’s the next eclipse? Where will the artillery shell land? Will this bridge withstand a magnitude 8 earthquake?

    The problem is that you are looking only at predictions of things that are mostly not predictable (inherently, as in predicting weather more than about 5 days in advance), and claiming those predictions represent all predictions.

    Thumb up 5 Thumb down 4
  3. Eric M. Jones. says:

    I predict that all old chubby guys like me who have a great IRA will suddenly become irresistible to young nubile women–so much so that we will be forced to take multiple wives and dozens of concubines and be treated like kings and spend our remaining years being showered with gifts and attending pleasure orgies greater that those of the Caesars of Rome.

    Thumb up 3 Thumb down 1
  4. Impossibly Stupid says:

    “This made the exercise incredibly difficult, as I have enough knowledge to give an extreme answer – true or false – but not enough to give a more subtle one.”

    This makes no sense to me. If you know enough (or think you do) to give a definite answer, that should make things *easier*, not harder. Sure, you might consequently be wrong in a big way, but that is the price you pay for proclaiming your expertise.

    “if I didn’t know about the world here and now, how could I possibly predict the future?”

    Adopt a perspective that reduces the weight you give to the “here and now” issues. After all, perfect prediction would mean that current affairs were *already* in a predictable state at some point in the past. Never-you-mind about the impossibility of tracking it all the way back to some initial state!

    Thumb up 0 Thumb down 3
  5. Frode Grøtheim says:

    Haha, random guessing might be less taxing and better for your self-esteem. ;P

    Thumb up 2 Thumb down 0
  6. FirstInsights says:

    Humbly, I suggest that whatever is hoped to be predicted by this effort is predictably flawed as not all predictions are equal. For starters, some degree of subjective measure is required to determine the magnitude and charateristics of the outcome. Add the idea that the predicted outcomes often have a concern for time that may be eventual in nature and thereby prove absolutely correct at some point. So an arbitrary time constraint must be imposed restricting the possible observations of predicted outcomes, thus making the conclusions themselves arbitrary. Then there’s semantics…such fun.

    Thumb up 0 Thumb down 0
  7. TAMCNEIL says:

    Let’s simply predict that semantics will have a large role to play in the exercise making the conclusions predictably arbitrary.

    Thumb up 0 Thumb down 0
  8. ryan says:

    Seems pretty heavy to do it with global economies etc plus it will take forever. Why not try sports- the answers will be much more immediate?

    Thumb up 0 Thumb down 0