Am I Good Enough to Compete In a Prediction Tournament?


Last spring, we posted on Phil Tetlock’s massive prediction tournament: Good Judgment.  You might remember Tetlock from our latest Freakonomics Radio podcast, “The Folly of Prediction.”  (You can download/subscribe at iTunes, get the RSS feed, or read the transcript here.)

Tetlock is a psychologist at the University of Pennsylvania, well-known for his book Expert Political Judgment, in which he tracked 80,000 predictions over the course of 20 years. Turns out that humans are not great at predicting the future, and experts do just a bit better than a random guessing strategy.

Good Judgment is Tetlock’s latest project, an ambitious plan to put 2,500 volunteers to the test in a forecasting tournament sponsored by the U.S. government. The Good Judgment  research team includes Barb Mellers and Don Moore, with an advisory board of Daniel Kahneman, Robert Jervis, Scott Armstrong, Michael Mauboussin, Carl Spetzler and Justin Wolfers. The criteria for selection is for those “who have a serious interest in and knowledge about world affairs, politics, and global economic matters and are interested in testing their own forecasting and reasoning skills.”

Considering myself someone who fits this description, I signed up to represent Team Freakonomics in the tournament. I have an econ degree from the University of Chicago (and consider myself a decent tarot card reader) so, why not? Plus it pays $150 a year to answer some questions online, and at worst I could use a “random guessing strategy” that would give me pretty good odds in this game.

The entry process started innocently enough, with a survey designed to gauge one’s interest in forecasting. Then came a test of world knowledge, which was hard for a couple of reasons. First, Google searching isn’t allowed; additionally, the test is timed, so you’d barely have time for Google in any case. Second, the test questions asked for a range of how true a given statement might be. For instance: a certain country’s GDP is $X in a given year; how true is that claim? This made the exercise incredibly difficult, as I have enough knowledge to give an extreme answer – true or false – but not enough to give a more subtle one.

I also knew about “anchoring” (which Richard Thaler speaks about in our “Mouse in the Salad” podcast). My mind was probably playing tricks on me with answers that seemed right, but probably weren’t.

This whole process did a number on my self-esteem: if I didn’t know about the world here and now, how could I possibly predict the future? I ended up completing the survey in segments; luckily there were some LSAT-type logic questions at the end and a fun IQ-shapes game.

I know what I was thinking by the end: boy, this should definitely pay more than $150! And I’m not even in the tournament yet!

Thankfully, I got notice yesterday that I’ve been accepted into the tournament (phew!). So let the games begin! Now all that’s left is for me to decide on a strategy: random guessing or actually trying to predict the future. Given what Tetlock’s research shows, that’s a tough call.


It was a bit different for me. The timing on the world knowledge test was not a time deadline, instead it just tracked how long it took you to answer, I assume I could have taken as long as I wanted answering without repercussion. I still ended up rushing through it.

...I think the directions said that there would be variation in the experience for many players. Good luck Bourree!

Joshua Northey

I was disappointed I never got a survey email. I am a bit of a polymath and extremely knowledgeable about the world and its history which are two of my stronger subjects.

Good luck to the volunteers. It will be interesting to see the results.


Why are you people so stuck on this "humans are not great at predicting the future" nonsense? In fact, humans have worked out a lot of very effective ways of predicting the future. Indeed, pretty much all of science and engineering are about making predictions of the future. When's the next eclipse? Where will the artillery shell land? Will this bridge withstand a magnitude 8 earthquake?

The problem is that you are looking only at predictions of things that are mostly not predictable (inherently, as in predicting weather more than about 5 days in advance), and claiming those predictions represent all predictions.

Eric M. Jones.

I predict that all old chubby guys like me who have a great IRA will suddenly become irresistible to young nubile women--so much so that we will be forced to take multiple wives and dozens of concubines and be treated like kings and spend our remaining years being showered with gifts and attending pleasure orgies greater that those of the Caesars of Rome.

Impossibly Stupid

"This made the exercise incredibly difficult, as I have enough knowledge to give an extreme answer – true or false – but not enough to give a more subtle one."

This makes no sense to me. If you know enough (or think you do) to give a definite answer, that should make things *easier*, not harder. Sure, you might consequently be wrong in a big way, but that is the price you pay for proclaiming your expertise.

"if I didn’t know about the world here and now, how could I possibly predict the future?"

Adopt a perspective that reduces the weight you give to the "here and now" issues. After all, perfect prediction would mean that current affairs were *already* in a predictable state at some point in the past. Never-you-mind about the impossibility of tracking it all the way back to some initial state!

Frode Grøtheim

Haha, random guessing might be less taxing and better for your self-esteem. ;P


Humbly, I suggest that whatever is hoped to be predicted by this effort is predictably flawed as not all predictions are equal. For starters, some degree of subjective measure is required to determine the magnitude and charateristics of the outcome. Add the idea that the predicted outcomes often have a concern for time that may be eventual in nature and thereby prove absolutely correct at some point. So an arbitrary time constraint must be imposed restricting the possible observations of predicted outcomes, thus making the conclusions themselves arbitrary. Then there's semantics...such fun.


Let's simply predict that semantics will have a large role to play in the exercise making the conclusions predictably arbitrary.


Seems pretty heavy to do it with global economies etc plus it will take forever. Why not try sports- the answers will be much more immediate?


My experience has been a bit different, but I am enjoying the experiment so far. The only question that has closed was one I answered almost randomly, and got wrong. With a couple of questions I've gone from being pretty certain in one direction to being pretty certain in the opposite direction after new information comes in, in an embarrassingly short period of time. Quite easy to sit in one's armchair and pontificate, quite humbling to be confronted with one's inaccuracies.


I was taken aback by how little I (thought I) knew about actual current affairs - too much reliance on being able to look up facts! I am curious to see how this plays out, though.
As an afterthought: any body have any idea on the gender breakdown of the participants?

Don Moore

Without intending it to be so, we found our sample to be male-skewed. So while we value everyone's participation, a special shout-out of appreciation to all the women who are taking part.


I'm also participating in this event, and so far I have been quite impressed with the way they have it set up. I'm anxiously awaiting the results of the first round of questioning, so we can see who ends up being the best predictor. If the results turn out like I'd expect them to, I would love to see a similar system put in place for major media pundits to register their predictions for major events, so you could see who is the most trustworthy with their predictions. This would help give some accountability to the prediction market, as has been mentioned in several previous posts.

Don Moore

I completely agree that it would be great if professional prognosticators, especially whom the media give a high public profile, would make their predictions clear enough that we could actually be able to score their accuracy. One way to accomplish that would be if their media hosts would push for precisely such clarity.

Phil Tetlock and Barb Mellers have written quite persuasively on the value of such clear scorable predictions from the intelligence community when it comes to national security. But for them, having the courage to be clear means putting their heads on the political chopping block.

Terry Murray

As the Project Manager for the Good Judgment Project, I want to let Joshua (and anyone else interested), know that we are still taking volunteers. Our waiting list is about to be cleared again, so there's an excellent chance that those who sign up now will be admitted to the study soon. To sign up, see