The Folly of Prediction

Stephen J. DUBNER: What does it mean to be a witch exactly in Romania? Are these people that we know here as psychics or fortunetellers, or are they different somehow?

Vlad MIXICH: I don’t know who is the fortuneteller in the United States. But here, generally, they are a woman of different ages. They can–they say they can cure some diseases. They can bring back your husband or your wife. Or they can predict your future.

DUBNER: Who is a typical client for a witch?

MIXICH: There are quite a lot of politicians who are going to witches. You know, the French president, Nicolas Sarkozy, he went to witches last year. And our president in Romania, and very important politicians from different parties, they are going to witches. Some of them, they were obliged to recognize they went to witches. Some of them, it’s off-the-record information. But me being a journalist, I know that information.

Vlad Mixich is a reporter in Bucharest, the capital of Romania. He knows a good bit about the witches there.

MIXICH: Quite a lot of them, they are quite rich. They have very big houses with golden rooftops. A lot of the Romanians, they are living in small apartments in blocks. So, just going into such a building will give you a sense of majesty and respect.

But the Romanian witch industry has been under attack. First came a proposed law to regulate and tax the witches. It passed in one chamber of Parliament before stalling out. But then came another proposal arguing that witches should be penalized if the predictions they make don’t turn out to be true.

MIXICH: So if you are one of my clients, and if I’m a fortune teller, if I fail to predict your future, I pay a quite substantial fine to the state, or if this happens many times, I will even go to jail. The punishment is between six months and three years in jail.

DUBNER: What’s being proposed in Romania is revolutionary. It strikes me because we typically don’t hold anybody accountable for bad predictions. So, I’m wondering in Romania, let’s say, if a politician makes a bad prediction, do they get fined or penalized in any way?

MIXICH: No, not at all. In fact this is one of the hobbies of our president. He’s doing a lot of predictions, which are not coming true, of course. And after that he is reelected! Or his popularity is rising, like the sun in the morning, you know? No, anyone can do publicly a lot of predictions here in eastern Europe and not a single hair will move from his or her head.

C’mon people, that doesn’t seem fair, does it? I don’t care if you’re anti-witch or pro-witch or witch-agnostic. Why should witches be the only people held accountable for bad predictions? What about politicians and money managers and sports pundits? And what about you?

*      *      *

All of us are constantly predicting the future, whether we think about it or not. Right now, some small part of your brain is trying to predict what this show is going to be about. How do you do that? You factor in what you’ve heard so far. What you know about Freakonomics. Maybe you know a lot, maybe you’ve never heard of it, you might think it’s some kind of communicable disease! When you predict the future, you look for cognitive cues, for data, for guidance. Here’s where I go for guidance:

Steven LEVITT: I think to an economist, the best explanation for why there are so many predictions is that the incentives are set up in order to encourage predictions.

That’s Steve Levitt. He’s my Freakonomics friend and co-author, an economist at the University of Chicago.

LEVITT: So, most predictions we remember are ones which were fabulously, wildly unexpected and then came true. Now, the person who makes that prediction has a strong incentive to remind everyone that they made that crazy prediction which came true. If you look at all the people, the economists, who talked about the financial crisis ahead of time, those guys harp on it constantly. “I was right, I was right, I was right.” But if you’re wrong, there’s no person on the other side of the transaction who draws any real benefit from embarrassing you by bringing up the bad prediction over and over. So there’s nobody who has a strong incentive, usually, to go back and say, “Here’s the list of the 118 predictions that were false.” I remember growing up, my mother, who is somewhat of a psychic–

DUBNER: Wait, somewhat of a psychic?

LEVITT: She’s a self-proclaimed psychic. And she would predict a stock market crash every single year.

DUBNER: And she’s been right a couple times.

LEVITT: And she has been. She’s been right twice in the last 15 years, and she would talk a lot about the times she was right. I would have to remind her about the 13 times that she was wrong. And without any sort of market mechanism or incentive for keeping the prediction makers honest, there’s lots of incentive to go out and to make these wild predictions. And those are the ones that are remembered and talked about. Think about one of the predictions that you hear echoed more often than just about any one — it’s Joe Namath’s famous pronouncement about how the Jets were going to win the Super Bowl. And it was unexpected. And it happened. And if the Jets had lost the Super Bowl, nobody would remember that Joe Namath made that pronouncement.

DUBNER: And conversely, you can probably find at least one player on every team that’s lost the Super Bowl in the last forty years that did predict that his team would win.

LEVITT: That’s probably right. That’s exactly right. Now, the flip side, which is perhaps surprising, is that in many cases, the goal of prediction is to be completely within the pack. And so I see this a lot with pension fund managers, or endowment managers, which is if something goes wrong, then as long as everybody else made the same prediction, you can’t be faulted very much.

Pension managers. Football players. Psychic moms. Romanian witches. Who doesn’t try to predict the future these days? And you know the worst thing? There’s almost nobody keeping track of all those predictions! Nobody, except for this guy:

Philip TETLOCK: Well, I’m a research psychologist, who —.

DUBNER:Don’t forget your name, though.

TETLOCK: I’m Phil Tetlock, and I’m a research psychologist. I spent most of career at the University of California, Berkeley, and I recently moved to the University of Pennsylvania, where I’m cross- appointed in the Wharton School and the psychology department.

Philip Tetlock has done a lot of research on cognition and decision-making and bias, pretty standard stuff for an Ivy League psych PhD. But what really fascinates him is prediction.

TETLOCK: There are a lot of psychologists who believe that there is a hard-wired human need to believe that we live in a fundamentally predictable and controllable universe. There’s also a widespread belief among psychologists that people try hard to impose causal order on the world around them, even when those phenomena are random.

This hardwired human need, as Tetlock puts it, has created what he calls a prediction industry. Now, don’t sneer. You’re part of it, too.

TETLOCK: I think there are many players in what you might count the prediction industry. In some sense, we’re all players in it. Whenever we go to a cocktail party, or a colloquium, or whatever where opinions are being shared, we frequently make likelihood judgments about possible futures and the truth or falsity of particular claims about futures. The prediction business is a big business on Wall Street, and we have futures markets and so forth designed to regulate speculation in those areas. Obviously, government has great interest in prediction. They create large intelligence agency bureaucracies and systems to help them achieve some degree of predictability in a seemingly chaotic world.

DUBNER: Let me read something that you have said or written in the past. “This determination to ferret out order from chaos has served our species well. We’re all beneficiaries of our great collective successes in pursuit of deterministic regularities in messy phenomena — agriculture, antibiotics, and countless other inventions.” So talk to me for a moment about the value of prediction. Obviously, there’s much to be gained. Do we overvalue prediction though, perhaps?

TETLOCK: I think there’s an asymmetry of supply and demand. I think there is an enormous demand for accurate predictions in many spheres of life in which we don’t have the requisite expertise to deliver. And when you have that kind of gap between demand and real supply, you get the infusion of fake supply.

“Fake supply.” I like this guy, this Philip Tetlock. He’s not an economist, but he knows the laws of supply and demand can’t just be revoked. So if there’s big demand for prediction in all realms of life, and not enough real supply to satisfy it, what does this “fake supply” sound like? There’s a punditocracy out there, a class of people who predict ad nauseam, often on television. They can be pretty good at making their predictions tough to audit.

TETLOCK: It’s the art of appearing to go out on a limb without actually going out on a limb. For example, the word “could.” Something could happen, the room you happen to be sitting in could be struck by a meteor in the next 23 seconds. That makes perfect sense, but the probability of course is point zero, zero, zero, zero, et cetera, one. It’s not zero, but it’s extremely low. In fact, the word could, the possible meanings people attach to it range from a 0.01 to a .6, which covers more than half the probability scale right there.

Look, nobody likes a weasel. So more than 20 years ago, Tetlock set out to conduct one of the largest empirical studies, ever, of predictions. He chose to focus on predictions about political developments around the world. He enlisted some of the world’s foremost experts — the kind of very smart people who have written definitive books, who show up on CNN or on The New York Times’s op-ed page.

TETLOCK: In the end, we had close to three hundred participants. And they were very sophisticated political observers. Virtually all of them had some post-graduate education. Roughly two-thirds of them had PhDs. They were largely political scientists, but there were some economists and a variety of other professionals as well.

DUBNER: And they all participated in your study anonymously, correct?

TETLOCK: That was a very important condition for obtaining cooperation.

DUBNER: Now, if they were not anonymous, then presumably, we would recognize some of their names — these are prominent people at political science departments, economics departments at I’m guessing some of the better universities around the world — is that right?

TETLOCK: Well, I don’t want to say too much more, but I think you would recognize some of them, yes. I think some of them had substantial Google counts.

The study became the basis of a book Tetlock published a few years ago, called “Expert Political Judgment.” There were two major rounds of data collection, the first beginning in 1988, the other in 1992. These nearly 300 experts were asked to make predictions about dozens of countries around the world.The questions were multiple choice. For instance: In Democracy X — let’s says it’s England — should we expect that after the next election, the current majority party will retain, lose, or strengthen its status? Or, for Undemocratic Country Y — Egypt, maybe — should we expect the basic character of the political regime to change in the next five years? In the next 10 years? And if so, in what direction? And to what effect? The experts made predictions within their areas of expertise, and outside; and they were asked to rate their confidence for their predictions. So after tracking the accuracy of about 80,000 predictions by some 300 experts over the course of 20 years, Philip Tetlock found:

TETLOCK: That experts thought they knew more than they knew. That there was a systematic gap between subjective probabilities that experts were assigning to possible futures and the objective likelihoods of those futures materializing.

Let me translate that for you. The experts were pretty awful. And you think: awful compared to what? Did they beat a monkey with a dartboard?

TETLOCK: Oh, the monkey with a dartboard comparison, that comes back to haunt me all the time. But with respect to how they did relative to, say, a baseline group of Berkeley undergraduates making predictions, they did somewhat better than that. Did they do better than an extrapolation algorithm? No, they did not. They did for the most part a little bit worse than that. How did they do relative to purely random guessing strategy? Well, they did a little bit better than that, but not as much as you might hope.

That “extrapolation algorithm” that Tetlock mentioned? That’s simply a computer programmed to predict “no change in current situation.” So it turned out these smart, experienced, confident experts predicted the political future about as well, if not slightly worse, than the average daily reader of The New York Times.

TETLOCK: I think the most important takeaway would be that the experts are, they think they know more than they do. They were systematically overconfident. Some experts were really massively overconfident. And we are able to identify those experts based on some of their characteristics of their belief system and their cognitive style, their thinking style.

DUBNER: OK. So now we’re getting into the nitty-gritty of what makes people predict well or predict poorly, what are the characteristics then of a poor predictor?

TETLOCK: Dogmatism.

DUBNER: It can be summed up that easily?

TETLOCK: I think so. I think an unwillingness to change one’s mind in a reasonably timely way in response to new evidence. A tendency, when asked to explain one’s predictions, to generate only reasons that favor your preferred prediction and not to generate reasons opposed to it.

DUBNER: And I guess what’s striking to me — and I’d love to hear what you had to say about this — is that it’s easy to provide one word, prediction, to many, many, many different realms in life. But those realms all operate very differently — so politics is different from economics, and predicting a sports outcome is different than predicting, you know, an agricultural outcome. It seems that we don’t distinguish so much necessarily and that there’s this modern sense almost that anything can be and should be able to be predicted. Am I kind of right on that, or no?

TETLOCK: I think there’s a great deal of truth to that. I think it is very useful in talking about the predictability of the modern world to distinguish those aspects of the world that show a great deal of linear regularity and those parts of the world that seems to be driven by complex systems that are decidedly nonlinear and decidedly difficult if not impossible to predict.

DUBNER: Talk to me about a few realms that generally are very, very hard to predict, and a few realms that generally are much easier.

TETLOCK: Predicting Scandinavian politics is a lot easier than predicting Middle Eastern politics.

DUBNER: Yes, that was the first one that came to my mind too! All right, but keep going.

TETLOCK: The thing about the radically unpredictable environments is that they often appear for long periods of time to be predictable. So, for example, if you had been a political forecaster predicting regime longevity in the Middle East, you would have done extremely well predicting in Egypt that Mubarak would continue to be the president of Egypt year after year after year in much the same way that if you had been a Sovietologist you would have done very well in the Brezhnev era predicting continuity. There’s an aphorism I quote in the “Expert Political Judgment” book from Karl Marx. I’m obviously not a Marxist but it’s a beautiful aphorism that he had, which was that, “When the train of history hits a curve, the intellectuals fall off.”

Coming up: Who do you predict we’ll hear from next — a bunch of people who are awesomely good at predicting the future? Yeah, right. Maybe later. First, we’ll hear from some more duds — from Wall Street, the NFL, and … the cornfield.

*      *      *

So Phillip Tetlock has sized up the people who predict the future–geopolitical change, for instance–and determined that they’re not very good at predicting the future. He also tells us that their greatest flaw is dogmatism–sticking to their ideologies even when presented with evidence that they’re wrong. You buy that? I buy it. Politics is full of ideology; why shouldn’t the people who study politics be a least a little bit ideological? So let’s try a different set of people, people who make predictions that, theoretically at least, have nothing to do with ideology. Let’s go to Wall Street.

Christina FANG: I’m Christina Fang, a Professor of Management at New York University’s business school.

Christina Fang, like Philip Tetlock, is fascinated with prediction:

FANG: Well, I guess generally forecasting about anything, about technology, about a product, whether it will be successful, about whether an idea, a venture idea could take off, a lot of things, not just economic but also business in general.

Fang wasn’t interested in just your street-level predictions, though. She wanted to know about the Big Dogs, the people who make bold economic predictions that carry price tags in the many millions or even billions of dollars. Along with a fellow researcher, Jerker Denrell, Fang gathered data from the Wall Street Journal’s Survey of Economic Forecasts. Every six months, the paper asked about 50 top economists to predict a set of macroeconomic numbers — unemployment, inflation, gross national product, things like that. Fang audited seven consecutive surveys, with an eye toward a particular question: when someone correctly predicts an extreme event — a market crash, maybe, or a sudden spike in inflation — what does that say about his overall forecasting ability?

FANG: In the Wall Street Journal survey, if you look at the extreme outcomes, either extremely bad outcomes and extremely good outcomes, you see that those people who correctly predicted either extremely good or extremely bad outcomes, they’re likely to have overall lower level of accuracy. In other words, they’re doing poorer in general.

Uh-oh. You catching this?

FANG: Those people who happen to predict accurately the extreme events, we also look at their–they happen to also have a lower overall level of accuracy.

DUBNER: So I can be right on the big one, but if I’m right on the big one, I generally will tend to be more often wrong than the average person.

FANG: On average –.

DUBNER: On average.

FANG: Across everyday predictions as well. And our research suggests that for someone who has successfully predicted those events, we are going to predict that they are not likely to repeat their success very often. In other words, their overall capability is likely to be not as impressive as their apparent success seems to be.

So the people who make big, bold, correct predictions are in general worse than average at predicting the economic future. Now, why is this a problem? Maybe they’re just like home-run hitters — y’know, a lot of strikeouts but a lot of power too. All right, I’ll tell you why it’s a problem. Actually, I’ll have Steve Levitt tell you.

LEVITT: The incentives for prediction makers are to make either cataclysmic or utopian predictions. Right? Because you don’t get attention if I say that what’s going to happen tomorrow is exactly as what’s going to happen today …

DUBNER: You don’t get on TV.

LEVITT: I don’t get on TV. If it happens to come true, who cares? I don’t get any credit for it coming true either.

There’s a strong incentive to make extreme predictions, because, seriously, who tunes in to hear some guy say that “Next year will be pretty much like last year”? And once you have been right on an extreme forecast — let’s say you predicted the 2008 market crash and the Great Recession — even if you were predicting it every year, like Steve Levitt’s mother, you’ll still be known as ‘The Guy Who Called the Big One.’ And even if all your followup predictions are wrong, you still got the ‘Big One’ right. Like Joe Namath.

All right, look. Predicting the economy? Predicting the political future? Those are hard. Those are big, complex systems with lots of moving parts. So how about football? If you’re an NFL expert, how hard can it be to forecast, say, who the best football teams will be in a given year? We asked Freakonomics researcher Hayes Davenport to run the numbers for us:

Hayes DAVENPORT: Well, I looked at the past three years of expert picking from the major NFL prediction outlets, which are USA Today, SportsIllustrated.com and ESPN.com. We looked at a hundred and five sets of picks total. They’re picking division winners for each year, as well as the wild card for that year. So they’re basically picking the whole playoff picture for that year.

DUBNER: So talk about just kind of generally, the degree of difficulty of making this kind of a pick.

DAVENPORT: Well, if you’re sort of an untrained animal making NFL picks, you’re going to have about a 25 percent chance of picking each division correctly, because there are only four teams.

DUBNER: All right, so Hayes, you’re saying that an untrained animal would be about twenty five percent accurate if you pick one out of four. But what about a trained animal, like me, a casual fan? How do I do compared to the experts?

DAVENPORT: Right. So if you’re cutting off the worst team in each division, if you’re not picking among those, you’ll be right 33 percent of the time, one in three, and the experts are right about 36 percent of the time, so just a little better than that.

DUBNER:OK, so if you’re saying they’re picking about thirty-six percent accuracy, and I or someone by chance would pick at about thirty three-percent accuracy. So that’s a three percentage point improvement, or about a 10 percent better, maybe we should say, you know, that’s not bad. If you beat the stock market by 10 percent every year you’d be doing great. So are these NFL pundits being 36 percent right being really wonderful or –.

DAVENPORT: I wouldn’t say that because there’s a specific fallacy these guys are operating from, which is they tend to rely much too heavily on the previous year’s standings in making their picks for the following year. They play it very conservatively. But there’s a very high level of parity in the NFL right now, so that’s not exactly how it works.

DUBNER: Tell me some of the pundits who — whether by luck or brilliance and hard work — turn out to be really, really good.

DAVENPORT: Sure. There are two guys from ESPN who are sort of far ahead of the field. One is Pat Yasinskas, and the other is John Clayton, who is pretty well known; he makes a lot of appearances on SportsCenter and he’s kind of a, nebbish-y professorial type. And they perform much better than everyone else because they’re excellent wild-card pickers. They’re the only people who have correctly predicted both wild card teams in a conference in a season. But they’re especially good because they actually play it much safer than everyone else.

DUBNER:Now, you say that they are very good. Persuade me that they’re good and not lucky.

DAVENPORT:I can’t do that. There’s a luck factor involved in all of these predictions. For example, if you pick the Patriots in 2008 and Tom Brady gets injured, and they drop out of the playoffs, there’s very little you can do to predict that. So injuries will mess with prediction all the time. And other turnover rates in football that are sort of unpredictable. So there’s a luck factor to all of this.

So whether it’s football experts calling Sunday’s game or economists forecasting the economy, or political pundits looking for the next revolution, we’re talking about accuracy rates that barely beat a coin toss. But maybe all these guys deserve a break. Maybe it’s just inherently hard to predict the future of other human beings. They’re so malleable; so unpredictable! So how about a prediction where human beings are incidental to the main action?

Joe PRUSACKI: I’m Joe Prusacki and I am the Director of Statistics Division with USDA’s National Agricultural Statistics Service, or NASS for short.

DUBNER: You grew up on a farm, yeah?

PRUSACKI: Uh-huh, yep. I grew up in – I always call it “deep southern” Illinois. I’m sitting here in Washington D.C., and where I grew up in Illinois is further south than where I’m sitting today. We raised, we had corn, soybeans and raised hogs.

You’ve heard of Anna Wintour, right? The fabled editor of Vogue magazine? Joe Prusacki is kinda like Anna Wintour for farmers. He puts out publications that are read by everyone who’s anyone in the industry — titles like “Acreage” and “Prospective Plantings” and “Crop Production.” Prusacki’s reports carry running forecasts of crop yields for cotton, soybeans, wheat and corn.

PRUSACKI: Most of the time, our monthly forecasts are probably within, I can guarantee you, within five percent. And most of the time, I can say within two to three percent of the final. And someone would say that seems very good. But in the agricultural world, the users expect us to be much more precise in our forecasts.

So how does this work? How does the USDA forecast something as vast as the agricultural output of American farmers?

PRUSACKI: Like at the beginning of March, we will conduct a large survey of farmers and ranchers across the United States and sample size this time, this year, was about 85,000.

The farmers are asked how many acres they plan to devote to each crop. Corn, let’s say. Then, in late July, the USDA sends out a small army of “enumerators” into roughly 1,900 cornfields in 10 states. These guys mark off plots of corn, 20 feet long by two rows across.

PRUSACKI: They’re randomly placed. We have randomly selected fields, in random location within field. So you may get a sample that’s maybe 20 paces into the field and 40 rows over and you may get one that’s 250 paces into the field and 100 rows over.

The enumerators look at every plant in that plot.

PRUSACKI: And then they’ll count what they see or anticipate to be ears based on looking at the plant.

A month later, they go back out again and check the cornstalks, check the ears.

PRUSACKI: Well, you could have animal loss, animals might chew the plant off, the plant may die. So all along, we’re updating the number of plants, all along we’re updating the number of ears. The other thing we need, you need an estimate of ear weight or fruit weight.

So they go out again, cut off a bunch of ears and weigh them. But wait: still not done. After the harvest, there’s one more round of measurement.

PRUSACKI: Once the field is harvested, and the machine has gone through the field, the enumerator will go back out to the field, they’ll lay out another plot – just beyond the harvest area where we were – and they will go through and pick up off the ground any kernels that are left on the ground, pieces of ears of corn and such on the ground so we get a measure of harvest loss.

So this sounds pretty straightforward, right? Compared to predicting something like the political or economic future, estimating corn yield based on constant physical measurements of corn plants is pretty simple. Except for one thing: It’s called the weather. Weather remains so hard to predict in the long term that the USDA doesn’t even use forecasts; it uses historic averages instead.

DUBNER: So Joe, talk to me about what happened last year with the USDA corn forecast. You must have known this was coming from me. So the Wall Street Journal’s headline was: “USDA Flubs in Predicting Corn Crops.” Explain what happened.

PRUSACKI: Well, this is the weather factor that came into play. It turned out pretty hot and pretty dry in most of the growing region. And I had asked a few folks that are out and about in Iowa what happened. They said this is just a really strange year. We just don’t know. Now, when, if someone says “Did we flub it?” I don’t know. It was the forecast based on the information I had as for August 1. Now, September 1, I had a different set of information. October 1, I had a different set of information. Could we have done a better job?

A lot of people thought they could have. Last June, the USDA lowered its estimate of corn stockpiles; and in October, it cut its estimate of corn yield. After the first report, the price of corn spiked 9 percent. The second report? Another 6 percent. Joe Prusacki got quite a few e-mails:

PRUSACKI: OK, the first one is, this was: “Thanks a lot for collapsing the grain market today with your stupid … and the word is three letters, begins with an “a” and then it has two dollar signs … USDA report.

“As bad as the stench of dead bodies in Haiti must be, it can’t even compare to the foul stench of corruption emanating from our federal government in Washington D.C.”

DUBNER: It strikes me that there’s room for trouble here in that your forecasts are used by a lot of different people who engage in a lot of different markets, and your research can move markets. I’m wondering what kind of bribes maybe come your way?

PRUSACKI: It’s interesting, I have people that call, we call them ‘fishers.’ They call maybe a day or two days before, when we’re finishing our work, and it’s like I tell them, I say, “Why do you do this? We’ve had this discussion before.” There’s a couple things, one, I sign a confidentiality statement every year that says I shall not release any information before it’s due time or bad things happen. It’s a $100,000 fine or time in prison. It’s like the dollar fine, OK. It’s the prison part that bothers me!

DUBNER: But there’s got to be a certain price at which – so let’s say I offered you, I came to you and I said “Joe, $10 million for a 24-hour head start on the corn forecast.”

PRUSACKI: I’m not going to do it. Trust me, somebody would track me down.

DUBNER: I hear you.

PRUSACKI: Again, the prison time, it bothers me.

All right, so Joe Prusacki probably can’t be bought. And the USDA is generally considered to do a pretty good job with crop forecasts. But, look how hard the agency has to work, measuring corn fields row by row, going back to look for animal loss and harvest loss. And still, its projection — which is looking only a few months into the future — can get thrown totally out of whack by a little stretch of hot, dry weather. That dry spell was essentially a random event, kind of like Tom Brady’s knee getting smashed. I hate to tell you this but the future is full of random events. That’s why it’s so hard to predict. That’s why it can be scary. Do we know this? Of course we know it. Do we believe it? Mmmmm.

Some scholars say that our need for prediction is getting worse, or, more accurately, that we get more upset now when the future surprises us. After all, as the world becomes more rational and routinized, we often know what to expect. I can get a Big Mac not only in New York but in Beijing, too — and they’ll taste pretty much the same. So when you’re used to that, and when things don’t go as expected — watch out.

Our species has been trying to foretell the future forever. Oracles and goat entrails and roosters pecking the dirt. The oldest religious texts are filled with prediction. I mean, look at the afterlife! What is that if not a prediction of the future? A prediction that, as far as I can tell, can never be categorically refuted or confirmed. A prediction so compelling that it remains all these years later a concept around which billions of people organize their lives. So what do you see when you gaze into the future? A yawning chasm of random events? Or, do you look for a neat pattern, even if no such pattern exists?

Nassim TALEB: It’s much more costly for someone to not detect a pattern.

That’s Nassim Taleb, the author of Fooled By Randomness and The Black Swan.

TALEB: It’s much costlier for us — as a race, to make the mistake of not seeing a leopard than having the illusion of pattern and imagining a leopard where there is none. And that error, in other words, mistaking the non-random for the random, is what I call the “one-way bias.” Now, that bias works extremely well, because what’s the big deal of getting out of trouble? It’s not costing you anything. But in the modern world, it is not quite harmless. Illusions of certainty makes you think that things that haven’t exhibited risk, for example the stock market, are riskless. We have the turkey problem — the butcher feeds the turkey for a certain number of days, and then the turkey imagines this is permanent.

“The butcher feeds the turkey and the turkey imagines this is permanent.” So you’ve got to ask yourself: who am I? The butcher? Or the turkey? Coming up: hedgehogs and foxes — and a prediction that does work. Here’s a hint: if you like this song, you’ll probably like this one too.

*      *      *

Hey, guess what, Sunshine? Al Gore didn’t win Florida. Didn’t become president either. Try walking that one back. So we are congenital predictors, but our predictions are often wrong. What then? How do you defend your bad predictions? I asked Philip Tetlock what all those political experts said when he showed them their results. He had already stashed their excuses in a neat taxonomy:

TETLOCK: So, if you thought that Gorbachev, for example, was a fluke, you might argue, well, my understanding of the Soviet political system is fundamentally right, and the Soviet Politburo, but for some quirky statistical aberration of the Soviet Politburo would have gone for a more conservative candidate. Another argument might be, well, I predicted that Canada would disintegrate, that Quebec would secede from Canada, and it didn’t secede, but the secession almost did succeed because there was a 50.1 percentage vote against secession, and that’s well within the margin of sampling error.

DUBNER: Are there others you want to name?

TETLOCK: Well, another popular prediction is “off on timing.” That comes up quite frequently in the financial world as well. Many very sophisticated students of finance have commented on how hard it is, saying, “the market can stay irrational longer than you can stay liquid,” I think is George Soros’s expression. So, “off on timing” is a fairly popular belief-system defense as well. And I predicted that Canada would be gone. And you know what? It’s not gone yet. But just hold on.

DUBNER: You answered very economically when I asked you what are the characteristics of a bad predictor; you used one word, dogmatismm. What are the characteristics, then, of a good one?

TETLOCK: Capacity for constructive self-criticism.

DUBNER: How does that self-criticism come into play and actually change the course of the prediction?

TETLOCK: Well, one sign that you’re capable of constructive self-criticism is that you’re not dumbfounded by the question: What would it take to convince you you’re wrong? If you can’t answer that question you can take that as a warning sign.

In his study, Tetlock found that one factor was more important than any other in someone’s predictive ability: cognitive style. You know the story about the fox and the hedgehog?

TETLOCK: Isaiah Berlin tells us that the quotation comes from the Greek warrior poet Archilichus 2,500 years ago. And the rough translation was the fox knows many things, but the hedgehog knows one big thing.

DUBNER: So, talk to me about what the foxes do as predictors and what the hedgehogs do as predictors.

TETLOCK: Sure. The foxes tend to have a rather eclectic, opportunistic approach towards forecasting. They’re very pragmatic. A famous aphorism by Deng Xiaoping was, he “didn’t care if the cat was white or black as long as it caught mice.” And I think the attitude of many foxes is they really didn’t care whether ideas came from the left or the right, they tended to deploy them rather flexibly in deriving predictions. So they often borrowed ideas across schools of thought that hedgehogs viewed as more sacrosanct. There are many subspecies of hedgehog. But what they have in common is a tendency to approach forecasting as a deductive, top-down exercise. They start off with some abstract principles, and they apply those abstract principles to messy, real-world situations, and the fit is often decidedly imperfect.

So foxes tend to be less dogmatic than hedgehogs, which makes them better predictors. But, if you had to guess, who do you think is more likely to show up TV or in an op-ed column, the pragmatic, nuanced fox, or the know-it-all hedgehog? You got it!

TETLOCK: Hedgehogs, I think, are more likely to offer quotable sound bites, whereas foxes are more likely to offer rather complex, caveat-laden sound bites. They’re not sound bites anymore if they’re complex and caveat-laden.

DUBNER: So, if you were to gain control of let’s say a really big media outlet, The New York Times, or NBC TV, and you said, you know, I want to dispense a different kind of news and analysis to the public, what would you do? How would you suggest building a mechanism to do a better job of keeping all this kind of poor expert prediction out of the, off the airwaves.

TETLOCK: I’m so glad you asked that question. I have some specific ideas about that. And I don’t think they would be all that difficult to implement. I think they should try to keep score more. I think there’s remarkably little effort in tracking accuracy. If you happen to be someone like Tom Friedman or Paul Krugman, or someone who’s at the top of the pundit pecking order, there’s very little incentive for you to want to have your accuracy tested because your followers are quite convinced that you’re extremely accurate, and it’s pretty much a game you can only lose.

Can you imagine? Every time a pundit appeared on TV, the network would list his batting average, right after his name and affiliation. You think that might cut down on blowhard predictions just a little bit? Looking back at what we’ve learned so far, it makes me wonder: maybe the first step toward predicting the future should be to acknowledge our limitations. Or – at the very least – let’s start small. For instance: if I could tell you what kind of music I like, and then you could predict for me some other music I’d want to hear. That actually already exists. It’s called Pandora Radio. Here’s co-founder Tim Westergren.

Tim WESTERGREN: So, what we’ve done is, we’ve broken down recordings into their basic components for every dimension of melody, harmony, and rhythm, and form, and instrumentation, down into kind of the musical equivalent of primary colors.

The Pandora database includes more than a million songs, across every genre that you or I could name. Each song is broken down into as many as 480 musical attributes, almost like genetic code. Pandora’s organizing system is in fact called the “Music Genome Project.” You tell the Pandora website a song you like, and it rummages through that massive genetic database to make an educated guess about what you want to hear next. If you like that song, you press the thumbs-up button, and Pandora takes note.

WESTERGREN: I wouldn’t make the claim that Pandora can map your emotional persona. And I also don’t think frankly that Pandora can predict a hit because I think it is very hard, it’s a bit of magic, that’s what makes music so fantastic. So, I think that we know our limitations, but within those limitations I think that we make it much, much more likely that you’re going to find that song that just really touches you.

DUBNER: So Tim, you were good enough to set up a station for me here. It’s called “Train in Vain Radio.” So the song we gave you was “Train in Vain.” So let me open up my radio station here and I’ll hit play and see what you got for me.

DUBNER: Oh yeah. Yeah I like them, that’s The Jam, so I’m going to give it a thumbs up. I like “Town Called Malice.” on my little window here. I think there are a couple more songs in my station here.

“Television” by Tom Verlaine, he was always too cool for me. I can see why you would think that I would like them, and I appreciate your effort, Mr. Pandora. How about you, were you a “Television” fan?

WESTERGREN: Yeah, yeah. And you know, one thing of course is that the songs are all rooted in guitar riffs.

DUBNER: Yep.

WESTERGREN: There’s a repetitive motif played on the guitar. And a similar sound and they’ve got a little twang– and they’re played kind of rambly, a little bit rough, there’s a sort of punk element in there. The vocals have over twenty attributes just for the voice. In this case these are pretty unpolished vocal deliveries.

DUBNER: I got to tell you that even though when this song came up, and I’ve heard this song a few times, and I told you I didn’t like Television very much, this song, I’m kind of digging it now.

WESTERGREN: See, there you go, that’s exactly what we’re trying to do.

DUBNER: So, it’s a really great thing to do, but it’s not really predicting the future the way most people think of it as predicting the future, is it?

WESTERGREN: Well, I certainly wouldn’t have put our mission in the same category as predicting the economy, or, you know, geopolitical futures. But you know, the average American listens to 17 hours of music a week. So, they spend a lot of time doing it, and I think that if we can make that a more enjoyable experience and more personalized, I think maybe we’ll make some kind of meaningful contribution to culture.

So Pandora does a pretty good job of predicting the music you might want to hear, based on what you already know you like. But again, look how much effort that takes — 480 musical attributes! And it’s not really predicting the future, is it? All Pandora does is breaks down the confirmed musical preferences of one person today and comes up with some more music that’ll fulfill that same person’s preferences tomorrow. If we really want to know the future, we probably need to get much more ambitious. We probably need a whole new model. Like, how about prediction markets?

Robin HANSON: A prediction market is basically like a betting market or a speculative market, like orange juice futures or stock markets, things like that. The mechanics is that there’s a — an asset of some sort that pays off if something’s true, like whether a person wins the presidency or a team wins a sporting contest. And people trade that asset and the price of that asset becomes then a forecast of whether that claim is likely to be true.

That’s Robin Hanson, an economics professor at George Mason University and an admitted advocate of prediction markets. As Hanson sees it, a prediction market is far more reliable than other forecasting methods because it addresses the pesky incentive problems of the old-time prediction industry.

HANSON: So a prediction market gives people an incentive, a clear, personal incentive to be right and not wrong. Equally important, it gives people an incentive to shut up when they don’t know, which is often a problem with many of our other institutions. So, if you as a reporter call up almost any academic and and ask them vaguely related questions, they’ll typically try to answer them, just because they want to be heard. But in a prediction market most people don’t speak up. Every one of your listeners today had the right to go speak up on orange juice futures yesterday. Every one of you could have gone and said, orange juice futures forecasts are too low or too high, and almost no one did. Why? Because most of you don’t think you know. And that’s just the way we want it. So in most of these prediction markets what we want is the few people who know the best to speak up and everybody else to shut up.

Prediction markets are flourishing. Some of them are private — a multinational firm might set up an internal market to try to forecast when a big project will be done. And there are for-profit prediction markets like InTrade, based in Dublin, where you can place a bet on, say, whether any country that currently uses the Euro will drop the Euro by the end of the year. (As I speak, that bet has a 15% chance on InTrade.) Here’s another InTrade bet: whether there’ll be a successful WMD terrorist attack anywhere in the world by the end of 2013. (That’s got a 28% chance.) Now that’s starting to sound a little edgy, no? Betting on terrorism? Robin Hanson himself has a little experience in this area, on a U.S. government project he worked on.

HANSON: All right, so — back in 2000, DARPA, the Defense Advanced Research Projects Agency, had heard about prediction markets, and they decided to fund a research project. And they basically said, listen, we’ve heard this is useful for other things, we’d like you to show us that this can be useful for the kind of topics we are interested in. Our project was going to be forecasting geopolitical trends in the Middle East. We were going to show that prediction markets could tell you about economic growth, about riots, about perhaps, wars, about whether the changes of heads of state, and how these things would interact with each other.

In 2003, just as the project was about to go live, the press heard about it.

HANSON: On Monday morning, two senators had a press conference where they declared that the DARPA, the — and the military were going to have a betting market on terrorism. And so, there was a sudden burst of media coverage, and by the very next morning, the head of the military basically declared before the Senate that this project was dead, and there was nothing more to worry about.

DUBNER:What do you think you — we collectively, you, in particular — would know now about that part of the world, let’s say, if this market had been allowed to take root?

HANSON: Well, I think we would have gotten much earlier warning about the revolutions we just had. And if we would have had participants from the Middle East forecasting those markets. Not only we would get advanced warning about which things might happen, but then how our actions could affect those. So, for example, the United States just came in on the side of the Libyan rebels, to support the Libya rebels against the Qaddafi regime. What’s the chances that will actually help the situation, as opposed to making it worse?

DUBNER: But give me an example of what you consider among the hardest problems that a prediction market could potentially help solve?

HANSON: Who should — not only who should we elect for president but whether we should go to war here, or whether we should begin this initiative? Or should we approve this reform bill for medicine, etc.

DUBNER:So that sounds very logical, very appealing. How realistic is it?

HANSON: Well, it depends on there being a set of customers who want this product. So, you know, if prediction markets have an Achilles heel, it’s certainly the possibility that people don’t really want accurate forecasts.

Prediction markets put a price on accountability. If you’re wrong, you pay, simple as that. Just like the proposed law against the witches in Romania. Maybe that’s what we need more of. Here’s Steve Levitt again:

LEVITT: When there are big rewards to people who make predictions and get them right, and there are zero punishments for people who make bad predictions because they’re immediately forgotten, then economists would predict that’s a recipe for getting people to make predictions all the time.

DUBNER: Because the incentives are all encouraging you to make predictions.

LEVITT: Absolutely.

DUBNER: If you get it right, there’s an upside, and if you get it wrong, there’s almost no downside.

LEVITT: Right, if the flip side were that if I make a false prediction I’m immediately sent to prison for a one-year term, there would be almost no prediction.

And all those football pundits and political pundits and financial pundits wouldn’t be able to wriggle out of their bad calls — saying “My idea was right, but my timing was wrong.” Maybe that’s how everybody does it. That big storm the weatherman called but never showed up? “Oh, it happened all right,” he says, “but two states over.” Or how about those predictions for the End of the World — the Apocalypse, the Rapture, all that? “Well,” they say, “we prayed so hard that God decided to spare us.”

Remember back in May, when an 89-year-old preacher named Harold Camping declared that the Earth would be destroyed at 5:59 p.m. on a Saturday, and only the true believers would survive? I remember it very well because my 10-year-old son was petrified. I tried telling him that Camping was a kook — that anybody can say pretty much anything they want about the future. It didn’t help; he couldn’t get to sleep at night.

And then the 21st came and went and he was psyched. “I knew it all along, Dad,” he said.

Then I asked him what he thought should happen to Harold Camping, the false Doomsday prophet. “Oh, that’s easy,” he said. “Off with his head!”

My son is not a bloodthirsty type. But he’s not a turkey either.