What's Wrong With Punishing Bad Predictions?

(Photo: Alessandro Giangiulio)

In the heat of a Presidential campaign, it can be hard to pay attention to other news. But a small-seeming story out of Italy yesterday has, to my mind, the potential to shape the future as much as a Presidential election.

As reported by ABC, the BBC, the Wall Street Journal, the New York Times, and elsewhere, an Italian court has convicted seven earthquake experts of failing to appropriately sound the alarm bell for an earthquake that wound up killing more than 300 people in L’Aquila in 2009. The experts received long prison sentences and fines of more than $10 million. (Addenum: Roger Pielke Jr. discusses the “mischaracterizations” of the verdict.)

There is of course the chance that the verdict will be thrown out upon appeal, discredited as an emotional response to a horrible tragedy. 

But let’s consider what the verdict represents as of today: scientists and government officials are being held personally accountable for the failure to adequately predict the scale of a natural disaster.

If you ever heard our “Folly of Prediction” podcast, you might think we would embrace this verdict. Our main point, after all, was that the world is awash in prediction, much of it terrible, in large part because there are strong incentives to make bold predictions and weak penalties for bold predictions don’t come true.

For example: I could write here today that the Dow will reach 35,000 by the end of 2013. In the very unlikely event that this prediction comes true, I would be lauded and rewarded to no end. And in the likely event my prediction doesn’t come true, it will be pretty much forgotten.

But the Italian earthquake story is in fact the opposite of this common scenario. By not predicting an anomalous event which then did occur, the scientists are being penalized for not sounding an alarm bell loudly enough.

If a child of mine had been killed in the Italian earthquake, I might well feel entitled to vengeance. But, in a world where experts in all realms are constantly sounding alarm bells that turn out to be meaningless, how does this verdict potentially shape the future? I can see at least two effects:

  1. Scientists and others might become less willing to predict the future — which, given our skill at prediction, might not be a terrible thing but which, given the fact that future-gazing is often an excuse for useful research, might be a terrible thing.
  2. A message has been sent that it is better to predict a calamity than to fail to predict a calamity. If you fail to predict a calamity, as anomalous as it may be, you risk being sent to jail and fined. So the incentive to predict a calamity has just been significantly increased.

Does this mean I can have my financial planner sent to jail every time the market tanks? Can I have my kid’s guidance counselor sent to jail if anthropology doesn’t turn out to be the hot employment field of the future?

Am I wrong in thinking this is a big, big story?

FWIW, we did another podcast about how hard it is to predict earthquakes.

(HT: inter alia, Steven Merahn and Marginal Revolution)


"For example: I could write here today that the Dow will reach 35,000 by the end of 2013. In the very unlikely event that this prediction comes true, I would be lauded and rewarded to no end. And in the likely event my prediction doesn’t come true, it will be pretty much forgotten."

Is that a reference to Kevin Hassett? He wrote "Dow 36,000." He's still a major GOP economics adviser despite being so terribly wrong with his DJIA prediction (which was mainly the result of a basic math error and a misunderstanding of what numbers you plug into pricing models).


When I heard this news yesterday, your "Folly of Prediction" podcast was the first thing that came to mind. Much like the fortune teller who was held legally responsible for her predictions, it's equally ridiculous that the earthquake-predicting experts should be held responsible for the reciprocal omissive error of "failing to alert the public loud enough".

Maybe the solution here is to use an exact system of measuring predictability - so if the chances of damaging earthquake reach a set probability, the public is alerted. If this system is impossible, then the prediction is not based on solid science and therefore should not be punishable.


If this judgment stands, they probably would need to have been fined as well if it was the other way around, predicting a calamity that didn't happen, due to associated costs of evacuation. So the risk of making such predictions (if they can be held liable) very high.

I would like to know whether they have signed a convenant or contract in which it is specified that they can be held liable for their predictions. First of all I don't think they would have signed such a contract due to the difficulty of predicting such an event exactly (how bad), timely (when) and correctly (where). Secondly, such a contract should then also have a high compensation for correct predictions in order for them to be able to pay the liability claims ($10mln divided by seven scientists is a lot of money).

As someone once said: "there are no mistakes in science, only lessons". How can you sentence seven scientist to six years in prison for not being able to to what is generally agreed to be very, very difficult and in the past has rarely been done exactly correct?

Shouldn't the local government bear the resposibility for not earthquake-proofing their community in an area prone to earthquakes? Or didn't they want to change its historic center (which would seem quite logical)?

Or should the people who choose to live in an area prone to earthquakes bear the risk?



Wait! Wait! You mean anthropology isn't the hot employment field of the future? Sheesh... and here I thought I might be able to get a life.

Howard Tayler

I think the big problem here is that the scientific method requires scientists to be wrong, and to be wrong a LOT. Earthquake science, even for as much as we know, is still in its infancy. Jailing scientists for being wrong is a direct attack upon the methods that allow them to improve.

As was pointed out elsewhere (I forget where -- my numbers could be off) something like half of all big quakes are preceded by small temblors, but only around 2% of small temblors are followed by big quakes. Based on that information, and the best science we've got available, the big quake was pretty unlikely.


There is just one big prediction that is a no-win, predicting the end of the world... If you are wrong everybody laughs at you, if you are right, there's nobody there to congratulate you.


Like the MAN-made Global Warming disaster prediction. Over 22 billion has been spent on this end-of-the world calamity. They predicted 50 million climate refugees by 2010; they were wrong. They predicted snow-less winters in England; they were wrong. They predicted an accelerated rising sea-level, it's been decelerating. They predicted "free" energy from wind, but every country that puts the turbines up sees skyrocketing electricity prices. The global temp averages haven't risen for 16 years, in spite of the steady rise of CO2. In fact, the temps have been LOWER than their scenario for if we froze the CO2 concentrations at 16 years ago, showing their models are flawed.

When it is determined that the the prognosticators are wrong, can we sue?

Or will they constantly say "recalculating" . . . it's still coming . . . later . . . when the generation that paid for this folly is dead, and so are the responsible (irresponsible) scientists. Who, by the way, benefit from billions in public money for predicting/"studying" this apocalypse.



Based upon Roger Pielke Jr's article, the predications were not based upon science but rather the predications were to designed to support their own personal & political agendas. Maybe punishment is warranted when this occurs

"Does this mean I can have my financial planner sent to jail every time the market tanks?" It might if all of the financial planner's recommendations were only focused upon maxmizing the planner's commission income.

"Can I have my kid’s guidance counselor sent to jail if anthropology doesn’t turn out to be the hot employment field of the future?" It might be fitting if the guidance counselor was getting a kick back for every that took an anthropology course at the local college.


Precisely. If scientists can't predict earthquakes, then they shouldn't be assuring people that there will NOT be an earthquake. At most, they should say, "The likelihood of an earthquake of magnitude X on any given day in this region is Y%; given the unusual circumstances of [numbers of tremors, etc.], I calculate the current likelihood at Z%."

And if a judge/jury could find that the scientists were motivated by factors other than duty to the public, then I can well understand a finding that the scientists are liable.

That said ....

Under what circumstances is it a legitimate function of government to calm public fears? People praise Roosevelt and Churchill for calming public fears during the Depression and WWII. But I suspect some people consequently let down their guard against further economic woe or blitz attack, and paid a price as a consequence.

In the case of the Italian earthquake, a series of tremors prompted a physics professor to predict a major earthquake. The public became alarmed. Assuming the Italian seismologists sincerely believed that the physicist's predictions were unjustified, was it inappropriate for them to join in the public campaign to calm people's fears?

Compare: What role should scientists play in the climate change debate? Should they simply offer scholarly pronouncements without regard to how those statements get spun in popular discourse? Or should they tailor their remarks to make them more accessible to the public -- even if this will expose the scientists to the charge that they're propagandizing as part of a political agenda?



Your recent guest Nate Silver wrote about this earthquake in his book. It's downstairs and I don't have time to go get it here, but I think the issue here wasn't that the scientists failed to predict the earthquake, which, no human can predict earthquakes, but that they publicly said that there wouldn't be one or something. I'm sure I'm getting it slightly wrong, and I still don't think the punishment is fair, but it's not just that they failed to predict the earthquake, it's that they did make a prediction, and it was a bad one.


I can't help thinking that this is the same country that imprisoned Galileo for his work in science.

John Lister

(Following up on Bobby's post)

Just a point of clarification: despite the widespread reporting, the convictions were not because the scientists failed to predict the earthquake. Rather, they were because a government official who spoke to the scientists then held a press conference incorrectly stating they'd concluded there was nothing to worry about. The court held the scientists then failed to publicly correct the official.

In other words, the issue wasn't not predicting a danger, but rather a false assertion of safety. While it doesn't change the insanity of the prosecution and conviction, the offense was about communication of risk, not the forecast of risk itself.


Kathie Clohessy

This story left me speechless. Fortunately I now know that I am not the only person who foresees a world where mass evacuations take place for the most paltry and barely comprehensible reasons, leaving economic chaos, civil unrest, and throngs of fleeing residents killed in their wake.

Fortunately I think this ridiculous example of the Italian legal system's many short-comings (vis-s-vis the Amanda Knox tragedy) will be overturned on appeal.

Dr. Constantinos Charalambous

Bad predictions are primarily based on false assumptions. The impact of these false assumptions can be disastrous as can be seen from this blog post.

Real fire caused by imaginary cigarette


Julien Couvreur

There are cases where bad predictions could incur liability, but it depends on contracts and defaults (norms).
I thought the default for private opinions was "caveat emptor" (buyer beware) unless an explicit commitment was made (directly to a person, or indirectly through a contract with an employer).

Now for government officials the question is a bit different. Since government taxes people to offer them "protection services", it is only fair that the people getting taxed have a say in what the expectations of service should be. The people who took charge of this responsibility should be accountable. This is a simple consequence of a non-voluntary arrangement, which can backfire.


What I read about this said the inhabitants of L'Aquila were specifically told by the scientist(s) in question they would never see an earthquake of any consequence. While I agree with your premise as you stated it, I think there's a difference between not predicting a tragedy and telling someone a tragedy absolutely won't happen.

All that said, I also think a manslaughter conviction sounds harsh - I don't know what the ramifications are for the scientist(s) in question, but prison time would be a bit much. Gut reaction.


We'll end up with more TSA type government agencies... Better to waste enormous resources "protecting the people" from unlikely events than very rarely have something bad happen.

Andreas Moser

To understand this case, you have to understand the basic principles of Italian criminal law: http://andreasmoser.wordpress.com/2012/10/24/italian-criminal-law/

Simone Calamai

Quoting John Lister a few comments above. Sentence was not about the scientists' failure to predict earthquake. It was about their failure to correct miscommunication of their results. Even in this case, the prosecution and conviction for manslaughter in pure insanity. I hope the decision will be overturned on appeal.

Eric M. Jones.

I encourage anyone to study the "Predictions" of the Central Intelligence Agency regarding a host of geopolitical events, e.g.: The CIA predicted that the Soviets were fat and happy and could soon dominate the world economically...30 days before it collapsed.

The list of similar errant CIA predictions is endless.