What’s Wrong With Punishing Bad Predictions?

(Photo: Alessandro Giangiulio)

In the heat of a Presidential campaign, it can be hard to pay attention to other news. But a small-seeming story out of Italy yesterday has, to my mind, the potential to shape the future as much as a Presidential election.

As reported by ABC, the BBC, the Wall Street Journal, the New York Times, and elsewhere, an Italian court has convicted seven earthquake experts of failing to appropriately sound the alarm bell for an earthquake that wound up killing more than 300 people in L’Aquila in 2009. The experts received long prison sentences and fines of more than $10 million. (Addenum: Roger Pielke Jr. discusses the “mischaracterizations” of the verdict.)

There is of course the chance that the verdict will be thrown out upon appeal, discredited as an emotional response to a horrible tragedy. 

But let’s consider what the verdict represents as of today: scientists and government officials are being held personally accountable for the failure to adequately predict the scale of a natural disaster.

If you ever heard our “Folly of Prediction” podcast, you might think we would embrace this verdict. Our main point, after all, was that the world is awash in prediction, much of it terrible, in large part because there are strong incentives to make bold predictions and weak penalties for bold predictions don’t come true.

For example: I could write here today that the Dow will reach 35,000 by the end of 2013. In the very unlikely event that this prediction comes true, I would be lauded and rewarded to no end. And in the likely event my prediction doesn’t come true, it will be pretty much forgotten.

But the Italian earthquake story is in fact the opposite of this common scenario. By not predicting an anomalous event which then did occur, the scientists are being penalized for not sounding an alarm bell loudly enough.

If a child of mine had been killed in the Italian earthquake, I might well feel entitled to vengeance. But, in a world where experts in all realms are constantly sounding alarm bells that turn out to be meaningless, how does this verdict potentially shape the future? I can see at least two effects:

  1. Scientists and others might become less willing to predict the future — which, given our skill at prediction, might not be a terrible thing but which, given the fact that future-gazing is often an excuse for useful research, might be a terrible thing.
  2. A message has been sent that it is better to predict a calamity than to fail to predict a calamity. If you fail to predict a calamity, as anomalous as it may be, you risk being sent to jail and fined. So the incentive to predict a calamity has just been significantly increased.

Does this mean I can have my financial planner sent to jail every time the market tanks? Can I have my kid’s guidance counselor sent to jail if anthropology doesn’t turn out to be the hot employment field of the future?

Am I wrong in thinking this is a big, big story?

FWIW, we did another podcast about how hard it is to predict earthquakes.

(HT: inter alia, Steven Merahn and Marginal Revolution)

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.



View All Comments »
  1. Michael says:

    “For example: I could write here today that the Dow will reach 35,000 by the end of 2013. In the very unlikely event that this prediction comes true, I would be lauded and rewarded to no end. And in the likely event my prediction doesn’t come true, it will be pretty much forgotten.”

    Is that a reference to Kevin Hassett? He wrote “Dow 36,000.” He’s still a major GOP economics adviser despite being so terribly wrong with his DJIA prediction (which was mainly the result of a basic math error and a misunderstanding of what numbers you plug into pricing models).

    Thumb up 9 Thumb down 5
  2. Kellan says:

    When I heard this news yesterday, your “Folly of Prediction” podcast was the first thing that came to mind. Much like the fortune teller who was held legally responsible for her predictions, it’s equally ridiculous that the earthquake-predicting experts should be held responsible for the reciprocal omissive error of “failing to alert the public loud enough”.

    Maybe the solution here is to use an exact system of measuring predictability – so if the chances of damaging earthquake reach a set probability, the public is alerted. If this system is impossible, then the prediction is not based on solid science and therefore should not be punishable.

    Thumb up 2 Thumb down 0
  3. Nanno says:

    If this judgment stands, they probably would need to have been fined as well if it was the other way around, predicting a calamity that didn’t happen, due to associated costs of evacuation. So the risk of making such predictions (if they can be held liable) very high.

    I would like to know whether they have signed a convenant or contract in which it is specified that they can be held liable for their predictions. First of all I don’t think they would have signed such a contract due to the difficulty of predicting such an event exactly (how bad), timely (when) and correctly (where). Secondly, such a contract should then also have a high compensation for correct predictions in order for them to be able to pay the liability claims ($10mln divided by seven scientists is a lot of money).

    As someone once said: “there are no mistakes in science, only lessons”. How can you sentence seven scientist to six years in prison for not being able to to what is generally agreed to be very, very difficult and in the past has rarely been done exactly correct?

    Shouldn’t the local government bear the resposibility for not earthquake-proofing their community in an area prone to earthquakes? Or didn’t they want to change its historic center (which would seem quite logical)?

    Or should the people who choose to live in an area prone to earthquakes bear the risk?

    Well-loved. Like or Dislike: Thumb up 36 Thumb down 0
  4. Bart says:

    Wait! Wait! You mean anthropology isn’t the hot employment field of the future? Sheesh… and here I thought I might be able to get a life.

    Thumb up 4 Thumb down 0
  5. Howard Tayler says:

    I think the big problem here is that the scientific method requires scientists to be wrong, and to be wrong a LOT. Earthquake science, even for as much as we know, is still in its infancy. Jailing scientists for being wrong is a direct attack upon the methods that allow them to improve.

    As was pointed out elsewhere (I forget where — my numbers could be off) something like half of all big quakes are preceded by small temblors, but only around 2% of small temblors are followed by big quakes. Based on that information, and the best science we’ve got available, the big quake was pretty unlikely.

    Well-loved. Like or Dislike: Thumb up 28 Thumb down 0
  6. Pepe says:

    There is just one big prediction that is a no-win, predicting the end of the world… If you are wrong everybody laughs at you, if you are right, there’s nobody there to congratulate you.

    Well-loved. Like or Dislike: Thumb up 16 Thumb down 0
    • TexCIS says:

      Hidden due to low comment rating. Click here to see.

      Disliked! Like or Dislike: Thumb up 13 Thumb down 23
  7. Jake says:

    Based upon Roger Pielke Jr’s article, the predications were not based upon science but rather the predications were to designed to support their own personal & political agendas. Maybe punishment is warranted when this occurs

    “Does this mean I can have my financial planner sent to jail every time the market tanks?” It might if all of the financial planner’s recommendations were only focused upon maxmizing the planner’s commission income.

    “Can I have my kid’s guidance counselor sent to jail if anthropology doesn’t turn out to be the hot employment field of the future?” It might be fitting if the guidance counselor was getting a kick back for every that took an anthropology course at the local college.

    Well-loved. Like or Dislike: Thumb up 11 Thumb down 0
    • nobody.really says:

      Precisely. If scientists can’t predict earthquakes, then they shouldn’t be assuring people that there will NOT be an earthquake. At most, they should say, “The likelihood of an earthquake of magnitude X on any given day in this region is Y%; given the unusual circumstances of [numbers of tremors, etc.], I calculate the current likelihood at Z%.”

      And if a judge/jury could find that the scientists were motivated by factors other than duty to the public, then I can well understand a finding that the scientists are liable.

      That said ….

      Under what circumstances is it a legitimate function of government to calm public fears? People praise Roosevelt and Churchill for calming public fears during the Depression and WWII. But I suspect some people consequently let down their guard against further economic woe or blitz attack, and paid a price as a consequence.

      In the case of the Italian earthquake, a series of tremors prompted a physics professor to predict a major earthquake. The public became alarmed. Assuming the Italian seismologists sincerely believed that the physicist’s predictions were unjustified, was it inappropriate for them to join in the public campaign to calm people’s fears?

      Compare: What role should scientists play in the climate change debate? Should they simply offer scholarly pronouncements without regard to how those statements get spun in popular discourse? Or should they tailor their remarks to make them more accessible to the public — even if this will expose the scientists to the charge that they’re propagandizing as part of a political agenda?

      Thumb up 4 Thumb down 2
      • nobody.really says:

        Future Tense basically gets it right:

        “[T]he whole mess could easily have been avoided if the scientists had clearly asserted the truth of the matter by proclaiming: ‘We just don’t know.’

        The matter is made a bit more complex because, according to media reports, part of the motivation for the l’Aquila Seven’s meeting in the first place was to discredit a non-expert ‘laboratory technician’ (as the Economist described him, thus making clear his lowly status in the hierarchy of expertise) who was scaring local residents by predicting that a big earthquake was about to occur. So there seems to have been some sense that the goal of the Seven’s meeting was to calm unnecessary anxieties that were being stoked by a charlatan. At the same time, quelling these fears meant being clear about where the expertise actually lay—with the Seven, not with the mere technician. In this case the right message to convey would have been: ‘We don’t know—and he doesn’t know either’—but that would seem to undercut the very expertise that gave the Seven their status and legitimacy to begin with.”

        See http://www.slate.com/blogs/future_tense/2012/10/30/l_aquila_earthquake_manslaughter_case_the_importance_of_scientists_saying.html

        Thumb up 1 Thumb down 0
  8. Bobby says:

    Your recent guest Nate Silver wrote about this earthquake in his book. It’s downstairs and I don’t have time to go get it here, but I think the issue here wasn’t that the scientists failed to predict the earthquake, which, no human can predict earthquakes, but that they publicly said that there wouldn’t be one or something. I’m sure I’m getting it slightly wrong, and I still don’t think the punishment is fair, but it’s not just that they failed to predict the earthquake, it’s that they did make a prediction, and it was a bad one.

    Thumb up 3 Thumb down 2