Bring Your Questions for FiveThirtyEight Blogger Nate Silver, Author of The Signal and the Noise

Nate Silver first gained prominence for his rigorous analysis of baseball statistics. He became even more prominent for his rigorous analysis of elections, primarily via his FiveThirtyEight blog. (He has also turned up on this blog a few times.)

Now Silver has written his first book, The Signal and the Noise: Why So Many Predictions Fail — But Some Don’t. I have only read chunks so far but can already recommend it. (I would like to think his research included listening to our radio hour “The Folly of Prediction,” but I have no idea.)

A section of Signal about weather prediction was recently excerpted in the Times Magazine. Relatedly, his chapter called “A Climate of Healthy Skepticism” has already been attacked by the climate scientist Michael Mann. Given the stakes, emotions, and general unpredictability that surround climate change, I am guessing Silver will collect a few more such darts. (Yeah, we’ve been there.)

In the meantime, he has agreed to field questions about his new book from Freakonomics readers. So feel free to post your questions in the comments section below, and we’ll post his replies in short course. Here, to get you started, is the book’s table of contents:

 

1. A CATASTROPHIC FAILURE OF PREDICTION

2. ARE YOU SMARTER THAN A TELEVISION PUNDIT?

3. ALL I CARE ABOUT IS W’S AND L’S

4. FOR YEARS YOU’VE BEEN TELLING US THAT RAIN IS GREEN

5. DESPERATELY SEEKING SIGNAL

6. HOW TO DROWN IN THREE FEET OF WATER

7. ROLE MODELS

8. LESS AND LESS AND LESS WRONG

9. RAGE AGAINST THE MACHINES

10. THE POKER BUBBLE

11. IF YOU CAN’T BEAT ’EM . . . 

12. A CLIMATE OF HEALTHY SKEPTICISM

13. WHAT YOU DON’T KNOW CAN HURT YOU

 This post is no longer accepting comments. The answers to the Q&A can be found here.

COMMENTS: 54


  1. David L says:

    Do you see a media bias toward (presidential) polls that demonstrate the narrowest results? (in other words, would an EIC/ME prefer to publish a 45-45 poll over a 48-42 poll)?

    The cynic in me argues that keeping the rhetorical election close will attract more readers/viewers.

    Well-loved. Like or Dislike: Thumb up 8 Thumb down 1
  2. paul clark says:

    There are so many polls produced during the two-year-plus American presidential election cycle. Do the polls really help educate voters or are they mainly an easy prop for media and pundits to create a short-term news burst?

    Thumb up 4 Thumb down 0
  3. Camille Sweeney says:

    What’s the biggest mistake that major pollsters make?
    Camille Sweeney
    Twitter @artofdoingbook

    Thumb up 4 Thumb down 0
  4. Quin says:

    How would you compare the accuracy of polls vs. prediction markets for American elections? Why don’t you include the latter in your estimates on FiveThirtyEight?

    Well-loved. Like or Dislike: Thumb up 7 Thumb down 1
  5. Seminymous Coward says:

    Which party would win a baseball game with only its national-level politicians as players?

    Well-loved. Like or Dislike: Thumb up 15 Thumb down 2
  6. Ryan says:

    What’s your response to the “attacks” by Michael Mann on your climate science chapter?

    Well-loved. Like or Dislike: Thumb up 12 Thumb down 1
    • James says:

      Hidden due to low comment rating. Click here to see.

      Disliked! Like or Dislike: Thumb up 3 Thumb down 9
      • J1 says:

        Orbital mechanics problems aren’t predictions; they’re (not particularly complicated) math problems that assume compliance with known laws of physics. Determining the path of a spacecraft after launch is not analogous to predicting the weather.

        Well-loved. Like or Dislike: Thumb up 8 Thumb down 1
      • James says:

        They certainly are predictions: at time t you predict an event that will take place at time t+dt.

        You’re right that they’re predictions made by doing computations based on known laws of physics. So are many other sorts of predictions, for instance those involving weather and climate. Indeed, weather prediction is not particularly complicated, physically or mathematically. The difficulty in making long-term weather predictions stems (as Silver himself discusses in the linked NYT article) from the inherent sensitivity to initial conditions of the physics involved.

        That is where the Freakonomics guys display their ignorance: they make blanket statements about the impossibility of prediction, when in fact many problems (like orbital mechanics) are quite predictable, others (like weather) are predictible within calculable limits, still others may not be predictable at all. Rather than dismiss all prediction as folly, it would seem sensible to try to understand what can and can’t be predicted, and why.

        Thumb up 5 Thumb down 6
      • Ryan says:

        I think this is a very important distinction that Mann bring up. Making predictions based on fundamental physical laws is very different than making predictions about, say, economic activity.

        However, it should be pointed out that the calculations behind weather prediction are not as settled as James says. While its true that we (physicists) have a very good handle on particle-particle interactions, weather simulations are not done by simulating trillions (or whatever) of individual atoms. They are done, out of necessity, by utilizing approximations to get the simulation down to a manageable scale.

        It’s these approximations, rather than the underlying laws of nature, that add an element of uncertainty to the predictions. The physics is well understood and relatively simple. The approximations are neither.

        I do agree though that to lump scientific weather models in with “less scientific” predictions and then try to make blanket statements about the entire group is folly.

        Thumb up 3 Thumb down 0
      • James says:

        Sorry, but it is the exponential nature of the physical laws that makes long-term weather prediction fundamentally impossible. Silver’s linked NYT article has a good explanation. The approximations would be good enough, if the physical laws allowed it. After all, we don’t have to simulate every atom of a space probe to predict its trajectory: the physical laws are such that we can approximate it very well as a single particle.

        Now in theory it would be predictible by particle simulation IF we could know the exact initial conditions of every single atom, but we can’t. Not can’t as in we don’t have the instruments & data storage needed, but can’t as in Heisenberg’s Uncertainty Principle is a fundamental law of nature.

        Thumb up 1 Thumb down 0
      • Ryan says:

        Surely chaos plays a role in the difficulty of long term weather prediction. But to pretend that there aren’t modeling issues or that the computational methods are somehow “settled” is disingenuous.

        Right now instrumentation, data storage and computational power are many orders of magnitude larger impediments to attempting such an ‘atomic’ calculation than fundamental limits are, so Heisenberg is irrelevant.

        Even without Heisenberg, such a simulation wouldn’t work because of finite precision. There are many sources of inaccuracy in these simulations.

        Thumb up 2 Thumb down 0
  7. Scott Blackburn says:

    The argument for aggregation is that if you ask individuals to estimate something, the average of their results is more accurate than any one individual predictor. This has led to successful poll based models.

    But polls are not completely independent variables like estimates. Do you think its possible that we will see an election, where aggregation will build in some methodology bias and lead to vastly inaccurate predictions?

    Thumb up 1 Thumb down 0
  8. Quin says:

    In the primaries, my preferred candidates never seem to win, but as I talk to people in my circle, they say they like my candidate best also but feel compelled to vote for one of the front-runners so as not to waste their vote. How would (did?) elections fare without polling? How do published polling results bias future polls? Can anything be done about it?

    Well-loved. Like or Dislike: Thumb up 18 Thumb down 1
  9. Dave M says:

    If the public had no knowledge of poll results would the election outcome be any different? Is seeing “your guy” falling behind a motivation to vote? Perhaps seeing him too far behind causes people to stay home since their vote won’t matter?

    Well-loved. Like or Dislike: Thumb up 7 Thumb down 0
  10. Jeff Bladt says:

    What industries do you see as next to revolutionized by data? I keep hearing that PR is particularly vulnerable, in the emperor-has-no-clothes sense.

    Thumb up 4 Thumb down 0
  11. Nick says:

    Who should I bet on to win the world series?

    Thumb up 3 Thumb down 3
  12. frankenduf says:

    do u vote?- and if so, does it produce cognitive dissonance if ur stats lead u to believe that ur vote wont change the outcome?

    Thumb up 1 Thumb down 2
  13. rempart says:

    I have the book on order, can’t wait for the 27th. How about some about a fantasy baseball question? In a draft league, do you think it is best to use a good prediction model (Pecota) for example and let it stand so to speak. I mean of course, just line them up top to bottom and only use minor manipulations ( something like inside info on an injury). The question implies the use of intuition comapred to the model. Daniel Kahneman refers to this in his book “thinking slow and fast”. The notion of models outperforming intuition. He also talks about checklists as being useful, if applied in a disciplined manner. What I’m getting at it is, a baseline Pecota type projection, followed up by a checklist of the top 5-6 things that move a player off his projection. Do you think this could help identify the Jose Bautista, Cliff Lee errors earlier. Or is this just noise?

    Thumb up 1 Thumb down 0
  14. Owinok says:

    Nate, could you let readers here know what statistical analysis package you use? Secondly, have any firms or collaborators asked you to evaluate any packages that they have made?

    Thumb up 0 Thumb down 0
  15. Lenny says:

    It appears to me that prediction markets such as Intrade are continuing to grow, so I’m assuming that the insight that one can get from them is changing as more people use them. What’s your opinion of the usefulness of prediction markets and how has that changed over the past few years? How do you feel they will change moving forward?

    Thumb up 0 Thumb down 0
  16. Dutch says:

    What’s your take on the public equity markets–are they efficient or nearly-efficient? How much predictive power does a change in a company’s stock price have on future earnings annoucement?

    Thumb up 1 Thumb down 0
  17. Dutch says:

    Pardon me if you mind me asking, but what do you do with your personal investments? Do you buy any individual stock names, or do you stick with indexing? Overall, how do you apply your work in predictions to your personal finances? To me, the prices of securities have some predictive power and I’d love to hear your take on how you think about this and apply it to yourself.

    Thumb up 2 Thumb down 0
  18. Devin says:

    For anyone interested in the subject of predictions, I’d recommend Future Babble by Dan Gardner. I haven’t read Silver’s book, but I would imagine that it would be similar to Gardner’s.

    Has Silver read Future Babble? Has anyone at Freakonomics read it?

    Does Silver’s book contain anything about Philip Tetlock?

    Thumb up 0 Thumb down 0
    • Eric M. Jones. says:

      Anyone interested in predictions MUST READ “The Book of Predictions” by Amy Wallace Irving Wallace David Wallechinsky 1980. You can always pick up a used copy someplace. I keep mine in a velvet-lined box inside a safe.

      BTW:

      I always like to do surveys by asking 1000 people with IQs of 150 and 1000 people with IQ’s of 50. That way I get a population average IQ of 100. My surveys describe the way the world works very well.

      Thumb up 1 Thumb down 1
    • Alan T. says:

      My kindle software tells me that the name “Tetlock” appears 38 times in the book.

      Thumb up 1 Thumb down 0
  19. RGJ says:

    Hi Nate,

    Baseball question, if you don’t mind.

    Why would a National League team, particularly one with financial constraints, not consider the following approach?

    Sign no high priced starting pitchers. Fill your pitching roster with middle reliever/reliever pitchers with different attributes — junk ballers, power pitchers, sidearmers — with a lefty/righty balance.

    Balance your roster with righty and lefty position players.

    In the course of a game, substitute (except for blowouts) for the pitcher when his place comes up in the order, and do so with a righty or lefty batter depending on the pitcher.

    Advantages:

    – You bring an American League offense to the National League.
    – In key situations, you have a disproportionate number of righty/lefty pitching matchups.
    – In key situations, you have a disproportionate number of righty/lefty fielding matchups.
    – Rather than seeing the same pitcher four or five times in a row, an opposing batter may face batters of different styles and handedness (sic?) in a game.
    – Keep payroll down by avoiding high priced starting pitchers.
    – Arguable, but may have an injury benefit by avoiding 100 plus pitch outings to pitchers.
    – keeps the entire roster active

    Disadvantages/Challenges

    – will have trouble attracting pitchers focused on traditional counting stats and Ws.
    – will pose a pitching management challenge in rest and IP.

    Naturally, in a blow out situation, you could simply have a young pitcher stretch out for 7 or 8 innings and save everyone else’s arm. And it is important to note that now, when there is an upside blowout with a team’s ace, those usual innings are pitched by him in search of the almighty W.

    I ran this by Bill James on his website. His comment was that the only downside he saw was the part about attracting top starting pitchers. He also said that, in reality, baseball has been moving this way, albeit glacially, with the growth of middle relievers and closers and emphasis on righty/lefty situational matchups.

    Thoughts?

    Thumb up 0 Thumb down 0
  20. Laura Harrison McBride says:

    Hidden due to low comment rating. Click here to see.

    Disliked! Like or Dislike: Thumb up 0 Thumb down 8
  21. Stuart says:

    Are polls a very good predictor of actual behavior? It seems to me they are lousy indicators of future behavior and there are much better models of behavior (starting a diet, quitting smoking, beginning an exercise program, etc).

    If I am right that polling is not a very good indicator of future behavior, then why do otherwise reputable organizations rely and amplify their “findings” in such polls and why do scientific journals publish polls in the same manner they publish true economic studies or models?

    For instance, the Legacy Foundation recently conducted a major poll where 40 percent of respondents said if the FDA were to ban menthol cigarettes they would quit smoking. It would seem that an economic model based on cigarette taxes going up or some other model would do a much better job of predicting actual quitting smoking behavior in the event of such an FDA ban.

    “A new study released today in the American Journal of Public Health (AJPH) presents the first peer-reviewed data on menthol smokers’ behavioral intention if menthol cigarettes were taken off the market, a decision pending with the U.S. Food and Drug Administration (FDA). The results of the national survey show that nearly 40 percent of menthol smokers say they would quit if menthol cigarettes were no longer available.”

    http://www.prnewswire.com/news-releases/many-menthol-smokers-say-they-would-quit-if-menthols-were-no-longer-available-170563516.html

    Thumb up 0 Thumb down 0
  22. James says:

    Nate,

    Is there any research on whether partisan political campaigning can prevent or delay an economic recovery? My theory is that “confidence” in the economy may have weakened as a direct consequence of the Rebublican primary season and heightened election-year politics in general. Republicans are very eager this year to tell us that the economy is bad under President Obama. Does that rhetoric actually cause the economy to be worse? Is economic growth harder to come by in election years in general? Maybe due to uncertainty?

    Thanks for your great work,
    James

    Thumb up 1 Thumb down 0
  23. mannyv says:

    Q: what factors can help people see which predictions won’t fail?

    As an example, you use various models for 538. How do you know which set of models is correct at any given time?

    Thumb up 0 Thumb down 0
  24. David says:

    Is there any circumstance where an event would shake up the political landscape so much you would pull the 538 forecast, and if so, can you give me a sense of what kind of magnitude it would need to be?

    Thumb up 0 Thumb down 0
  25. Bob says:

    You mentioned in a recent blog that certain states are often highly correlated in elections. Do you specifically model these correlations or are the states uncorrelated in your simulations?

    Thumb up 0 Thumb down 0
  26. Michael J says:

    How much of Romney’s current deficit is due to factors beyond his control, and how much of it do you think is due to problems of his own making? Where do you think the forecast would stand right now if Republicans had managed to find a nominee with political skill along the lines of Clinton or Reagan?

    Thumb up 0 Thumb down 0
  27. Fionn O'Donovan says:

    I’d like to hear Nate’s view on this question: how might we reliably distinguish between accurate predictions that are accurate because their perpetrators used correct scientific methodology from those that appear simply to be lucky guesses?

    I find this particularly interesting because philosophers of science since Imre Lakatos have attempted to show that we can tell if a theory is scientific if it can predict future events accurately – early physicists, for example, were able to predict the occurence of certain solar events. But it still seems possible, of course, that a person with no knowledge at all of a subject could still make an accurate prediction, or even a series of accurate predictions, simply by chance. How do we know where to draw the line?

    Thumb up 1 Thumb down 0
  28. Chris B. says:

    The ability to make an accurate prediction given a random, representative sample is mathematically well known and turns out (as I remember from too many years ago of math) to be around 600 “samples” (people, microbes, division III referees). This group should tell us within some reasonable accuracy what an outcome should be (Akin pulls out a win, the African subcontinent suffers longer with malaria, three of Golden Tate’s fingers on a ball constitutes simultaneous possession).

    In practice, this just never seems to work out correctly. If it did, I believe all pollsters would hover around the exact same data points with fluctuations account for in their error-bars. So, why doesn’t it?

    Selection bias (“We sampled more than 5,000 married, white 2 children stay-at-home mothers with husbands earning over $350,000″)
    Question bias (“You say your a democrat? Do you think it’s more likely Obama will raise taxes, or pull the still beating heart out of a human Mola Ram style?”)
    Mathematical bias (“I count in base 10, but then I add (unconverted) in base 12″).

    Why do polls vary significantly? Why are exit polls so dicey? And why, given what we know about the demographics of voters and their alignments in exit polls and their makeup of the electorate so hard to extrapolate to better data?

    P.S. The next time you predict my Cleveland Indians to do well I wish you would bury that feeling way down deep and leave it unexpressed. You are to lucky talismans what the rabbit is to the rabbit’s foot.

    Thumb up 1 Thumb down 0
  29. Morgan Warstler says:

    Sept 4th 2010, Nate told voters (via NYT) that the odds of a 60 seat gain by the GOP was only 1:4.

    How badly does Obama lose, if Nate is exactly as wrong in the same direction as Nate was wrong in 2010?

    And since Nate was that wrong once, IF Nate is that wrong again in 2012, shouldn’t we deduce Nate has dramatically refactor his model to favor the right for than it currently does?

    Thumb up 2 Thumb down 1
  30. Dan Schroeder says:

    What’s your assessment of economics as a discipline, judged in terms of its ability to make politically useful predictions? For example, can economists predict with any reliability what the economic impact of a tax cut or a government spending program will be?

    Thumb up 0 Thumb down 0
  31. Grossamer says:

    Is it possible that the Republicans redistricting has become so effective that it is backfiring on moderate republicans and has allowed the far right to become more powerful? If this is the case and Obama wins, it would seem that redistricting was a factor. Yes? No?

    Thumb up 0 Thumb down 0
  32. J. Cross says:

    Way back in 2005, Steven Levitt said: “My contention is that the secret to Oakland’s success has little to do with the things described in Moneyball, such as the emphasis on finding the skills in baseball that are good at producing runs, but not properly valued by the market.”

    To support this statement he said: “The reason the A’s win, year after year, is because they have better pitchers than anyone else. the 2004 season is typical: the A’s were ninth out of fifteen teams in the American League in scoring runs, but had the second lowest ERA.”

    Did Levitt forget about park adjustments? Who was right Lewis or Levitt?

    Thumb up 0 Thumb down 0
  33. Liza Brings says:

    Every election cycle has something unique that could be huge. 2008 the Bradley effect. 2012 stock mkt up, emp down. How do you factor these in?

    Thumb up 1 Thumb down 0
  34. John says:

    - anxiously waiting for release tomorrow on Amazon /Kindle so haven’t read yet but wondering:

    When \ predictions involve human ‘systems’ & behavior ( social, economic, political etc) that are by their very nature ‘adaptive’, how do you deal with the tricky “Heisenberg Principle” -like effect where the very act of predicting itself becomes a factor that adds information that alters the system and influences individual and/or collective behavior?

    Thumb up 0 Thumb down 0
  35. Grenouille says:

    Given that your political predictions rely on polling, how much faith do you have in pollsters asking good questions? Granted the question “who would you vote for…” is straight forward enough, but questions like “do you feel better off than you were 4 years ago…” and “who do you trust…” etc. can all be very biased or vague questions when asked in a certain way. I guess what I am asking is, how far can we reasonably trust in the accuracy of a poll when there may be flaws in the delivery of the survey that either purposely or accidentally sway the response?

    Thumb up 0 Thumb down 0
  36. Adam says:

    I’ve been a fan since your BP days. For a seemingly mild- mannered numbers guy you run into a lot of controversy – what gives you the confidence to take on ‘establishment’ figures, be they Joe Morgan or Dick Morris?

    Thumb up 0 Thumb down 0
  37. Martin Schwimmer says:

    Does Mr Silver have any idea as to the scope pf the possible effect of the alleged voter suppression laws? Does the 538 model takes this into account?

    Thumb up 1 Thumb down 0
  38. Adam says:

    I find people, myself included, have trouble with probabilities. I get 50% – that’s a coin flip, and I get 0% and 100%, but the further away from those three points the less I feel I truly understand. Should I give up if chances are 19%? Should I count my winnings at 81%?

    Thumb up 0 Thumb down 0
  39. John Hogan says:

    Explain the effects of cell phone data (or lack of) in today’s polls…

    Thumb up 0 Thumb down 0
  40. Neil says:

    Has there been a presidential election where the strategy of “rallying the base” has ever worked? No matter the rhetoric and cable news polarization, it seems that the candidate that does the best job of moving to the center always wins, and turnout never plays as big of a role as analysts make it out to be.

    Thumb up 0 Thumb down 0
  41. Daniel says:

    For about as long as you’ve been releasing your predictions for 2012, the Electoral College Distribution chart in your forecast displays a big spike somewhere in the 330s, at a much higher probability than anything else around it. What combination(s) of states lead to that outcome, and why is it so likely?

    Thumb up 1 Thumb down 0
  42. Robin Vernooij says:

    On basis of what methodology criteria do you rate the polls on bias? Which kind of specific methodology features of the polls do you take into account? Data collection, data extraction and sample size? How willing are all the polls to share this with you?

    Thanks for your great work and greetings from Barcelona, your work is appreciated here as well.

    Well-loved. Like or Dislike: Thumb up 6 Thumb down 0
  43. Paul Kim says:

    How much wood could a woodchuck chuck if a woodchuck could chuck wood? Follow-up question: why is it so frickin’ hard to predict the outcome of a week’s slate of NFL football games (large group of actors) and the outcome of certain, high-profile Supreme Court cases (small group of actors)?

    Thumb up 0 Thumb down 0
  44. Alan T says:

    Under what circumstances will a voter actually change his/her mind about whom to vote for? (I’m sure every politician in the world would like to know this, too.) I understand that this rarely happens (see, for example, http://themonkeycage.org/blog/2012/09/21/there-go-the-undecided-voters/), and that most of the action involves undecided voters deciding whom to vote for.

    Also, if political scientist are right that voters rarely change their minds, how can a large swing in the polls ever occur? A classic example that your briefly mention in your book is that of Michael Dukakis, who was ahead of GHW Bush by 10% at one point in 1988.

    Thumb up 0 Thumb down 0
  45. Mike says:

    What is your take on the theories of election rigging in Ohio during the 2004 presidential race? Is this just a case of people not realizing the inherent biases of polls and assuming that polling results should tie directly to election results?

    Thumb up 0 Thumb down 0