Picking the NFL Playoffs: How the Experts Fumble the Snap

Photo: Ed Yourdon

Our latest Freakonomics Radio podcast, “The Folly of Prediction,” is built around the premise that humans love to predict the future, but are generally terrible at it. (You can download/subscribe at iTunes, get the RSS feed, or read the transcript here.) But predictions about world politics and the economy are hard — there are so many moving parts. What about something with less variables, like football for example?

In the podcast, you’ll hear from Freakonomics researcher Hayes Davenport, who ran the numbers for us on how accurate expert NFL pickings have been for the last 3 years. He put together a guest post for us on football predictions.

Picking the NFL Playoffs: How the Experts Fumble the Snap

As careers in journalism go, making preseason NFL predictions is about as safe as they come these days. The picks you make in August can’t be reviewed for four months, and by that time almost nobody remembers or cares what any individual picker predicted. So when Freakonomics asked me to look at the success rate of NFL experts in predicting division winners at the beginning of the season, I was excited to look back at the last few years of picks and help offer this industry one of its first brushes with accountability.

For my sample data, I looked at a total of 101 sets of predictions over the course of three seasons from pundits representing the three major sports journalism outlets: ESPN, Sports Illustrated, and USA Today. Overall, I found that NFL pickers chose the division winner correctly about 36% of the time. If they were picking at random, they’d get it right 25% of the time, so they’re certainly performing above the level of pure chance. Some years were better than others, of course: in 2010, the pickers actually did predict division winners with 25% accuracy, exactly as well as they’d do with a blindfold and a dartboard.

But how good is that 36% overall figure, really? Think of it this way: if the pickers were allowed to rule out one team from every division and then choose at random, they’d pick winners 33% of the time. So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.

How is it possible, then, that paid experts pick at about the same level of accuracy as any armchair prognosticator? Looking deeper into the predictions, one statistically derived explanation stands out: widespread risk aversion. See, football pickers tend to rely rather heavily on the previous year’s playoff picture when making their predictions: on average, they include 8.13 teams that made the playoffs the year before, and only 3.83 new teams. You can see the appeal of a strategy like that: if a team was good last season, shouldn’t they be good again?

But compared to any other major professional sport, NFL standings are the most volatile from year to year, and picking eight-plus teams to return to the playoffs just doesn’t make sense in a league with such parity. Over the last fifteen seasons, the NFL has averaged exactly six new teams in the playoffs every year, meaning that half of the playoff picture is completely different from the year before. The turnover is pretty consistent from year to year: not once in that decade-and-a-half have there been fewer than five new playoff teams from the season before.

Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. But in the 105 sets of picks we looked at over the last three years, pickers chose six or more new teams only 11 times. Some pickers actually predicted that there would be only one new team in the playoffs, which has never happened in the modern NFL.

There are plenty of individual cases that demonstrate how picking based on the previous season has led predictors astray. The Dallas Cowboys, for example, were a “hot” team in 2008. They’d won the hugely competitive NFC East the season before, and were selected by 29 of 35 pundits to repeat this. They ended up finishing third. So the next season, the pundits turned their back on the Cowboys, reluctant to get burned again. Only six of the 35 chose the Cowboys to win the division…which, of course, they did. In 2010, the pundits leapt back onto the bandwagon, with 26 of the 35 picking the Cowboys to take the NFC East. Once again, they were a year behind the trend: the Cowboys finished third again.

There is, of course, an obvious reason why pickers are inclined to play it safe: if you go with the crowd and pick a team that was successful last year, you’re unlikely to get completely embarrassed. Fans won’t remember if you picked the failed 10-6 Giants to make the playoffs, but they might remember the guy who picked the 2-14 Panthers to take their division. And if you do miss on all your picks for a season, you’ve got an easy out if everyone else was picking the same way you did. The football prediction industry, like the game it covers, is a lot easier when all the players are working together.

Hayes Davenport is a writer for Allen Gregory, which premieres in October on Fox. He also blogs about basketball at CelticsHub.com.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 39


  1. Mitch says:

    I think that predictions are bad so much as some things are impossible to predict. There’s too much randomness, particularly star players getting injured, to foresee the narrative that actually plays out, and sometimes the successful prediction isn’t the right one, given the information that’s available at the time.

    Well-loved. Like or Dislike: Thumb up 7 Thumb down 0
  2. Peter says:

    I think your assertion that “a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs” is silly. That’s kind of like saying that if Tiger Woods had a 49% chance of winning golf tournaments a few years ago, if you had to pick one player you would pick someone in the field because the field wins 51% of the time.

    Well-loved. Like or Dislike: Thumb up 27 Thumb down 7
    • Jesse says:

      Hidden due to low comment rating. Click here to see.

      Disliked! Like or Dislike: Thumb up 9 Thumb down 23
      • Peter says:

        Can you explain why you think it’s a bad analogy?

        Well-loved. Like or Dislike: Thumb up 9 Thumb down 0
      • Jon says:

        Hidden due to low comment rating. Click here to see.

        Disliked! Like or Dislike: Thumb up 3 Thumb down 16
      • Peter says:

        Jon,

        The two outcomes analogy is exactly the point. The situation the author presents is a two-outcomes situation: previous winners vs. new teams, and he suggests picking one team from the group of new teams, which would be equivalent to in the Tiger Woods example choosing one player from the group of other players (the field). While everyone else in the field together have a better shot of winning, any player in the field individually would be a worse choice than Tiger. In the NFL example while the group of new teams together have a better shot of winning than the previous winner, any individual new team would likely be a worse choice than the previous winner.

        Well-loved. Like or Dislike: Thumb up 20 Thumb down 3
    • Steve says:

      Exactly. It really depends on if you are rewarded for a better average, or if you are only rewarded for getting 100%. If you know that approximately half the teams repeat their division, your best strategy is to pick every team to repeat, that way you will get 50%. Much better than picking 50% of teams to repeat (of which you only get 50% right), and then picking another team for the remaining spots, which you will presumably get less than 50% right. However, if you are only rewarded for a 100% accurate prediction, then yeah, you need to take some chances.

      Well-loved. Like or Dislike: Thumb up 12 Thumb down 1
      • Joshua Northey says:

        Well the smart move really depends on if you are trying to maximize: the accuracy of your prediction, or its uniqueness, or its shock value (to drive traffic), or more likely what weight you put on those three things.

        Honestly most of them are just relatively unsophisticated and are trying to predict something that is relatively unpredictable. Case close.

        If you want to ding them on something I would say they are weakest on differentiating between which things in the previous year’s performance that led to wins and losses tend to correlate with the the following year’s performance, and which do not.

        As an example: It is a very bad habit to predict good teams to repeat who recovered say 75% of opponent fumbles when opponent fumble recovery is in fact very random and there is no reason to expect a repeat of that good fortune.

        So you re-evaluate the teams based on what elements of their performance tend to correlate year over year and then use last years performance on those measures as your guide, instead of using the aggregate performance on all measures which includes a lot of statistical noise.

        Well-loved. Like or Dislike: Thumb up 6 Thumb down 1
  3. YX says:

    I think most of the expert would do much better picking teams if their goal is to get as many right as possible.
    As Vegas sharps who bring money in year after year shown, it is possible to be right more often than wrong.
    But since their goal is to generate viewer-ship (interesting but not controversial) rather than be correct, the result is greatly skewed.

    Thumb up 4 Thumb down 0
    • ktb says:

      I was thinking the exact same thing! It’s not necessarily an issue of correctly determining the winner(s) – it’s an issue of determining which team has the most followers who want to hear that you think their team will win.

      It would be interesting to see a study that compared playoff predictions to demand for articles about those teams. The Cowboys example above would probably fit that very well.

      Thumb up 2 Thumb down 0
  4. RickD says:

    If you know that 6/12 playoff teams will return and 6/20 non-playoff teams will make it to the following year’s playoffs, it’s not necessarily savvy to pick a slate of playoff teams including the previous year’s non-playoff teams. Your likelihood of error increases every time you include a non-playoff team to join the set.
    Of course, this is an overly simplistic way to view the problem.

    Well-loved. Like or Dislike: Thumb up 18 Thumb down 1
    • Howard says:

      True, but if you’re trying to paint an accurate picture of the upcoming season’s playoffs, you should produce one that accounts for the high parity of the NFL. If you choose the entire playoff slate from the previous season your odds max out at 60% or so.

      Thumb up 1 Thumb down 2
    • Will W says:

      If you eliminate 8 teams that you conclude don’t really have a shot at the playoffs (this is subjective but like he mentioned there are some that don’t have chance), you’re really just picking 12/24 with six old teams and six new ones. Then it really just becomes about risk aversion: if you assume that about six teams are going to turn over, then you have a chance at getting zero playoff teams correct but you also increase your odds to a possible 100% accuracy, which you almost definitely don’t have if you just pick the entire slate from the year before.

      Thumb up 1 Thumb down 1
  5. Adam Y says:

    Playoff predictions are bad for the same reason that soccer players kick to the side on penalty kicks way more often than they should. Its because a successful penalty kick is not the only objective. The objective is to have a successful penalty kick, but also not look stupid if you fail. If you kick it down the middle and the goalie stops it, then you’re perceived as an idiot. In the same way, if you go out on a limb and you’re really wrong with your picks then you look like an idiot.

    Thumb up 3 Thumb down 2
    • alex in chicago says:

      As a soccer player I never understood this sentiment. While there are a great many statistical analyses about Penalty Kicks, they all ignore reality. The fact is that if you kick to the side, and kick it well, you are guaranteed a goal. The only reason to kick to the middle is if you aren’t good enough to not hit your spot.

      I’m 4/5 on PKs in my career, the only “miss” was when I went “down the middle” and luckily it was during the game and the keeper deflected it right to me…I didn’t make the same mistake on the rebound shot.

      Thumb up 3 Thumb down 2
      • Kyle says:

        While I understand your reasoning of why you shouldn’t kick a penalty kick right down the middle, a statistical sample of 5 kicks hardly seems like a reliable sample size from which to draw a conclusion. I’m willing to stick with statistical analysis based on all penalty kicks over multiple seasons.

        Thumb up 1 Thumb down 0
    • Meghan says:

      The intention of a PK is to score…missing makes you look like an idiot UNLESS the goalie makes a save that seems impossible…kicking the ball furthest from a standing goalie makes them choose the correct direction, height, and speed at which a ball is traveling…kicking it at a goalie makes it easier to save…getting a goalie to have to leave their feet removes control and strength…

      Thumb up 1 Thumb down 0
  6. Greg says:

    I like the article! But I’m not sure that picking 6 last-year playoff teams and 6 last-year non playoff teams for this year’s playoffs is “savvy”. If 6 of each (say) make the playoffs this year, then each of last year’s playoff teams have a 50% chance of making this year’s playoffs, while only 30% of last year’s non-playoff teams will make the playoffs this year. If the goal is to maximize the number of correct predictions this year, why not just go with last year’s entire playoff slate – knowing that there will be mistakes but also knowing that trying to predict the mistakes will probably result in a worse showing?

    Well-loved. Like or Dislike: Thumb up 8 Thumb down 0
  7. Caleb b says:

    It works the same way in finance too. Being the hero vs the goat. If you buy Apple and get burned, no big deal, everyone liked Apple. But go out on a limb with some Brazilian copper mine and you had better be right or you’re gone.

    Thumb up 2 Thumb down 0
  8. Caleb b says:

    Sorry, should have mentioned…if everyone bets on that obscure Brazilian copper mine and blows up, the Fed sweeps in with a bunch of free money and you don’t have to cancel that fancy European ski vacation you had planned.

    For example: see Sub-prime MBS/ CDOs.

    Thumb up 2 Thumb down 0
  9. Joshua Northey says:

    “Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. ”

    This whole section shows you don’t understand statistics, which is kind of sad and shocking for someone posting on this site.

    Even if on average half of the teams each year are new, the best prediction could very well be that all of last year’s teams repeat under a wide array of assumptions.

    NFL Pundits are bad at picking NFL division winners mostly because it is unpredictable. That is all that really needs to be said on that subject.

    Thumb up 4 Thumb down 1
    • AL says:

      Also because they overrate wins, which are very poorly predictive of future success.

      Thumb up 0 Thumb down 0
    • Mike says:

      Agreed. Sad and very shocking.

      Here is a far superior take on pre-season predictions from someone that knows their stats:

      here: http://www.advancednflstats.com/2010/07/pre-season-predictions-are-still.html

      and here: http://fifthdown.blogs.nytimes.com/2009/07/22/an-antidote-to-pessimism-on-the-jets/

      Thumb up 1 Thumb down 0
    • Ryan says:

      This seems similar to picking NCAA bracket entries. If there is no incentive to pick upsets, it makes the most sense to go with the better seeds, which correlate with better past performance.

      Some posts have talked about how best to predict all 12 teams. Even if you know for a fact that exactly 6 of last year’s teams will make this year’s playoffs, you still have to determine WHICH six those will be. If my math skills serve me well, that’s 924 combinations in choosing 6 of last year’s 12 teams. There are 38,760 combinations in choosing 6 of the remaining 20 teams. That makes almost 36 million combinations of 12 teams. If you relax the constraint that 6 of last year’s teams must make it, that boosts the number of combinations to over 285 million (assuming one winner from each division and 2 wild cards from each conference).

      Thumb up 0 Thumb down 0
  10. Devin says:

    And how about all those idiots who pick all the one seeds in the NCAA tournament? Don’t they know how unlikely it is for that to happen?

    Thumb up 0 Thumb down 1
  11. MattieShoes says:

    “Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. ”

    No… Given that information, a savvy picker would predict all 12 playoff teams to repeat their playoff appearances, giving them 1 in 2 rather than 1 in 3 like the other “experts”.

    Well-loved. Like or Dislike: Thumb up 6 Thumb down 0
  12. Meghan says:

    Well, I challenge you Freakonomics to set up the challenge to choose the playoff teams for next season. Compare who is the most accurate…NFL analysts, statisticians, economists, and a set of fans…I’m in as a fan!!!!

    Thumb up 0 Thumb down 0
  13. Steve says:

    “So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.”

    The, of course, is misleading. Before the season started we knew the Bengals would be bad and probably the Dolphins, Broncos and Panthers too. But we didn’t know Manning was hurt, or that Jamal Charles and Eric Berry would get hurt, so Kansas City and Indiapolis were actually popular playoff picks. Meanwhile you’ve left out teams there were surefire cellar-dwellers two weeks ago, like Buffalo, which “traded its best player away” and Oakland (no comment needed).

    So I think it’s probably fair to say that ruling out one team in each division isn’t that easy, even with a lot of football knowledge.

    Well-loved. Like or Dislike: Thumb up 9 Thumb down 0
  14. Liaham Vaginakins says:

    Where’s my cupcakes!? They bloody well better taste like VIOLENCE!!! (with inspirational icing please :D)

    Thumb up 1 Thumb down 0
  15. China Pants says:

    My Cupcakes bring all the freakonomics to the yard, damn right, they’re better than yours.

    Thumb up 1 Thumb down 0
  16. Caleb b says:

    I want to see a comparison of superbowl predictions by experts vs vegas odds-makers.

    Thumb up 0 Thumb down 0
  17. BSK says:

    “So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.”

    Why would you rule out those particular teams? The Chiefs won the division just last year. The Vikings won it two years ago. There is nothing in the data that would suggest those teams are highly unlikely to win it this year, unless you are using data that includes each team’s 0-2 start, something the predictors did not know in August.

    The bigger issue is that there is often a razor thin margin between making the playoffs and not. If someone picked the Giants to make the playoffs last year, they would have been wrong. But the Giants finished tied for a playoff spot and missed out due to tiebreaker. I wouldn’t really classify that as a botched pick. Because the NFL is limited to 16 games, one game going differently than expected can have huge results. The Chiefs surprised the Chargers early last year. The Chiefs ended up winning the division over the Chargers by one game. Had that game gone the opposite way, as predicted, then the Chargers are in and the Chiefs are out and most predictors get those two right instead of those two wrong.

    If we look at predicted record and order of finish, my guess is that the legit predictors (and not all the guys who work for the major outlets are legit) are much better.

    This is a failed method of assessment, plain and simple.

    Thumb up 0 Thumb down 0
  18. BSK says:

    “So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.”

    Why would you rule out those particular teams? The Chiefs won the division just last year. The Vikings won it two years ago. There is nothing in the data that would suggest those teams are highly unlikely to win it this year, unless you are using data that includes each team’s 0-2 start, something the predictors did not know in August.

    The bigger issue is that there is often a razor thin margin between making the playoffs and not. If someone picked the Giants to make the playoffs last year, they would have been wrong. But the Giants finished tied for a playoff spot and missed out due to tiebreaker. I wouldn’t really classify that as a botched pick. Because the NFL is limited to 16 games, one game going differently than expected can have huge results. The Chiefs surprised the Chargers early last year. The Chiefs ended up winning the division over the Chargers by one game. Had that game gone the opposite way, as predicted, then the Chargers are in and the Chiefs are out and most predictors get those two right instead of those two wrong.

    If we look at predicted record and order of finish, my guess is that the legit predictors (and not all the guys who work for the major outlets are legit) are much better.

    This is a failed method of assessment, plain and simple.

    Thumb up 1 Thumb down 0
  19. john says:

    Diamond Mind Baseball used to look at this for annual baseball predictions, though the summaries appear to have been removed from the page.

    Thumb up 0 Thumb down 0
  20. placebro says:

    If prognosticators are barely getting 1/3 right, why on earth would you suggest that a savvy picker would choose six new teams? He’d then have to correctly guess which six teams miss the playoffs for that strategy to be remotely savvy.

    A savvy picker would pick the same 12 teams who made the playoffs the previous year, get 50% right, and dominate a field that only gets 36% correct.

    Thumb up 0 Thumb down 0
  21. G says:

    It would be great if you addressed the sample size issues with a 16 game season when addressing variability in postseason picks. One lucky bounce in a 16 game football season is equivalent to 10 lucky bounces in a 162 game baseball season. This could significantly affect the accuracy of predictions.

    “(this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example)”

    This is hindsight bias.

    Most of this year’s predictions were made with the expectation of a healthy Manning, the Colts were not obviously eliminated in August.
    The Chiefs are an incumbent division winner. Now they’re easy to eliminate? You criticize these experts for being risk averse because they pick too many incumbents but none of them picked the Chiefs. Most didn’t pick the Bears either.

    Thumb up 0 Thumb down 0
  22. Jeff says:

    The NFL has become so competitive, it’s become almost impossible to pick a team as the Super Bowl winner. Wild card teams being the winner has become common place. More often than not the team with the worse record wins against the team with the better record. Who would think a 9-7 team would ever be the champs,but it happened last year. I don’t even try anymore,and I laugh when my friends think they know how the playoffs are going to turn out. This year they were wrong most of the time. I know the 49ers are favored to win it this year, but I wouldn’t count the Ravens out by any means. They’ve been playing some inspired football at the right time. I’m not picking, just gonna watch and hope for a good game.

    Thumb up 0 Thumb down 0