Picking the NFL Playoffs: How the Experts Fumble the Snap

Photo: Ed Yourdon

Our latest Freakonomics Radio podcast, “The Folly of Prediction,” is built around the premise that humans love to predict the future, but are generally terrible at it. (You can download/subscribe at iTunes, get the RSS feed, or read the transcript here.) But predictions about world politics and the economy are hard — there are so many moving parts. What about something with less variables, like football for example?

In the podcast, you’ll hear from Freakonomics researcher Hayes Davenport, who ran the numbers for us on how accurate expert NFL pickings have been for the last 3 years. He put together a guest post for us on football predictions.

Picking the NFL Playoffs: How the Experts Fumble the Snap

As careers in journalism go, making preseason NFL predictions is about as safe as they come these days. The picks you make in August can’t be reviewed for four months, and by that time almost nobody remembers or cares what any individual picker predicted. So when Freakonomics asked me to look at the success rate of NFL experts in predicting division winners at the beginning of the season, I was excited to look back at the last few years of picks and help offer this industry one of its first brushes with accountability.

For my sample data, I looked at a total of 101 sets of predictions over the course of three seasons from pundits representing the three major sports journalism outlets: ESPN, Sports Illustrated, and USA Today. Overall, I found that NFL pickers chose the division winner correctly about 36% of the time. If they were picking at random, they’d get it right 25% of the time, so they’re certainly performing above the level of pure chance. Some years were better than others, of course: in 2010, the pickers actually did predict division winners with 25% accuracy, exactly as well as they’d do with a blindfold and a dartboard.

But how good is that 36% overall figure, really? Think of it this way: if the pickers were allowed to rule out one team from every division and then choose at random, they’d pick winners 33% of the time. So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.

How is it possible, then, that paid experts pick at about the same level of accuracy as any armchair prognosticator? Looking deeper into the predictions, one statistically derived explanation stands out: widespread risk aversion. See, football pickers tend to rely rather heavily on the previous year’s playoff picture when making their predictions: on average, they include 8.13 teams that made the playoffs the year before, and only 3.83 new teams. You can see the appeal of a strategy like that: if a team was good last season, shouldn’t they be good again?

But compared to any other major professional sport, NFL standings are the most volatile from year to year, and picking eight-plus teams to return to the playoffs just doesn’t make sense in a league with such parity. Over the last fifteen seasons, the NFL has averaged exactly six new teams in the playoffs every year, meaning that half of the playoff picture is completely different from the year before. The turnover is pretty consistent from year to year: not once in that decade-and-a-half have there been fewer than five new playoff teams from the season before.

Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. But in the 105 sets of picks we looked at over the last three years, pickers chose six or more new teams only 11 times. Some pickers actually predicted that there would be only one new team in the playoffs, which has never happened in the modern NFL.

There are plenty of individual cases that demonstrate how picking based on the previous season has led predictors astray. The Dallas Cowboys, for example, were a “hot” team in 2008. They’d won the hugely competitive NFC East the season before, and were selected by 29 of 35 pundits to repeat this. They ended up finishing third. So the next season, the pundits turned their back on the Cowboys, reluctant to get burned again. Only six of the 35 chose the Cowboys to win the division…which, of course, they did. In 2010, the pundits leapt back onto the bandwagon, with 26 of the 35 picking the Cowboys to take the NFC East. Once again, they were a year behind the trend: the Cowboys finished third again.

There is, of course, an obvious reason why pickers are inclined to play it safe: if you go with the crowd and pick a team that was successful last year, you’re unlikely to get completely embarrassed. Fans won’t remember if you picked the failed 10-6 Giants to make the playoffs, but they might remember the guy who picked the 2-14 Panthers to take their division. And if you do miss on all your picks for a season, you’ve got an easy out if everyone else was picking the same way you did. The football prediction industry, like the game it covers, is a lot easier when all the players are working together.

Hayes Davenport is a writer for Allen Gregory, which premieres in October on Fox. He also blogs about basketball at CelticsHub.com.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 39

View All Comments »
  1. Joshua Northey says:

    “Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. ”

    This whole section shows you don’t understand statistics, which is kind of sad and shocking for someone posting on this site.

    Even if on average half of the teams each year are new, the best prediction could very well be that all of last year’s teams repeat under a wide array of assumptions.

    NFL Pundits are bad at picking NFL division winners mostly because it is unpredictable. That is all that really needs to be said on that subject.

    Thumb up 4 Thumb down 1
    • AL says:

      Also because they overrate wins, which are very poorly predictive of future success.

      Thumb up 0 Thumb down 0
    • Mike says:

      Agreed. Sad and very shocking.

      Here is a far superior take on pre-season predictions from someone that knows their stats:

      here: http://www.advancednflstats.com/2010/07/pre-season-predictions-are-still.html

      and here: http://fifthdown.blogs.nytimes.com/2009/07/22/an-antidote-to-pessimism-on-the-jets/

      Thumb up 1 Thumb down 0
    • Ryan says:

      This seems similar to picking NCAA bracket entries. If there is no incentive to pick upsets, it makes the most sense to go with the better seeds, which correlate with better past performance.

      Some posts have talked about how best to predict all 12 teams. Even if you know for a fact that exactly 6 of last year’s teams will make this year’s playoffs, you still have to determine WHICH six those will be. If my math skills serve me well, that’s 924 combinations in choosing 6 of last year’s 12 teams. There are 38,760 combinations in choosing 6 of the remaining 20 teams. That makes almost 36 million combinations of 12 teams. If you relax the constraint that 6 of last year’s teams must make it, that boosts the number of combinations to over 285 million (assuming one winner from each division and 2 wild cards from each conference).

      Thumb up 0 Thumb down 0
  2. Devin says:

    And how about all those idiots who pick all the one seeds in the NCAA tournament? Don’t they know how unlikely it is for that to happen?

    Thumb up 0 Thumb down 1
  3. MattieShoes says:

    “Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. ”

    No… Given that information, a savvy picker would predict all 12 playoff teams to repeat their playoff appearances, giving them 1 in 2 rather than 1 in 3 like the other “experts”.

    Well-loved. Like or Dislike: Thumb up 6 Thumb down 0
  4. Meghan says:

    Well, I challenge you Freakonomics to set up the challenge to choose the playoff teams for next season. Compare who is the most accurate…NFL analysts, statisticians, economists, and a set of fans…I’m in as a fan!!!!

    Thumb up 0 Thumb down 0
  5. Steve says:

    “So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.”

    The, of course, is misleading. Before the season started we knew the Bengals would be bad and probably the Dolphins, Broncos and Panthers too. But we didn’t know Manning was hurt, or that Jamal Charles and Eric Berry would get hurt, so Kansas City and Indiapolis were actually popular playoff picks. Meanwhile you’ve left out teams there were surefire cellar-dwellers two weeks ago, like Buffalo, which “traded its best player away” and Oakland (no comment needed).

    So I think it’s probably fair to say that ruling out one team in each division isn’t that easy, even with a lot of football knowledge.

    Well-loved. Like or Dislike: Thumb up 9 Thumb down 0
  6. Liaham Vaginakins says:

    Where’s my cupcakes!? They bloody well better taste like VIOLENCE!!! (with inspirational icing please :D)

    Thumb up 1 Thumb down 0
  7. China Pants says:

    My Cupcakes bring all the freakonomics to the yard, damn right, they’re better than yours.

    Thumb up 1 Thumb down 0
  8. Caleb b says:

    I want to see a comparison of superbowl predictions by experts vs vegas odds-makers.

    Thumb up 0 Thumb down 0