Picking the NFL Playoffs: How the Experts Fumble the Snap

Photo: Ed Yourdon

Our latest Freakonomics Radio podcast, “The Folly of Prediction,” is built around the premise that humans love to predict the future, but are generally terrible at it. (You can download/subscribe at iTunes, get the RSS feed, or read the transcript here.) But predictions about world politics and the economy are hard — there are so many moving parts. What about something with less variables, like football for example?

In the podcast, you’ll hear from Freakonomics researcher Hayes Davenport, who ran the numbers for us on how accurate expert NFL pickings have been for the last 3 years. He put together a guest post for us on football predictions.

Picking the NFL Playoffs: How the Experts Fumble the Snap

As careers in journalism go, making preseason NFL predictions is about as safe as they come these days. The picks you make in August can’t be reviewed for four months, and by that time almost nobody remembers or cares what any individual picker predicted. So when Freakonomics asked me to look at the success rate of NFL experts in predicting division winners at the beginning of the season, I was excited to look back at the last few years of picks and help offer this industry one of its first brushes with accountability.

For my sample data, I looked at a total of 101 sets of predictions over the course of three seasons from pundits representing the three major sports journalism outlets: ESPN, Sports Illustrated, and USA Today. Overall, I found that NFL pickers chose the division winner correctly about 36% of the time. If they were picking at random, they’d get it right 25% of the time, so they’re certainly performing above the level of pure chance. Some years were better than others, of course: in 2010, the pickers actually did predict division winners with 25% accuracy, exactly as well as they’d do with a blindfold and a dartboard.

But how good is that 36% overall figure, really? Think of it this way: if the pickers were allowed to rule out one team from every division and then choose at random, they’d pick winners 33% of the time. So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.

How is it possible, then, that paid experts pick at about the same level of accuracy as any armchair prognosticator? Looking deeper into the predictions, one statistically derived explanation stands out: widespread risk aversion. See, football pickers tend to rely rather heavily on the previous year’s playoff picture when making their predictions: on average, they include 8.13 teams that made the playoffs the year before, and only 3.83 new teams. You can see the appeal of a strategy like that: if a team was good last season, shouldn’t they be good again?

But compared to any other major professional sport, NFL standings are the most volatile from year to year, and picking eight-plus teams to return to the playoffs just doesn’t make sense in a league with such parity. Over the last fifteen seasons, the NFL has averaged exactly six new teams in the playoffs every year, meaning that half of the playoff picture is completely different from the year before. The turnover is pretty consistent from year to year: not once in that decade-and-a-half have there been fewer than five new playoff teams from the season before.

Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. But in the 105 sets of picks we looked at over the last three years, pickers chose six or more new teams only 11 times. Some pickers actually predicted that there would be only one new team in the playoffs, which has never happened in the modern NFL.

There are plenty of individual cases that demonstrate how picking based on the previous season has led predictors astray. The Dallas Cowboys, for example, were a “hot” team in 2008. They’d won the hugely competitive NFC East the season before, and were selected by 29 of 35 pundits to repeat this. They ended up finishing third. So the next season, the pundits turned their back on the Cowboys, reluctant to get burned again. Only six of the 35 chose the Cowboys to win the division…which, of course, they did. In 2010, the pundits leapt back onto the bandwagon, with 26 of the 35 picking the Cowboys to take the NFC East. Once again, they were a year behind the trend: the Cowboys finished third again.

There is, of course, an obvious reason why pickers are inclined to play it safe: if you go with the crowd and pick a team that was successful last year, you’re unlikely to get completely embarrassed. Fans won’t remember if you picked the failed 10-6 Giants to make the playoffs, but they might remember the guy who picked the 2-14 Panthers to take their division. And if you do miss on all your picks for a season, you’ve got an easy out if everyone else was picking the same way you did. The football prediction industry, like the game it covers, is a lot easier when all the players are working together.

Hayes Davenport is a writer for Allen Gregory, which premieres in October on Fox. He also blogs about basketball at CelticsHub.com.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 39

View All Comments »
  1. BSK says:

    “So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.”

    Why would you rule out those particular teams? The Chiefs won the division just last year. The Vikings won it two years ago. There is nothing in the data that would suggest those teams are highly unlikely to win it this year, unless you are using data that includes each team’s 0-2 start, something the predictors did not know in August.

    The bigger issue is that there is often a razor thin margin between making the playoffs and not. If someone picked the Giants to make the playoffs last year, they would have been wrong. But the Giants finished tied for a playoff spot and missed out due to tiebreaker. I wouldn’t really classify that as a botched pick. Because the NFL is limited to 16 games, one game going differently than expected can have huge results. The Chiefs surprised the Chargers early last year. The Chiefs ended up winning the division over the Chargers by one game. Had that game gone the opposite way, as predicted, then the Chargers are in and the Chiefs are out and most predictors get those two right instead of those two wrong.

    If we look at predicted record and order of finish, my guess is that the legit predictors (and not all the guys who work for the major outlets are legit) are much better.

    This is a failed method of assessment, plain and simple.

    Thumb up 0 Thumb down 0
  2. BSK says:

    “So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.”

    Why would you rule out those particular teams? The Chiefs won the division just last year. The Vikings won it two years ago. There is nothing in the data that would suggest those teams are highly unlikely to win it this year, unless you are using data that includes each team’s 0-2 start, something the predictors did not know in August.

    The bigger issue is that there is often a razor thin margin between making the playoffs and not. If someone picked the Giants to make the playoffs last year, they would have been wrong. But the Giants finished tied for a playoff spot and missed out due to tiebreaker. I wouldn’t really classify that as a botched pick. Because the NFL is limited to 16 games, one game going differently than expected can have huge results. The Chiefs surprised the Chargers early last year. The Chiefs ended up winning the division over the Chargers by one game. Had that game gone the opposite way, as predicted, then the Chargers are in and the Chiefs are out and most predictors get those two right instead of those two wrong.

    If we look at predicted record and order of finish, my guess is that the legit predictors (and not all the guys who work for the major outlets are legit) are much better.

    This is a failed method of assessment, plain and simple.

    Thumb up 1 Thumb down 0
  3. john says:

    Diamond Mind Baseball used to look at this for annual baseball predictions, though the summaries appear to have been removed from the page.

    Thumb up 0 Thumb down 0
  4. placebro says:

    If prognosticators are barely getting 1/3 right, why on earth would you suggest that a savvy picker would choose six new teams? He’d then have to correctly guess which six teams miss the playoffs for that strategy to be remotely savvy.

    A savvy picker would pick the same 12 teams who made the playoffs the previous year, get 50% right, and dominate a field that only gets 36% correct.

    Thumb up 0 Thumb down 0
  5. G says:

    It would be great if you addressed the sample size issues with a 16 game season when addressing variability in postseason picks. One lucky bounce in a 16 game football season is equivalent to 10 lucky bounces in a 162 game baseball season. This could significantly affect the accuracy of predictions.

    “(this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example)”

    This is hindsight bias.

    Most of this year’s predictions were made with the expectation of a healthy Manning, the Colts were not obviously eliminated in August.
    The Chiefs are an incumbent division winner. Now they’re easy to eliminate? You criticize these experts for being risk averse because they pick too many incumbents but none of them picked the Chiefs. Most didn’t pick the Bears either.

    Thumb up 0 Thumb down 0
  6. Jeff says:

    The NFL has become so competitive, it’s become almost impossible to pick a team as the Super Bowl winner. Wild card teams being the winner has become common place. More often than not the team with the worse record wins against the team with the better record. Who would think a 9-7 team would ever be the champs,but it happened last year. I don’t even try anymore,and I laugh when my friends think they know how the playoffs are going to turn out. This year they were wrong most of the time. I know the 49ers are favored to win it this year, but I wouldn’t count the Ravens out by any means. They’ve been playing some inspired football at the right time. I’m not picking, just gonna watch and hope for a good game.

    Thumb up 0 Thumb down 0