Picking the NFL Playoffs: How the Experts Fumble the Snap

Photo: Ed Yourdon

Our latest Freakonomics Radio podcast, “The Folly of Prediction,” is built around the premise that humans love to predict the future, but are generally terrible at it. (You can download/subscribe at iTunes, get the RSS feed, or read the transcript here.) But predictions about world politics and the economy are hard — there are so many moving parts. What about something with less variables, like football for example?

In the podcast, you’ll hear from Freakonomics researcher Hayes Davenport, who ran the numbers for us on how accurate expert NFL pickings have been for the last 3 years. He put together a guest post for us on football predictions.

Picking the NFL Playoffs: How the Experts Fumble the Snap

As careers in journalism go, making preseason NFL predictions is about as safe as they come these days. The picks you make in August can’t be reviewed for four months, and by that time almost nobody remembers or cares what any individual picker predicted. So when Freakonomics asked me to look at the success rate of NFL experts in predicting division winners at the beginning of the season, I was excited to look back at the last few years of picks and help offer this industry one of its first brushes with accountability.

For my sample data, I looked at a total of 101 sets of predictions over the course of three seasons from pundits representing the three major sports journalism outlets: ESPN, Sports Illustrated, and USA Today. Overall, I found that NFL pickers chose the division winner correctly about 36% of the time. If they were picking at random, they’d get it right 25% of the time, so they’re certainly performing above the level of pure chance. Some years were better than others, of course: in 2010, the pickers actually did predict division winners with 25% accuracy, exactly as well as they’d do with a blindfold and a dartboard.

But how good is that 36% overall figure, really? Think of it this way: if the pickers were allowed to rule out one team from every division and then choose at random, they’d pick winners 33% of the time. So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark.

How is it possible, then, that paid experts pick at about the same level of accuracy as any armchair prognosticator? Looking deeper into the predictions, one statistically derived explanation stands out: widespread risk aversion. See, football pickers tend to rely rather heavily on the previous year’s playoff picture when making their predictions: on average, they include 8.13 teams that made the playoffs the year before, and only 3.83 new teams. You can see the appeal of a strategy like that: if a team was good last season, shouldn’t they be good again?

But compared to any other major professional sport, NFL standings are the most volatile from year to year, and picking eight-plus teams to return to the playoffs just doesn’t make sense in a league with such parity. Over the last fifteen seasons, the NFL has averaged exactly six new teams in the playoffs every year, meaning that half of the playoff picture is completely different from the year before. The turnover is pretty consistent from year to year: not once in that decade-and-a-half have there been fewer than five new playoff teams from the season before.

Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. But in the 105 sets of picks we looked at over the last three years, pickers chose six or more new teams only 11 times. Some pickers actually predicted that there would be only one new team in the playoffs, which has never happened in the modern NFL.

There are plenty of individual cases that demonstrate how picking based on the previous season has led predictors astray. The Dallas Cowboys, for example, were a “hot” team in 2008. They’d won the hugely competitive NFC East the season before, and were selected by 29 of 35 pundits to repeat this. They ended up finishing third. So the next season, the pundits turned their back on the Cowboys, reluctant to get burned again. Only six of the 35 chose the Cowboys to win the division…which, of course, they did. In 2010, the pundits leapt back onto the bandwagon, with 26 of the 35 picking the Cowboys to take the NFC East. Once again, they were a year behind the trend: the Cowboys finished third again.

There is, of course, an obvious reason why pickers are inclined to play it safe: if you go with the crowd and pick a team that was successful last year, you’re unlikely to get completely embarrassed. Fans won’t remember if you picked the failed 10-6 Giants to make the playoffs, but they might remember the guy who picked the 2-14 Panthers to take their division. And if you do miss on all your picks for a season, you’ve got an easy out if everyone else was picking the same way you did. The football prediction industry, like the game it covers, is a lot easier when all the players are working together.

Hayes Davenport is a writer for Allen Gregory, which premieres in October on Fox. He also blogs about basketball at CelticsHub.com.


Mitch

I think that predictions are bad so much as some things are impossible to predict. There's too much randomness, particularly star players getting injured, to foresee the narrative that actually plays out, and sometimes the successful prediction isn't the right one, given the information that's available at the time.

Peter

I think your assertion that "a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs" is silly. That's kind of like saying that if Tiger Woods had a 49% chance of winning golf tournaments a few years ago, if you had to pick one player you would pick someone in the field because the field wins 51% of the time.

Jesse

Worst analogy ever…

Peter

Can you explain why you think it's a bad analogy?

Jon

I agree with Jesse in that it's a terrible analogy. If you could bet on the rest of the field or not Tiger to win, you'd be right more often than not. Using a two outcomes analogy to explain a multi-variable outcome is misleading.

Peter

Jon,

The two outcomes analogy is exactly the point. The situation the author presents is a two-outcomes situation: previous winners vs. new teams, and he suggests picking one team from the group of new teams, which would be equivalent to in the Tiger Woods example choosing one player from the group of other players (the field). While everyone else in the field together have a better shot of winning, any player in the field individually would be a worse choice than Tiger. In the NFL example while the group of new teams together have a better shot of winning than the previous winner, any individual new team would likely be a worse choice than the previous winner.

YX

I think most of the expert would do much better picking teams if their goal is to get as many right as possible.
As Vegas sharps who bring money in year after year shown, it is possible to be right more often than wrong.
But since their goal is to generate viewer-ship (interesting but not controversial) rather than be correct, the result is greatly skewed.

ktb

I was thinking the exact same thing! It's not necessarily an issue of correctly determining the winner(s) - it's an issue of determining which team has the most followers who want to hear that you think their team will win.

It would be interesting to see a study that compared playoff predictions to demand for articles about those teams. The Cowboys example above would probably fit that very well.

RickD

If you know that 6/12 playoff teams will return and 6/20 non-playoff teams will make it to the following year's playoffs, it's not necessarily savvy to pick a slate of playoff teams including the previous year's non-playoff teams. Your likelihood of error increases every time you include a non-playoff team to join the set.
Of course, this is an overly simplistic way to view the problem.

Howard

True, but if you're trying to paint an accurate picture of the upcoming season's playoffs, you should produce one that accounts for the high parity of the NFL. If you choose the entire playoff slate from the previous season your odds max out at 60% or so.

Adam Y

Playoff predictions are bad for the same reason that soccer players kick to the side on penalty kicks way more often than they should. Its because a successful penalty kick is not the only objective. The objective is to have a successful penalty kick, but also not look stupid if you fail. If you kick it down the middle and the goalie stops it, then you're perceived as an idiot. In the same way, if you go out on a limb and you're really wrong with your picks then you look like an idiot.

alex in chicago

As a soccer player I never understood this sentiment. While there are a great many statistical analyses about Penalty Kicks, they all ignore reality. The fact is that if you kick to the side, and kick it well, you are guaranteed a goal. The only reason to kick to the middle is if you aren't good enough to not hit your spot.

I'm 4/5 on PKs in my career, the only "miss" was when I went "down the middle" and luckily it was during the game and the keeper deflected it right to me...I didn't make the same mistake on the rebound shot.

Kyle

While I understand your reasoning of why you shouldn't kick a penalty kick right down the middle, a statistical sample of 5 kicks hardly seems like a reliable sample size from which to draw a conclusion. I'm willing to stick with statistical analysis based on all penalty kicks over multiple seasons.

Greg

I like the article! But I'm not sure that picking 6 last-year playoff teams and 6 last-year non playoff teams for this year's playoffs is "savvy". If 6 of each (say) make the playoffs this year, then each of last year's playoff teams have a 50% chance of making this year's playoffs, while only 30% of last year's non-playoff teams will make the playoffs this year. If the goal is to maximize the number of correct predictions this year, why not just go with last year's entire playoff slate - knowing that there will be mistakes but also knowing that trying to predict the mistakes will probably result in a worse showing?

Jared

"I was excited to look back at the last few years of picks and help offer this industry one of its first brushes with accountability."

You must not read ESPN's TMQ. Specifically his annual bad predictions review:
http://sports.espn.go.com/espn/page2/story?page=easterbrook/110215_tuesday_morning_quarterback&sportCat=nfl

Caleb b

It works the same way in finance too. Being the hero vs the goat. If you buy Apple and get burned, no big deal, everyone liked Apple. But go out on a limb with some Brazilian copper mine and you had better be right or you're gone.

Caleb b

Sorry, should have mentioned...if everyone bets on that obscure Brazilian copper mine and blows up, the Fed sweeps in with a bunch of free money and you don't have to cancel that fancy European ski vacation you had planned.

For example: see Sub-prime MBS/ CDOs.

Joshua Northey

"Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. "

This whole section shows you don't understand statistics, which is kind of sad and shocking for someone posting on this site.

Even if on average half of the teams each year are new, the best prediction could very well be that all of last year's teams repeat under a wide array of assumptions.

NFL Pundits are bad at picking NFL division winners mostly because it is unpredictable. That is all that really needs to be said on that subject.

AL

Also because they overrate wins, which are very poorly predictive of future success.

Devin

And how about all those idiots who pick all the one seeds in the NCAA tournament? Don't they know how unlikely it is for that to happen?

MattieShoes

"Given that information, a savvy picker relying on statistical precedent would choose six new teams when predicting the playoffs. "

No... Given that information, a savvy picker would predict all 12 playoff teams to repeat their playoff appearances, giving them 1 in 2 rather than 1 in 3 like the other "experts".

Meghan

Well, I challenge you Freakonomics to set up the challenge to choose the playoff teams for next season. Compare who is the most accurate...NFL analysts, statisticians, economists, and a set of fans...I'm in as a fan!!!!

Steve

"So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark."

The, of course, is misleading. Before the season started we knew the Bengals would be bad and probably the Dolphins, Broncos and Panthers too. But we didn't know Manning was hurt, or that Jamal Charles and Eric Berry would get hurt, so Kansas City and Indiapolis were actually popular playoff picks. Meanwhile you've left out teams there were surefire cellar-dwellers two weeks ago, like Buffalo, which "traded its best player away" and Oakland (no comment needed).

So I think it's probably fair to say that ruling out one team in each division isn't that easy, even with a lot of football knowledge.

Read more...

Liaham Vaginakins

Where's my cupcakes!? They bloody well better taste like VIOLENCE!!! (with inspirational icing please :D)

NIKOOKOO CIKESAURUS

Yum, Shirwa A

China Pants

My Cupcakes bring all the freakonomics to the yard, damn right, they're better than yours.

Caleb b

I want to see a comparison of superbowl predictions by experts vs vegas odds-makers.

BSK

"So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark."

Why would you rule out those particular teams? The Chiefs won the division just last year. The Vikings won it two years ago. There is nothing in the data that would suggest those teams are highly unlikely to win it this year, unless you are using data that includes each team's 0-2 start, something the predictors did not know in August.

The bigger issue is that there is often a razor thin margin between making the playoffs and not. If someone picked the Giants to make the playoffs last year, they would have been wrong. But the Giants finished tied for a playoff spot and missed out due to tiebreaker. I wouldn't really classify that as a botched pick. Because the NFL is limited to 16 games, one game going differently than expected can have huge results. The Chiefs surprised the Chargers early last year. The Chiefs ended up winning the division over the Chargers by one game. Had that game gone the opposite way, as predicted, then the Chargers are in and the Chiefs are out and most predictors get those two right instead of those two wrong.

If we look at predicted record and order of finish, my guess is that the legit predictors (and not all the guys who work for the major outlets are legit) are much better.

This is a failed method of assessment, plain and simple.

Read more...

BSK

"So if you consider that most NFL divisions include at least one team with no hope of finishing first (this year’s Bengals, Chiefs, Dolphins, Panthers, Broncos, Vikings, and Manning-less Colts, for example), the pickers only need a minimum of NFL knowledge before essentially guessing in the dark."

Why would you rule out those particular teams? The Chiefs won the division just last year. The Vikings won it two years ago. There is nothing in the data that would suggest those teams are highly unlikely to win it this year, unless you are using data that includes each team's 0-2 start, something the predictors did not know in August.

The bigger issue is that there is often a razor thin margin between making the playoffs and not. If someone picked the Giants to make the playoffs last year, they would have been wrong. But the Giants finished tied for a playoff spot and missed out due to tiebreaker. I wouldn't really classify that as a botched pick. Because the NFL is limited to 16 games, one game going differently than expected can have huge results. The Chiefs surprised the Chargers early last year. The Chiefs ended up winning the division over the Chargers by one game. Had that game gone the opposite way, as predicted, then the Chargers are in and the Chiefs are out and most predictors get those two right instead of those two wrong.

If we look at predicted record and order of finish, my guess is that the legit predictors (and not all the guys who work for the major outlets are legit) are much better.

This is a failed method of assessment, plain and simple.

Read more...

john

Diamond Mind Baseball used to look at this for annual baseball predictions, though the summaries appear to have been removed from the page.

placebro

If prognosticators are barely getting 1/3 right, why on earth would you suggest that a savvy picker would choose six new teams? He'd then have to correctly guess which six teams miss the playoffs for that strategy to be remotely savvy.

A savvy picker would pick the same 12 teams who made the playoffs the previous year, get 50% right, and dominate a field that only gets 36% correct.