A Football Outsider Answers Your Questions

We recently solicited your questions for Bill Barnwell, a Football Outsider and one of the many authors of the new Football Outsiders Almanac. Here are his replies, which cover everything from miracle turnarounds to the role of injuries to his own background. (I have a strong prediction as to which will be his least-popular answer; can you guess what I’m guessing?) Thanks to all, and especially Bill, for participating.

As a Bills fan, I’m interested to hear your thoughts on perennial cellar dwellers. Have you found any statistical indicators of pending turnarounds? Also, I’m interested in the specifics of the looming lockout. – Scott

The miracle turnaround you speak of is often accompanied by a remarkable, unexpected, drastic shift in team health. We’ve found in the past that about 25 percent of a team’s year-to-year change in wins is accounted for by the change in their injury rate.
As examples, consider the biggest swings in win-loss record from each of the past two seasons. In 2007, the Miami Dolphins went 1-15. By Adjusted Games Lost (our proprietary injury metric and a variant on the HGL metric that is mentioned in the link above), they were the eighth most-injured team in football. A year later, they went 11-5, shocking everyone en route to an AFC East title. While much of the credit was given to the team’s quickly-overrated “Wildcat” offensive scheme, there was one patently obvious factor driving their success: the Dolphins had become the league’s fourth-healthiest team.
The easiest way to crystallize that information is to look at, arguably, the team’s two best players from that season, quarterback Chad Pennington and halfback Ronnie Brown. Each played a full 16-game season. Combined, they’d played 11 NFL seasons before the 2008 campaign, and in those 11 seasons they had a total of one season uninterrupted by injury (Pennington in 2006). Both subsequently suffered injuries during the 2009 season, and the Dolphins went 7-9.
In 2008-09, that team was the Cincinnati Bengals. The 2008 Bengals were, by our metrics, the most-injured team of the decade. In 2009, they were still the ninth-most injured team in the league, but the difference in AGL between the two seasons was still enough to rank as the largest improvement in health across the league. They improved from 4.5 wins (four wins and a tie against Philadelphia) to ten, and won one of the league’s toughest divisions.
Both the Dolphins and the Bengals rank among the ten largest year-to-year positive swings in health over the past decade. Those ten teams improved by an average of more than four games.
Now, a change in health wasn’t the only thing going these teams’ ways. The Dolphins had Pennington — among the league’s best quarterbacks when healthy — fall into their lap when the Jets traded for Brett Favre. They also played the league’s fourth-easiest schedule, and went 7-2 in games decided by a touchdown or less. That’s a total fluke, and they consequently went 6-4 in those games a year ago. The Bengals got Carson Palmer back from injury, and he’s one of the league’s better quarterbacks when healthy. They played an average schedule, but they went 6-3 in those close games after going 7-11-1 in such games over the previous three seasons.
So, who is the team most likely to suddenly get much healthier in 2010? Would you believe the Buffalo Bills? Sure enough, the Bills were the most-injured team in football by a fair margin last year; in fact, only three teams accrued more AGL in a given season over the past decade than the Bills did in 2009. The Bengals were one of those teams. The 2006 Browns improved by six wins in the subsequent season, while the 2008 Lions went from zero wins to two in 2009. The Buffalo Bills have been a reasonably healthy team in the past, with four consecutive top-ten (healthy) finishes in AGL from 2003-2006, and a 13th-place finish in 2008. Some teams have a habit of finishing towards the top or bottom of the injury tables, but the Bills are not likely to be particularly injury-riddled again in 2009. If they were a team with even average health in 2010, they would likely have the largest positive swing forward of any team.
I also believe that we’re probably underestimating how hard it is for a team like Buffalo to compete while beset by so many injuries. One of the issues we have to deal with in projecting and analyzing football performance is that context affects performance much more dramatically than it does in baseball. If a baseball team has an injured right fielder, they can call up or acquire a replacement-level right fielder and immediately insert him into the lineup. The actual level of his performance is subject to variance and park effects, of course, but he’s not going to change his batting stance or throw with the opposite arm because he’s playing in a different town.
That’s just not the case in football. A team that’s forced to sign a free agent off the street is not just settling for a player whose talent is not up to the level of an average NFL player. The team needs to teach that player their playbook; what usually takes an entire summer has to happen in a matter of days. The interaction effects of having an underprepared or significantly below-average player are far greater; the worst thing a hitter can do in most at-bats is strikeout or ground into a double play, but a right tackle that doesn’t know his assignment can get his quarterback severely injured. Twelve of the 22 players who started for the Bills in Week 1 were not starting by Week 17; four of the guys in the starting lineup that week were street free agents that were cut by NFL teams and signed by the Bills during the season. Even a below-average NFL starter that knows the playbook is a massive upgrade on players like that.
So, in short, the path to a Bills playoff run amounts to a dramatic increase in health and an improvement on their 2-5 record in games decided by a touchdown or less. Unearthing a quarterback might also help.

How did you wind up where you are? We know that Bill James did a lot of his early work while he was a night watchman at a bean factory, but what’s the Bill Barnwell story? – Jon

About all I can really say is that I had no intention of ever becoming a football writer, that’s for sure. I went to Northeastern University in Boston as a Computer Science major before I realized I hated programming, and ended up majoring in Media Studies. Although I liked to write, I actually consciously avoided journalism — I went to visit The Boston Globe for an interview and they explained to me that one of the last sports interns had to sit outside a hospital overnight and wait for Terry Francona to leave.
While I was in Boston, I wanted to join a fantasy baseball league so I found one on Baseball Primer, the sabermetric community site, and joined shortly thereafter. The commissioner of that league happened to be Aaron Schatz, who created Football Outsiders a year later. He’s now my boss. I started at the bottom of the FO organizational chart, doing data entry as an unpaid intern (although admittedly while I was sitting around doing nothing at my “real” paid internship), and I’ve seen my role in the company steadily expand from there. I graduated from NU in 2006 and took on a variety of post-grad liberal arts jobs — I was a concierge to the really rich and entitled holders of a particularly exclusive credit card, a tech copywriter, and I even wrote about video games for a while. I moved into a full-time role with FO in 2008.
I think what’s far more interesting, though, is that my job — and Football Outsiders — couldn’t have existed as recently as ten years ago. No magazine or newspaper would have underwritten the work that we do, even though there’s clearly a national market of people who are interested in our work and the concepts behind it. Whenever I read those articles about the death of the American newspaper, I always wonder about the jobs that have been created by the changes in the news media.

Do you account for personality and psychological effect?
For example:
1) Peyton Manning basically ended his season with an interception in the big game.
2) Aaron Rodgers‘ season ended on what could have instead been ruled a face-mask penalty.
Do these situations burn inside the players and create a bigger force this season, or alternatively tear them up, never to be what they once were? Or does all that “effect” even out amongst professionals?
Are there reliable statistics for these special circumstances?
– Mr. Frost

Although I don’t think anyone doubts that there are intangible effects and they affect both the motivation and performance of players, I don’t think we can ascribe statistical significance to those ideas. We could probably come up with motivations for every quarterback in the league: Eli Manning wants to get back to the Super Bowl and erase his team’s humiliating finish to last season. Donovan McNabb wants to prove that the Eagles shouldn’t have traded him. Kevin Kolb wants to prove that the Eagles were right. Tony Romo wants to stop the chatter about how he can’t get it done in big games. There’s the NFC East right there.
You’ll also note that only skill position players (quarterbacks, running backs, wideouts, and tight ends) and the occasional prominent defensive player are ascribed motivations. You’ll never read a game story about how the left guard was clutch and willed his team to win.

To what degree does luck play in determining the outcome of a game? – Mark

When it comes to an individual game, luck plays a far bigger role than anyone cares to admit. We know that the act of recovering a fumble is almost entirely luck, and that the distance of a fumble (or interception) return is mostly random, but a game can very well come down to who recovers the fumbles or whether a player slips on a return.
There are also factors that we see regress towards the mean on a seasonal level that can drive huge single-game swings in performance. Research published by Football Outsiders in the Times suggests that a kicker’s field goal accuracy is mostly random from year-to-year. We’ve also found that teams have no ability to influence the success rate of field goals taken against them.
The Jets were lucky enough to exploit this during the first two rounds of the playoffs this year, when they were the beneficiaries of five consecutive missed field goals. Special teams coach Mike Westhoff — one of the best in the league — suggested that it could be a product of their style, saying: “Our guys really rush hard. It’s an extension of our defense. We rush hard every single time. Subconsciously, you watch it and go ‘Wait a minute. I don’t have all day.’ I think – and this is my opinion; I could be wrong – it affects sometimes a rhythm where [the kickers] are a little bit quicker than what they’d normally be.”
Meanwhile, the Jets “allowed” 19 of the 23 field goals taken against them in the regular season to go through the uprights. After factoring in the distance of those field goals and the stadiums they took place in, that’s actually 3.3 points more than expected in an average performance. At that rate, the odds of the Jets fading five consecutive field goals were 5,292:1. It was the first time a team had enjoyed five consecutive misses against them in a decade.

Do you like the way games tied at the the end of four quarters get resolved? If not, do you have any suggestion about how best to handle this? Is a coin flip a fair way to start sudden death? – Ian Callum

I’ve never understood the NFL’s obsession with a sudden death system. Baseball, basketball and soccer manage to avoid sudden death systems (although FIFA flirted with one long enough to come to regret it), and there’s not many complaints about how they decide games.
Overtime should be one 15-minute period. No sudden death. In the regular season, if the fifth quar— … er, period ends deadlocked, then it’s a tie. If it’s a playoff game, then you can move to sudden death without worrying about someone not getting a chance to score.
There are television concerns with such an arrangement — namely, TV networks showing an out-of-market 1 PM ET game have to cut away from that game at 4:15 PM ET if they have a “local” team taking part in a game then. Fans of 60 Minutes on the East Coast might also be frustrated by a 4:15 game that runs late. If the goal is to find the most equitable solution for the game itself, I think that works.
An alternate proposal was made by a former colleague of mine, Michael David Smith, in a piece he called “Splitting The Overtime Pizza.”
Michael’s idea was to essentially turn the idea into an auction. The winner of the overtime coin toss would be able to choose a yard line from which the opening kickoff would take place. The loser of the coin toss would then be allowed to kick or receive.
This would lead to a fundamental change in strategy for those kickoffs. On the standard kickoff, the kicker’s goal is to simply boot the ball as far as possible; since most returnable kicks will be run back past the 20-yard line, about the best a kicker can do is kick the ball as deep into the end zone as possible and force the returner into a touchback.
With kickoffs now taking place far closer to the end zone, teams would attempt to pin the return man as close to his goal line as possible with a high, short kick. That would turn giving the ball away into a potential positive: Teams that take possession of the ball inside their own 20-yard line actually have a negative point expectation for that drive because of the likelihood of turnovers and slim chance of actually moving the ball far enough to score. As an example, teams that take over on their own 10-yard line score, on average, -0.92 points on said drive.
At the very least, it’d be a fun way to see whether coaches can understand game theory.

I would like to get your thoughts on the optimal use of onside kicks before the fourth quarter and going for it on fourth down. My own back-of-the-napkin figures have long led me to believe that offenses should be less risk-averse in that regard. In both cases, your maximum downside is generally a loss of field position — the net kickoff yardage in the case of the onside kick, and net punt yardage in the case of the fourth down try — so maybe 50 and 30 yards, respectively. While you of course never want to give up yards, the upside seems comparatively massive — either an extra possession, or the continuation of the current possession, plus the psychological deflation that it must cause to the opponent. And both tactics work with enough regularity that I really wonder why they aren’t used a whole lot more. I’ve heard it argued that coaches are afraid they’ll get inordinately blamed if it goes sideways (vs. credited if it works), but at the end of the day the coach gets to keep his job if and only if he wins football games, so to that end, you would expect him to do anything that increases the team’s win expectancy. Then again, maybe rational expectations is too generous an assumption. Would love to get your take on this phenomenon! – David L

This is going to be a long-winded answer, but I think it probably does the best job I can of addressing your issue and the integration of statistical analysis within the league.
The keynote panel at the 2010 MIT Sloan Sports Conference was called “What Geeks Don’t Get: The Limits Of Moneyball.” Among the folks on the panel was Bill Polian, the brilliant general manager of the Indianapolis Colts. For those unfamiliar with Polian, it’s hard to find someone who’s been more successful in the NFL — Polian created the Bills dynasty that made it to four Super Bowls in the early nineties, built the expansion Carolina Panthers into a championship contender in its second season, and then took over the Indianapolis Colts. His first pick was Peyton Manning. Things have been OK in Indianapolis since then.
When a discussion of Bill Belichick‘s famous decision to go for it on fourth-and-two against the Colts during the 2009 season came up, though, Polian proceeded to denigrate the statistical analyses he had seen of the decision, most of which revolved around the idea of win probability and applying historical probabilities to this specific NFL situation. (You can see the video of the panel.) The whole video is worth watching, but I’m going to quote the article written by Brian MacPherson on the conference here:

Polian broke down all the different aspects of that play — the Colts’ hot offense, the Patriots’ banged-up defense, the tendency of the Patriots to run quarterback sneaks in short-yardage situations rather than throwing the type of pass that was called — and backed up the decision.
Here’s the funny thing: Polian agreed with the decision not because he believed the statistical data that supported it, he said, but because it made sense in that specific moment with those specific players on the field.
“Was it the right call? In my opinion, it was 100 percent the right decision,” Polian said. “All of the statistical analysis that’s done over the course of the season means nothing. The situation on the field at the time dictates the decision.”

I was in the crowd at the time, but I resisted the urge to yell “Confirmation bias!” at the stage. Of course, the examples Polian gives of his reasoning are exactly those which justify Belichick’s decision, including several soft factors that had little to do with the actual success of the situation. (I strongly doubt that the absence of journeyman linebacker Tully Banta-Cain had more than a negligible impact on the chances of the Colts scoring. The Patriots’ pass rushers were tired from rushing the passer, but the Colts ends had also been on the field for 42 pass attempts, fewer than the 40 pass attempts the Patriots had been through. I could go on.)
At the time of the decision, I wrote a piece that shared Polian’s sentiment about statistical analyses published elsewhere failing to take into account the unique nature of the game situation and the two teams. My best estimate of the situation was that it was too close to call; Belichick’s win probability when choosing to punt or go for it was reasonably similar, and well within the margin of error we could consider in this specific situation.
On the other hand, suggesting that “…statistical analysis over the course of a season means nothing” is throwing out the baby with the bathwater. Ironically, the situation that served as an ideal counter-example to Belichick’s decision had happened only two months earlier; it was Sean Payton‘s decision-making in the Super Bowl against Polian’s Colts.
In the game, Payton made two decisions that can be justified using probability analysis, regardless of the game situation. The more obvious decision was actually the one that didn’t work out. At the end of the second quarter, Payton chose to run the ball on fourth-and-goal from the Colts one-yard line, only for his sweep to be stuffed. Now, we don’t even need to consider that the Saints were seventh when running in short-yardage, although the Colts were sixth at stopping teams running in such situations. If we just use the historical probability of converting such carries — an extremely conservative figure considering the players on the field — the Saints would score a touchdown 54.3 percent of the time, yielding an expectation of 3.80 points. They could convert a 18-yard field goal about 98.7 percent of the time, which yields an expectation of 2.96 points. The other factors related to the game situation also favor the decision to go for it: failing to score would force the Colts to drive about 70 yards to have any shot of scoring themselves — the point expectation for even the Colts scoring on the subsequent drive was below zero, and the Saints had a shot at stopping the Colts and regaining possession before halftime. The latter is exactly what happened, and the Saints ended up with a field goal. The decision to go for it was an absolute no-brainer.
The more celebrated decision was the one that worked out, Payton’s famous choice to employ an onside kick to open the third quarter. The margin for that decision was smaller, but it was the perfect melding of statistics and scouting. Our research suggests that historically “unexpected” onside kicks have been recovered by the kicking team about 70 percent of the time. Assuming that the kick would be recovered on the Saints’ 42-yard line by whichever team was able to fall on the ball, the Saints would expect to score .75 points if they recovered the kick and allow .59 points if they failed to.
It’s a slightly positive move, but the ability of the Colts’ offense with a short field probably turns it into a virtual pick-em. Payton undoubtedly needed a little more to convince himself that it was the right move, and that’s where the factor Polian ignores in his criticism comes in: You pay your coaches a lot of money to find the ideal situation to exploit the opposition. That’s exactly what Payton and his staff did. During film analysis during the week and observations of the Colts’ kickoff coverage unit during the game, Payton had seen the Colts’ gunners routinely turning around to sprint back towards the return man too early, creating a window for an onside kick to successfully take place.
And as for Polian’s chatter that statistical analysis done over the course of a season means nothing? There was someone who estimated during the week before the Super Bowl that the probability of the Saints recovering a surprise onside kick was in the 60-70 percent range, a figure he quoted to the media after the game. I suspect that the man in question — Sean Payton — might disagree with Polian.

How valuable (in wins, and therefore is dollars) is Darelle Revis?
The salary average and variance varies by position. So too does the impact on the game. Which positions are overpaid and underpaid relative to the average or variance?
How well can you isolate the play of any particular player (e.g., the RB) from that of his teammates (e.g., the O-line)? Who contributes more to a 2000-yard season, the RB or the line? – Erik

We’re nowhere near the point of valuing a player as being worth a number of wins, because we’re light-years away from quantifying all the things a player does. I doubt we’ll ever have a reliable “wins” metric because there are too many interactions between positions that we can’t account for in football. Take a quarterback, for example: even if we were to develop a measure of performance that stripped out the effects of his receivers and offensive line and placed his passing performance in a perfect, league-average context, we’d have to account for how he read defenses and called audibles at the line, how effective he was in setting up defenses on the play-fake, whether he had any impact on the running game versus an average quarterback … it’s not a realistic goal.
By the statistics we do have, though, Darrelle Revis was quite the cornerback in 2009. Our Game Charting Project at Football Outsiders uses a flotilla of volunteers to track a variety of things that aren’t contained in the official NFL play-by-play, like how many blitzers there were on a given pass play, whether there was play-action, or whether a defender rushed the quarterback into a throw.? For pass plays, we use the angles provided on TV broadcasts to guesstimate who the receiver in coverage was. (For plays where there’s a blown coverage, our charters can list that the catch came in a “Hole in Zone”; or with clear double coverage, they can list more than one defender.) It’s an inexact science, since it’s difficult to diagnose certain coverage schemes from that camera angle, but it’s a big step in the right direction for analyzing the play of defensive backs.
Two of the primary stats we track for cornerbacks are Yards per Attempt (YPA) and Success Rate. YPA is, very simply, the number of yards a receiver gave up divided by the number of times he was in coverage. Success Rate captures the percentage of the time that the offense threw a pass against a particular defender and gained 45 percent of the needed yards for a new set of downs on first down, 60 percent of the needed yards on second down, or 100 percent of the needed yards on third or fourth down. Completions that don’t meet these standards, incompletions, or interceptions are considered to be failures.
Darrelle Revis paced all starting cornerbacks in both YPA and Success Rate in 2009, and it wasn’t particularly close. His YPA was 2.9 standard deviations above the average performance by a qualifying cornerback, while his Success Rate was 3.0 standard deviations above that average corner.
To try and place that in context with more traditional statistics, I noted that Revis was thrown at 96 times, which was the fifth-most targets of any corner in the league. I split the difference between those two Z-scores, suggesting that Revis was playing at a level 2.95 standard deviations above the average starting corner, and then applied that level of performance on a per-play basis to those players that ranked fifth in the league in usage at their particular category.
The results were staggering. Peyton Manning threw the fifth-most passes of any quarterback in the league. He actually threw for 4,640 yards. If he was playing at a level 2.95 standard deviations above the mean, like Revis was, he would have thrown for 5,532 yards — that would be an NFL record. Maurice Jones-Drew was the running back with the fifth-most carries in the league. While he ran for 1,391 yards, a Revis-style performance would have seen him hit exactly 2,000 yards. Brandon Marshall picked up 1,127 receiving yards while finishing fifth in the league in catches; had he been 2.95 standard deviations above the mean on a per-play basis, he would’ve picked up 1,922 receiving yards, which would also have been a league record.
Placing a value on that sort of performance is another topic altogether, but I think it’s pretty clear that Darrelle Revis spent his 2009 season playing at a truly remarkable level.

If the NFL Rules Committee Genie appeared to you one Sunday and allowed you to immediately implement a single rules change, what would it be? – howlless

Wow. I only get to pick one? I asked my colleague Doug Farrar about this, and he had an answer that I liked: make everything reviewable. Teams would still only get two challenges, but every play would be reviewable for whichever rules infraction or misinterpretation the coach wanted to challenge. It would lead to some judgment calls, but that’s not the worst thing in the world: A referee making a judgment call with a dozen replays is usually going to be better than a referee making a judgment call on the fly.

How important do you think it is to have a good backup QB? I hear all the time about how important it is, but it seems to me that the teams that win aren’t teams with good backups at QB but simply teams whose QB’s stay healthy. If QB is really so important, then there is no way a team can have much success with a QB who in theory is no better than 33rd best in the league (as a rough approximation). – Steve Nations

I think you’re probably underestimating how good the league’s best backup quarterback is. At the moment, I’d say it’s Chad Pennington, and I’d probably take Chad Pennington over about half the league’s starting quarterbacks. (If I were playing to win one game and didn’t have to worry about Pennington’s propensity for injury, he’s probably in the top six or seven.)
It’s an interesting question, though. We know that injuries to the starting quarterback bear the strongest relationship to a decline in team success by virtually any metric I can come up with. It even affects the performance of other players in ways that you might not imagine. As an example, while there’s no correlation whatsoever between injuries to a team’s starting running back and a decline in the performance of their running game (or passing offense), there is a slight correlation (around .3) between injuries to a team’s starting quarterback and a drop in rushing performance.
So, this might not be the most satisfying answer, but I think you’re part-right. Keeping a quarterback healthy is the easiest way to ensure optimal performance, but you also have to consider that injuries to the starter can create opportunities for a backup that might not otherwise have seen the field. Remember, Tom Brady only got his chance to play when Drew Bledsoe went down with an injury. The same is true, to one extent or another, for Matt Schaub, David Garrard, Kyle Orton, Matt Cassel, Kevin Kolb, and Matt Moore. James Harrison — the 2008 AP Defensive Player of the Year — was languishing on the Steelers bench and about to retire at age 26 to become a veterinarian when the starter at his position suffered an injury lifting weights. For every Brady and Harrison, there has to be another 3-4 elite players lurking at the bottom of a roster league-wide. And for every Brady that got their chance, there has to be a quarterback or so a decade that could have been an elite player and never got the opportunity.