Search the Site

Talent Evaluation is Different in the NFL and NBA

By Dave Berri

The sudden emergence of Jeremy Lin has led people to wonder about talent evaluation in the NBA. Two recent examples — from Stephen Dubner in this forum and from Jonah Lehrer at Wired Science — both take similar approaches.  Both begin with the story of Lin, and then pivot to a discussion of the National Football League.  In essence, each writer argues that talent evaluation in basketball and football is similar.

In my next two posts, I wish to address why I think talent evaluation in the NBA and the NFL is quite different.

(Photo: killthebird)

Let’s start with the NFL (my next post will focus primarily on the NBA). Brian Burke (of Advanced NFL Stats) and I have a chapter in The Economics of the National Football League: The State of the Art (Sports Economics, Management and Policy). In this chapter, we examine a variety of metrics designed to evaluate football players.  For the purpose of this discussion, I want to focus on the quarterback position (a position with an abundance of performance statistics).

In our discussion of NFL signal callers, we noted the percentage of a quarterback’s current season performance that could be explained by what the quarterback did last year.  Here is a sample of what we found for a variety of different performance measures:

To put these numbers in perspective, for many statistics in the NBA, season-to-season explanatory power exceeds 70 percent.  As one can see –relative to NBA players — NFL quarterbacks are quite inconsistent from season to season.  This is especially true for interceptions per play — a factor that has a big impact on outcomes — where explanatory power is below 1 percent.  In other words, just because a quarterback avoided interceptions in the past, it doesn’t mean the pattern will continue in the future.

All of this tells us that even if you get to a see a quarterback play in the NFL, you are going to have a hard time knowing exactly what you will get from that same quarterback next season.  This is not surprising since much of what a quarterback does depends upon his offensive line, running backs, wide receivers, tight ends, play calling, opposing defenses, etc.  Given that many of these factors change from season to season, we should not be surprised that predicting the performance of veteran quarterbacks is difficult.

With this result in mind, let’s think about the draft.  Imagine a team decides it needs to select a quarterback this upcoming April.  A number of quarterbacks both excelled in college and are now available to enter the NFL.  At the top of the list are Andrew Luck and Robert Griffin III, the two players who led the Heisman vote. After these two, we see names like Ryan Tannehill, Brock Osweiler, Nick Foles, Brandon Weeden, Kirk Cousins, Kellen Moore, Russell Wilson, and Case Keenum (among others).  Of these, which player will be the best quarterback in the NFL?

To sort through these players, the NFL holds a combine (which begins yesterday). This event will provide measures of each player’s height, weight, intelligence (i.e. Wonderlic score), and speed (40-yard dash time).  All of this is added to what the quarterback actually did on the college football field to determine which quarterback will be the “best.”

So how much does all this information matter?  There are two issues to consider:

It was these two questions that Rob Simmons and I addressed in a paper published last year in the Journal of Productivity Analysis.  This paper looked at which factors explain where a quarterback is drafted, and furthermore, how those same factors relate to future NFL performance.

When it comes to predicting draft position, we found that a quarterback’s height, his Body Mass Index, his Wonderlic score, his time in the 40-yard dash, whether or not he played in the top division of college football, and various on-field statistical measures are related to where a quarterback is drafted.

So the information the NFL gathers does impact evaluations.  But when we turned to the second question, we found a result that may seem surprising.  None of the factors that were related to where a quarterback was taken in the draft predicted future NFL performance (and we considered a variety of per-play measures across a variety of time periods in a player’s career).   We did find that completion percentage in college — a factor that wasn’t related to where a quarterback was selected in the draft — did predict completion percentage in the NFL.  But our analysis indicated that less than 20 percent of completion percentage in the NFL can be explained by college completion percentage.

In sum, it is clear that people in the NFL do consider a variety of factors when choosing among NFL quarterbacks.  But the factors considered don’t predict what we see in the NFL.

Consequently, when we look at the NFL draft — with respect to quarterbacks — we should expect to be surprised.  Right now, people are convinced that Andrew Luck is a bit better than Robert Griffin III. And both of these players will be better than Ryan Tannehill.  And from what I can see from looking at various on-line draft evaluations, Tannehill is considered a better prospect than Russell Wilson and Case Keenum.

We should certainly expect that if Luck and Griffin III are taken in the first few picks of the draft, they will get to play more than those taken later.   In other words, draft position – as Rob and I noted in our paper – does impact playing time (just consider the number of chances Alex Smith of the San Francisco 49ers has gotten to play in his career).  But when we consider per-play performance (or when we control for the added playing time top picks receive), where a quarterback is drafted doesn’t seem to predict future performance.

And given what we see with respect to NFL veterans, this shouldn’t be surprising. It is difficult to predict the future for quarterbacks who have already played in NFL games.  Given that difficulty, we should not expect to have much success when we try and predict wh
at a player who has never played in the NFL will accomplish once they start playing on Sundays.

Fortunately for teams at the top of this year’s draft, how much they have to pay these prospects has declined considerably.  Teams no longer have to give large guaranteed contracts to players like JaMarcus Russell, Matt Leinart, David Carr, Joey Harrington, Tim Couch, Akili Smith, and Ryan Leaf.   Each of these quarterbacks was highly touted on draft day.  And each disappointed the people who gave them many millions of dollars to try and play in the NFL.

With the new Collective Bargaining Agreement, top draft picks will now receive far less than what they were paid in the past. So when teams miss on these quarterbacks – and given what we have seen in the past, we can predict with confidence that teams will miss on some of these players – the cost is going to be much lower.

Okay, let’s now pivot back to the NBA.  We have just seen that predicting performance in the NFL is difficult.  And that confirms part of the story told by Dubner and Lehrer.  But does that really help us understand Jeremy Lin?  In my next post, I am going to discuss what the published research tells us about player evaluation in the NBA.  And that story is going to demonstrate that the NBA is not really the same as the NFL.


Comments