Why Don’t Sports Teams Use Randomization? A Guest Post

Here’s the latest guest post from Yale economist and law professor Ian Ayres. His past posts can be found here and here.

In a recent post, I mentioned that when playing poker, I use my watch as a crude random number generator to tell me when to bluff. While there are lots of sports in which it’s best to play a somewhat random strategy, that doesn’t mean that every possible play is equally likely. But it does mean, for example, that when it’s third-and-2 in football, the offense wants to have some possibility of passing to keep the defense honest.

Levitt and others have tested the degree to which professional tennis and soccer players are successful at playing randomized strategies. But it remains a mystery to me why coaches don’t have random number generators (any laptop would do) to help them pick the next pitch in baseball, or the next play they will call in football. Norv Turner would pick the probability of running or passing, and then let the computer decide which it would be.

But an even bigger puzzle is why teams don’t exploit the other powerful use of randomization. To my knowledge, no sports team in the history of humankind has ever run a random control trial to figure out which strategies work the best. (I make this extravagant claim in hopes of provoking you all into providing some counterexamples.) Randomized studies are the gold standard of medical testing, and they’re now the hottest thing in Internet ads.

Want to know whether your Web banner for beer should say “Tastes Great” or “Less Filling”? Run a randomized test in which half the people see one and half the people see the other at random, and then sit back and watch whether one ad generates more sales. I ran just this kind of test on Google Adwords to help choose the title of a book (shameless pitch) I was writing. When I started writing it, I loved the title, “The End of Intuition.” But in a randomized test, “Super Crunchers” had a 63 percent higher click-through rate.

So why don’t sports teams run (more) randomized experiments? The Boston Red Sox are famous for relying on number crunching to gain a competitive edge. But why don’t they proactively make some powerful data by creating randomized treatment and control groups? They could use their minor league teams, for instance, to figure out whether catchers or pitchers make better calls.

They could even have a randomized trial of randomization — they could randomly assign the pitches for half the at-bats to be called in the traditional way (by the coach or the catcher) and the other half could be called by a random strategy established in advance. It would be a double-blind study, because neither the pitcher nor the hitter would need to know which system called the pitch.

If it turned out that the random strategy reduced the batting average of your opponents, that would be pretty strong evidence that it was a better strategy.

Or you could run an experiment to find out whether football teams should go for it more often on fourth down. Economist David Romer has crunched numbers to suggest that professional football teams should go for it fourth down a lot more than they currently do. His proposed optimal strategy is summarized in the following graph, found at the end of his paper:

graph

Amazingly, the data suggests that if it’s fourth down and your team has the ball on the opponent’s 33-yard line, you should go for it even if you have 9 yards left for a first down. NFL coaches have resisted Romer’s advice (though Pulaski Academy has started acting on it). But this is another area in which a little randomized testing could go a long way to help figure out what works. There are thousands upon thousands of college and high school games, but we collectively go for decades without figuring out whether simple changes in strategy could really produce better outcomes.

If you know of any randomized tests of sports strategy, please let me know. And if you are a coach and want to run a test, feel free to contact me. I’d be happy to help design and evaluate a test.


mike kagan

The Red Sox DO use randomization, in the form of Tim Wakefield. When a pitcher throws a screwball he has no better an idea where the ball will go than does the batter.

Tim

Doug - to make that argument you have to prove that a batter's performance is truly random. What the coaches belief is that batter has some sort of advantage versus that pitcher - he knows how to read his opponent's delivery or is adept at hitting the pitches that pitcher prefers.

Gary

I am more inclined to believe coaches are reluctant to try randomization in sports out of arrogance. They are so convinced that its their own brilliant play calling responsible for beating the defense, that randomization could only possibly fail.

In their defense, there are so many variables in sports that randomizing play calls would be a nightmare. Willie Parker has 4 fumbles today, its the 4th quarter, its still raining, and the computer wants me to give him the ball on 3rd and 3 from our 15 yard line? No chance.

David

Marty tried this in last years playoff game against New England. It was a 4th and 10ish from the 30 or so yard line. Mary elected to go for it. Rivers could not make the play (I think he may have been sacked). The PATS took the ball and drove down the field for a TD. Of course the Chargers ended up losing the game by 3.

John

Most likely the reason these aren't tested is because coaches & managers are all in CYA mode.

If they do the same thing as every other coach & fail, it's easier to pin blame on environment, players or other organizational issues. Thus preventing a bad season from killing his career.

However, if a coach were to start going for it on fourth downs in "ill-advised" situations, calling pitches or plays at random & fail, he would suffer irreversable damage to his career.

Ian

@KevinQ: The video game simulation sounds like an interesting and fun test to run, but I would question its value. Video games are designed to be entertaining, not to provide a good model of actual strategy. Now, most sports games do focus on making realistic models because sports fans care about that sort of realism in their sports entertainment. But those models are expected to be realistic only so far as people play the games in a way similar to actual teams. A winning strategy in the game could just as easily be a case of the model not fitting reality as it could the discovery of a superior real-life strategy.

Doug

Very interesting post. But I have always been even more baffled not by how they don't use statistics, but how they use them when they do - particularly in baseball. For example, a coaches propensity to use a player based on their most recent statistics - they are currently hot or cold - instead of their life long statistics. To me, a 200 life time hitter who's gone 4 for 5 in say his last five at bats, against a specific pitcher, etc. is most likely due for a string of failures, not successes.

..Doug

Charles

I'm sure that coaches do not want to test if randomization is true just like stock brokers don't want to believe in the random walk theory.

ElSupreme

I really like this article. I am a avid, and knowledgeable fan of American Football. I did however do most of my learning about the game with Madden '96 when I was about 13. I found many circumstances in the game where going for it, going for 2, and letting the other team score (the so called 'Sega Defense', due to the offense being unstoppable in early Nintendo/Sega games the person scoring last usually won) that while not a great model for real football still have some grounding and should be thought about in the real game.

If you are up by one at the end of a game (preferably a high scoring one) and the other team is well within field goal range: let them walk into the end zone. Then you have more time to get your 8 needed to send the game to OT. I have seen this NOT happen in many a college football game, some of which have had over 4 TDs (combined both teams) in the fourth quarter.

JSN

I think George has made some good points. The effectiveness of a play can depend on the what players are in the game and their physical condition.

Joe

I thought the Red Sox were famous for spending lots of money to gain a competitive edge?

KevinQ

I saw this on an earlier post about clock management, but it seems like video games, such as Madden, would be a perfect place to study this possibility in earnest. No simulation is perfect, but the benefit in increased number of games to test on is impossible to ignore.

I'm no economist, but couldn't you run 10000 simulation games of identical teams playing each other (how about Vikings vs. Vikings, so Adrian Peterson is always on the field)? One team is programmed never to punt, the other to follow its built-in ultra-conservative mindset. Couldn't we expect some interesting results?

Nathan Whitehead

Wow, that is a thought provoking post. I think the actual answer as to why teams don't use randomization or A/B testing is that they don't understand the benefits. Humans have an innate bias against randomness and the scientific method. We like to control things and find patterns that make sense, not add randomness and perhaps discover facts that we don't understand fully.

TaintMasterMitch

Really neat post, I'd like to see some tests in sports come out of it. Thanks for putting the time and effort into a thought-provoking read.

Barb

Maybe Mike Tomlin could use this advice... Anyone know how to reach him? He's not hiding from his fans yet, is he?

Craig Kocur

This post, and last week's about football coaches not going for it on 4th down, are doing a lot for my ego. For years while watching games I've daydreamed about being an NFL coach and going for it on every 4th down (except in the most dire circumstances, 4th and 26 from your own 10 for example). It always struck me that any NFL team should be able to manage a 2.5 yard per play average and longer and sustained drives would do a lot to wear down opposing defenses. Also, offenses would play looser knowing they have a larger margin for error.

I've also thought that sports teams don't take advantage enough of computers. I've had this image in my head for awhile of 2 coaches in the coaches' box, one pulling randomized plays from a laptop and relaying them to the field and the other inputting the results of plays in another laptop to help make the randomizations smarter by accounting for the variables, down and yardage, defensive formations, wind speed and direction, day or night, real or artificial grass, jersey colors (you never know what influences outcomes until you test for them), etc.

Read more...

George

Randomization has it's limits though, doesn't it? Each play is not necessarily as effective as others and therefore, by opting for a random alternative instead of a strategic one could illicit negative results that could cost the game.

In football, going for it on fourth down in your opponents territory would be a good thing because, the net gain from punting would be minimal and the more opportunities you have to score 7 points rather than three would more than likely increase thus increasing your chances to win. But going for it in your territory could reward your opponent with prime field position. I think a team like New England - with it's high powered offense - can go for it on 4th down (and this year they have) a lot more than most. A struggling offensive team might not be so successful. If you've gained 6 yards on your first three plays - you might not want to risk gaining the needed 4 on one final play.

But in baseball, most starting pitchers have three strong pitches and two that aren't as strong. While some hitters are guess hitters, most adapt to the actual pitch based on release point, speed, etc - which means that the effectiveness of the pitch isn't based as much on the element of surprise as it is on the actual strength of the pitchers skill level. Having it go at random would leave a pitcher more likely to give up hits and runs, leaving his team more likely to lose.

In basketball - the current Golden State Warriors basically play as random as you can get on offense. They just shoot whenever they can - bunking conventional wisdom (ie: shooting in a 1 on 4 situation, shooting within 8 seconds on the shot clock, forcing the tempo when they don't have "numbers", etc) . They are currently 11-9, although 8-2 in their last 10.

Read more...

Jonathan Potts

Good point Barb. Given what everyone knows about the Patriots' offense, why did Tomlin kick a field goal when the Steelers were down 14-10 in the first half of the Patriots' game? Did he have that much faith in his defense? In fact, consensus is that the Patriots have a defense that is merely good, with aging players who might not stand up to really getting pushed around in a game. So why kick a field goal when your offensive line was stilling show signs of winning the line-of-scrimmage battle?

Ari

It seems to me that you don't really need prospective, double-blind studies to answer most of the questions that you'd want to ask. That is, unless you're trying something completely new.

The incredible wealth of retrospective data available in sports would make most controlled studies an expensive artifact. It can even be argued that, with enough variables, the controls available using retrospective data are more reliable than the contrived controls of a small prospective study. If you allow for Bayesian logic, the potential for insight from existing data becomes virtually limitless.

Mind you, I've advocated for this approach for many medical studies as well(those testing effectiveness and outcomes of existing treatments) and met up with significant resistance from the establishment.

Dan

A former Patriots couch (I believe it was Pete Carroll) would used to select the first 15 plays before the game started and would not waiver despite the situation. Although their selection was not random they were blind to the situation in which they would be used. If it was Pete Carroll he failed miserably as the Patriots head coach before going on to USC and winning a couple of national championships.