Breaking Down the Clemens Report: A Guest Post

Sports fans will probably be aware that Roger Clemens is currently before Congress, arguing that the Mitchell Report wrongly tagged him as having used performance-enhancing drugs. And last week, his agents released the “Clemens Report,” arguing that his career statistics somehow exonerate him. The full marketing spin is available here.

I was interested in understanding how they could “prove” his innocence by crunching numbers, and in an effort to make sense of it all, I sat down in a Wharton conference room with three fellow data hounds – Eric Bradlow, Shane Jensen, and Adi Wyner. With two stats professors and a stats/marketing prof in the room, I felt a bit outmatched. But it sure was fun to work through the issues together.

The main argument in the Clemens Report is that there is nothing unusual in his career numbers, and, in fact, his performance is quite similar to that of Nolan Ryan. But we note the following:

[S]uch comparisons tell an incomplete story. By comparing Clemens only to those who were successful in the second act of their careers, rather than to all pitchers who had a similarly successful first act, the report artificially minimizes the chances that Clemens will look unusual.

There’s a pretty neat trick at work here: if you compare Clemens only to those who had a terrific last decade of their careers, then the last decade of Clemens’ career doesn’t look that unusual. To sidestep this, we suggest that “[a] better approach to this problem involves comparing the career trajectories of all highly durable starting pitchers.”

So we put together data on all 31 other pitchers since 1968 who started at least 10 games in at least 15 seasons and have pitched at least 3,000 innings. This broader comparison group yields some pretty different conclusions than the Clemens v. Ryan contrasts.

A picture is worth a thousand words, and here we show simple quadratic fits to the data for Clemens v. controls:

roger clemens report

The Clemens Report is also notable for its near-exclusive focus on his ERA. Now, any Sabermetrician will tell you that this is not a particularly reliable statistic, and that it bounces around a lot more than a pitcher’s true performance. This is a problem because noisy data can obscure an underlying pattern. So we supplemented our analysis by examining a range of alternative indicators, including walks and hits per inning pitched (see right panel, above).

We conclude that “the available data on Clemens’s career strongly hint that some unusual factors may have been at play in producing his excellent late-career statistics.”

To be clear, we don’t know whether Roger Clemens took steroids or not. But to argue that somehow the statistical record proves that he didn’t is simply dishonest, incompetent, or both. If anything, the very same data presented in the report – if analyzed properly – tends to suggest an unusual reversal of fortune for Clemens at around age 36 or 37, which is when the Mitchell Report suggests that, well, something funny was going on.

You can read our full analysis in today’s Times, here.

UPDATE: Roger Clemens’ crisis management consultants have just released a rejoinder to our analysis, available here. Further coverage: Lester Munson at ESPN.com, a less flattering analysis at MLB.com, and another piece at ESPN.com (leading to hundreds of comments).


TB

I'm not sure if comparing him to pitchers from 1968 is accurate. There have been advancements in exercise science in general that would give him an advantage over pitchers from the 60s.

Mike

Kyle, very interesting. Part of Randy and Curt's late career worsening is due to moving from NL to AL... but not that much.

Nolan's WHIP goes down over time due to, I think, I a change in approach. He realized walks were killing him, and drastically reduced them towards the end of his career.

mraver

Dr. Wolfers-

Your choice of WHIP as a useful statistic for comparing Mr. Clemens' career is odd. To paraphrase, any Sabermetrician will tell you that WHIP is not a particularly reliable statistic, and that it bounces around a lot more than a pitcher's true performance.

It seems like looking at Mr. Clemens' strikeout, walk, and home run rates would be the most useful.

Austin

"Don't you guys get it? The fact that you can't tell wether he took steroids or not exonerates him!"

Minus the part where they were arguing against the statistics being used to exonerate him, when they clearly, if one wants to look at all possible scenarios, demonstrate that no firm conclusion can be drawn.

The real issue is, ultimately, what other evidence will come to light on Clemens. I have no strong opinion either way, but the statistical data neither proves his innocence nor his guilt.

Absence of evidence is not evidence of absence.

t.c.

go tribe

Michael

Bill and Cyril,

The point is not that the data shows that he took (or didn't take) drugs--that would be completely impossible. It's that the data presented by the Clemens defense team is misleading. His incredible performance late in his career is quite atypical, not par for the course as Clemens' reports apparently suggests (I have not seen their analysis, I'm just going by what Justin Wolfers claims). Whether or not those stats are possible for someone who did not use drugs is completely irrelevant to this article.

Kyle S

Professor Wolfers, a few comments:

1) I believe there could be an error in your culling formulae. I find 34 pitchers with 15 or more seasons since 1968 who have at least 3,000 innings pitched in that timespan. Here's my list: http://sturgeongeneral.wordpress.com/files/2008/02/wolfers_list.pdf Who don't you include from that list, and why?

2) The list of comparables is interesting, but not particularly useful for making a very specific conclusion about the shape of Clemens' performance. Any pitcher who pitches 3,000 innings over a 15+ season career is in the far right tail of the distribution of major league pitchers, who themselves are in the far right tail of pitching talent across the world. You can find interesting things about each guy in your list, but the sample is just so small that drawing broad conclusions is very dangerous. As Cyril says, do you think Warren Spahn was on steroids?

3) Clemens had the fourth highest (unadjusted) strikeout rate of the group I found (behind Randy Johnson, Nolan Ryan, and Curt Schilling). Right behind him are John Smoltz, Chuck Finley, Steve Carlton, Mike Mussina, and Tom Seaver. As you may know, high-strikeout pitchers tend to age better (show less of a dropoff in their performance) than do low-strikeout pitchers.

To test this, I broke the group of 33 (excluding Clemens) into four quartiles, and built a multiple regression model to predict WHIP by age and quartile. Here's a picture of the output: http://sturgeongeneral.files.wordpress.com/2008/02/wolfers-chart.png

Each curve looks much like the output of your model in the NYT article, which means I must at least have partially done something right :). However, if you'll notice, quartile one pitchers (of which Clemens would have been one, had he not been excluded) have the shallowest aging curve - they don't tend to increase their WHIP as much as they get older. This is exactly what we'd expect.

Clemens' career certainly has an unusual shape, and I myself believe he took steroids to achieve it. But I don't believe your analysis is particularly probative of that conclusion.

Read more...

Nikolet

Nice site!