Breaking Down the Clemens Report: A Guest Post

Sports fans will probably be aware that Roger Clemens is currently before Congress, arguing that the Mitchell Report wrongly tagged him as having used performance-enhancing drugs. And last week, his agents released the “Clemens Report,” arguing that his career statistics somehow exonerate him. The full marketing spin is available here.

I was interested in understanding how they could “prove” his innocence by crunching numbers, and in an effort to make sense of it all, I sat down in a Wharton conference room with three fellow data hounds – Eric Bradlow, Shane Jensen, and Adi Wyner. With two stats professors and a stats/marketing prof in the room, I felt a bit outmatched. But it sure was fun to work through the issues together.

The main argument in the Clemens Report is that there is nothing unusual in his career numbers, and, in fact, his performance is quite similar to that of Nolan Ryan. But we note the following:

[S]uch comparisons tell an incomplete story. By comparing Clemens only to those who were successful in the second act of their careers, rather than to all pitchers who had a similarly successful first act, the report artificially minimizes the chances that Clemens will look unusual.

There’s a pretty neat trick at work here: if you compare Clemens only to those who had a terrific last decade of their careers, then the last decade of Clemens’ career doesn’t look that unusual. To sidestep this, we suggest that “[a] better approach to this problem involves comparing the career trajectories of all highly durable starting pitchers.”

So we put together data on all 31 other pitchers since 1968 who started at least 10 games in at least 15 seasons and have pitched at least 3,000 innings. This broader comparison group yields some pretty different conclusions than the Clemens v. Ryan contrasts.

A picture is worth a thousand words, and here we show simple quadratic fits to the data for Clemens v. controls:

roger clemens report

The Clemens Report is also notable for its near-exclusive focus on his ERA. Now, any Sabermetrician will tell you that this is not a particularly reliable statistic, and that it bounces around a lot more than a pitcher’s true performance. This is a problem because noisy data can obscure an underlying pattern. So we supplemented our analysis by examining a range of alternative indicators, including walks and hits per inning pitched (see right panel, above).

We conclude that “the available data on Clemens’s career strongly hint that some unusual factors may have been at play in producing his excellent late-career statistics.”

To be clear, we don’t know whether Roger Clemens took steroids or not. But to argue that somehow the statistical record proves that he didn’t is simply dishonest, incompetent, or both. If anything, the very same data presented in the report – if analyzed properly – tends to suggest an unusual reversal of fortune for Clemens at around age 36 or 37, which is when the Mitchell Report suggests that, well, something funny was going on.

You can read our full analysis in today’s Times, here.

UPDATE: Roger Clemens’ crisis management consultants have just released a rejoinder to our analysis, available here. Further coverage: Lester Munson at ESPN.com, a less flattering analysis at MLB.com, and another piece at ESPN.com (leading to hundreds of comments).


Brent

I think it would be more interesting to look at the trend line as a % above/below the average era for the league that is being pitched in. Obviously, the move to the NL helped Clemens with the great years late in life. I'd also argue that his playing the shorter seasons later in his career helped him maintain his durability and not tire like many other pitchers have experienced.

I'm not saying I think he's innocent...just saying that the statistical analysis has a lot of other factors that aren't being accounted for here...

Jonah Keri

"So how does the graph of Nolan Ryan versus the other pitchers look?"

Glad you asked!

http://nymag.com/news/features/43591/

Chris Sokalofsky

After reading their rebuttal, it seems like the Clemens defense is basing their analysis on the fact that "Roger Clemens is an extraordinary player, and therefore cannot be compared to lesser individuals when analyzing their career statistics".

Also, their statement regarding his workout regime being key to his success seems irrelevant, since all players at the major league level have to follow pretty strict regimes simply to cope with the 162 game season. I doubt his regime is any different than the rest of the starting pitchers.

Len N

Hank Aaron was still among the league leaders in home runs at 37, 38 and 39. I guess, by this kind of analysis, he was on steroids or something.

The fact is that this is not the best way to analyze this. The Times had a similar analysis of the players in the Mitchell report and there was no firm conclusion that could be drawn. Some had better stats, others had worse.

The entire era was 'tainted', if you want to call it that, and the fault is not a bunch of men playing a kids game but the commissioner, the owners, the player's assn and the baseball writers who were all to happy to write about the 50-60 homer seasons and tape measure home runs.

Maybe steroids were just the level playing field of the time.

We'll never know the extent of all this. The ballplayers won't point fingers because those that didn't use knew that plenty of others did...and many may have benefited by playing on winning teams that had players who used steroids. We'll never know.

You can't rewrite history. The Marion Jones mess shows the path that re-arranging stats and medals can lead to. Relay partners, who devoted their lives for a moment of well-deserved glory and did nothing wrong, had their gold medal revoked. Is that fair? And the Olympic committee, wanting to cascade the medals downward, is in the unenviable position of giving a medal to someone who, herself, was tainted - and which they won't do. No sense in all that.

The hearings are nonsense. Congress forced baseball to put in meaningful testing and substantial penalties. They did their job. They have better things to spend the taxpayers money on than to bring in ballplayers for a big photo-op and get autographed pictures and balls.

The only path is forward. You can't turn the hall of fame into a hall of asterisks. Athletes of every era have always looked for an edge. Aaron admitted to using greenies (amphetamines). Willie Mays had his 'red juice' as prominent in his locker as McGwire's andro. Who knows what substances earlier ballplayers used because no one cared to know.

This will all just be a footnote sooner than we think. Enough is enough.

Read more...

Jim Strathmeyer

"And last week, his agents released the "Clemens Report," arguing that his career statistics somehow exonerate him."

"To be clear, we don't know whether Roger Clemens took steroids or not."

Don't you guys get it? The fact that you can't tell wether he took steroids or not exonerates him!

What on earth is all of this number crunching supposed to go? I can see it gets much farther than I could've imagined, but in the end doesn't it just accuse the best players of being steroid user. The Clemmens-charts sure look out of the ordinary, but they don't really prove anything.

paul

I'm just saying:

People seem much more willing to jump to Clemen's defense than, say, Barry Bonds.

Hmmm.

Ryan

Since you refer to a comparison between Nolan Ryan and Roger Clemens, your graph should plot Nolan Ryan's performance (in addition to the control group).

Caleb Standafer

What are the curves like for the other successful late-career pitchers, such as Johnson, Schilling, and Ryan? Comparing their curves to Clemens would be useful, as it would show if the path he took to his late-career success is an anomaly. While it certainly is compared to most pitchers with early-career success, the more relevant question it seems is if the path he took is different than that taken by similarly successful late-career pitchers, like those mentioned before.

Additionally, do you control for other pitchers accused of using steroids? Charting their career paths and comparing them to Clemens's may be useful, as well.

Kyle S

Here's what I get when I filter to only include seasons of 90+ IP.

Schilling and Johnson have more typical aging curves, but Ryan's is even more of an outlier than is Clemens'.

http://sturgeongeneral.files.wordpress.com/2008/02/aging_curves.png

bert

Wouldn't it make more sense to do a seperate charting of how "performance enhanced" pitchers performed with age. If Clemens's trend matches more evenly with a juiced pitcher, than a typical "all-star" the trending with age would show it, especially in light of your ERA curve.

Caleb Powers

Kyle is certainly on the right track here. The specific comparison pointed out in the Clemens Report was between pitchers with a similar performance over their first ten years AND a similar late career performance. As Kyle points out, they are only half right. It seems to me, Kyle, that the true comparison would be to look at the career paths of pitchers with similar first ten year performances to Clemens, judged by the various standards you've developed. That is the specific claim that he makes, that is, that his own performance is not at odds with that of others who had similar beginnings to their career.

A statistical refutation of that point refutes his entire report.

Nathan

Just for kicks, I tried comparing Clemens' ERA to recent hall-of-famers (without smoothing), and the results are a little less compelling, but still kind of interesting:

http://flowingdata.com/2008/02/11/comparing-roger-clemens-to-hall-of-fame-pitchers/

Chris M

The statistics are telling..however the NY Times owns a part of the Red Sox...James Mitchell sits on the Red Sox Board of Directors... Clemens is arguably the greatest pitcher of the last 50 yrs and left the Red Sox after the 96' season.. Dan Duquette made the comment that Clemens was in the "twilight of his career" and the Red Sox were embarrassed as ROCKET went on to win multiple CY Youngs and 2 World Series victories with the hated Yankees.. What better way for the Sox to get even by having the Mitchell Report use Clemens as the lynchpin of the their investigation.. Why were there no prominent Red Sox players in this report?? Am I to believe that Varitek and Damon who were named in the preliminary report on CNBC had no links to PED use?? AM I to believe that David Ortiz aka Big Papi went from being waiver wire fodder in 02' to being the Dominican Babe Ruth just 2 yrs later?? I think the real investigation should be how Bud Selig steered the Red Sox to an ownership group that was outbidded by at least one other group and how the Mitchell report is skewed towards implicating major players on their most hated rivals while they remain relatively unblemished... and to make matters worse we need a congressional hearing ?? Something stinks!!

Read more...

Bill in Denver

I'm raising my eyebrows as much as anyone over Clemens' late career surge, but I have to take exception with the vague characterization that something unusual was at play. In this context, the implication is clearly that Clemens was up to something unsavoury. Keep in mind however, that the comparison pitchers used in the analysis compromised only 30 data points spread out over 40 years of baseball history. During the time in Clemens' career where he appears to have improved his stats unusually, what was happening in the rest of baseball? This period coincides, for example, with the new MLB drug testing poicy. Were all pitching stats improved over this time as power hitting numbers declined? I'd be more concerned for Clemens' stats if he dropped sharply again as we have seen with Bonds, and others who clearly benefited from drug use.

Cyril Morong

They don't seem to adjust for league averages or park effects. They also mention that ERA is not a good stat since it is affected by fielders. But the two graphs show his career trajectory in ERA and walks + hits per 9 IP, both affected by fielders. Sabermetricians for several years have been using an ERA based only on defense independent stats (DIPS ERA).

They show that Clemens has done better than a typical career trajectory would predict. But there are always going to be atypical pitchers. If he exceeded the normal career trajectory by alot more than anyone else ever has (as I think is the case for Bonds) then it would mean something. They don't show what other pitchers did. Warren Spahn kept pitching great )in the late 1950s and early 1960s) past the age of 40, presumably in an era before PEDs.

JC Bradbury, author of the book "The Baseball Economist," has a different analysis

http://www.sabernomics.com/sabernomics/index.php/2008/02/how-did-clemens-age-relative-to-other-pitchers/

Read more...

Alex

I agree with Bill that the implication is definitely anti-Clemens. It was my assumption that Clemens' analysis was to show that he wasn't a complete outlier. If Nolan Ryan, Johnson, or Schilling were visiting Congress, would your report have said that there was unusual activity in their late careers? To my knowledge, no one has accused any of them of using PEDs.

Dan

This analysis doesn't seem all that informative. Clemens has late career numbers that are better than the average of the other 31 pitchers, but in order to decide whether his performance is "unusual" among this group of 32 highly durable pitchers you have to look at the range of performances of the other 31 pitchers, not just the central tendency.

For instance, if you can summarize each pitcher's career trajectory with a single number, with (say) lower numbers representing better late-career performance, then you could rank the 32 pitchers and see where Clemens falls on that list. If that measure has him 1st in late-career performance by a wide margin, then that would be a very unusual performance. If he's 9th out of 32, and close to several other pitchers on both sides, then not so much.

Mike P.

If you look at Clemens' career you will see 6 unusual seasons - 1993, 1995, 1996, and 2004-2006. The first three were three of the last four of his years in Boston and were unusual because he was average instead of great. Ask any Boston fan about those years and they will say he was lazy, out of shape, and unmotivated - potentially due to him fighting with management. He then went to Toronto and had 2 great years at ages 34 and 35. These seasons were seen as a return to the Clemens of old and can't realistically be called unusual. Again, ask a Bostonian and you will hear that he was re-energized in Toronto and started working again. He then had 5 average years in New York, consistent with an aging pitcher. The eyebrows should only be raised about his 3 stellar years in Houston (2004-2006) at ages 41-43.
As to the analysis above and the Hendricks' analysis - they, of course, prove nothing. A cursory glance reveals that Clemens' was better after 35 than most, but he was also better before 32 than most. So comparing him to all pitchers with a lengthy record in valid, but comparing him to pitchers with elite early records similar to his is not invalid.
Really, the only seasons that seem unusual are his 2004-2006 (age 41-43) in Houston. As far as I can tell the only other pitcher, post Babe Ruth, with that level of success at 40 or over is Randy Johnson's age 40, 2004 season.

Read more...

Ted

I appreciate the in depth analysis, but it is not deep enough for conclusions. The following factors still need to be factored in: who was catching (catchers usually call the pitches, better catchers do better calling pitches); team, Clemens was on better teams later, meaning the better offensive players were playing with him, not against him.

Showing the lines of Clemens against 4 or 5 of the 31 would give an idea of true trajectory of the sample. Others could have similar paths, and Ryan must be close (you did not show the one they think is most comparable, and he among the few pitchers that do compare).

zai

So how does the graph of Nolan Ryan versus the other pitchers look? If his team is suggesting that he is normal within the pool of unusually long-playing pitchers, it is a valid argument to make: special, but not so special as to be singularly unique and thus, suspicious.