So Long and Thanks for All the F-Tests

I’ve been reading a truly excellent book by Joshua Angrist and Jorn-Steffen Pischke called Mostly Harmless Econometrics: An Empiricist’s Companion. It’s not written for a general audience, but if you pulled an A- or better on a college-level econometrics course (and if you love Freakonomics), then this is the book for you. It should be required reading for anyone who is trying to write an applied dissertation. It is the rare book that captures the feeling of how to go about trying to attack an empirical question; and it does this by working through two or three dozen of the neatest empirical papers of the last decade (often coauthored by Angrist). It is also peppered with references to Douglas Adams‘s writing — so what’s not to like?

Here’s a fine example, in plain English, explaining how econometricians think about what they are doing:

[Something that distinguishes] the discipline of econometrics from the older sister field of statistics … is a lack of shyness about causality. Causal inference has always been the name of the game in applied econometrics. Statistician Paul Holland (1986) cautions that there can be “no causation without manipulation,” a maxim that would seem to rule out causal inference from nonexperimental data. Less thoughtful observers fall back on the truism that “correlation is not causality.” Like most people who work with data for a living, we believe that correlation can sometimes provide pretty good evidence of a causal relation, even when the variable of interest has not been manipulated by a researcher or experimenter. (p. 133)

The book backs up this assertion by teaching the reader to think carefully about what assumptions about the counter-factual are necessary to make a causal inference. I was thinking about the book a couple of weeks ago when reading a New York Times article discussing the college and law-school years of Supreme Court nominee Sonia Sotomayor. The article in the second paragraph claims that Judge Sotomayor “benefited from affirmative action policies.” To me, this is pretty clearly a causal claim and this claim is not well supported by the subsequent evidence in the article.

At least one relevant counterfactual question to ask is “What would have happened to Judge Sotomayor in applying to college and law school in a world without affirmative action?” We are told that Ms. Sotomayor was an honors student in high school and that she graduated near the top of her class in college. James A. Thomas, a former dean of admissions, concluded that “Ms. Sotomayor’s background had little role in her acceptance to [Yale Law] school.” This is hardly strong evidence for claiming that she was a beneficiary of affirmative action. The article shows that it is not just econometricians who can mistake correlation for causation. It is a mistake that a reader of Angrist and Pischke is less likely to make.


Caliphilosopher

Counterfactual analyzation isn't the only way to sift around for causation. There are other ways to try to ascertain causal mechanisms - interventionistic approaches and probabilistic approaches.

I'm curious to know what econometricians have to say regarding probabilistic causation and interventionist causation, especially when it seems as if economic policies affect outcomes more closely to those two than through a counterfactual process.

Tyler

I agree that the causal relationship implied may be a bit unfounded, but what do you expect the dean of admissions to say about one of the school's most distinguished alumni, "We only let her in because she was a hispanic woman"? Just saying that we have to take both sides of the argument with a grain of salt, especially when the "evidence" is subjective interviews done ex post facto.

MikeM

Getting accepted to Yale law school is still a long, looooong way from getting nominated to the Supreme Court. And one of these selections is almost certainly more merit-based than the other.

While maybe not technically affirmative action, as soon as Souter announced he would step down, it was immediately declared that Obama should choose a woman, preferably a Latina, to replace him. Heck of a way to get your name on the short list if you're already an appeals judge.

SpecialK

Sotomayor is on record admitting she was a "product of affirmative action". She admitted her standardized test scores were lower than her peers. Based on her own views I think we can safely assume what would have happened to this "perfect affirmative action baby" had there not been affirmative action. She wouldn't have had nearly the same resume and she wouldn't be a potential lifetime member to the most powerful court in the most powerful country.

Othere than that I liked the econometrics stuff.

athelas

Well, Sotomayor did not clarify the issue much when she proudly declared herself an "affirmative action baby" even as her grades were "highly questionable."

"My test scores were not comparable to that of my colleagues at Princeton or Yale," Sotomayor once said on a discussion panel during an event sponsored by a non-profit law organization in the 1990s.

charles

"Like most people who work with data for a living, we believe that correlation can sometimes provide pretty good evidence of a causal relation..."

I think this is really funny. Undoubtedly, he meant to imply a separation in expertise between those who work with data "for a living" and those who do not but what he said is:

"Like most dairy farmers we believe milk is critical to the human diet."

And, I'll warn, you often do not know what you do not know.

Lastly, the devil is hiding in the "sometimes", a word, those who work with data for a living would notice, was carefully chosen.

Caliphilosopher

I think that a number of comments on this thread are missing the point. Here's how:

1 - Besides interviews, what other evidence does one propose to analyze counterfactual claims? It cant be (solely or primarily) empirical evidence, since the claim is about a situation that has not happened and (nomologically speaking) cannot happen (i.e., no Back To The Future scenarios).

2 - Since when were supreme court nominations meritocratic? it's not like the political process is a meritocratic one all the time. Regardless of party affiliation, there have been a number of questionable nominations.

3 - Are test scores the only thing that matters to law school admissions? Unless we have some actual admissions deans on this blog, I would strongly advise against the conflation of test scores with what matters for law school admissions.

jdiec

Shame on all of you for taking a perfectly good opportunity to quote Douglas Adams and turning it into another squabble about Supreme Court nominees!

I'd just like to know if there are any questions econometricians have worked on where the answer is 42?

SpecialK

@Caliphilosopher

2 - Of course the political process isn't meritocratic. I was merely providing additional evidence, not cited in the post, to support the claim that she was helped by AA. After the admissions office, I think the student is in the best position to know whether or not they were given a leg up by AA.

3 - Again I'd point out the student is in a very good position to know how they stack up against their peers. Whatever the criteria for admission, Sotomayor believes she was helped by AA. I don't see how I could possibly doubt her on that point.

Levis A. Kochin

Testimony has a context. Sotomayor was affirming that suppor fpr Affimative Actiont when she claimed to be a beneficiary of Afirmative Action at Princeton and Yale. Those Justices appointed for their exceptional legal expertise were seldom marginal admits to Law School.

Caliphilosopher

@Special K

I think that your comments were furthest from missing the mark. I agree with you that the student would be in the best position to know, but I think that would be the case only if accurate grades were made public and non-anonymous.

@ jdiec

You mean to tell me that my very first comment wasn't on the mark as far as causation in economics? I thought that it was rather relevant to the attempt to measure causal influences on social/economic events, which is much harder to do than merely physical or chemical events. There is plenty to be said regarding issues surrounding measuring causation in economics (methodological individualism, participant-observation, unreasonable assumptions about the model used to predict whether or not something is "caused", etc.)

Richard

Darn you. Add one more book to the reading list. Thanks.

Jim

Always keep in mind that 108 of the 112 previous Supreme Count members were white men. I'm sure every one of them was a meritocratic decision.

Amin

Well I was C- so I guess I am excluded from this

Cheers!!

Allison

Joshua Angrist taught my Econometrics class last semester at MIT, and MHE was on the reading list. I dare say I did not read the whole thing, but I learned a ton from his class. He definitely knows his stuff! I recommend reading some of his papers too, especially if you're into labor economics.

PaulD

It does strike me as interesting that race is deemed NOT a suitable entry criterion for firefighters, but one can get away with nominating an hispanic woman for the Supreme Court even after leaking to the press that an hispanic female candidate would be politically expedient. One possible defense of race-based nominations to the Supreme Court might be that there are very few of them, and hence don't impact employment statistics much.

Bob Nease

There are also certain aspects of natural experiments that lend themselves to wringing causality from association. A particularly elegant and compelling example was an analysis of the effect of flexible sigmoidoscopy on colon cancer mortality. Selby et al used a case-control approach, which allows the determination of the relative magnitude of an association but not causality. The trick is that the flexible sig can only visualize the lower portion of the colon. What they found was that flex sig reduced mortality, but only for cancers located in the part of the colon that the sigmoidoscope could reach - making causality much more likely than mere association.

Y11

Prof. Ayres, I think the whole purpose of Mostly Harmless Econometrics was to make it, uh, mostly harmless for a broader audience that is interested in applied econometrics but can't bring itself to read dense textbooks.