So Long and Thanks for All the F-Tests

I’ve been reading a truly excellent book by Joshua Angrist and Jorn-Steffen Pischke called Mostly Harmless Econometrics: An Empiricist’s Companion. It’s not written for a general audience, but if you pulled an A- or better on a college-level econometrics course (and if you love Freakonomics), then this is the book for you. It should be required reading for anyone who is trying to write an applied dissertation. It is the rare book that captures the feeling of how to go about trying to attack an empirical question; and it does this by working through two or three dozen of the neatest empirical papers of the last decade (often coauthored by Angrist). It is also peppered with references to Douglas Adams‘s writing — so what’s not to like?

Here’s a fine example, in plain English, explaining how econometricians think about what they are doing:

[Something that distinguishes] the discipline of econometrics from the older sister field of statistics … is a lack of shyness about causality. Causal inference has always been the name of the game in applied econometrics. Statistician Paul Holland (1986) cautions that there can be “no causation without manipulation,” a maxim that would seem to rule out causal inference from nonexperimental data. Less thoughtful observers fall back on the truism that “correlation is not causality.” Like most people who work with data for a living, we believe that correlation can sometimes provide pretty good evidence of a causal relation, even when the variable of interest has not been manipulated by a researcher or experimenter. (p. 133)

The book backs up this assertion by teaching the reader to think carefully about what assumptions about the counter-factual are necessary to make a causal inference. I was thinking about the book a couple of weeks ago when reading a New York Times article discussing the college and law-school years of Supreme Court nominee Sonia Sotomayor. The article in the second paragraph claims that Judge Sotomayor “benefited from affirmative action policies.” To me, this is pretty clearly a causal claim and this claim is not well supported by the subsequent evidence in the article.

At least one relevant counterfactual question to ask is “What would have happened to Judge Sotomayor in applying to college and law school in a world without affirmative action?” We are told that Ms. Sotomayor was an honors student in high school and that she graduated near the top of her class in college. James A. Thomas, a former dean of admissions, concluded that “Ms. Sotomayor’s background had little role in her acceptance to [Yale Law] school.” This is hardly strong evidence for claiming that she was a beneficiary of affirmative action. The article shows that it is not just econometricians who can mistake correlation for causation. It is a mistake that a reader of Angrist and Pischke is less likely to make.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.



View All Comments »
  1. Caliphilosopher says:

    Counterfactual analyzation isn’t the only way to sift around for causation. There are other ways to try to ascertain causal mechanisms – interventionistic approaches and probabilistic approaches.

    I’m curious to know what econometricians have to say regarding probabilistic causation and interventionist causation, especially when it seems as if economic policies affect outcomes more closely to those two than through a counterfactual process.

    Thumb up 0 Thumb down 0
  2. Tyler says:

    I agree that the causal relationship implied may be a bit unfounded, but what do you expect the dean of admissions to say about one of the school’s most distinguished alumni, “We only let her in because she was a hispanic woman”? Just saying that we have to take both sides of the argument with a grain of salt, especially when the “evidence” is subjective interviews done ex post facto.

    Thumb up 0 Thumb down 0
  3. MikeM says:

    Getting accepted to Yale law school is still a long, looooong way from getting nominated to the Supreme Court. And one of these selections is almost certainly more merit-based than the other.

    While maybe not technically affirmative action, as soon as Souter announced he would step down, it was immediately declared that Obama should choose a woman, preferably a Latina, to replace him. Heck of a way to get your name on the short list if you’re already an appeals judge.

    Thumb up 0 Thumb down 0
  4. SpecialK says:

    Sotomayor is on record admitting she was a “product of affirmative action”. She admitted her standardized test scores were lower than her peers. Based on her own views I think we can safely assume what would have happened to this “perfect affirmative action baby” had there not been affirmative action. She wouldn’t have had nearly the same resume and she wouldn’t be a potential lifetime member to the most powerful court in the most powerful country.

    Othere than that I liked the econometrics stuff.

    Thumb up 0 Thumb down 0
  5. athelas says:

    Well, Sotomayor did not clarify the issue much when she proudly declared herself an “affirmative action baby” even as her grades were “highly questionable.”

    “My test scores were not comparable to that of my colleagues at Princeton or Yale,” Sotomayor once said on a discussion panel during an event sponsored by a non-profit law organization in the 1990s.

    Thumb up 0 Thumb down 0
  6. charles says:

    “Like most people who work with data for a living, we believe that correlation can sometimes provide pretty good evidence of a causal relation…”

    I think this is really funny. Undoubtedly, he meant to imply a separation in expertise between those who work with data “for a living” and those who do not but what he said is:

    “Like most dairy farmers we believe milk is critical to the human diet.”

    And, I’ll warn, you often do not know what you do not know.

    Lastly, the devil is hiding in the “sometimes”, a word, those who work with data for a living would notice, was carefully chosen.

    Thumb up 0 Thumb down 0
  7. Caliphilosopher says:

    I think that a number of comments on this thread are missing the point. Here’s how:

    1 – Besides interviews, what other evidence does one propose to analyze counterfactual claims? It cant be (solely or primarily) empirical evidence, since the claim is about a situation that has not happened and (nomologically speaking) cannot happen (i.e., no Back To The Future scenarios).

    2 – Since when were supreme court nominations meritocratic? it’s not like the political process is a meritocratic one all the time. Regardless of party affiliation, there have been a number of questionable nominations.

    3 – Are test scores the only thing that matters to law school admissions? Unless we have some actual admissions deans on this blog, I would strongly advise against the conflation of test scores with what matters for law school admissions.

    Thumb up 0 Thumb down 0
  8. jdiec says:

    Shame on all of you for taking a perfectly good opportunity to quote Douglas Adams and turning it into another squabble about Supreme Court nominees!

    I’d just like to know if there are any questions econometricians have worked on where the answer is 42?

    Thumb up 0 Thumb down 0