Back to the drawing board for our latest critics…and also the Wall Street Journal and (Oops!) the Economist.

Thanks to articles in the Wall Street Journal and the Economist, a working paper by Chris Foote and Chris Goetz that is sharply critical of John Donohue and me has gotten an enormous amount of attention.

In that working paper Foote and Goetz criticized the analysis underlying one of the tables in our original article that suggested a link between legalized abortion and crime. (It is worth remembering that the approach they criticize was one of four distinct pieces of evidence we presented in that paper…they offer no criticisms of the other three approaches.)

Foote and Goetz made two basic changes to the original analysis we did. First, they correctly noted that the text of our article stated that we had included state-year interactions in our regression specifications, when indeed the table that got published did not include these state-year interactions. Second, they correctly argue that without controlling for changes in cohort size, the original analysis we performed provided a test of whether cohorts exposed to high rates of legalized abortion did less crime, but did not directly afford a test of whether “unwantedness” was one of the channels through which this crime reduction operated. (Note: we didn’t claim that this particular analysis was a direct test of the “unwantedness” hypothesis. This last section of the paper was the most speculative of analysis of all that we did and frankly we were surprised it worked at all given the great demands it put on the data.) They found that once you made those changes, the results in our original Table 7 essentially disappear.

There is, however, a fundamental problem with the Foote and Goetz analysis. The abortion data that are available are likely to be quite noisy. As one adds more and more control variables (e.g. nearly 1,000 individual state-year interactions), the meaningful variation in abortion rates gets eaten away. The signal-to-noise ratio in what remains of the variation in measured abortions gets worse and worse. That will lead the measured impact of abortions on crime to dwindle. Because the analysis is carried out with a unit of analysis of a state-year-single year of age, the analyses performed are highly saturated with interactions: state-age interactions, age-year interactions, and state-year interactions. Together, these interactions account for over 99% of the variance in arrest rates and over 96% of the variation in the abortion proxy. It is an exercise which is incredibly demanding of the data.

In light of this, it seems uncontroversial that one would want to do the best one could in measuring abortion when carrying out such an exercise.

The abortion measure used by Foote and Goetz is one that is produced by the Alan Gutmacher Institute. The Alan Gutmacher Institute makes estimates based on surveys of abortion providers of the number of abortions performed per live birth in each state and year.

To proxy for the abortion exposure of, say, 19 year olds arrested in California in 1993, Foote and Goetz use the abortion rate in California in 1973. This is not an unreasonable first approximation (and indeed is the one we used in most parts of our original paper because it is simple and transparent), but it is just an approximation for a number of reasons:

1) There is a great deal of cross-state mobility. So many of the 19 year olds arrested in California in 1993 were not born in California. They were born in other states, or possibly other countries. Indeeed, I believe that recent figures suggest that over 30% of those in their late teens do not reside in the state in which they were born.

2) Using a date of 20 years earlier to proxy for the abortion exposure of a 19 year old induces an enormous amount of noise. If I am a 19 year old sometime in 1993, I may have been born as early as Jan. 2, 1973 (that would make me still 19 on Jan. 1, 1993) or as late as Dec. 31, 1974 (that would have me turning 19 on Dec. 31, 1993). Abortions occur sometime in advance of birthdays, typically about 13 weeks into a pregnancy. So the relevant date (roughly) of when those who are 19 in 1993 would have been exposed to legalized abortion is about six months before they were born, or July 2, 1972-June 30, 1974. While that window overlaps with the year 1973 (which is what Foote and Goetz use as their time period of abortion exposure), note that it also includes half of 1972 and half of 1974!

3) A non-trivial fraction of abortions performed in the United States, especially in the time when legalization was taking place, involved women crossing state lines to get an abortion. As a consequence, measuring abortions in terms of the state in which the abortion is performed (which is what the data Foote and Goetz use does), rather than the state of residence of the woman getting the abortion, induces further measurement error into their abortion proxy.

4) The Alan Gutmacher abortion numbers are, even by the admission of the people who collect the data themselves, far from perfect. Indeed, the correlation between these abortion estimates and another time series collected by the CDC is well below one, suggesting that even if problems (1)-(3) did not exist, there would be substantial measurement error. The correlation between the Alan Gutmacher measure and the CDC measure, not surprisingly, gets lower and lower the more control variables that are included. This is exactly what one would expect if the controls are taking the signal out of the abortion measures and leaving behind mostly noise.

What John Donohue and I have done (with fantastic research assistance from Ethan Lieber), is attempted to address as best we can these four problems with the abortion measure that Foote and Goetz are using. In particular, we do the following:

1) As we describe in our original paper on abortion, one can deal with cross-state mobility by using the decennial censuses to determine the state of birth for the current residents of a state (the results from carrying out this correction in our crime regressions are reported in Table 5 of the original 1999 paper). This is possible to do because the census micro data reports the state of birth and current state of residence for a 5% sample of the U.S. population. Note that the correction we are able to make is unlikely to be perfect, so it may not fully solve the problem, but it clearly moves us in the right direction.

2) Given that the window of abortion exposure that 19 year olds in 1993 spans from 1972 to 1974, the obvious solution to this problem is to allow abortions performed in 1972, 1973, and 1974 to influence arrests of 19 year olds in 1993. It is straightforward to work out roughly the weights that one wants to put on the different years’ abortion rates (or one can do it non-parametrically and let the data decide — the answers are virtually identical).

3) In order to deal with the fact that many women were crossing state lines to get abortions in the 1970s, we use the Alan Gutmacher Institute’s estimates of abortions performed on women residing in a state relative to live births in that state. (We were unaware of the existence of these better data when we wrote the initial paper, otherwise we would have used them at that time.) There is little question that measuring abortions by state of residence is superior to measuring them by where the procedure is performed.

4) The standard solution to measurement error is to perform instrumental variables in which one uses one noisy proxy of the phenomenon that is poorly measured as an instrument for another noisy proxy. (I recognize that most readers of this blog will not understand what I mean by this.) In this setting, the CDC’s independently generated measure of legalized abortions is likely to be an excellent instrument. Because there is so much noise in each of the measures, the standard errors increase when doing this IV procedure, but under a standard set of assumptions, the estimates obtained will be purged of the attentuation bias that will be present due to measurement error.

I think that just about any empirical economist would tend to believe that each of these four corrections we make to the abortion measure will lead us closer to capturing a true impact of legalized abortion on crime. So, the question becomes, what happens when we replicate the specifications reported in Foote and Goetz, but with this improved abortion proxy?

The results are summarized in this table, which has two panels. The top panel are the results for violent crime. The bottom panel corresponds to property crime.

Starting with the first panel, the top row reports the same specifications as Foote and Goetz (I don’t bother showing their estimates excluding state-age interactions because it makes no sense to exclude these and they themselves say that their preferred specifications include state-age interactions). We are able to replicate their results. As can be seen, the coefficients shrink as one adds state-year interactions and population controls.

The second row of the table presents the coefficients one obtains with our more thoughtfully constructed abortion measure (changes 1-3 above having been made to their abortion measure). With a better measure of abortion, as expected, all the estimated abortion impacts increase across the board. The results are now statistically significant in all of the Foote and Goetz specifications. Even in the final, most demanding specification, the magnitude of the coefficient is about the same as in the original results we published that didn’t control for state-year interactions or population. The only difference between
what Foote and Goetz did and what we report in row 2 is that we have done a better job of really measuring abortion. Everything else is identical.

The third row of the table reports the results of instrumental variables estimates using the CDC abortion measure as an instrument for our (more thoughtfully constructed) Alan Gutmacher proxy of abortions. The results all get a little bigger, but are more imprecisely estimated.

The bottom panel of the table shows results for property crime. Moving from Foote and Goetz’s abortion measure in the top row to our more careful one in the second row (leaving everything else the same), the coefficients become more negative in 3 of the 4 specifications. Doing the instrumental variables estimation has a bigger impact on property crime than on violent crime. All four of the instrumental variables estimates of legalized abortion on property crime are negative (although again less precisely estimated).

The simple fact is that when you do a better job of measuring abortion, the results get much stronger. This is exactly what you expect of a theory that is true: doing empirical work closer to the theory should yield better results than empirical work much more loosely reflecting the theory. The estimates without population controls, but including state-year interactions, are as big or bigger than what is in our original paper. As would be expected (since the unwantedness channel is not the only channel through which abortion is acting to reduce crime), the coefficients we obtain shrink when we include population controls. But, especially for violent crime, a large impact of abortion persists even when one measures arrests per capita.

The results we show in this new table are consistent with the impact of abortion on crime that we find in our three other types of analyses we presented in the original paper using different sources of variation. These results are consistent with the unwantedness hypothesis.

No doubt there will be future research that attempts to overturn our evidence on legalized abortion. Perhaps they will even succeed. But this one does not.