How to Spot Advocacy Science: John Taylor Edition

Photo: iStockphoto

Sometimes you see the perfect piece of evidence. The scatter plot that is just so. The data line up perfectly. And then you realize, perhaps they’re just too perfect. What you are seeing is advocacy, dressed up as science. Here’s an example, provided by John Taylor (via Greg Mankiw):

 

 

Taylor’s conclusion:

The data on spending shares show that the most effective way to reduce unemployment is to raise investment as a share of GDP.

But why begin the scatter plot in 1990? There’s no good reason. In fact, most folks typically download the entire history of available macro data. So let’s see what happens if we extend it back to, say, 1970:

 

Hmm… What conclusions should we draw about this relationship? And now why do you think Taylor began his sample in 1990?

Actually, we should use all the available data. The chart below goes back to 1948, when these series—in their current form—began:

 

Now what’s your conclusion?

Here’s Mankiw’s assessment of Taylor’s claim:

There’s no doubt that the strength of the correlation is impressive.


But when you look beyond the cherry-picked sample, the correlation is a decidedly unimpressive -0.14.

Here’s my conclusion: On balance, times in which the investment share is higher, are slightly more likely to be good times. But I’m not sure why. Is it—as Taylor asserts—that high investment shares create good times? Or is it that good times encourage investment? Or is it a third factor—perhaps in good times the government doesn’t need to prime the fiscal pump, and so the investment share is higher? Or is it something else?

Be wary of economists wielding short samples.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 37

View All Comments »
  1. Mark A. Sadowski says:

    Not to excuse Taylor by any means (I ‘m normally suspicious of his intents) but NAIRU fell sharply during the 1990s. What if you graphed the relationship between investment and the gap betweeen unemployment and NAIRU?

    Thumb up 0 Thumb down 0
  2. Mark Brucker says:

    I would think that you would have to look for lags in the effect of investment on employment for it to have any value at all!

    Thumb up 0 Thumb down 0
  3. Scott says:

    Use of the smaller dataset could be justified if we have a good reason to believe that something important changed in or about 1990. Indeed, your second graph seems to show that there are two distinct periods with respect to the data– One period prior to 1990 where the relationship is loosely correlated and maybe slightly positive, and a second period after 1990 where the relationship is strongly negatively correlated. It’s not cherry-picking to think that something may have changed in the broader economy to cause this change in correlation.

    Thumb up 0 Thumb down 0
  4. James says:

    Though I do agree with some parts of what you are saying I will say that to:

    “Be wary of economists wielding short samples.”

    I counter: “Be wary of economists who believe economies are the same over long periods of time”

    Thumb up 1 Thumb down 0
    • JJ says:

      +10^10^10
      Still the most important conclusion would be: don’t call it a science. Have 10 economists look at it and you’ll get 10 different “opinions”/”theories”.

      Thumb up 0 Thumb down 0
  5. Andrew Berman says:

    There is no problem with the original plot. Even cherry-picked, it’s an impression basket of cherries. The question is– what does the plot mean. It could mean that during those years, something was happening to put them in correlation. It could be the reverse of the natural explanation, i.e., perhaps higher employment resulted in higher investment..

    Considering that it’s doubtful that investment would reduce unemployment the very same year, perhaps one could plot investment vs. employment 2 years later. Or, even better, just do a time series with unemployment and investment as two plots. Any correlations would show up there as well.

    Thumb up 0 Thumb down 0
  6. Michael says:

    How to spot Advocacy Science: Freakonomics Edition:

    Note that a chart uses limited data, but DO NOT attempt to determine reasons that data may be limited. Question only the completeness, not the methodology. For example, don’t mention that the confidence/accuracy of the numbers on the scatter chart is much lower when the GDP per capita ratio was much lower, and the post-1993 numbers represent much more robust comparison.

    No, don’t do that. Just point out that earlier numbers are different, then vaguely insult the person who compiled them.

    Good work, Freakonomics! You’ve taught me another skill!

    Thumb up 0 Thumb down 0
    • O says:

      —For example, don’t mention that the confidence/accuracy of the numbers on the scatter chart is much lower when the GDP per capita ratio was much lower, and the post-1993 numbers represent much more robust comparison.

      What is the reason to believe in this claim?

      Thumb up 0 Thumb down 0
  7. fresno dan says:

    in any bunch of data, you can find correlations. Just as with coin flips, the vast majority of the time it means nothing.

    Thumb up 0 Thumb down 0
  8. Mike MacDonald says:

    Perhaps your post should be titled “How to miss paradigm shifts, Justin Wolfers edition. I’m an Electronics Engineer, not an Economist, but I collect experimental data, I analyze it, and I can unequivocally state that noise does not look like your first plot. That’s signal. Perhaps your data is best presented in the form “Post stagflation”, “Stagflation” and “Rebuilding from WWII”. Those were economic phenomena, were they not? And are we rebuilding from war still? If not, why use that data? What’s the “misery index”? If it’s low, why include the seventies and eighties?

    Thumb up 1 Thumb down 0