How to Spot Advocacy Science: John Taylor Edition

Photo: iStockphoto

Sometimes you see the perfect piece of evidence. The scatter plot that is just so. The data line up perfectly. And then you realize, perhaps they’re just too perfect. What you are seeing is advocacy, dressed up as science. Here’s an example, provided by John Taylor (via Greg Mankiw):

 

 

Taylor’s conclusion:

The data on spending shares show that the most effective way to reduce unemployment is to raise investment as a share of GDP.

But why begin the scatter plot in 1990? There’s no good reason. In fact, most folks typically download the entire history of available macro data. So let’s see what happens if we extend it back to, say, 1970:

 

Hmm… What conclusions should we draw about this relationship? And now why do you think Taylor began his sample in 1990?

Actually, we should use all the available data. The chart below goes back to 1948, when these series—in their current form—began:

 

Now what’s your conclusion?

Here’s Mankiw’s assessment of Taylor’s claim:

There’s no doubt that the strength of the correlation is impressive.


But when you look beyond the cherry-picked sample, the correlation is a decidedly unimpressive -0.14.

Here’s my conclusion: On balance, times in which the investment share is higher, are slightly more likely to be good times. But I’m not sure why. Is it—as Taylor asserts—that high investment shares create good times? Or is it that good times encourage investment? Or is it a third factor—perhaps in good times the government doesn’t need to prime the fiscal pump, and so the investment share is higher? Or is it something else?

Be wary of economists wielding short samples.


Anna

The NYT publishes advocacy, junk science polls and studies all the time, unfortunately. Everybody in the media has an agenda to push, to the detriment of our country's future well-being.

Joshua Northey

Why single out the NYT (which I am not huge fan of personally).? Many journals publish "advocacy science" (I am not sure that is a useful category). It is the nature of the beast. If you don't include narrative urgency into your articles how on earth are you going to drum up more funding for your next round of research? If journals don't publish articles with narrative urgency how on earth are they going to sell copies?

So much science that is published takes the form of:

wild speculation based on conclusion A
evidence A
argument/science
valid conclusion A
wild speculation based on conclusion A

You find out that polar bear populations are decreasing. You use science to discover this is due to the ice receding. Then you couch the whole thing in a narrative about how climate change is going to destroy the world's habitability. Which is a complete nonsequitur because it is not at all clear that this disappearance of polar bears is even going to noticeably harm the productivity of arctic ecosystem, much less make the world uninhabitable.

Most science I encounter is great, but the conclusions and speculations that accompany that science are as biased as the ones you get from any random person.

Read more...

Michael Fisk

Could part of the difference be a function of differences in monetary policy and money demand from various eras? There is the possibility that, due to changes in capital markets, a correlation could exist now that didn't exist before. But yes, the correlation is almost suspicious, even more so when there's no firm conclusions to be drawn from it.

Clancy

Wow, I just read Krugman’s equally thorough take-down of Taylor’s graph. http://krugman.blogs.nytimes.com/2011/03/30/whats-behind-low-investment/

But Krugman takes a different angle and points out the reason for the drop in investment is mostly the real-estate bust.
Guess today was a bad day to post misleading charts.

Wendy

I am not an economist so I don't really know anything about spending shares but as a scientist I would say that something really interesting happened in the 70's and 80's. The rest of the data basically fits Taylor's pattern so a more interesting question would be: why was there such high unemployment in the late 1970's and 1980's despite the fact that the investment to GDP ratio was relatively high? This isn't cherry picking it is just figuring out when a pattern holds and when it doesn't.

Joshua

Taylor might well be right if his statement is appropriately restricted. It looks like there is very little, if any, correlation prior to 1990, followed by phenomenal correlation. Given that stark contrast it might well be that something changed around 1990 that led to high correlation between GDP and investment.

Of course, even under this interpretation Taylor would still be at fault for presenting the data as though they were representative.

Wendy

Alternatively you could ask what the heck happened in 1990, because before that it looks like the pattern might have been the opposite of what Taylor describes. Simple linear regression isn't always the solution, sometimes you have to break the data into chunks to figure out what happened when.

jonathan

Putting aside Taylor's conclusion, which is silly because he ignores housing investment's cliff dive, I'd love to see this kind of data mapped against traditional correlations for investment, notably interest rates and inflation. What I see in Taylor's bit and then your graphs is another example of the oddity of the current situation, with zero rates not creating investment - despite the theoretical arguments of Austrians et al - and thus this weird kind of wiggling around by people unwilling to accept that the data says their models are wrong. Thus, we aren't seeing investment so therefore rather than accept our models are wrong, we blame something inchoate floating in the air. That isn't economics. We can translate that kind of thing into economics through surveys that measure confidence and those say that demand worries (sales) are 3x normal while government-related issues (regulation, taxation) are roughly the same as normal. But rather than accept that data, the choice is to pick that inchoate thing for which there is only hand-picked anecdotal evidence.

Read more...

serge d'agostino

these stuffs show that economics are historical science: which is (maybe) true from 1990 to 2010 (or which is a good correlation), can be false (or bad correlation) between 1948 and 1990, etc.

Mark A. Sadowski

Not to excuse Taylor by any means (I 'm normally suspicious of his intents) but NAIRU fell sharply during the 1990s. What if you graphed the relationship between investment and the gap betweeen unemployment and NAIRU?

Mark Brucker

I would think that you would have to look for lags in the effect of investment on employment for it to have any value at all!

Scott

Use of the smaller dataset could be justified if we have a good reason to believe that something important changed in or about 1990. Indeed, your second graph seems to show that there are two distinct periods with respect to the data-- One period prior to 1990 where the relationship is loosely correlated and maybe slightly positive, and a second period after 1990 where the relationship is strongly negatively correlated. It's not cherry-picking to think that something may have changed in the broader economy to cause this change in correlation.

James

Though I do agree with some parts of what you are saying I will say that to:

"Be wary of economists wielding short samples."

I counter: "Be wary of economists who believe economies are the same over long periods of time"

JJ

+10^10^10
Still the most important conclusion would be: don't call it a science. Have 10 economists look at it and you'll get 10 different "opinions"/"theories".

Andrew Berman

There is no problem with the original plot. Even cherry-picked, it's an impression basket of cherries. The question is-- what does the plot mean. It could mean that during those years, something was happening to put them in correlation. It could be the reverse of the natural explanation, i.e., perhaps higher employment resulted in higher investment..

Considering that it's doubtful that investment would reduce unemployment the very same year, perhaps one could plot investment vs. employment 2 years later. Or, even better, just do a time series with unemployment and investment as two plots. Any correlations would show up there as well.

Michael

How to spot Advocacy Science: Freakonomics Edition:

Note that a chart uses limited data, but DO NOT attempt to determine reasons that data may be limited. Question only the completeness, not the methodology. For example, don't mention that the confidence/accuracy of the numbers on the scatter chart is much lower when the GDP per capita ratio was much lower, and the post-1993 numbers represent much more robust comparison.

No, don't do that. Just point out that earlier numbers are different, then vaguely insult the person who compiled them.

Good work, Freakonomics! You've taught me another skill!

O

---For example, don’t mention that the confidence/accuracy of the numbers on the scatter chart is much lower when the GDP per capita ratio was much lower, and the post-1993 numbers represent much more robust comparison.

What is the reason to believe in this claim?

fresno dan

in any bunch of data, you can find correlations. Just as with coin flips, the vast majority of the time it means nothing.

Mike MacDonald

Perhaps your post should be titled "How to miss paradigm shifts, Justin Wolfers edition. I'm an Electronics Engineer, not an Economist, but I collect experimental data, I analyze it, and I can unequivocally state that noise does not look like your first plot. That's signal. Perhaps your data is best presented in the form "Post stagflation", "Stagflation" and "Rebuilding from WWII". Those were economic phenomena, were they not? And are we rebuilding from war still? If not, why use that data? What's the "misery index"? If it's low, why include the seventies and eighties?

Gary

This technique is at the core of so much "climate science." When you peel back the covers and look at the data - especially it's quality - and how the global warming story is presented, you find exactly this kind of cherry-picking in way too many peer-reviewed papers, not to mention the IPCC assessments. It's a disservice to the public to lie by omission, but that's what happens when a cause overtakes integrity.

Joshua Northey

That may be true, but I hope you admit that the world is nevertheless warming, and that people are causing this? It is very clear that is the case.

Now personally I actually think that warming is actually a good thing, and most of the harms scientists' predict from it are just a bunch of henny-penny scare mongering (I would bet we will eventually decide to melt the ice sheets when we begin looking into terraforming other worlds (it will be so much easier/cheaper to terraform earth than it will be to terraform mars)).

But none of that changes the fact that warming is happening.

Nosybear

Can it be the lurking variable is the prosperity itself? One might ask, too, investment in what. There's a logical (unproven) connection between improving the infrastructure of the country, investment in basic research (as long as it's not handedness in polar bears) and investment in the human capital of the country - economic theory would indicate these will increase GDP.

Keith Eubanks

Even with the new data, I'd say something was there. The question is why would the 1970s and 1980s present a different relationship. If there is a causal relationship between investment and employment, what other factors enter the picture?

Since 1990, the correlation seems strong. What about the 1970s and 1980s might have changed this relationship? Are there any longer term trends changing this picture? Is the data the same: definition wise? How did the price controls and inflation of the 1970s affect this? How about deregulation? How about regulations? How about population growth?

The economy is not just a random walk. Many causal factors drive employment; certainly investment would be a primary factor. What this points out though is that our understanding is still limited.

My question would be are public polices (including the Federal Reservess) founded on a better understanding or are people taking actions for which they truly do not understand the likely outcomes? Or perhaps their understanding is sufficiently limited that their actions are potentially damaging?

Read more...

Keith Eubanks

Any idea what the picture would look like if investment were limited to private investment?