Detecting Political Momentum Is Harder Than You Think

Over at?FiveThirtyEight.com, Nate Silver has a post attempting to debunk the idea that there is momentum in political campaigns.? But I think he’s wrong.? And his post provides a fun opportunity for a simple statistics lesson on the difficulty of discovering momentum.

Here’s what Nate does: He compares the change in polls between successive periods.? If there were momentum, he argues, a rise in the polls this month would make it more likely there is another rise next month, while a fall in this month’s polls would likely yield another fall next month.? That is, he believes that momentum will cause successive changes in a time series to be positively correlated. Instead, he finds the opposite to be true. (His graph is?here.)

But this isn’t a fair test of the momentum hypothesis.? Here’s the problem: Imagine that there is some measurement error in the poll taken this month – perhaps the pollsters just interviewed too many Democrats, or too many Republicans. If there’s any kind of measurement error, this could drive Nate’s findings.? There are two cases to consider:

  • If the estimate for this month is too high, it will make the change between this month and last month look too large, and it will make the change from this month to next month look too small (or perhaps negative).
  • Or, if the estimate for this month is too low, it will make the change between this month and last month look too small, and it will make the change from this month to next month look too large.

Either way, if there’s any kind of noise in your estimate for this month, you get a?negative correlation between adjoining changes in polls.? But in my example, this negative correlation is driven by measurement error in the polls, not by the presence or absence of momentum.

The problem is that he’s using the same measurement – this month’s poll – in constructing both of the variables he’s analyzing.? And this is likely responsible for much of the (negative) correlation he observes.? So perhaps this negative correlation is in fact disguising true momentum in political races.

Perhaps a simple example will help.? I’m going to use data on the black unemployment rate to make my point, just because these are the data I have handy:

It’s crystal clear that there’s substantial momentum here: When black unemployment has been rising for a few months, it usually continues to rise; when it has been falling, it usually continues to fall for a few more months.

But let’s see what happens if we perform Nate Silver’s test, analyzing successive monthly changes in this black unemployment rate?

Instead of finding positive momentum, we find that this month’s change is negatively correlated with next month’s change in unemployment! This would lead Nate’s approach to (wrongly) conclude there’s no momentum.? The reason is that the black unemployment rate is measured with error, and by construction, these errors cause this month’s change and last month’s change to move in opposite directions.

There’s a simple solution: Analyze changes where you don’t use the same measure – this month’s unemployment rate – in constructing both your dependent and independent variables.? I try this alternative test in the next graph, showing the change in the black unemployment rate between next month and this month versus the change between last month and the previous month:

And now this analysis shows what was obvious from the first graph: Yes, there is momentum in the black unemployment rate.

Is this what is going on with Nate’s analysis of polling data?? I don’t know for sure, because I don’t have his database to test this idea on.? But I’m willing to bet it is.? Nate: Here’s my proposal.? Re-run your analysis, but instead of analyzing the relationship between changes over periods A and B as a function of changes between periods B and C, analyze them as a function of changes between periods C and D.? I’m confident that you’ll find less evidence of negative correlation; you may even find evidence of a positive correlation.

In fact, let’s bet a fancy dinner on the outcome – I reckon you’ll find that non-overlapping changes in polling are in fact positively related. That is, there probably is momentum in political races.

Do we have a bet?

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 15

View All Comments »
  1. Fly on the wall. says:

    Why don’t you be quite specific as to which races we can watch to see who is buying whom dinner?

    Thumb up 0 Thumb down 0
  2. Ari Caroline says:

    Another possible solution is to use a long-term weighted trend such as that created in exponential smoothing forecasts.

    Granted, if you define momentum as month-to-month, this doesn’t help as much. Still, I think most of us looking at these charts have a longer term view of momentum.

    Thumb up 0 Thumb down 0
  3. jim says:

    If its a question of noisy data, wouldn’t a smoothing method be more appropriate?

    Thumb up 0 Thumb down 0
  4. Tristan says:

    When I read Nate’s post I didn’t take away that his conclusion was that there was negative correlation (even though his graph shows a slight negative pattern), but that he said there was no impact from one period to the next.

    I would agree that not using independant variables would cause a slight negative correlation, but it appeared to be at a level that wasn’t significant.

    I believe that the point Nate is trying to make is that when news personalities report on polling data they make a claim for a strong effect of momentum, one that’s been shown not to exist, certainly not in the magnitude that they imply.

    Even looking at the data you use to show an “obvious” case of momentum I struggle to see it in the first graph and remain unconvinced that it’s an important factor after looking at the scatter plot. If I were to report on black unemployment numbers and talk only about the momentum effect that would miss out on the real story, which is most likely much to complex to discover from just looking at these graphs. In the same way that trying to tell the story about popular support for politicians is much too complex a story to tell just based on looking at graphs of polling numbers. In both cases it’s relatively easy to fall in to the trap of assigning too much importance to momentum just because there’s no other background available.

    Thumb up 0 Thumb down 0
  5. Andrew says:

    A few points:
    1. The point of the piece was not to show that there was a negative correlation for momentum between two periods. Rather, the point was that the correclation was insignificant.

    2. Nate does not compare strictly month-to-month. Period A (right before the election) is only 10 days long, whereas Perid D (two months out+) is 90 days long.

    3. His data points are not based on single polls, but rather the average of multiple polls. He “require[d] that at least 2 polls were conducted in each period for the race to be included.”

    Also note that the relevant timescales for elections (a few months) and the black unemployment rate (a few years) are very different.

    Thumb up 0 Thumb down 0
  6. Mantonat says:

    I think a definition of what you mean by momentum is in order here. Momentum in terms of polls doesn’t just mean a change over time, at least not to me. What I think people mean when they say momentum is that there is change that accelerates as the deadline approaches, change that becomes harder and harder for the opposition to overcome. So I don’t see how the word momentum even applies to the graph of unemployment. There’s no deadline or goal date, there’s no opposition. It’s simply raw numbers. When coming up a with a statistical measure of momemtum, one must also be able to measure the impact of obstacles. After all, doesn’t momentum mean how hard an object is to stop once it gets going? In electoral races, someone may have so much momemtum that only the worst news about the candidate will change the polls. Others may have just a little momentum, so that the opposition can use strategy to slow popularity. With unemployment statistics, momentum can’t really bowl over mitigating factors. Unemployment increases or decreases because of external factors, not in spite of them.

    Thumb up 0 Thumb down 0
  7. Dr J says:

    The correlations on both those plots look terrible – as a physicist I wouldn’t claim any particular significance or usefulness in either of them – you claim 12%(!) of the month to month variation can be attributed to momentum – that is very little – clearly, month to month variations in unemployment are dominated by noise, hence to claim momentum based on month to month variations (as our news media often does) is specious. I think Nate has a point – even though there is an underlying signal (obviously) you can not tell much from the month to month variations – maybe you should do a fourier analysis and look at the power spectrum for various frequencies to get a feeling for the time scale that the large scale variations are taking place – or do your correlation analysis over various time scales if you don’t understand fourier transforms, like averaging over a several month’s worth of data… so far the depth of your analysis does not impress me.

    Thumb up 0 Thumb down 0
  8. Peter R says:

    Another area where people obviously search for “momentum” is the stock market. Isn’t there some method we can borrow from them to do this a bit more rigorously?

    Thumb up 0 Thumb down 0