Archives for data analysis

Are Predictions Getting Better?

If you’re the kind of person who cares about “The Folly of Prediction” and The Signal and the Noise, you may want to read Amy Zegart‘s Foreign Policy piece about predictions. Making predictions within the intelligence community, for example, is a different game than betting on basketball:

In March Madness, everyone has access to the same information, at least theoretically. Expertise depends mostly on how geeky you choose to be, and how much time you spend watching ESPN and digging up past stats. In intelligence, however, information is tightly compartmented by classification restrictions, leaving analysts with different pieces of data and serious barriers to sharing it. Imagine scattering NCAA bracket information across 1,000 people, many of whom do not know each other, some of whom have no idea what a bracket is or the value of the information they possess. They’re all told if they share anything with the wrong person, they could be disciplined, fired, even prosecuted. But somehow they have to collectively pick the winner to succeed.

In other spheres, however, predictions just keep getting better. “Smart people are finding clever new ways of generating better data, identifying and unpacking biases, and sharing information unimaginable 20 or even 10 years ago,” writes Zegart.

FiveThirtyEighter Nate Silver Answers Your Questions About Politics, Baseball, and The Signal and the Noise

We recently solicited your questions for Nate Silver regarding his new book The Signal and the Noise: Why So Many Predictions Fail — But Some Don’t. Not too surprisingly, a lot of the questions were about politics and baseball. Below are Nate’s answers to some of them. Thanks to him for playing along and to all of you (as always) for sending in the excellent questions.

Q. Under what circumstances will a voter actually change his/her mind about whom to vote for? I understand that this rarely happens (this study for example), and that most of the action involves undecided voters deciding whom to vote for.

Also, if political scientist are right that voters rarely change their minds, how can a large swing in the polls ever occur? A classic example that your briefly mention in your book is that of Michael Dukakis, who was ahead of GHW Bush by 10% at one point in 1988. -Alan T

A. We see more big shifts in the primaries, when voters don’t have that much information about the candidates. Dukakis was a relative unknown at the start of the 1988 race, before the two parties could advance their own narratives. You rarely see big swings in voter conversion in late stage presidential races, though. If I knew how to cause such a swing, I’d be drawing a big salary from one of the campaigns right now. Read More »

The Economics Revolution Will Be Televised

There’s a revolution underway in economics. It’s not due to the financial crisis, but rather something more mundane: Data, and computing power. At least that’s the claim that Betsey Stevenson and I make in our latest Bloomberg View column:

“Consider the stream of data you will create today. Your metro card will record what time you caught the train. Your Web browser will note how you go about your job, and how much you procrastinate. A mid-afternoon purchase at Starbucks will reveal your penchant for lattes and the occasional cookie. Your flow of e-mail traffic will trace out your professional and personal networks.

At the same time, computing power has made it extremely easy and cheap to analyze all the data you produce. An economist with a laptop can, in a matter of seconds, do the kind of number crunching it used to take a roomful of Ph.D.’s weeks to achieve. Just a few decades ago, economists used punch cards to program data analysis for their empirical studies.”

Two weeks ago, Harvard’s Raj Chetty gave a spectacular talk at the National Bureau of Economic Research, about what he called “The Transformative Potential of Administrative Data.” He documented that today’s cutting-edge research is based on crunching newly-available data from the vast databases which underlay our schools, welfare state and tax systems.  I’m just as optimistic that new data coming online from the private sector will prove to be just as useful. Read More »

The Life of the Number-Crunching Analyst

Thousands of economics majors head off to industry each year to work as analysts. They’re lured by the promise that they’ll learn a lot, work hard, play hard and get ahead.  But is it true?  Who better to ask than the brilliant young analyst Elisabeth Fosslien.  And as a good young analyst, she’s distilled her portrait of life as an analyst into charts.  Having once lived the analyst life—my first job out of college was at the Reserve Bank of Australia, crunching numbers and making charts—all of these resonated with me. Read More »

Is Higher Income Inequality Associated with Lower Intergenerational Mobility?

A lot of our political debate boils down to questions about equality of outcomes versus equality of opportunity. But it turns out that they’re pretty closely related. Take a look at the chart below, which is from a terrific recent speech (with charts!) by Alan Krueger:

The horizontal axis shows the Gini coefficient, which is a summary of the degree of income inequality for each country. I think of this as a measure of inequality of outcomes. The United States sits out there on the right, which says that we have high inequality, which I bet that doesn’t surprise you. Read More »

The Economic Value of (Very) Personal Data

Graphic designer Nicholas Felton keeps track of how many miles he walks each day. He also records how many book pages he reads, how many work e-mails he sends, and what songs he listens to. Felton’s become somewhat famous for his obsessive self-tracking, and the slick info-graphics he produces on himself each year. Both the Wall Street Journal and Slate have made videos about him, here and here.

Felton began tracking his daily habits and compiling a Personal Annual Report in 2005, available at his website. Read More »

Visualizing Mortality History

Hans Rosling, a guru of data animation, is at it again. Read More »

How Does Your Kid's School Rank?

Katherine Mangu-Ward at Reason writes that interested parties can now find out how their school stacks up against other schools using the Education Nation Scorecard. Read More »