Search the Site

Posts Tagged ‘Data Analysis’

Are Predictions Getting Better?

If you’re the kind of person who cares about “The Folly of Prediction” and The Signal and the Noise, you may want to read Amy Zegart‘s Foreign Policy piece about predictions. Making predictions within the intelligence community, for example, is a different game than betting on basketball:

In March Madness, everyone has access to the same information, at least theoretically. Expertise depends mostly on how geeky you choose to be, and how much time you spend watching ESPN and digging up past stats. In intelligence, however, information is tightly compartmented by classification restrictions, leaving analysts with different pieces of data and serious barriers to sharing it. Imagine scattering NCAA bracket information across 1,000 people, many of whom do not know each other, some of whom have no idea what a bracket is or the value of the information they possess. They’re all told if they share anything with the wrong person, they could be disciplined, fired, even prosecuted. But somehow they have to collectively pick the winner to succeed.

In other spheres, however, predictions just keep getting better. “Smart people are finding clever new ways of generating better data, identifying and unpacking biases, and sharing information unimaginable 20 or even 10 years ago,” writes Zegart.

FiveThirtyEighter Nate Silver Answers Your Questions About Politics, Baseball, and The Signal and the Noise

We recently solicited your questions for Nate Silver regarding his new book The Signal and the Noise: Why So Many Predictions Fail — But Some Don’t. Not too surprisingly, a lot of the questions were about politics and baseball. Below are Nate’s answers to some of them. Thanks to him for playing along and to all of you (as always) for sending in the excellent questions.

Q. Under what circumstances will a voter actually change his/her mind about whom to vote for? I understand that this rarely happens (this study for example), and that most of the action involves undecided voters deciding whom to vote for.

Also, if political scientist are right that voters rarely change their minds, how can a large swing in the polls ever occur? A classic example that your briefly mention in your book is that of Michael Dukakis, who was ahead of GHW Bush by 10% at one point in 1988. –Alan T

A. We see more big shifts in the primaries, when voters don’t have that much information about the candidates. Dukakis was a relative unknown at the start of the 1988 race, before the two parties could advance their own narratives. You rarely see big swings in voter conversion in late stage presidential races, though. If I knew how to cause such a swing, I’d be drawing a big salary from one of the campaigns right now.

The Economics Revolution Will Be Televised

There’s a revolution underway in economics. It’s not due to the financial crisis, but rather something more mundane: Data, and computing power. At least that’s the claim that Betsey Stevenson and I make in our latest Bloomberg View column:

“Consider the stream of data you will create today. Your metro card will record what time you caught the train. Your Web browser will note how you go about your job, and how much you procrastinate. A mid-afternoon purchase at Starbucks will reveal your penchant for lattes and the occasional cookie. Your flow of e-mail traffic will trace out your professional and personal networks.

At the same time, computing power has made it extremely easy and cheap to analyze all the data you produce. An economist with a laptop can, in a matter of seconds, do the kind of number crunching it used to take a roomful of Ph.D.’s weeks to achieve. Just a few decades ago, economists used punch cards to program data analysis for their empirical studies.”

Two weeks ago, Harvard’s Raj Chetty gave a spectacular talk at the National Bureau of Economic Research, about what he called “The Transformative Potential of Administrative Data.” He documented that today’s cutting-edge research is based on crunching newly-available data from the vast databases which underlay our schools, welfare state and tax systems.  I’m just as optimistic that new data coming online from the private sector will prove to be just as useful.

The Life of the Number-Crunching Analyst

Thousands of economics majors head off to industry each year to work as analysts. They’re lured by the promise that they’ll learn a lot, work hard, play hard and get ahead.  But is it true?  Who better to ask than the brilliant young analyst Elisabeth Fosslien.  And as a good young analyst, she’s distilled her portrait of life as an analyst into charts.  Having once lived the analyst life—my first job out of college was at the Reserve Bank of Australia, crunching numbers and making charts—all of these resonated with me.

Is Higher Income Inequality Associated with Lower Intergenerational Mobility?

A lot of our political debate boils down to questions about equality of outcomes versus equality of opportunity. But it turns out that they’re pretty closely related. Take a look at the chart below, which is from a terrific recent speech (with charts!) by Alan Krueger:

The horizontal axis shows the Gini coefficient, which is a summary of the degree of income inequality for each country. I think of this as a measure of inequality of outcomes. The United States sits out there on the right, which says that we have high inequality, which I bet that doesn’t surprise you.

The Economic Value of (Very) Personal Data

Graphic designer Nicholas Felton keeps track of how many miles he walks each day. He also records how many book pages he reads, how many work e-mails he sends, and what songs he listens to. Felton’s become somewhat famous for his obsessive self-tracking, and the slick info-graphics he produces on himself each year. Both the Wall Street Journal and Slate have made videos about him, here and here.
Felton began tracking his daily habits and compiling a Personal Annual Report in 2005, available at his website.

How Does Your Kid's School Rank?

Katherine Mangu-Ward at Reason writes that interested parties can now find out how their school stacks up against other schools using the Education Nation Scorecard.

Better Golfing Through Data

The game of golf has in many ways retained its down-to-earth origins. So what happens when a gaggle of statisticians and mathematical theorists bearing GPS and laser surveyors descend on the links?

Harnessing Google to Solve Parkinson's

In Wired, Thomas Goetz profiles Sergey Brin’s search for a cure for Parkinson’s disease: “Brin proposes a different approach, one driven by computational muscle and staggeringly large data sets. It’s a method that draws on his algorithmic sensibility-and Google’s storied faith in computing power-with the aim of accelerating the pace and increasing the potential of scientific research.”

ClimateGate as Rorschach Test

In the 10 days since we first blogged about “ClimateGate” – the unauthorized release of e-mails and other material from the Climate Research Unit (C.R.U.) at East Anglia University in Norwich, England – it’s become strikingly clear that one’s view of the issue is deeply colored by his or her incoming biases. No surprise there, but still, the demarcation is clear. One of the best indicators: when you stumble onto a blog post about the topic, you can tell which way the wind is blowing simply by looking at the banner ad at the top of the site: if it’s for an M.B.A. in Sustainable Business, you’re going to hear one thing about ClimateGate; if the ad shows Al Gore with a Pinocchio nose, meanwhile – well, you get the idea.

On the Prevalance of H1N1

In Seattle recently, I met a pulmonologist who said that the H1N1 virus has him busier than he’s ever been, his hospital beds full of flu patients. The uptick hit particularly hard about 10 days ago, he said.
How has the flu been playing out across the country?

Multidecadal Fantasy Baseball

Barry Bonds, Todd Helton, and Mickey Mantle are the top three batters in baseball history … well, according to a new study that used network science to rank players by analyzing the outcome of every at-bat from 1954 to 2008.

What Does the Human Development Index Measure?

There’s been some interesting recent commentary on the Human Development Index. But first, some background. This index is calculated each year by the U.N. Development Program as a summary indicator of “Human Development,” combining data on life expectancy at birth, adult literacy, educational enrollment, and average income (measured as G.D.P. per capita). And earlier this week, Catherine Rampell noted a recent effort by the SSRC-funded American Human Development Project to develop a Human Development Index, for U.S. states. Philosophically, it is an attempt to broaden the development debate beyond G.D.P. But does it succeed?

I Fell for Their Data

I fell for a stupid article and turned off my home PC last night. The article says that Americans who leave computers on overnight are wasting $2.8 billion on energy costs per year. It ignores the cost of turning computers off — and having to turn them on again the next morning. Let’s say that process takes five minutes per . . .

The Downside of Google's Data Obsession

| He didn’t announce it via cake, but Doug Bowman quit his job as head of Google’s visual design team last week, citing the company’s “reliance on data” for design decisions as the main reason for his departure. Bowman writes on his blog that he’ll miss Google’s “incredibly smart and talented people” and the “occasional massage,” but not “a design . . .

When Winning Leads to Winning: A Response

Here are two interesting follow-ups to Tuesday’s post, in which I described how basketball teams who are behind at half-time fare a bit better than might be expected. First, my friend Lionel Page points me to a related study of his, which analyzes tennis. Lionel uses a similar approach to arrive at a different conclusion, but I think his results . . .

Our Daily Bleg: What to Do About Too Much Data?

A reader named Evan Schumacher wrote in with an interesting bleg. (Read about blegs here and send your own here.) Tucked inside his bleg is the part that tickled me the most: a website Evan created to tell him whether it’s worth it to watch a basketball game he’d recorded. Anyway, I’ll give my answer below, after his bleg. I . . .

Amazing New Trade Data

Wow. We really do live in the midst of a tidal wave of more detailed and interesting data. The latest:, the brainchild of brothers Ryan and David Petersen, with Michael Kanko. They exploit customs reporting obligations and Freedom of Information requests to organize and publish — in real-time — the contents of every shipping container entering the United States. . . .

Love Data? Zillow Wants You

We’ve blogged a few times about Zillow (here and here), a website that is trying to shake up the real estate industry. I’ve made radical predictions about the future of the real estate industry. I’m hoping that Zillow will help make those prophesies come true. So to do my part (and because I am as susceptible to flattery as the . . .

Gapminder Is Cool

When I first stumbled onto the name voyager at the Baby Name Wizard a few years back, I felt like I was seeing the future. It was the sort of web tool that folks dream about. I had exactly the same feeling when I recently visited They have an interactive data tool called “gapminder world” that is truly remarkable . . .

Freakonomics in The Times Magazine: Hoop Data Dreams

Levitt and I have a column in this Sunday’s Times Magazine about the attempt to bring to the sport of basketball the intense statistical analysis that Bill James has made popular throughout baseball. The column centers on the Boston Celtics, who have just completed the best-ever turnaround in N.B.A. history, winning 66 games this year after winning just 24 last . . .

More Analysis of the Environmental Impact of Walking vs. Driving

Last month I blogged about Chris Goodall‘s claim that walking could exacerbate global warming more than driving if the person doing the walking gets his or her calories from foods like beef or milk. A group called the Pacific Institute has done some further analysis of the data. Their analysis suggests that for most reasonable assumptions about the diet of . . .

160,000 Four-Leaf Clovers?

This doesn’t really seem possible, but Edward Martin has found 160,000 four-leaf clovers. I’ve been looking my whole life and never found one. Trying to find one was my main reason for playing Pee Wee Baseball, but then I got moved from outfield to shortstop and my baseball career ended shortly thereafter. How fast does Martin find them? He is . . .

Misreporting on Divorce

Today is apparently D-Day here at Freakonomics — the “D” stands for divorce. Along with Hamermesh’s earlier post and this post by Wolfers, there’s one more on the way. One of the most frustrating things about doing research on families is seeing how often even the simple facts are misreported in the press. And Sue Shellenbarger, writing in this morning’s . . .

The Racial Tipping Point

A few years back, I got interested in taxicab tipping – and what influences how much people tip. So together with Fred Vars and Nasser Zakariya, I collected data on more than 1,000 cab rides in New Haven, CT and crunched the numbers. The study (published in The Yale Law Journal) found — after controlling for a host of other . . .