How Do We Know What Really Works in Healthcare? (Ep. 201)

Listen now:
(Photo: Dedric Cherry)

(Photo: Dedric Cherry)

Our latest Freakonomics Radio episode is called “How Do We Know What Really Works in Healthcare?” (You can subscribe to the podcast at iTunes or elsewhere, get the RSS feed, or listen via the media player above. You can also read the transcript, which includes credits for the music you’ll hear in the episode.) The gist of the episode: a lot of the conventional wisdom in medicine is nothing more than hunch or wishful thinking. A new breed of data detectives is hoping to change that.

Imagine there’s one elementary school in one district where the kids do much better than all the other nearby schools. This also happens to be the only school that serves its kids breakfast every day (in addition to lunch). It’d be tempting to conclude that the school’s good grades are due to the breakfast — and that if you simply started serving breakfast at all the other schools, their grades would also shoot up. But how can you tell for sure? Maybe breakfast is one of 10 things this school does differently — or maybe the kids are different, or the parents, or the teachers, or the curriculum. Maybe this is the only school where dodge ball is played every day at recess. So how do you find out; how do you isolate the effect of the breakfast? You set up an experiment – a randomized controlled trial or RCT, like the ones traditionally used in bench science, in drug studies, and elsewhere. You take one population, randomly divide it into groups, and give some groups a treatment that the others don’t get. Then you can measure whether the treatment group came out any differently than the control group. Here’s what Steve Levitt has to say in the podcast about the use of RCTs in research:

LEVITT: So I think the randomized trial is the very best way to learn about the world around us. And that’s for a couple of reasons. One is because randomization is just your best friend when you’re trying to find causality. Because absent randomization, you always have to tell stories about why what we observe in the world — which are correlations — actually can be mapped into causal relationships. But the beauty of randomization, if done well, and at least in large numbers with large samples, is that because you’ve randomized, on average, you expect the outcomes to be exactly the same for the treatment group and the control group.

As we’ve regularly noted in the past, economists and other academic researchers have increasingly been using RCTs to study all sorts of things, including how to best fight poverty. At the forefront of this movement is J-PAL, or the Abdul Latif Jameel Poverty Action Lab, at MIT. The award-winning economist Esther Duflo, one of J-PAL’s founders, has helped run many RCTs in India, Kenya, and elsewhere, trying to learn how best to prevent teen pregnancy and anemia, and drunk driving; and how to better incentivize nurses, small-business growth, and modern farming techniques.

Amy Finkelstein, MIT economist, advocates the use of randomized controlled trials in healthcare delivery.

In this episode, we turn our attention to the U.S. and J-PAL’s efforts to learn about what really works in healthcare delivery. We focus on research done by the MIT economist Amy Finkelstein and several colleagues, whose growing body of work in this realm is fascinating. As Finkelstein tells us in the podcast, RCTs are far too rare in healthcare delivery — which is a shame, for the link between healthcare and poverty is strong:

FINKELSTEIN: We take a rather broad view of poverty alleviation. And so anything that improves the efficiency of healthcare delivery, I think is important for the public for two reasons. First, you know, the poor are disproportionately unhealthy and therefore have the burden of healthcare relative to less poor people. Also, given that healthcare spending is currently about a fifth of public-sector budgets at the state and federal level, anything one can do to improve the efficiency of healthcare delivery frees up more money to spend on other programs as well. Or to spend on, you know, getting even better health.

You will hear about Finkelstein’s research on a Medicaid expansion plan in Oregon. While there was no RCT attached to this project, Oregon did use a lottery to determine who would and wouldn’t receive healthcare coverage, so the effect was essentially the same. Finkelstein and her colleagues looked into how this new supply of healthcare coverage affected clinical outcomes, emergency-room use, and employment. (Perhaps not surprisingly, liberals and conservatives all leaped at the chance to cherry-pick and spin these findings. Here, for instance, are left and right views of the findings on clinical outcomes; and left and right views of the finding that Medicaid led to a rise, not the suspected fall, in ER visits.)

(Photo: John D. & Catherine T. MacArthur Foundation)

Jeffrey Brenner, family doctor and healthcare revolutionary, has strong and bracing views on the medical business model. (Photo: John D. & Catherine T. MacArthur Foundation)

We also talk about an RCT that Finkelstein and J-PAL are currently working on with a New Jersey group called the Camden Coalition of Healthcare Providers. Its focus is low-income “super utilizers,” the kind of patients who might show up in ERs dozens or even hundreds of times a year. The mission is to help them get better treatment while also cutting through some of the grotesquely inflated costs that come with modern healthcare. The Coalition’s executive director and founder is Jeffrey Brenner, a family doctor, healthcare maverick, and MacArthur “genius” who has strong and bracing views on the medical business model:

BRENNER: So we learned that 1 percent of the patients is 30 percent of the payments to the hospitals, and that 5 percent of the patients is about 50 percent of the payments to the hospital. So a very small sliver of patients are driving all of the revenues to the system. … And you know, the question really is this the fault of the patients or is this a system failure? And I think our journey over the last couple of years has really demonstrated to use that it’s a system failure and that we could be doing much, much better for these patients.

The conversation with Brenner was so fascinating that we will put out a follow-up episode next week that continues some of the themes he raises here. We’ll look into why Americans are consuming more and more healthcare; whether all that extra care is actually improving outcomes; and what happens when a significant portion of American cardiologists go away at the same time to a medical conference. Do you think there is a huge increase in heart deaths during their absence — or maybe, just maybe, the opposite?


Ummm... I live in Eastern Europe and health economics and evidence informed policy making is starting to gain awareness even here. I find it hard to imagine that these concepts and disciplines would be so unheard-of in the USA.

Alex Jennings

To answer your question, I don't believe that there is a huge increase in heart deaths, while they're gone. It's a conference, not a long, sustained vacation that they're all going on. Unfortunately, I cannot go see a cardiologist because the costs are too high, and I don't have very good health insurance.


I found the 4/2 podcast on health care very interesting and made me continue to think about the 'unintended consequences' of policies or actions that we make.
During that podcast you used data that showed an increase in emergency care after formally uninsured people had access to health care insurance. Has any follow-up studies shown that the increase has continued or was that just a 'lagging indicator' that corrected once people became adjusted to their new status.
In other words; Over time have people found medical care in the normal clinical setting?

Ronald Koretz, M.D.

Dr. Brenner described a large Medicare-sponsored experiment about health care delivery in which a number of different agencies participated. Over time, all but one were eliminated for various reasons (mostly because expense was increased), but one program appeared to show good results. With regard to study design, this appears to be equivalent to assessing multiple outcomes; in such situations "significance" become less reliable because the investigator is taking multiple bites out of the apple to find one with a low p value (e.g., < 0.05). The probability of finding a difference with an associated p value < 0.05 is not less than 1 in 20 if multiple analyses are done. How can we be sure that the single successful program was not just a statistical fluke (non-reproducible outlier)? At the very least, the exact methodology that that one organization was employing has to be validated with a second experiment.



Great podcast. So great to use data to pinpoint and solve healthcare problems. Also it is not surprising that politicians try to support their argument or that poor people use the emergency room more with insurance. There's a shortage of doctors in poor areas and a lot of doctors won't take Medicare. Also it may take a long time to get an appointment. It is much easier to go to the emergency room. It would not take much to change this behavior. For example charge a penalty for non emergencies and give patients directions to a nearby walk in clinic.

Jill Herendeen

Or, raise Medicare/Medicaid payments, and/or pay doctor extra to work in poor areas, and/or make it cheaper to be a doctor (free higher education, free med school, socialized malpractice insurance should do the trick).


I found it odd (and interesting and telling) that Prof. Levitt implicitly pooh-poohed stories early in the show (something to the effect that without RCT, "all you have are stories"), only to find that later in the show the success of the health care initiatives seemed so linked to people being considered as people rather than numbers. In other words, when people have a (hi)story, their lives have greater meaning. Maybe stories aren't so bad.

Jill Herendeen

You almost got it. The reason there are so few RCTs about healthcare delivery is that nobody's making money off of delivery; and the whole point of healthcare in America--which nobody wants to admit-- is PROFIT. Americans have no right to health care; they have the right to buy it IF they have the money, or to go bankrupt. Medicare/Medicaid are regressively-funded programs to provide payments to HOSPITALS and other facets of the for-profit medical-industrial complex. Making people healthier, using less hospitalization and fewer drugs, would cut into their profits. By contrast, countries which provide universal health care--especially those which fund this care progressively--DO research what makes people healthier and implement it, possibly only because their millionaires don't want to be paying more taxes than necessary or because their gov'ts cap their profits from their medical industries, or both.



The question that is not answered with respect to the Oregon Medicaid experience is how many of the emergency room visits were either not necessary or could have been prevented had people utilized preventive, or early illness intervention, medical care. The rationale driving the expectation that there would be fewer emergency room visits was that better preventive primary care, or primary care early in the course of an illness, would reduce the need to visit an emergency room; but there has not been sufficient analysis of the data, nor longer term review of the data, possibly over a couple of years, to determine whether this rationale is valid.