How Do We Know What Really Works in Healthcare? (Ep. 201)

Listen now:
(Photo: Dedric Cherry)

(Photo: Dedric Cherry)

Our latest Freakonomics Radio episode is called “How Do We Know What Really Works in Healthcare?” (You can subscribe to the podcast at iTunes or elsewhere, get the RSS feed, or listen via the media player above. You can also read the transcript, which includes credits for the music you’ll hear in the episode.) The gist of the episode: a lot of the conventional wisdom in medicine is nothing more than hunch or wishful thinking. A new breed of data detectives is hoping to change that.

Imagine there’s one elementary school in one district where the kids do much better than all the other nearby schools. This also happens to be the only school that serves its kids breakfast every day (in addition to lunch). It’d be tempting to conclude that the school’s good grades are due to the breakfast — and that if you simply started serving breakfast at all the other schools, their grades would also shoot up. But how can you tell for sure? Maybe breakfast is one of 10 things this school does differently — or maybe the kids are different, or the parents, or the teachers, or the curriculum. Maybe this is the only school where dodge ball is played every day at recess. So how do you find out; how do you isolate the effect of the breakfast? You set up an experiment – a randomized controlled trial or RCT, like the ones traditionally used in bench science, in drug studies, and elsewhere. You take one population, randomly divide it into groups, and give some groups a treatment that the others don’t get. Then you can measure whether the treatment group came out any differently than the control group. Here’s what Steve Levitt has to say in the podcast about the use of RCTs in research:

LEVITT: So I think the randomized trial is the very best way to learn about the world around us. And that’s for a couple of reasons. One is because randomization is just your best friend when you’re trying to find causality. Because absent randomization, you always have to tell stories about why what we observe in the world — which are correlations — actually can be mapped into causal relationships. But the beauty of randomization, if done well, and at least in large numbers with large samples, is that because you’ve randomized, on average, you expect the outcomes to be exactly the same for the treatment group and the control group.

As we’ve regularly noted in the past, economists and other academic researchers have increasingly been using RCTs to study all sorts of things, including how to best fight poverty. At the forefront of this movement is J-PAL, or the Abdul Latif Jameel Poverty Action Lab, at MIT. The award-winning economist Esther Duflo, one of J-PAL’s founders, has helped run many RCTs in India, Kenya, and elsewhere, trying to learn how best to prevent teen pregnancy and anemia, and drunk driving; and how to better incentivize nurses, small-business growth, and modern farming techniques.

Amy Finkelstein, MIT economist, advocates the use of randomized controlled trials in healthcare delivery.

In this episode, we turn our attention to the U.S. and J-PAL’s efforts to learn about what really works in healthcare delivery. We focus on research done by the MIT economist Amy Finkelstein and several colleagues, whose growing body of work in this realm is fascinating. As Finkelstein tells us in the podcast, RCTs are far too rare in healthcare delivery — which is a shame, for the link between healthcare and poverty is strong:

FINKELSTEIN: We take a rather broad view of poverty alleviation. And so anything that improves the efficiency of healthcare delivery, I think is important for the public for two reasons. First, you know, the poor are disproportionately unhealthy and therefore have the burden of healthcare relative to less poor people. Also, given that healthcare spending is currently about a fifth of public-sector budgets at the state and federal level, anything one can do to improve the efficiency of healthcare delivery frees up more money to spend on other programs as well. Or to spend on, you know, getting even better health.

You will hear about Finkelstein’s research on a Medicaid expansion plan in Oregon. While there was no RCT attached to this project, Oregon did use a lottery to determine who would and wouldn’t receive healthcare coverage, so the effect was essentially the same. Finkelstein and her colleagues looked into how this new supply of healthcare coverage affected clinical outcomes, emergency-room use, and employment. (Perhaps not surprisingly, liberals and conservatives all leaped at the chance to cherry-pick and spin these findings. Here, for instance, are left and right views of the findings on clinical outcomes; and left and right views of the finding that Medicaid led to a rise, not the suspected fall, in ER visits.)

(Photo: John D. & Catherine T. MacArthur Foundation)

Jeffrey Brenner, family doctor and healthcare revolutionary, has strong and bracing views on the medical business model. (Photo: John D. & Catherine T. MacArthur Foundation)

We also talk about an RCT that Finkelstein and J-PAL are currently working on with a New Jersey group called the Camden Coalition of Healthcare Providers. Its focus is low-income “super utilizers,” the kind of patients who might show up in ERs dozens or even hundreds of times a year. The mission is to help them get better treatment while also cutting through some of the grotesquely inflated costs that come with modern healthcare. The Coalition’s executive director and founder is Jeffrey Brenner, a family doctor, healthcare maverick, and MacArthur “genius” who has strong and bracing views on the medical business model:

BRENNER: So we learned that 1 percent of the patients is 30 percent of the payments to the hospitals, and that 5 percent of the patients is about 50 percent of the payments to the hospital. So a very small sliver of patients are driving all of the revenues to the system. … And you know, the question really is this the fault of the patients or is this a system failure? And I think our journey over the last couple of years has really demonstrated to use that it’s a system failure and that we could be doing much, much better for these patients.

The conversation with Brenner was so fascinating that we will put out a follow-up episode next week that continues some of the themes he raises here. We’ll look into why Americans are consuming more and more healthcare; whether all that extra care is actually improving outcomes; and what happens when a significant portion of American cardiologists go away at the same time to a medical conference. Do you think there is a huge increase in heart deaths during their absence — or maybe, just maybe, the opposite?


Did anyone notice the irony of how such "world class" experts somehow have not yet figured how to conclude their experiments within the budgeted grant period? When asked to actually substantiate their work (i.e., the results) beyond intellectual curiosity, the one speaker's response was that he likely needed more time and grant money to find out. I chuckled. That sounds just like big government, not problem solving.

John Steward

Did I hear an F-bomb on this pod cast? Think so... Careful guys.

Anyway, another reason to go to the emergency room is that you get to see a doctor NOW. "Now" might be 4 hours after you walk through the door, but it's still only 4 hours. When you have a busy schedule and poor peoples schedules can be extremely busy, getting medical help NOW makes a difference. As urgent care becomes more prevalent, I suspect the emergency room visits will THEN drop.
Great show! Can't wait for part 2.

Mitchell A

I work in healthcare and would love to know the name of the company that this refers to:

"Two of them dropped out because it was so hard to do. And there was only one still left standing in Doylestown, Pennsylvania."

It would be interesting to actually see a workflow/ description of how they are operating.

Does anyone have that company's name?


Kieron George

I know everyone is going to love me for this and I'm going to end up on the NSA fan list, but let's imagine a hypothetical "experiment" where;

9/11 happens on the final day of the Cardiologists' conference, and

The 10 percent point reduction in mortality held for the following 3 days that planes were grounded.

Would there have been a Net gain in American lives?

3000 people died in the 911 terrorist attacks.

6500 people died from heart disease on the 3 days that followed in the US.

If those deaths result from a mortality reduced from 25% to 15%, only 4000 people would've died, halving the net lives lost.

Obviously that's a ridiculous scenario. The mortality reduction wouldn't nearly be that smooth or universal, and it ignores the long-term & tertiary deaths caused by 9/11 and the cultural, political, and social harm caused by 9/11.

But I think the biggest point I want to make here is that scientist shouldn't need to wait for a "convenient" terrorist attack to get good medical data. There should be a system in place that allows for large scale medical research.


Bill McA

Two possibilities to consider.
1. The doctors that stay behind are hand picked by the hospitals to be able to manage the workload. Test by doing a similar study looking at the results of the subset that stayed behind at other times when the cohort that goes to the conference are all around.

2. More experienced doctors may be more aggressively pushing higher risk procedures in cases where it would be better to wait and see. Unfortunately doctors are likely to get into more trouble by doing nothing than by doing something. Older doctors may be more sensitive to this special kind of pressure. Much more difficult to test for this...

Henry Wei

Randomized controlled trials are random, well, sort of. There's a *non*-random element to them which is quite problematic, which is that, in the realm of drug trials for example, we might not know if the type of individual who chooses to participate in a clinical trial looks like the real types of patients out in the real world.

Why does this matter? Well, one of the trickiest areas to recruit patients for trials is in clinical studies of medication adherence -- where the intervention isn't the drug itself, but rather what we can do to help patients reliably take the drug. And the issue is that in clinical trials, a lot is done to ensure that the patient will be taking the drug at least 80% of the time (a B-minus grade, by the way), because we can't just study patients who aren't taking the drugs after all. In the real world, though, let's say we were recruiting for patients to do this study -- wait a second, don't we actually want the people who'd ignore the study? Who don't care about taking their medication? And then on top of this, does it distort the results if the clinical trial is paid, because after all, we generally don't pay people for taking medications in real life (quite the opposite actually).

So from a biochemical standpoint, clinical trials make sense to test whether, in a human animal, this compound produces some effect over time relative to a set of well-matched control comparators. But from a real-world standpoint, getting that study population -- and figuring out if they look like real world decision makers -- is a whole other bag of hurt.

More recently, in Sweden, the SWEDEHEART registry (get it?) was a cardiology-focused effort that may prove to be a fascinating answer. In this case, *every* patient in the country was added to a database and then randomization and enrollment could happen much faster and more easily... although the details of informed consent are still tricky and a manual but necessary step. Experts like Dr. Lauer at the NHLBI commented that this may dramatically bring down the cost of clinical trials by an order of magnitude, by simplifying recruitment and accelerating it as well.



Fuck was the perfect word to vent his frustration.

Oliver H

That would be the same Amy Finkelstein who tried to pass off a change in cigarette consumption below year-to-year variance and a clinically nonsignificant change in body weight gain during pregnancy as the consequence of changes in Medicare? Forgive me if I'm skeptical about any data this lady produces. In sciences, she likely would not hold her position any longer.

jim lampi

I was disappointed that when the idea of a free market alternative to the current system was brought up, the doctor stated that it failed for him when he provided care under a Medicaid system. Medicaid is pretty much the antithesis of the free market. His clinic failing due to low Medicaid reimbursements says nothing about the free market.

Kathleen Sheridan, MD

I am an obstetrician, not a cardiologist. When you discussed the study about improved outcomes during cardiology meetings, you theorized that more experienced cardiologists attend the meetings. I would suppose the opposite-- perhaps cardiology fellows are more likely to attend the conference and patients are getting more experienced physicians during the conference. Cardiology fellows are presenting their research, networking for faculty positions, etc. at those meetings.

Matt B.

Great show! It gave me an idea for a randomized trial. Take 100 of the people who go to urgent care/emergency room for a head cold. For 50% of them don't change anything and for the other 50%, have the doctor tell them that there's nothing he can do, it just has to run it's course, they can buy something over the counter to alleviate the symptoms. Then the doctor can explain all the symptoms in detail and tell them not to come back for head colds. Then you compare the repeat customers between the two groups.

In my experience, the doctor always writes a prescription, no matter what. This makes the patient feel like they made the right choice to go in because they couldn't have gotten the medication otherwise. The doctors are the problem. Or at least a big part of it.

Jon H

Love the podcast. I am listener for couple years.

Just give me a heads up for F-bomb. Something in introduction of on the description That strong language is part of episode.

I don't want to here it and would gladly skip section if needed to not here it and I can focus on the rest of the episode(s) that do not have it.

Thanks for producing great and very interesting shows.


PLEASE put an explicit tag in iTunes when inappropriate language is used. I listen in the car and often have a three year old with me. I enjoy your podcast and I do not mine poor language used in good context BUT I prefer my daughter not hear it. Just a warning would be greatly appreciated. Thank you


This and the next related podcast were two of the most interesting and potentially important that Freakonomics has produced. I hope policy makers and those in the healthcare field are listening. Many thanks.