How Do We Know What Really Works in Healthcare? (Ep. 201)

Listen now:
(Photo: Dedric Cherry)

(Photo: Dedric Cherry)

Our latest Freakonomics Radio episode is called “How Do We Know What Really Works in Healthcare?” (You can subscribe to the podcast at iTunes or elsewhere, get the RSS feed, or listen via the media player above. You can also read the transcript, which includes credits for the music you’ll hear in the episode.) The gist of the episode: a lot of the conventional wisdom in medicine is nothing more than hunch or wishful thinking. A new breed of data detectives is hoping to change that.

Imagine there’s one elementary school in one district where the kids do much better than all the other nearby schools. This also happens to be the only school that serves its kids breakfast every day (in addition to lunch). It’d be tempting to conclude that the school’s good grades are due to the breakfast — and that if you simply started serving breakfast at all the other schools, their grades would also shoot up. But how can you tell for sure? Maybe breakfast is one of 10 things this school does differently — or maybe the kids are different, or the parents, or the teachers, or the curriculum. Maybe this is the only school where dodge ball is played every day at recess. So how do you find out; how do you isolate the effect of the breakfast? You set up an experiment – a randomized controlled trial or RCT, like the ones traditionally used in bench science, in drug studies, and elsewhere. You take one population, randomly divide it into groups, and give some groups a treatment that the others don’t get. Then you can measure whether the treatment group came out any differently than the control group. Here’s what Steve Levitt has to say in the podcast about the use of RCTs in research:

LEVITT: So I think the randomized trial is the very best way to learn about the world around us. And that’s for a couple of reasons. One is because randomization is just your best friend when you’re trying to find causality. Because absent randomization, you always have to tell stories about why what we observe in the world — which are correlations — actually can be mapped into causal relationships. But the beauty of randomization, if done well, and at least in large numbers with large samples, is that because you’ve randomized, on average, you expect the outcomes to be exactly the same for the treatment group and the control group.

As we’ve regularly noted in the past, economists and other academic researchers have increasingly been using RCTs to study all sorts of things, including how to best fight poverty. At the forefront of this movement is J-PAL, or the Abdul Latif Jameel Poverty Action Lab, at MIT. The award-winning economist Esther Duflo, one of J-PAL’s founders, has helped run many RCTs in India, Kenya, and elsewhere, trying to learn how best to prevent teen pregnancy and anemia, and drunk driving; and how to better incentivize nurses, small-business growth, and modern farming techniques.

Amy Finkelstein, MIT economist, advocates the use of randomized controlled trials in healthcare delivery.

In this episode, we turn our attention to the U.S. and J-PAL’s efforts to learn about what really works in healthcare delivery. We focus on research done by the MIT economist Amy Finkelstein and several colleagues, whose growing body of work in this realm is fascinating. As Finkelstein tells us in the podcast, RCTs are far too rare in healthcare delivery — which is a shame, for the link between healthcare and poverty is strong:

FINKELSTEIN: We take a rather broad view of poverty alleviation. And so anything that improves the efficiency of healthcare delivery, I think is important for the public for two reasons. First, you know, the poor are disproportionately unhealthy and therefore have the burden of healthcare relative to less poor people. Also, given that healthcare spending is currently about a fifth of public-sector budgets at the state and federal level, anything one can do to improve the efficiency of healthcare delivery frees up more money to spend on other programs as well. Or to spend on, you know, getting even better health.

You will hear about Finkelstein’s research on a Medicaid expansion plan in Oregon. While there was no RCT attached to this project, Oregon did use a lottery to determine who would and wouldn’t receive healthcare coverage, so the effect was essentially the same. Finkelstein and her colleagues looked into how this new supply of healthcare coverage affected clinical outcomes, emergency-room use, and employment. (Perhaps not surprisingly, liberals and conservatives all leaped at the chance to cherry-pick and spin these findings. Here, for instance, are left and right views of the findings on clinical outcomes; and left and right views of the finding that Medicaid led to a rise, not the suspected fall, in ER visits.)

(Photo: John D. & Catherine T. MacArthur Foundation)

Jeffrey Brenner, family doctor and healthcare revolutionary, has strong and bracing views on the medical business model. (Photo: John D. & Catherine T. MacArthur Foundation)

We also talk about an RCT that Finkelstein and J-PAL are currently working on with a New Jersey group called the Camden Coalition of Healthcare Providers. Its focus is low-income “super utilizers,” the kind of patients who might show up in ERs dozens or even hundreds of times a year. The mission is to help them get better treatment while also cutting through some of the grotesquely inflated costs that come with modern healthcare. The Coalition’s executive director and founder is Jeffrey Brenner, a family doctor, healthcare maverick, and MacArthur “genius” who has strong and bracing views on the medical business model:

BRENNER: So we learned that 1 percent of the patients is 30 percent of the payments to the hospitals, and that 5 percent of the patients is about 50 percent of the payments to the hospital. So a very small sliver of patients are driving all of the revenues to the system. … And you know, the question really is this the fault of the patients or is this a system failure? And I think our journey over the last couple of years has really demonstrated to use that it’s a system failure and that we could be doing much, much better for these patients.

The conversation with Brenner was so fascinating that we will put out a follow-up episode next week that continues some of the themes he raises here. We’ll look into why Americans are consuming more and more healthcare; whether all that extra care is actually improving outcomes; and what happens when a significant portion of American cardiologists go away at the same time to a medical conference. Do you think there is a huge increase in heart deaths during their absence — or maybe, just maybe, the opposite?

Howard Brazee

The idea of testing economic theories makes sense.

Currently, progressives are using examples in our federal system to compare similar states with different economic policies (Wisconsin vs Minnesota) to bolster their arguments. It still has the same problems as with the medical comparisons - the states aren't identical. Progressives claim that differences between states with oil and states without oil invalidate comparisons that don't bolster their claims.

Both conservatives and progressives discount real world failures by saying the policies just weren't extreme enough. (although they don't use that word).

Mark Dennis

Hey! What's up with broadcasting Jeffrey Brenner's f-bomb in this week's podcast? Let's keep it classy and professional, Freakonomists. Please?


Agree. You could have easily captured Brenner's conviction/passion without including the f-bomb. Unnecessary and disappointing.

Shani Burke Specht

Guys, the f-word really. Your not that kind of show.

Diego Calvo

You're* so right.


That f-bomb was necessary to showing the frustration of healthcare professionals. I felt Brenner's pain. Bravo


Just listened to this weeks show, definitely a fan. Its so frustrating to witness our government officials push policy that makes more impact on the nightly talking points than it actually does for the average american tax payer. I definitely understand and share Mr. Brenner's frustrations and appreciate his candor and commitment.

Josh G.

I'm a fairly liberal guy and an ACA supporter. This episode rocked my beliefs about the US Healthcare system to my core. Absolutely fascinating, well done!


As a fully credentialed health actuary I loved this podcast. When studying for my fellowship exams the concept of trying to calculate the savings of a particular treatment plan was one that seemed fairly rudimentary for the time we are in. Nowhere in my studies was this random control test even talked about hinted at. I would be very interested in learning more about Brenner's work and results, and maybe even talking to them at some point.

I work for one of the largest health insurance companies in the country and was always amazed at how many disease management companies there are saying how many dollars they are saving the company. It would be good to see something that has scientifically proven that rather than just theoretically.

Andrew Ekstrom

A new area of research in the medical field is "Evidence Based Medicine". Why? A lot of the things doctors believed for years/decades, is WRONG!!!!!!!!!!!!!! You can use Multiple Regression methods or Multiple Regression Methods with Randomized Control Trials to show that a lot of what we "know" in the medical field is garbage.

As a Statistician/Industrial Engineer, I LOVE IT!!! Unfortunately, a lot of doctors and hospitals DON'T want to know that they have been screwing up over the last few decades. This should be a boom time for medical statisticians. Yet, most hospitals only hire "business analysts" to analyze their data. The difference between a business analyst and a statistician is a business analyst is only required to take 3-4 data analysis classes for their master's degrees. A statistician takes 6-10 stats classes for their master's degree. (Why would you want to hire someone that is good at what they do, when you can pay more for someone that doesn't know as much?)


Nicole Ausmus

For a stunning lack of RCTs in an area of healthcare that claims huge ROI, check out corporate wellness program vendors. Some really astounding claims that are largely not questioned.


I have not heard about the finding from the Oregon Healthcare Experiment that the poor with health insurance go to the ER much more. That is very surprising to me.

However, I wonder ... If the poor's main (or perhaps only) way of receiving healthcare has been via the ER, do they simply continue to do what they have always done (that is go to the ER), but since it is now cheaper than just do it more often. In other words, is there an element of education or training that is missing. Perhaps it just takes a long time for people to learn and adapt to a new way of doing things.

I'd be interested in hearing about the rates of ER visits of the uninsured "poor" vs the insured "poor" vs the rest of the insured population.

Lonely Libertarian

Since few if any doctors make house calls these days - the ER is the place to go if you have a problem outside of business hours. The answering machine at my doctor's is very explicit - if I call outside of business hours I should do one of two things...
1. Call 911
2. Go to the ER.

I have also seen some data that suggest that Doctors like having their patients admitted via the ER - it is easier for everyone if the condition being dealt with is headed for a hospital stay.


For some of us, at least, there's also a problem with the whole idea of "your doctor". I'm basically healthy, so my only reason for visiting a doctor regularly is to get my biannual FAA medical exam. (Which has to be done by an FFA-certified doctor, and last I checked there were two in about a 100 mile radius of my home.) So if I had some urgent medical condition, would I spend time trying to locate a doctor, or go to an ER?

Ff this is the case for a fairly prosperous person like me, how much more difficult for a poor person to maintain a relationship with "their doctor".

Lonely Libertarian

Next tine you visit your Doctor ask him to point you to....

1. The evidence that a BMI of 25 is the "right" boundary between good and bad - over is bad - under is good. And the studies that found that 30 was a good point for obese - and 35 was a good point for severely obese [does anyone wonder how these nice round numbers ended up defining our healthiness?]
2. What is the "right" blood pressure for you - and why - are there studies that you might read that support that number.
3. Should you eat more or less salt - and why - and what studies support that conclusion
4. Is drinking two to three drinks a day good for you or bad for you and where can you find the definitive answer to this question.
5. What are trans fats and why are they bad for me.
6. Is dietary cholesterol in any way related to blood cholesterol levels? And what is the evidence to this?

I could go on - but you get the point...



As you probably know, the BMI is a first-cut sorting device, which applies to the population average. Beyond that, you look at things like body fat percentage.

For the rest, you can use PubMed as well as I can.

Andrew Cannon

Dubner, please keep beating the healthcare drum.

As Brenner says, any helpful significant change will require advocacy. And your show is very persuasive.


Interesting programme. I don't think it's just in healthcare where the evidence from well founded studies fails to change policy. What's interesting in fact is that so many professionals and academics still persist in conducting such rigorous research when the evidence is that their studies don't appear to have any effect !

One minor criticism, which i hope will be addressed in the 2nd episode. An official statement from a hospital, insurer or federal official on why such evidence isn't used would be much more informative than the impassioned advocacy of Dr Brenner.

Anyway, looking forward to it