How Do We Know What Really Works in Healthcare?

Listen now:
(photo: Images Money)

(photo: Images Money)

Season 5, Episode 2

In part one (“How Do We Know What Really Works in Healthcare?”), Freakonomics co-author Steve Levitt discussed the randomized control trial, or RCT, which he calls “the very best way to learn about the world around us.” Then Amy Finkelstein, a professor of economics at MIT, talks about using RCTs to explore healthcare delivery — and the “accidental” RCT she discovered when Oregon expanded Medicaid. We also hear from Jeffrey Brenner, an M.D. and executive director of the Camden Coalition of Healthcare Providers. He has teamed up with Finkelstein to run an RCT on a program designed to help the poorest, sickest patients. In part two (“How Many Doctors Does it Take to Start a Healthcare Revolution?”), the M.D./economist Anupam Jena discusses research that defies our most basic understanding of medicine. Don’t miss next week’s episode, when we continue our conversations with Jena and Brenner.

You can subscribe to the Freakonomics Radio podcast at iTunes or elsewhere, or get the RSS feed.


Re Medicaid patients using the ER more -- don't blame the patients. A relative w/serious medical problems was frequently told by her primary care physician to go to the ER rather than going into the DR's. office. She refused to do so, preferring to stay home and see what happened. One time she was given the reason tha she would get admitted faster if she went through the ER than if being admitted by her physician.

Michael Arrighi

Are randomized controlled trials the best way to learn about the world around us?

As an epidemiologist who has worked for a long time in the pharmaceutical industry, the simple implication that randomization and RCTs are all revealing, as implied, is a vast oversimplification and, inherently, incorrect. First, my apologies for such an epistle but the topic is extremely complex and, even here, I’m only able to scratch the surface.

I refer you to a few articles: Seven myths of randomization, Stat Med;2013;32:1439, the role of randomization in clinical trials: myths and beliefs J Clin Epidemiol 1999;52:463.

One all studies, including RCT, have limitation. Secondly, improper application may introduce substantial bias and third generalization beyond the design should be avoided.
With respect to your broadcast, one of the potential biases introduced using the lottery was that persons were not blinded to the ‘intervention’, nor could they be. People knew if they one or not and this may introduce substantial bias; how does knowledge that you ‘won the lottery’ opposed ‘to losing the lottery’ would alter one’s behavior? Even if the two groups had similar characteristics (or statistically adjusted) has the process of an unblinded approach introduced any change to the groups’ behaviors? Second, RCT have a design and analysis, with have assumptions. The design imposes structure on the question but even something such as timing of the intervention, dose and frequency, may be incorrect. A challenge is to enroll the proper participants in RCTs, to assess efficacy, opposed to effectiveness. In the lottery example, individuals had to voluntarily participate in the lottery, thus, are those who participated similar to those who did not? If there was a difference then the result are not generalizable. Also, the authors analyzed data from the Portland region of Oregon and not all of Oregon. Are result generalizable to the broader group, ie those in more rural parts of Oregon, where access to care, attitudes and behaviors may differ? What about participant withdrawals, informed censoring, which is a particular issue with RCT and, often ignored with intent to treat approach to analysis. Medicaid determines eligibility on monthly basis, which introduces a variety of potential confounding and behavioral issues. So, what was the design of the ‘experiment’? The two groups were separated on the basis of a lottery to determine eligibility to Medicaid and one should be concerned about generalizing this to other methods of providing insurance coverage and other types of coverage. For example, data from the NHIS (National Health Interview Survey) indicate that people on public programs, eg Medicaid, report that finding primary care providers is as difficult as those without any insurance, and nearly twice as a problem as those with private insurance. Thus, if coverage were provide for everyone regardless of one’s ability to pay would these individuals behave differently from a program such a Medicaid where eligibility is determined every 30 days? Randomized studies by their very nature often impose substantial eligibility criteria and, often are not, generalizable to the broader population.

The report from the authors’ survey that the diagnosis of hypertension did not differ, was not a surprise. Data from NHANES (National Health and Nutrition Examination Survey) show that more than 80% of people who were identified as hypertensive were aware of this status. The authors findings for diabetes were supportive access to Medicaid through the lottery was associated with improved outcomes and, overall, across nearly every measure, the outcomes were improved among with access opposed to those without. While the ‘magical’ p-value of 0.05 was not met in all circumstance; the use of p-values to determine a difference or not raises another area for discussion.

What is the question, is one interested in efficacy (does the intervention work under so-called ‘ideal’ circumstances) or effectiveness (does the intervention work in the so-called ‘real world’.) Thus, even the ‘simple’ question of ‘efficacy’ is confounded by factors that influence ‘effectiveness’. Others argue, well I want to know effectiveness. However, I first want to know if the intervention works or not. If the factors that influence effectiveness are modifiable then there is path if the therapy work. If all I know is that therapy is not effective then I’m still at the beginning of process, I do not know if the therapy would work if the barriers to utilization were reduced. As a real example in the area of pharmaceuticals, route of administration matters and is a cultural issue for some therapies. Thus, a particular therapy with a specific route may be very effective in one group and not in another. But if the route could be modified for the other group then perhaps they would be equally effective, albeit using two different routes administration. In biology where know symbiotic associations exist, if one did not have the concept of symbiosis and attempted to examine each factor in its purest isolation then one may never learn that legumes and rhizobia together are able to fix nitrogen.

The challenge is that we often do not know the complexity the beginning of the process and when one assumes, well I’ve randomized, now I know the ‘truth’ and generalize well beyond the data then one is bound to reach invalid conclusions.


Chuck Cutler

The episode highlights two critical elements in managing care for populations - data and aligned incentives. Health plans have had the data and financial incentives to engage members in way Dr. Brenner described, but have usually had to do so remotely by telephone. Over the last few years, health plans have developed at least two new approaches. They hired nurses to engage patients directly, especially to support and evaluate members in transitions between hospital and home; and have placed care managers in physicians offices. They also provided incentives for providers to engage members and shared data that allow them to identify the members who would most benefit from outreach. These efforts have produced the same kinds of results highlighted in the last broadcast. These are the kinds of activities that make managed care successful in improving cost and quality.

The Commonwealth Fund has reported studies about these efforts

Other publications including those from the University of Pennsylvania show the effectiveness of these programs -