This Vaccine Lottery Seemed Like a Great Idea. Why Didn’t it Work?

In the spring of 2021, the behavioral scientist Katy Milkman received an unexpected message.

MILKMAN: The vaccine rollout in the U.S. was just starting to gain real speed. And we had moved beyond the oldest, most vulnerable Americans and policymakers suddenly started to recognize there was going to be a cliff in terms of demand, that soon the supply would exceed the demand. So, I actually got an email, from two Nobel laureates, which doesn’t happen to me every day. I don’t know about you.

The two Nobel laureates were Richard Thaler from the University of Chicago, and Daniel Kahneman, now a professor emeritus at Princeton University. Like Katy, their work has focused on the intersection of economics, behavioral science, and psychology. Now, they were thinking about COVID vaccines. Katy works on getting people to change their behavior, and they wanted to pick her brain.

MILKMAN: So, you know, classic question — two Nobel laureates email you and ask if you want to hop on the phone, what do you say? So, I got on that phone call. And we started talking and trying to figure out, “Okay, this is a good idea, we should try a vaccine lottery.”

They could’ve just entered everyone who got the vaccine into a lottery, as a reward. They tried something else.

MILKMAN: We designed this as what’s called a “regret lottery.” So, a regret lottery is a lottery where everybody is entered. and we then draw a name. And say, we draw your name, Bapu, we call you and say, “Hey, you won $50,000, but only if you can provide proof that you were vaccinated before today.” And so, if you haven’t been vaccinated and you get that phone call, you have to politely decline. That’s why it’s called a regret lottery.

The purpose of this regret lottery, which took place in the city of Philadelphia, was to incentivize people to get their COVID vaccines. Who would want to hear they could have won a big cash prize — if only they’d already gotten vaccinated?

Historically, regret lotteries have been powerful tools in public health, and research has shown that they can actually get people to make better medical decisions. Katy and the Nobel Laureates were hopeful they’d see the same outcomes.

MILKMAN: So, the first thing we find: that was a complete bust, total waste of money, had absolutely no effect.

From the Freakonomics Radio Network, this is Freakonomics, M.D. I’m Bapu Jena. Today on the show, we’re going to talk with Katy Milkman about why the regret lottery approach didn’t work when it came to COVID vaccines—and what it can teach us about how and when to use incentives in medicine.

MILKMAN: We do better with these efforts to help people change when we actually match the barrier they’re up against with the solution we’re offering.

And later, economist Tom Chang will explain why sometimes, not only do incentives fail —they actually backfire.

CHANG: By offering people money for something that you’re saying is good, that makes people think, well, why are you offering me money for this, since it’s something that you’re saying is already good for me?

MILKMAN: My name is Katy Milkman, and I am a professor at the Wharton School at the University of Pennsylvania. I’m a behavioral scientist, and I co-direct the Behavior Change for Good Initiative, which I run with Angela Duckworth. That’s been covered a lot actually on your sister/brother show Freakonomics Radio. I don’t know if you call it your sister or your brother.

JENA: Step-sibling.

MILKMAN: Step-sibling. Okay.

If there’s a Freakonomics Radio Network family, then Katy Milkman is at least a cousin. She works closely with Angela Duckworth, who co-hosts one of our other podcasts, No Stupid Questions, with Stephen Dubner, and she’s been a guest on Freakonomics Radio a few times, most recently in Episode 455, “Are You Ready for a Fresh Start?”

Katy’s research focuses a lot on behavioral change and health, specifically: how can we get people to change their behaviors to become healthier? She stumbled on this question accidentally.

MILKMAN: I got to this topic in part, because of wandering over to a seminar at the University of Pennsylvania’s medical school a little over a decade ago and a researcher put up a pie chart showing the proportion of premature deaths in the United States that are due to different causes. And it was broken down — things like accidents, genetics, the environment. And then one category was decisions that could be changed. Daily decisions about things like what you eat and drink, whether you smoke, whether you buckle up in cars. And what blew my mind was that the biggest wedge in that pie chart, the biggest contributor to premature deaths was those daily decisions. 40 percent of premature deaths are the result of daily decisions we could change. And realizing how big the problem was — how big the opportunity was — that’s what led me to what I do today.

Two mechanisms that researchers like Katy use to get people to change their behaviors are incentives and nudges. An incentive refers to a reward for engaging in a certain behavior or a penalty for not. A nudge is different. It’s a tool that subtly encourages a behavior change, for instance by simply changing the way a choice is presented.

Studies have shown that both approaches can work. How well they work often depends on the situation. So, when Katy and her colleagues started to think about how to encourage people to get vaccinated in the spring of 2021, they tested a few approaches. That’s when they came up with the idea of the regret lottery.

MILKMAN: So, not only did we run a regret lottery that was city-wide, but we also designed a feature of this lottery that we thought was very clever. We’ll see if you agree, or if you say immediately, “That was a silly design decision.” But we thought, we can’t experimentally evaluate this lottery and its effect when we have to roll it out to the whole city. And one thing we could change is we could target certain zip codes, where there have not been as many vaccinations as in others. And you won’t be shocked to hear these are zip codes that generally are underserved. And we could randomly select from those zip codes, one for each lottery drawing that’s going to have vastly superior chances of a win. So, we’ll randomly draw one zip code and announce it two weeks before each drawing and really market like crazy that, if you’re in this zip code, you’re really likely to get a call so get out and get your vaccine.

JENA: And, just by the way, what do we know about regret lotteries? Do they generally work? Are they better than normal lotteries?

MILKMAN: Regret lotteries tend to be very powerful because of a force called loss-aversion, which is losses tend to loom much larger than gains psychologically. So, losing $20 on the street feels a lot worse than finding a $20 bill feels good. So, we had a database of people’s contact information, names that we’re drawing from, and we said “You don’t have to register, you’re already registered. But here’s a website if you want to just make absolutely certain your name and contact info is all correct. You may register if you’d like.” And people flocked to that website and they flocked at higher rates, vastly higher rates in these selected zip codes. So, it’s not like we didn’t get the word out and these folks weren’t excited about it. In fact, the day we announced that this lottery was taking place, 2 percent of Philadelphians, spontaneously went to our website and entered their information. A total of 6 percent did over the six-week lottery. We were getting the word out, they’re excited, but it had no impact. Just zero, a very precisely estimated zero. We can in fact rule out with 90 percent confidence that we generated even one extra vaccine per $4,700 spent on this targeting that we were so excited about.

JENA: Are there things that are specific about the COVID-19 vaccine that may not generalize? Put a different way: if you were to do the same approach with another health behavior, do you think you’d see different results?

MILKMAN: Yeah, absolutely. First of all, there’s the point in time when this all happened. Everybody was behind the eight-ball scrambling to roll out these incentives really after demand had plummeted. It was also a politically-heated issue, unfortunately, it should never have been a political issue, this is a health issue, but it was. And issues of identity and politics make, us dig in our heels sometimes even more than, we would in other circumstances. So, if it had been a different moment and a less political hot-button decision, sure, we have lots of evidence from past research that these tools can be useful. And I think with incentives—If you think about it, right, it’s a cost-benefit analysis that we’re trying to change. There’s some cost to taking the action. Like, this goes against my belief system, or this is, just not attractive to me, or I’m nervous about it, or it’s going to take a lot of time. And then there’s the benefit. And you’d think hopefully the reduced risk of death and hospitalization is enough, but let’s add a sweetener. How about a hundred dollars? How about a thousand dollars? And then the question is, you know, how big does the sweetener have to be? And none of the sweeteners tested here were all that large. So, I’m pretty sure if every American had been offered a million dollars to get a COVID vaccine that would’ve worked.

The regret lottery study was sort of a mixed bag. As Katy told us, it did not increase vaccination in the targeted zip codes that were given greater odds of winning. But some of her data suggests that the lottery did increase vaccination across the entire city of Philadelphia. The jury is still out, but that wasn’t Katy’s only attempt at encouraging vaccination. Even before COVID vaccines were approved and available, she and her colleagues suspected it would be difficult to get people to take them, like it can be for the flu shot. So, they tried a few other things.

MILKMAN: I’m going to take you back in time again to the beginning of the COVID pandemic, back in March of 2020.

JENA: Sounds like 16 years ago.

MILKMAN: I know, or like 100 years ago. So, back in March of 2020, my team and I at the Behavior Change for Good Initiative — we know a thing or two about human behavior. It’s not just going to be a problem of developing the vaccines. We were starting to jump up and down and say, “oh my goodness, we’re going to have to get people to actually take the things. And we don’t know nearly enough given the importance of this problem about how to do that.” So, we recruited Walmart to partner with us. They have a massive, pharmacy footprint in the U.S., and we were already working with Penn Medicine. We also looped in Geisinger Health, another large healthcare system in the Northeast. And we have this team of 160 scientists and we said, “Okay, we’re going to run three randomized controlled trials, encouraging vaccination at these three sites. Come up with your best ideas for messages we can send to encourage people to get a flu vaccine in Fall 2020,” knowing that the end goal is to develop communications that will be powerful and persuasive when the COVID vaccine comes out. So, we’re treating this as a test case. Not surprisingly people were really excited to try to help. So, it ended up being — we call it a mega study. Lots of, different people submitted their best ideas. And instead of testing just one idea, we tested dozens of ideas all at once. And, with Walmart, we had, I think 680,000 patients who got different text communications encouraging them to come and get a flu shot at Walmart pharmacy. And the formulations were really different. So, we tested a joke, you know, “Have you heard the one about the flu? Don’t spread it around.” We tested, rates — “More and more Americans are getting these vaccines than ever come and get yours. Don’t you want to do what everyone else is doing?” We tested, “Protect other people, protect your family and loved ones, do this for others.” A huge number of ideas and what rose to the top very clearly was telling people “There’s a vaccine waiting for you. We have one waiting for you, set aside for you specifically.”

JENA: It’s so interesting. What I liked about this study was not just the particular finding that you obtained about saying that there’s a vaccine waiting for you, but it’s that you tried just so many different approaches and you’re able to do that because you have so many patients over which you can randomize.

MILKMAN: I think of it as massively collaborative behavioral science on a scale that’s more like what you often see in natural sciences when there’s a major challenge, like a moonshot, right? But normally researchers work in silos and isolation on their own project, on their single hypothesis. And when there’s a pressing policy challenge, there’s so much that can be gained in terms of efficiency and, rapid knowledge creation by running these mega studies. And now we can make apples to apples comparisons about the cost effectiveness of different communication strategies and say, it’s a tournament, it was head-to-head, and this is what won, so let’s deploy that. So, we’re continuing to do a lot of work that uses this methodology.

JENA: It’s like, it’s almost like a Shark Tank, for behavioral economics.

MILKMAN: I’ve never used quite that way of describing it, but I like that. Yes. Matesh Patel, also a key leader of the mega study vaccination projects we did, ran a trial, right when mandates were being rolled out for vaccination among healthcare workers at Ascension hospital systems, and found that using this reserve-for-you language increased take-up as well. So, we’ve seen a couple of successful deployments, and that was really exciting. One caveat is that there was another test, in Rhode Island in May of 2021, right around the time when a lot of vaccine lotteries were rolled out using exactly this “reserved-for-you” language that didn’t produce much benefit. So, one interesting dynamic that came out was, these nudges can be useful in some context, but not in all contexts, not at all moments, and not when the people we need to persuade are really unpersuadable.

Coming up after the break, we’ll look at what can happen when you try to persuade the really unpersuadable, and how COVID has challenged what a lot of experts thought they knew about incentives and nudges.

CHANG: Maybe this event, as singular as it is, has shifted the way the public interacts with health. In which case, now we might need to reconsider how all health outreach needs to be messaged.

I’m Bapu Jena, and this is Freakonomics, M.D.

*          *          *

JENA: Where did you go to college?

CHANG: M.I.T.

JENA: I knew the answer to that question. That’s why I asked you. I went to M.I.T. I will say that whenever anybody asked me what was one of the hardest things I did in my life, probably the hardest thing was my first year in college.

CHANG: Yeah, I don’t know. I liked it. I liked my first year.

JENA: So, you’re smarter than I was.

CHANG: Or more relaxed?

After he finished up at M.I.T., Tom Chang entered the even more relaxing field of economics. He’s now an Associate Professor at the University of Southern California, where his work has focused on the intersection of behavioral economics and health policy.

CHANG: I fell into this because it’s where there’s a lot of interesting data and a lot of interesting questions.

In the spring of 2021, many of those questions focused on COVID vaccines, and how to get people to take them. Like Katy Milkman, Tom wanted to see if incentives would do the trick.

CHANG: When I present to non-economists, I used to have this slide, it’s a little obnoxious, but I would talk about their problem. And then I would say, “We as economists have the solution to your problem.” And then I’d switch to a slide which had a picture of money in it. And if you think about how economists think about stuff, it’s all about incentives. We think incentives work. And, from this sort of mindset, I thought sitting at home like everybody was, during COVID we, started thinking about: what can we do here?

Tom and his colleagues at U.S.C. teamed up with Contra Costa County, in Northern California. The county runs the Contra Costa Health Plan, a public, Medicaid-managed care plan. Contra Costa County has a population about the size of Rhode Island, and vaccine uptake there had, so far, been good. So, Tom and his team focused on how to incentivize the people who were still holding out on getting their COVID vaccines — despite being eligible, despite ample vaccine supply in the county. They tried three different approaches.

CHANG: We tried financial incentives, we tried different sorts of public health messaging, and we tried lowering what we would call the “frictions” of setting of appointment. And then we would just follow them to see what worked and what didn’t in terms of increasing vaccination uptake.

JENA: And then what was the size of the financial incentives that people were randomized to?

CHANG: We offer $10 and $50.

JENA: What did you do to make it easier to get the vaccine? You mentioned lowering the barrier cost, what did that look like?

CHANG: Instead of taking you, to California state’s vaccine scheduling system, Contra Costa has developed their own system that was much easier and simpler to set up appointments.

JENA: So, what were the outcomes that you looked at?

CHANG: We looked at two outcomes. The main outcome that we looked at was simply whether or not you got vaccinated within 30 days of, being messaged or being offered a financial incentive. So, to be clear, the way it worked is we sent out invitations and then people would take a survey about COVID, telling us some of their beliefs and as part of the survey, they would be randomly exposed to one of the messages and randomly assigned to receive a financial incentive or to see the link at the end to the new scheduling system.

JENA: And how many patients were randomized approximately?

CHANG: At the end of the day, we had just over 2,700 people who completed the survey and met all the requirements.

JENA: Okay. And what did you find?

CHANG: We found that nothing seemed to really work. I mean, I was personally a little surprised by this, but nothing really moved the needle in terms of actual vaccinations. The other outcome we looked at is their self-stated vaccine intentions. That is, how likely are you to get vaccinated in the next 30 days? And there, we saw big effects. We saw that, the messaging had rather large effects on people’s self-stated probability of getting vaccinated. But then when we followed those same people to see if they followed through, they didn’t. There was nothing there.

JENA: So, you randomize people to these interventions, financial incentives, public health messages, an appointment scheduler, and there was no effect on actual rates of vaccinations. But the people who received the messages who were randomized to receive these sort of pro-public health messages about why it’s important to get vaccinated, they said that they wanted to get vaccinated. So, there’s an increase in their intentions, but it did not translate to actual overall increased vaccinations?

CHANG: That’s absolutely right.

JENA: And then you found something that was really interesting to me, that I’d probably describe as backlash.

CHANG: Yeah. I mean, this was a study in which, I was wrong in, more places than one. I was actually convinced the finance incentives would work. It was just sort of a matter of whether or not they would, move the needle enough, what was the right value — the right amount to sort of offer people. And, around this time, when people were sort of promoting financial incentives, you had people worried about unintended consequences of offering financial incentives in this particular situation could backfire. And among certain groups, this looks like what happened here. So, for the older people in our sample and for people who stated that they supported Trump in the last presidential election, offers of money actually seemed to backfire and reduce significantly the share of them that got vaccinated over the next 30 days. So, the public health message didn’t do it. It was the financial incentive. So, if you offered someone $50 to get vaccinated over the next two weeks, and they were in one of these two groups, if you followed them over the next 30 days, they were actually less likely to get vaccinated.

JENA: And so, why do you think that is?

CHANG: One hypothesis that people put forth, even before we had these, results was that by offering people money for something that you’re saying is good, that sort of makes people, especially those who are distrustful think, “Well, why are you offering me money for this, since it’s something that you’re saying is already good for me? If you’re paying me for it, it must be because there’s something bad. Or there’s something that you are getting out of this.” And, for people who are older, our hypothesis is, older people, regardless of political affiliation, tended to have really high rates of COVID vaccinations. And part of that is because they bear the highest costs, right? They’re the ones who are most likely to get sick, most likely to die. And so, the few who remained unvaccinated at this point were the ones, who had really, really strong beliefs when it came to COVID. They had some story in their head as to why they shouldn’t get vaccinated. CHANG: And this seemed to confirm that story that made them double down even more and say, “Yeah, I was right not to get vaccinated.”

JENA: That’s really interesting, because what it’s basically saying is that the offer to compensate someone, to pay someone sent a signal to them that, “Wait, wait, wait, hold on. Why, why am I being paid to take something that’s supposed to be good for me?” It reminds me of this kind of related issue. In healthcare people have tried to use these price transparency tools to get patients to utilize lower cost lab tests or lower cost imaging. And the idea is if you just show them the price of a lower cost M.R.I., they will go get that lower cost M.R.I., rather than drive, across town to get the higher cost M.R.I. But it doesn’t happen that way. And one reason that we think that it doesn’t happen is because people infer something about the quality of that M.R.I. based on the price. Like why would I want to see a cheap orthopedic surgeon? That’s sort of the logic. So, it shares some similarity with what you found. Your story sort of feeds into this broader question though, of the way that we dealt with, the COVID-19 vaccine, it strikes me that it could have these downstream effects that could kind of last for a long time when we’re thinking about things like vaccine hesitancy, why it is that polio is now emerging again.

CHANG: I’m quite worried about this. And, just to take a step back, I think it’s not that they didn’t work, right? And the way I think about that is, you know, when we think about nudging people, you need that movable middle. You need the people who are sort of already on the fence about something. And then these small changes can get people on one side of the fence or the other. I think we need to first think about, who we’re targeting here and depending on who we’re targeting different things will work. Different strokes for different folks, right? But when it comes to the long run consequences, I’m very concerned about this. If you spent the last two years convincing yourself that your doctor is trying to get you to take this vaccine that’s not good for you or has a microchip in it, or they want to make some COVID money off of this —how do you reconcile that with, they want what’s best for me when they say, “oh, it’s time for you to get a cancer screening or it’s time for you to get, you know, a flu shot”? I worry that all this messaging, has created an increased distrust of the medical system that’s going to persist, over time.

JENA: When people like you, other researchers like you study the effect of things like public health messages, how important is it to figure out whether or not, the message is coming from a person who the patient actually trusts.

CHANG: We actually spent a lot of time with Contra Costra trying to figure out who would be trusted health messengers. And at the end of the day, the research out there seemed to show that, the most trusted health messenger is still your doctor. And so, one of the things we did, for our video messages is we had doctors record the different messages. Every message is recorded by a male white doctor, female white doctor, a male Hispanic doctor, and one of the things we had in there is we also checked the impact of race and gender concordance on messaging. And the evidence on this in the field is a bit mixed and, data sort of falls on the side of it doesn’t seem to matter much. And so, any additional trust that you might posit happens with race concordance or gender concordance doesn’t seem to survive, or work, if it’s a one-way communication. So, if you do it through video, as opposed to one-on-one interactions. I think if you lose trust a lot of the things that, we think normally work, a lot of mechanisms that sort of grease society, fall apart. It’s like when you go to the mechanic, right? So, you have to trust that the mechanic is actually telling you when you need this new part, that they actually do need this new part. And when that trust starts to break down, I think a lot of things start to break down. Maybe this event, as singular as it is, has shifted the way the public interacts with health. In which case, now we might need to reconsider how all health outreach needs to be messaged and how that needs to be incentivized. And so, figuring that out seems, a pretty good first step. 

These incentives and nudges that people like Tom Chang and Katy Milkman use to get people to make different decisions or modify their behaviors all have the same goal: change. As their work has shown, people seem to want to change. They say they’re excited about getting a vaccine, they make sure they’re enrolled in a lottery, but then they don’t follow through. Why not? Here’s Katy Milkman again.

MILKMAN: We are wired in a way that works against change. There’s a lot of different forces, internal forces pushing against us. And something that I think is so crucial to understand about nudging, about any kind of behavioral intervention is we do better with these efforts to help people change when we actually match the barrier they’re up against with the solution we’re offering. So, if someone is not taking their medication because, they keep forgetting, you need a really different solution than if they’re not taking their medication because they find the side effects unpleasant. So, if you just try to slap a universal solution on these problems, you don’t get very far.

That’s it for today’s show. I learned a lot. And on that note, I’d like to try something new. At the end of every episode, I’ll push myself a little and try to think of an idea, a question to study, based on what we talked about, and maybe a listener will, too. One idea that came to mind was based on Tom’s comment about how trust in our health care system could change as a result of the pandemic. It makes me wonder whether in areas or populations where vaccine uptake was low, we’ll see fewer visits to doctors in the future, less preventive care, less adherence to medications, all things that might suggest a breakdown in trust, a breakdown in people’s relationship with health care. If you decide to study that, let me know—and I’ll do the same.

I’d like to thank my guests, Katy Milkman and Tom Chang. And thanks to you, of course, for listening.

*          *          *

Coming up next week: incarceration is notoriously bad for people’s health. It increases their risk for a lot of medical problems and shortens their lives. But, could there be another side to the story?

WEAVER: It’s really much more about what’s happening outside of prison than what’s happening inside of it.

We’ll look at what happens when you give people a little more health care than they already have—and then, what happens if you take health care away.

 JACOME: To the extent that we can keep individuals out of the criminal justice system by extending eligibility, then perhaps we could also just reduce the likelihood these individuals ever come into contact with the criminal justice system.

That’s all coming up next week on Freakonomics, M.D. Thanks again for listening.

Freakonomics, M.D. is part of the Freakonomics Radio Network, which also includes Freakonomics Radio, No Stupid Questions, and People I (Mostly) Admire. All our shows are produced by Stitcher and Renbud Radio. You can find us on Twitter at @drbapupod. This episode was produced by Julie Kanfer and mixed by Eleanor Osborne, with help from Jasmin Klinger. We also had help this week from Lyric Bowditch. Our staff also includes Neal Carruth, Gabriel Roth, Greg Rippin, Rebecca Lee Douglas, Morgan Levey, Zack Lapinski, Ryan Kelley, Katherine Moncure, Jeremy Johnston, Daria Klenert, Emma Tyrrell, Jacob Clemente, Alina Kulman, and Stephen Dubner. Original music composed by Luis Guerra. If you like this show, or any other show in the Freakonomics Radio Network, please recommend it to your family and friends. That’s the best way to support the podcasts you love. As always, thanks for listening.

MILKMAN: I have to admit that I was teaching my 75 Wharton M.B.A. students about loss aversion, someone raised their hand and said, “Is that like F.O.M.O.?” And I had the awkward moment of saying, “I don’t know what that is, could you define it for me?”

JENA: Classic business school students, okay.

MILKMAN: But I confirmed that indeed does relate to loss aversion.