Farewell to a Generational Talent

Today on Freakonomics Radio, a very special episode — a conversation about the late Daniel Kahneman, whose insights into human behavior have been threaded through this show for years — ideas like confirmation bias and loss aversion and the planning fallacy. During this conversation, we also learn about a research paradigm that Kahneman embraced called adversarial collaboration, which means working shoulder-to-shoulder with your rivals.

Richard THALER: He felt that this is the right way to do science.

Kahneman was a phenomenally influential psychologist who won a Nobel prize in economics; wrote the best-selling book Thinking, Fast and Slow; and left behind an army of collaborators, mentees, and admirers. With them, we will take a careful look at the life and mind of Danny Kahneman.

*      *      *

Last month, in a sunlit auditorium overlooking the Chicago River, there was a gathering of psychologists, economists, and other social scientists: this was the Behavioral Decision Research in Management conference. The keynote event was supposed to be a conversation with Danny Kahneman, facilitated by Richard Thaler, his longtime friend and collaborator. Thaler is the University of Chicago economist who helped turn Kahneman’s insights into the field now known as behavioral economics. But when Kahneman died in March, at age 90, Thaler came up with a new plan for the conference; now it would pay tribute to Danny Kahneman. Freakonomics Radio was lucky enough to be asked along, to moderate a couple of panel discussions about his life and work. The episode you’re about to hear is a condensed version of those conversations; this all took place at the downtown outpost of the University of Chicago’s business school, in front of a couple hundred attendees. Some of the panelists had known Danny Kahneman for many decades — for instance, the psychologist Maya Bar-Hillel. Her father was a philosophy professor at Hebrew University in Jerusalem, where Kahneman got his undergraduate degree.

Maya BAR-HILLEL: My father taught Danny and gave him a lot of grief, but my father apparently gave just about everybody a lot of grief. He was a tough-minded philosopher.

And when Kahneman became a professor, one of his students was Maya Bar-Hillel — from generation to generation.

BAR-HILLEL: I met Danny in the first week of my first year at the Hebrew University. He gave the introductory statistics course. He looked at us and he pronounced right away, “You are the creme de la creme.” He said it in French. And we were.

Kahneman had grown up in France, during the Nazi occupation. He survived, barely, and lived for many years in Israel before coming to the U.S. to get his Ph.D., at U.C. Berkeley. His research partner, Amos Tversky, was another Israeli psychology professor who moved to the States. Tversky died young, in 1996, too early to share in what would surely have been a joint Nobel prize. Tversky was regarded as perhaps even more brilliant than Kahneman. The two of them published many papers on judgment and decision-making, but not just in psychology journals.

DUBNER: It always struck me, just in the story of how Kahneman and Tversky research was taken hold of by Thaler and others and turned into what we now call behavioral economics — that it was very, very important that these were two psychologists who were also very mathematically fluent. If that hadn’t been the case, and if the publication hadn’t been in, I guess, Econometrica and so on, was there a pretty good possibility for a counterfactual, where all that research might have stayed within the realm of psychology and never trickled over, and we might not have what we think of as behavioral economics?

BAR-HILLEL: It was not an accident. It’s not, “How fortunate that they went to Econometrica.” They realized that their work was attended to primarily by psychologists, and in fact they both considered themselves all their lives as psychologists. But they also realized that their research was perhaps more important outside of psychology. So the decision to publish their paper in Econometrica was a deliberate move. It was a strategic move to get the attention. They believed that that was the ticket. And without the ticket, they would not have been playing in that field. 

A lot of the early Kahneman-Tversky work centered around an observation that may seem obvious in retrospect, but at the time had not been explored with much rigor. The idea was that we are all constantly making decisions — personal, professional, political decisions — and then later, if we ask ourselves why we made a given decision (which, by the way, we usually don’t ask), we might tell ourselves a story about making the decision. But these stories are often not quite true. Why? One reason is that we want to appear to others, and maybe even ourselves, as smarter than we are. Here’s Richard Thaler:

THALER: I met Danny and Amos in 1977, and it was a transformative year for me. I decided to change jobs, and took a job at Cornell, and decided to offer a course in the thing I was now fascinated by, and called it “Behavioral Decision Theory,” which is kind of what the name of this field used to be. I got about eight students. So I had to think of something to do to increase the enrollment in the class. And so I changed the name to Managerial Decision Making. Fifty students show up. I began the class by asking, “How many of you signed up for this class because of the name?” No one raised their hand. I said, “Well, actually, nearly all of you.” That’s one illustration of what Danny is talking about. No one thinks they would be stupid enough to sign up for a class based on the name.

DUBNER: I believe it was in Thinking, Fast and Slow where Danny wrote, “Not only are we blind, but we are blind to our blindness.”

THALER: Yeah. And maybe one of the many secrets of the Kahneman-Tversky collaboration was they were not blind to those, and what they had was a mistake-detection facility. They had some radar where they could anticipate what the mistake is. And because they had this mistake-detection facility somewhere, they were able to figure out things. And then they were pretty good at predicting what people other than Amos and Danny would do. To me, the big point, the “aha” point that I got from reading their judgment papers was the idea of predictable bias. 

BAR-HILLEL: May I add something to that? Not only predictable. I think now I’m going to say something that is perhaps — I can’t say that I heard them say it, but I hope they would both agree — that the errors are not just predictable; the errors are smart. Our stupid mistakes are evidence of the wonderful human mind. That’s how normal cognition functions, and normal cognition is amazing.  

DUBNER: I love that. It does make me want to ask any — all of you — a question that’s fairly heretical, which is, you know, as a layperson, I read these findings — they’re attractive, they’re believable, they’re identifiable. But the thing that I always struggle with, or would like to understand better, is how you all can feel so confident that they’re generalizable? There must be people in the world who are not susceptible to loss aversion or recency bias or many, many of them. So really the question is, how much variance is there and how do you measure the variance?

THALER: You know, Amos used to have a joke, that there were species that didn’t exhibit loss aversion, and they’re now extinct. 

Think about what Maya Bar-Hillel was saying there, about the “smart errors” we all make. It would be easy to overlook the baseline insight that Danny Kahneman offered: that people make thinking mistakes all the time. Now, most of us, upon hearing this, might say, “No kidding! Our species is highly fallible — who doesn’t know that?” But it is our fallibility, Kahneman realized, that makes us interesting, and worthy of inspection. He accepted that humans are capable of wonderful things (as well as terrible ones), but that we are overconfident in our abilities; that we often have poor self-control; and that we employ an arsenal of mental shortcuts, or heuristics, to make decisions or reach judgments that often turn out poorly. And even when presented with evidence of our mistakes, we usually fail to change our minds. Here is Eldar Shafir, who runs the Kahneman-Treisman Center for Behavioral Science and Public Policy at Princeton. He and Kahneman used to co-teach a class.

SHAFIR: One lecture that I love that Danny used to give in our course: we would talk about people’s not-good sense about conditional probabilities. And then we had a clip of the O.J. Simpson trial, where Dershowitz explains to the jury that the probability that a beaten woman is going to be killed by her beating partner is extremely low‚ which is true. However, we’re not predicting the chance that Nicole Simpson will be killed. She has been killed. The probability that a beaten woman who has been killed was killed by her partner is immensely high. That nuance, of not being sensitive to those conditionals, has major implications. 

He was so genuinely curious and intellectually alive. He presented the same way to the Swedish monarchy and to an undergraduate. He just wanted to think about it. He wanted to struggle with the question. He wanted to listen and think of the best theory. That was what was so impressive. When we taught together, we took turns lecturing. Here was Danny — I mean, before and after the Nobel — and every class, he would come and sit down with the students and listen. He could easily not have shown up — it’s other people’s lecturing. He was always there, asking questions, answering questions. Devoted to understanding. Always had a new thought about something that we have done on slides for three years in a row.

In many things that were written about Kahneman, I’m always amused by the fact that they say he was aware of these quirks and fallacies, but he was aware that he has them too. And that sort of implies that since he was aware, it wasn’t as big a deal. There’s a couple of recorded interviews with Danny during Covid, where he predicts that Covid means life imprisonment for the more advanced in age. And you know, he was wrong. He lived a couple of wonderful years after Covid. There was a sense in which he really sank deep into the same error that he was able to predict and acknowledge. 

THALER: The other amazing thing about Danny is, when he won the Nobel Prize, you have to give a lecture, and he gave a lecture and decided he was going to give a new interpretation of all of his work with Amos, using this two-system approach. I kept saying, “Danny, you have two months to do this, why are you starting over? They gave you a Nobel Prize for what you did. You don’t need a new reinterpretation.” But that lecture in Stockholm was the beginning of what led to Thinking, Fast and Slow, because he was reinterpreting all the heuristics and biases stuff in the lens of system one and system two, which is your immediate reaction versus when you sit down and think about it.

DUBNER: What this story suggests to me — but tell me if I’m wrong — is that he understood that having an audience like the Nobel Committee audience and the king or queen or whatever, of Sweden, and that his voice was now amplified, that he wanted to take the work that he’d done and make it accessible, make it known to people who have some levers of power. He wanted to exploit a good opportunity. 

THALER: I don’t like the word “exploit.”

DUBNER: Uhh. Exploit, for pro-social reasons, a good opportunity. 

THALER: Thank you. Thank you. Yeah, I think he had an audience and wanted to get it out there, but it was torture for him. Absolute torture. I mean, during the writing of Thinking, Fast and Slow —

DUBNER: Which was a long gestation. 

THALER: Long gestation. And he decided absolutely positively that he was quitting two dozen times.  

SHAFIR: You know, Danny, in some sense, really was a bon vivant. I mean, he enjoyed life. He loved a good restaurant. He loved a good vacation. And I think he loved getting the prize.  

THALER: I don’t think that he changed. I mean, yes, the world was paying more attention to him, but he was always Danny. 

BAR-HILLEL: He changed.

SHAFIR: He was happier.

THALER: Okay.

BAR-HILLEL: No, I would like to say — 

THALER: If Maya says that, I accept.

BAR-HILLEL: I would like to say in what way: He relaxed. 

THALER: That was relaxed? 

BAR-HILLEL: Yes. Yes. 

DUBNER: What do you mean by that, Maya? 

BAR-HILLEL: He had made it. There was a common misperception that the Tversky-Kahneman collaboration was not symmetrical. The world seemed to think that Amos was the lead and Danny was an outside visitor. It was a symmetrical and equal collaboration, but the world didn’t know it. It was very important to Danny to be able to do good work after Amos dies, so that people won’t continue with this error that he was less than equal. So the prize told the world: this man is Nobel-worthy and he did excellent work because he had an excellent mind. But also it was important for him that Kahneman not die — the name — together with Tversky, and it didn’t.

As I said earlier, the main ideas of Danny Kahneman have been threaded through the fabric of Freakonomics Radio over the years. Episode 323, for instance, which was about the planning fallacy; it’s called “Here’s Why Your Projects Are Always Late (And What to Do About It).” Then there’s episode 271, “The Men Who Started a Thinking Revolution.” That was an interview with Michael Lewis about the book he wrote on the Kahneman-Tversky partnership; Lewis’s book is called The Undoing Project: A Friendship That Changed Our Minds. So, yes, you may be familiar with Kahneman’s greatest hits. But one of the conversations we had at this conference went into an area I knew very little about, and I’ll bet you don’t either. That’s coming up.

*      *      *

THALER: Good morning. Good morning. We have a session this morning, as you know, on adversarial collaboration.

Day two of the conference, with a panel devoted to a different way of doing business in the behavioral sciences. Let’s begin with a short clip from the late Danny Kahneman. This was recorded in 2022.

KAHNEMAN: Controversy is a terrible way to advance science. It’s normally conducted as a contest, and where the aim is to embarrass. The feature that makes most critiques intellectually useless is a focus on the weakest argument of the adversary. It is common for critics to include a summary caricature of the target position, refute the weakest argument in that caricature, and declare the total destruction of the adversary’s position. It’s rare for anyone to concede anything. Doing angry science is a demeaning experience.

“Doing angry science,” Kahneman came to believe, was a terrible thing. He knew this firsthand. Before the accolades, the prizes, the acceptance, Kahneman and Tversky came under a great deal of criticism.

KAHNEMAN: Gerd Gigerenzer published his first critique of our work 37 years ago, and he’s still not done with me.

And here, again, is Richard Thaler:

THALER: I’ve certainly had my share of angry science exchanges. They’re no fun and there’s never any light, only heat. 

And that’s why Kahneman came up with a different model, what he called an “adversarial collaboration.”

THALER: He felt that this is the right way to do science. All of the collaborations shed more light than heat. The typical debates in academia are exactly the opposite. So I think Danny has a good point. 

Kahneman was trying to solve a fundamental problem in social-science research: it’s natural for scientists to disagree with one another, but there’s no clear mechanism for resolving those disagreements. So what if researchers who disagree work together, in good faith, to resolve the issue? And what if they take on a neutral third party to serve as an arbiter? That is what Kahneman called an adversarial collaboration. He participated in a variety of them over the years. The conversation you’re about to hear includes some of the key players. Let’s start with introductions.

GILOVICH: I’m Tom Gilovich. I’m a social psychologist at Cornell University. 

MELLERS: I’m Barbara Mellers from the University of Pennsylvania, and I was fortunate to be the arbiter in two adversarial collaborations with Danny, 20 years apart. 

KILLINGSWORTH: I’m Matt Killingsworth. I’m also at the University of Pennsylvania. And I study human happiness and well-being. Like, what makes life worth living? And how do we collect data to try to understand what makes life better? 

FREDERICK: My name is Shane Frederick. I’m a professor at the Yale School of Management.

DUBNER: Tom Gilovich, let’s start with you. Let’s hear basically the story of the adversarial collaboration with Danny. It began quite early in your career. Give us the whole story. 

GILOVICH: I had published with Vicki Medvec a chapter in a book on counterfactual thinking. When I wrote the chapter, I don’t know if I was aware that Danny was going to write the wrap-up chapter, where he comments on all of the other ones. So I don’t know if his voice was in my head as we’re writing the chapter.

DUBNER: What’s the chapter about? What’s it cover? 

GILOVICH: A lot of it covers the subject of regret, what we regret most in life. Amos and Danny had published a study showing you regret things of action more than inaction. The example they used is, imagine you own stock in one company. You think, okay, that’s run its course, you switch to another, and it tanks, you lose a certain amount of money. Versus, you’re thinking about buying this stock, you decide not to, it takes off, and you lose the same amount of money. Which would you regret more? And almost everyone anticipates that you would regret the action the most. Nonetheless, if you ask people what are the biggest regrets in your life, what dominates are things they didn’t do. And so a lot of the chapter was, how do you reconcile those two things?

DUBNER: In approaching this riddle, or puzzle, in your mind, how did you measure — what was your actual research?  

GILOVICH: Our research was: what’s responsible for that difference? How do things change over time? Precisely because your regrets of action, really hurt, you do things about them. Sometimes you change your life accordingly. Or you certainly engage in lots of psychological work to achieve some level of peace. Because the inaction regrets gnaw at you less powerfully, you sort of leave them alone. The fun of that research was tracking down all of the different psychological processes that make those regrets of inaction hang around, or in some cases, even get more intense, whereas the regrets of action dissipate. 

DUBNER: How do you do that, though? Is it empirical? Is it theoretical?

GILOVICH: It’s empirical. For example, one of the mechanisms that we thought was especially interesting is when you don’t do something that you later on end up regretting, there are reasons for that. Like, I just don’t have the bandwidth right now to take this project on, etc. Those are compelling reasons in the moment for why you’re not doing this thing. You move along in time, you look back and you think, “Wait a minute, I could have done that.” So we would do studies where we’d ask Cornell current students, recent alums, or much older alums: Imagine there’s a class that you always wanted to take, but maybe you were a little afraid to. Suppose we added that to your curriculum this semester? The students are like, “Oh, I can barely hang on right now. You add that, it’s a catastrophe.” You ask the people who graduated many years ago, they think, “Oh, that’s just a piece of cake. It wouldn’t have interfered with my G.P.A., my social life, the amount of sleep I get,” etc.

DUBNER: Okay, so you write up the findings. At what point then and in what way do you hear from Danny? 

GILOVICH: It’s sort of a highlight of my career. I was on sabbatical. Check in your voice messages back at Cornell. And I get this message. “Tom, this is Danny Kahneman. I just read your paper with Medvec. It’s brilliant. It made my day. Just calling to say thank you. What a piece of work!” And then I stupidly press star-delete instead of saving that message forever to play to my grandkids. So I’m just on a high for a while. Two weeks later, there’s another phone call from Danny Kahneman. This time, live — “Oh, great, my new best friend is going to tell me about —”  “Tom, I’ve been thinking more about your chapter. It’s all wrong. I’ve heard you’re planning to publish this. You can’t do that. If you do, people will go after you. And if no one does, I’m going to go after you.” What happened to “brilliant”?

DUBNER: Did he, in that conversation, give you his argument for why he thought now you were wrong?

GILOVICH: Yes. He said that there really isn’t a change in the intensity of either action or inaction regrets. There’s just a substitution of the kinds of things that you’re regretting. And the regrets of inaction really aren’t serious regrets. You might not even call them regrets. They’re more wistful, like, “Oh, I wish I had learned to speak Esperanto.”

DUBNER: Said no one, ever. All right, so what happens next? 

GILOVICH: Well, lots of panic for a few days. Vicki Medvec and I went, “What do we do? It can’t be worse than this.” And I think — I forget what, there had been an example of an adversarial collaboration. I think it was Vicki’s idea — Hey, we should negotiate with him. Let’s see if we can do an adversarial collaboration. So we pitched it, and he was very receptive. 

DUBNER: This was in what year, Tom? 

GILOVICH: The chapter came out in ‘95, so it was a little before that.

DUBNER: So this was pre-Zoom, certainly. Early internet. How are you communicating, and how did you negotiate the negotiation, and then how did the actual negotiation happen? 

GILOVICH: I’m glad you asked that question, because it allows me to bring up a point about Danny that — you know, there’s been all this talk about his many qualities. And they’ve all been well-said. One thing kind of gets left out. I mean, to be at that level of success, you have to be brilliant and creative. And many people have spoken about that. You also just have to be a phenomenally hard worker. And man, he was on task all the time. It was amazingly easy to get that guy on the phone. I could get him on the phone more readily than I could get my coauthor, Vicki, on the phone. And something seemed wrong about that. He was always ready to engage. And so a lot of it happened over the phone. And the idea was, “No, these regrets of inaction that you say are important in people’s later lives, they’re just sort of wistful regrets.” And the intense, hot emotions that come from action, those are different kinds of things. So we designed some studies where we asked people to think about the biggest regret of action or inaction — 

DUBNER: You designed the studies together?

GILOVICH: And then we ran them together, yes. So think of your biggest regrets of action or inaction from the recent past or the distant past. How many of these emotions do they lead you to feel? We had a set of five hot emotions. A set of five wistful emotions. We thought that those long-term regrets of inaction, some of them are wistful — “I wish I had learned to play the piano” — but some of them are really intense — you met the right person at the wrong time, and so you let it pass, and now you’re looking back like, “Oh, my life would have been much better if I had acted on that.” We also asked people to rate whether a set of five, I think we called them emotions of despair — I’m depressed when I think about this, I feel empty when I think about this. And lo and behold, Danny was right. A lot of long-term regrets of inaction are kind of wistful. But we were right, too. They also produced these big, powerful feelings of, Oh, my life is not what it could have been. I feel empty and depressed thinking about these things that I didn’t do that now seem so easy to have done.

DUBNER: Did Danny, as a result, acknowledge that you were partially right as much as he acknowledged that he was partially right?

GILOVICH: Yeah, yeah. And that’s in the paper. 

DUBNER: So, truly a happy ending. 

GILOVICH: I believe so, yeah. Certainly more happy than that second phone call that I got.

DUBNER: You then went on to collaborate with Danny several times, correct? I’m just curious now that Danny is gone, and since your initial contact was this adversarial collaboration about regret, is there anything that you now regret not having worked on with Danny? 

GILOVICH: I wouldn’t say that there’s a particular topic in mind. But, you know, to have someone in your life like that and not take full advantage of it, I wish I had just reached out to him more to have more contact with him. One of the nice things about this event here is this family feeling of all the people he reached, some of whom I know well, some of whom I didn’t really know at all until now. They feel like family. So, you know, often with families, you wish you had spent more time with them. 

DUBNER: Not my family, but, you know.

The adversarial collaboration with Tom Gilovich was one of the earliest ones that Danny Kahneman undertook. Let’s take a look now at one of the most recent, with Matt Killingsworth, from the University of Pennsylvania. He studied engineering as an undergraduate, and got his Ph.D., in psychology, in 2012. Danny Kahneman got his psychology Ph.D. in 1961. So how did Matt Killingsworth end up in an adversarial collaboration with this giant in the field?

KILLINGSWORTH: Danny and Angus Deaton, both Nobel Prize winners, had this conclusion that there was sort of this plateau in people’s happiness once they reached $75,000 in income. And then I had published a paper that basically showed something completely different, and we tried to figure out who’s right and what’s the truth. 

DUBNER: You say it with this sort of sangfroid that, you know, “we published something completely different.” But as you noted, you were publishing a paper in opposition to the findings of not one but two Nobel laureates. Were you as calm and cool about it in the moment, when you decided to take up this route, as you seem to be now? 

KILLINGSWORTH: I mean, partly Danny is to blame, although he didn’t learn that before he passed away. I had written this as, like, a sub-point to a sub-point in another paper, and actually reading Michael Lewis’s book about Danny and Amos, they were talking about how they took on existing ideas, and that was an important part of their intellectual journey. And so I sort of rolled the dice and thought, “Well, I’ll try it.” But yeah, there’s definitely some part in the back of my mind of like, how is this going to go? 

DUBNER: I do have to ask, at this particular conference, which is held in celebration of Danny Kahneman’s work and life, do you feel a little bit like, you know, the wolf in the henhouse, or that someone’s going to come up and shiv you in the back at the buffet?  

KILLINGSWORTH: I hope not. No, I mean, we really had a wonderful collaboration. And I think part of what characterizes that is, we both just wanted to figure out the truth. I don’t think anyone was attached to any particular version of reality. My sense from him is that he was a little bit irritated. Not a lot. 

FREDRICK: A lot. It was a lot.

KILLINGSWORTH: I mean, he wanted to have a good conversation, but he wasn’t — 

DUBNER: For the record, that was Shane Frederick saying, “a lot.” Barb Mellers, let’s bring you in here. How did you get attached to this adversarial collaboration? 

MELLERS: Well, I was talking to Danny on the phone not too long after Matt’s paper came out, and he said, “Oh, by the way, do you happen to know a guy by the name of Matt Killingsworth?” And I said, “Well, as a matter of fact, I do. And if you’d like me to be the arbiter, I’d be happy to do so.” In adversarial collaborations, the arbiter is the research assistant, the tiebreaker, and occasionally the therapist.  

DUBNER: So Matt, how would you characterize the response to that original Kahneman-Deaton finding about the $75,000 happiness cutoff, whether inside academia or beyond? 

KILLINGSWORTH: I mean, it was very influential. And I mean, it’s probably one of the most visible kind of and exciting findings. I think a lot of us feel like money is sort of overemphasized in our daily lives, and it kind of gives a justification for maybe caring a little bit less about it, and focusing more on all the other stuff that’s also really important for happiness. Particularly because money is quantitative, you can kind of think of them as like points and like, “Well, I want to get more.” But you can easily imagine a sort of trap where you’re just continuously trying to get more and more and more of those points, ignoring all of this other stuff. And wouldn’t it be nice if you could kind of step out of that cycle? And at least that’s part of the reason that I think that that was so attractive.  

DUBNER: So, Matt, you’ve rehearsed very nicely the original finding. Now, bring you into the story here. What are you doing at the moment? 

KILLINGSWORTH: Sure. So — partly also due to Danny — my research program is really centered on large-scale experience sampling.

DUBNER: And to get at why that matters in the original study, those data were collected how?

KILLINGSWORTH: In the original study, they basically asked, “Did you experience a lot of the following emotion yesterday: Yes or no?” And then it had a series of emotions like sadness, happiness, stress, etc. People would either say “I did” or “I didn’t.”

DUBNER: And the technology or mechanism of harvesting those data was what?

KILLINGSWORTH: I wasn’t involved in that data collection. I believe those were verbal phone calls. And then sort of interviewing people over the phone.  

DUBNER: Does anyone here know anything different, or does that sound right, as far as we know?

MELLERS: That’s right. 

DUBNER: That was said with the voice of a referee, I have to say. Okay, Matt, continue, please.  

KILLINGSWORTH: So in my study, I’m essentially measuring people’s experience right in the moment. They’re carrying around on their phone an app, and I’m beeping them at random times. I’m asking them, “How do you feel right now?” And they’re responding on a scale that ranges from very bad to very good, and the scale is continuous. So there are a couple of things that distinguish that from the earlier paper. One is that it’s right at the time that people are feeling it, as opposed to the day as a whole, retrospectively. And the other is the gradations of feelings are on a continuous scale as opposed to something that’s binary or dichotomous. 

DUBNER: And the scale is what to what? It’s zero to 10, zero to 100? 

KILLINGSWORTH: Very bad to very good. And it’s just continuous. 

DUBNER: So you can rank it anywhere you want.

KILLINGSWORTH: There are hundreds of unique values.

DUBNER: Got it. And describe the differences between the pools of research subjects in the original case and in your case. 

KILLINGSWORTH: In the original case, it was a survey by Gallup. It was either representative or at least an attempt to be representative, whatever random digit-dialing was. That’s a relative strength of their paper, and of that study in general. My sample, in contrast, was really a convenient sample. It’s turned out to be an amazing sample of people that have really beautiful results, but it was essentially whoever was willing to sign up to try to understand their own happiness. 

DUBNER: What was the recruiting mechanism? 

KILLINGSWORTH: Largely thanks to folks like you. People wind up hearing about it in the media, on the radio, or reading about it. They think, “That’s interesting, I’d like to learn about that for myself, I’m willing to contribute to science.” And it turns out that when I look at what’s the distribution of incomes, for example, in my data, it’s almost a perfect match for the U.S. Census. I can replicate all kinds of things that we’ve seen in the literature for decades. So it turns out to be really good. But that’s kind of a lucky coincidence.

DUBNER: Okay, so you gather, assemble, analyze your data. Talk about from the analysis point to writing up the findings.

KILLINGSWORTH: What I find when I look at this relationship, plotting people’s happiness in the moment versus their income, it just keeps going up. This sort of critical point where they had found in the earlier paper this flatlining — I really see no difference at all. And when I write it up in the ultimate paper, I compare, well, what’s the slope below the point where they said it stops increasing and the point above that? The slopes differ by less than one percent. I really see no evidence for a difference at all.

DUBNER: Were you initially looking for that plateau in the data? 

KILLINGSWORTH: I was really just trying to understand what’s the relationship between these things that are obviously important? We’ve never had in-the-moment, continuous data. And so I want to see, what does this look like? And it turns out, well, it doesn’t look like what we thought it looked like.  

“When you strike at a king, you must kill him.” That’s from Ralph Waldo Emerson. Matt Killingsworth did strike at the king of his realm, Danny Kahneman. But what happened next? That’s coming up.

*      *      *

We’ve been hearing from Matt Killingsworth about his research on the relationship between income and happiness — research that disputed elements of earlier research on the same topic by Daniel Kahneman.

DUBNER: So you write up the paper, and the paper not quite directly, but quasi-directly addresses the fact that your finding is contra to a significant earlier finding by significant scholars. What happens next? 

KILLINGSWORTH: There’s a fair amount of attention about it and so forth. Maybe a month or two afterwards, I get a note from Barb that she’s been chatting with Danny.

He’s talking about Barb Mellers — a longtime friend of Danny Kahneman, as well as a University of Pennsylvania colleague of Matt Killingsworth.

KILLINGSWORTH: I think both of our attitudes was, we just want to find out the truth. I have no, like, personal attachment to what I found, particularly. And I don’t, other than his initial perhaps irritation, I don’t really think Danny did. It was really just, what do we think is going on? 

DUBNER: Did you envision that perhaps you would then collaborate with Danny on a joint study? 

KILLINGSWORTH: That certainly seemed to be the case once we got into it. But, you know, these kinds of data — took me many years to collect that data. For them, they were a client on an external organization with a lot of resources. And so we ended up doing it by going back and looking at my data, which existed, which made it tractable. I can sort of cut to the chase a little bit of how we did that, if you want. 

DUBNER: I want to go back to Barb for just a second here. Barb, if you could give Danny’s perspective or participation up to the point here, where we’re about to cut to the data. 

MELLERS: The starting point for Danny was the assumption that both data sets were valid. So how could they be so different? There must be a resolution in the data somewhere. And so the task was going to be — Matt became the research assistant, and started doing all the reanalyses on his data. 

DUBNER: How would you characterize the tenor of this adversarial collaboration?  

MELLERS: Really civil and nice. And you know, just the way it should go. 

DUBNER: So, Matt, you said you could cut to the chase. We’re at the chase now. Cut to it. 

KILLINGSWORTH: Essentially the resolution was, if we look at the range within my continuous happiness data, what if we look at the low end, which is the part where their measure would have been sensitive, can we find a similar pattern? And lo and behold, when you zero in on that — rather than looking up the average trend, which keeps going up —  you get not exactly but very, very close to what they found. Certainly much closer to what they found than to what the average trend looks like. We both found that pretty convincing evidence that what they found is true. There’s nothing wrong with the analysis at all. But it was really a question of how generally applicable is that? So, from low to medium incomes, the unhappiest part of the distribution rises a lot. And then for medium to high incomes, the unhappiest part of the distribution barely changes. But the rest of the distribution is rising steadily. And in fact — at the high end of the happiness distribution, you get an inversion of that. So rather than a rise and then a plateau, you have sort of a shallow slope and then an acceleration.

DUBNER: So in my lay-mind, as I’m trying to process all this, what I’m envisioning is that there is a cohort of people who are kind of cranky and they may experience a little bit less crankiness as income rises, but there is a range in which their temperament maybe overwhelms their income. 

KILLINGSWORTH: If anything, there’s one of the steepest slopes for the unhappiest people in the range of low income. So getting out of poverty if you’re really miserable is at least correlationally — 

DUBNER: But that’s almost a different story, because escaping poverty versus going from middle to upper —

KILLINGSWORTH: If you aren’t poor, and you’re really miserable, at least if you sort of extrapolate from there, it seems to not matter past that. And probably, as we speculate in the paper, at that point, if you have a decent amount of money and you’re really unhappy, whatever’s making you unhappy probably isn’t due to the lack of resources. It’s something else going on in your life. Maybe it’s challenges in a family relationship or maybe you’re depressed, or whatever it is, it’s something going on that perhaps money isn’t going to make a difference. 

DUBNER: Richard Thaler, let me ask you a question. You’ve just heard Matt’s presentation of his side of the original research, and then the adversarial collaboration, and then the conclusion, all of which I found honestly to be really fascinating. You know the literature well. You, too, have a Nobel Prize, although I have heard that was a clerical error. How would you, looking at this from above — Barbara was there as an arbiter, Matt was one side, Danny was another side, but is not present — if you could give an ultimate proclamation on not just where this finding arrived, but what the adversarial collaboration produced here, what would you say? 

THALER: Look, I think this is the way it should work. This is a good story. And the world would be a better place — academia would be a much better place if there were more of these kinds of collaborations. 

DUBNER: Would it affect what we sometimes callowly call “the replication crisis”?

THALER: Uhhh. Oof. 

DUBNER: Barbara says — Barbara’s nodding her head no. You say, oof!  

THALER: Yeah. Let’s move on to Shane.  

The discomfort you were hearing there, about the replication crisis — that discomfort is related to a two-part series we recently produced on academic fraud. Episodes 572 and 573. And while we’re on the topic of discomfort, let me add one thing I’m thinking about as we’re in the middle of this conversation about adversarial collaborations. Danny Kahneman is said to have introduced this concept into the realm of behavioral research. But if you’re not a behavioralist, you could look at this as yet another example of how academic researchers “discover” something that people in the real world have been doing forever. There are all sorts of arbiters and referees out there, all sorts of processes for compromise and negotiation — even war games, to test one plan against another, and come up with the best solution. In the same vein, some people look at the big insights that have come out of this behavioralist research as, essentially, Aesop’s fables, with a little bit of math. Earlier this year, we made a series about the late physicist Richard Feynman; he didn’t think the social sciences, like psychology and even economics, should be called “science” at all; he thought they were just too squishy to deserve that name. And as for psychologists discovering the idea of adversarial collaboration — well, Feynman was a junior member of the Manhattan Project, which in its quest to build an atomic bomb brought together the most brilliant and argumentative group of scientists ever assembled; that was adversarial and, yes, it was in the end a successful collaboration. Anyway, back to Chicago, as Richard Thaler suggests — let’s move on to Shane Frederick. He’s a behavioral scientist at Yale who collaborated with Danny Kahneman on several projects, and is cited throughout Kahneman’s book Thinking, Fast and Slow. Kahneman saw Frederick as a protégé, and they had a close relationship — sometimes a bit too close.

FREDERICK: So at like 10:00 on a Monday maybe like third quarter of Monday Night Football, I get a call: “Shane?” He said, “Are you watching the game?” It was one of the rare times I wasn’t. I said no. “Fantastic. I want to talk about heuristics.’” Another time, it was like 2:00 in the morning, we’re going back and forth, back and forth, back and forth, and eventually I’m getting, you know, sleepy. And so, I go to bed and I wake up in the morning, and the very next email from Danny: “Shane, don’t play dead on me.”

One idea they worked on together was the cognitive reflection test, or C.R.T.; it’s meant to measure a person’s ability to override their gut instinct and think more carefully about a problem. Here’s a famous example: A bat and a ball cost $1.10 in total. The bat costs $1 more than the ball. How much does the ball cost? If your first inclination is to say 10 cents — congratulations, you’re like just about everyone else. What the C.R.T. measures is: can you slow down and actually calculate the answer, rather than just go with your gut? If you can, you’d find that the answer is five cents. Remember: the bat costs $1 more than the ball. Shane Frederick has spent much of his career designing such tests.

FREDERICK: Some of the items are novel, some I invented, many of them I didn’t, I collect, some of them I revised. One goes back to at least 1919. There are variants of it.

DUBNER: And they were used in an academic setting, or like in employment? 

FREDERICK: No, just like in books of riddles. Stuff that Maya would read.  

DUBNER: But they weren’t stumpers.

FREDERICK: No, no, they weren’t, they weren’t stumpers.

BAR-HILLEL: I think I must tell the audience what a stumper is, okay? 

That, again, is Maya Bar-Hillel.

BAR-HILLEL: A one-way ride costs $10. A round trip costs $20. A passenger hands the cashier $20, saying absolutely nothing. The cashier knows right away that the passenger wants a round-trip rather than a one-way and change. And the question is: how did the cashier know this? 

DUBNER: Does anyone in the audience who does not know the answer to this stumper previously, want to take a guess to the answer? 

BAR-HILLEL: If you’re stumped, good. Because that’s why they’re called stumpers.

A stumper is unlike one of Shane Frederick’s CRT questions in that your intuition doesn’t produce an obvious but wrong answer.

BAR-HILLEL: It only has to do with my love of riddles, which I share with my partner in this research, Shane Frederick. And we really came to this out of love of riddles. But we’re both professional psychologists, so we felt like we had to approach it from the point of view of psychology. Now, Danny —

DUBNER: Wait, wait! Answer, please. 

BAR-HILLEL: Oh! No way. No, I’m not going to answer. That’s not what I was paid for.

THALER: Alright, I will answer — he handed the ticket agent two tens. 

DUBNER: Ah, very nice.

BAR-HILLEL: Did they pay you for that?

THALER: I’m getting paid the same as you, Maya. 

But there was something more at stake here than answering a riddle. Bar-Hillel, along with a co-author, wrote a paper arguing that cognitive reflection tests are not a good measurement of anything beyond math skills. And this argument led to an adversarial collaboration.

THALER: When I heard that Shane and Maya and Danny were having an adversarial collaboration, I didn’t know who was on which team. Maya and Danny go back 60 years. Shane is like a son for Danny.

The results of this collaboration have not yet been published — but Shane Frederick says that he and Kahneman essentially proved that the CRT is a good measure of cognitive abilities beyond just math skills.

FREDERICK: It seemed to do quite well. And it’s doing well consistently. Time preferences, risk preferences, other sorts of things.

DUBNER: And how would you characterize the nature of the collaboration? How adversarial was it?  

FREDERICK: It wasn’t so adversarial between the opposing groups. It’s just like everybody’s fighting with everybody else about everything.

THALER: It’s more like a family gathering, right?  

DUBNER: Does this suggest that an adversarial collaboration is not as useful as one might hope, if the actual adversaries are not often in the ring with the collaborators?  

THALER: I mean, I don’t I — I don’t think it will work. Do you, Barb? 

MELLERS: Do these things work? Well, what does “work” mean? It depends on your definition. If you think people change their minds all the way — no, it doesn’t work. If they change their minds a bit? That’s good. And it speeds up science, too. Somebody’s looking over your shoulder and making sure you’re not making mistakes. You’re defining variables more precisely. You’re designing an experiment that gets right at the core issue. It’s the way to go. 

DUBNER: Well, I could listen to the five of you talk for five hours. There are sessions that need to happen. This room is turning over. I thank you so much, this was a great conversation. Sorry, there is a clause in the contract that Thaler must always have every last word. So — 

THALER: I was just gonna say something nice about you. But if you insist, I’ll skip it.

And that was, again, Richard Thaler; I’d like to thank him for putting this event together and inviting us to join; thanks also to the other participants. It was a pretty wonderful event, and I’m glad we were able to share it with you. And if you need more Richard Thaler in your life — and who doesn’t need more Thaler in their lives? — we’re going to publish a bonus episode very soon, an update of a great conversation I had with Thaler a few years ago; it’s called “People Aren’t Dumb. The World Is Hard.” So keep your ears out for that.

*      *      *

Freakonomics Radio is produced by Stitcher and Renbud Radio. This episode was produced by Zack Lapinski, with live recording by Greg Rippin. A special thanks to conference organizers Amy Boonstra, Mark Temelko, and Chris Partridge, as well as the BlackOak AV team. Our staff also includes Alina Kulman, Augusta Chapman, Dalvin Aboagye , Eleanor Osborne, Elsa Hernandez, Gabriel Roth, Jasmin Klinger, Jeremy Johnston, Julie Kanfer, Lyric Bowditch, Morgan Levey, Neal Carruth, Rebecca Lee Douglas, Sarah Lilley, and Theo Jacobs.