Winners of Heart + Mind Donations Contest

(iStockphoto)

Earlier I ran a contest for two free copies of More Than Good Intentions. The quick summary: we ran a randomized trial to test whether employing the use of statistics, and worse yet scientific evidence, would raise more or less money when added to a standard emotional appeal in a direct mail marketing solicitation for donations for Freedom from Hunger (a charity I respect and do research with). We split the analysis by size of prior gift. The Freakonomics contest then asked two questions:

  • Guess the percentage point change for treatment relative to control for those who gave less than $100 in the past.
  • Guess the percentage point change for treatment relative to control for those who gave more than $100 in the past.

Before I get to the punchline, there’s an interesting result I see from looking at the guesses that were made. In part, this contest was inspired by the idea of creating a predictions market for randomized trials (or for any sort of well-defined scientific test, for that matter). This contest was not that, but it did provide me a simple way of learning what 73 individuals thought would happen.

The average guess for the small prior donors is an increase of 2.02 percentage points, i.e., people on average thought that mentioning scientific evidence would increase the response rate by 2.02 percentage points.

The average guess for the large prior donors is a smaller increase of just 1.37 percentage points.

So on average everyone thought that mentioning scientific evidence would increase the likelihood someone gives, and people thought that small prior donors would respond more positively than large prior donors.

The answer is the opposite.

Here are the results:

Small prior donors reduce giving when presented with scientific evidence by 0.9 percentage points (statistically significant at 90%). The winner is Will H, who guessed -1.0 percentage points, and with a reason that I think is dead-on too.

Large prior donors increase giving when presented with scientific evidence by 3.54 percentage points (significant at 99%). The winner here is Helen Law, who won by a coin toss over Peter who also guessed +4.0 percentage points).

In fairness to the collective wisdom of the readers, even though the average prediction got it backwards, if I tally the simple vote as to whether large or small donors would respond more positively, more people voted with large donors (i.e., the right answer):  34 out of 65 (52%) versus 23 out of 65 (35%).

One commenter, Aaron Miller, made a key point that should be noted when thinking about how much stock to put in these results when planning fundraising for other charities: Freedom from Hunger is known amongst its supporters and those in the microfinance world as being more focused on using evidence and research to guide their programs. Empirically, data even support this, as they have done more randomized trials than any other microfinance organization, to the best of my knowledge. And they were the first to attempt a randomized trial on microcredit (as far as I know) well before Innovations for Poverty Action (IPA) and the M.I.T. Jameel Poverty Action Lab got started.

Aaron’s point is a larger one about external validity, and is true of any empirical exercise. With some theory on why people give, and more replication (something not rewarded enough in academics, one of the motivations for starting IPA), we can and should tackle this. Personally I’m very interested in this issue of how we can help donors use evidence on effectiveness without turning off the emotional triggers that inspire us to give, and am always looking for nonprofit organizations game for doing tests to help learn how to do this.

Anyway, thanks for participating everyone. This was fun.

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 6

View All Comments »
  1. Beamish says:

    toin coss?

    I have certainly heard that mis-phrase spoken but I don’t think I have ever read outside of a specific discussion of that classic spoonerism. Intentional? Vindictive speech-to-text software?

    Thumb up 0 Thumb down 0
  2. rick davies says:

    Very interesting

    Could you now do a cost-benefit type calculation? What was the net extra income raised (4% up for big donors minus 0.9% down for small donors) and how does this compare to the cost of the original intervention RCT?

    [I recognise that there may be other benefits, and perhaps other costs, which we are not taking into account here]

    Thumb up 0 Thumb down 0
  3. rick davies says:

    See a related post by GiveWell on bigger donor behavior, at http://blog.givewell.org/2011/06/02/profile-of-a-givewell-customer/
    A quote: “Attitudes toward evidence seem less key than we would have guessed. When we started GiveWell, we and most of our supporters imagined that new customers could be found in certain industries where people are accustomed to using measurement to evaluate and learn from their decisions. We hoped these people would resonate with our desire to bring feedback loops into areas where feedback loops don’t naturally exist. But we’ve found that a lot of them don’t, largely because impact isn’t the main thing they’re aiming for when they give. People give for many reasons – to maintain friendships, to overcome guilt and cognitive dissonance, to achieve recognition – and they’re not going to be interested in GiveWell unless achieving impact is at the top of their list.”

    Thumb up 0 Thumb down 0
  4. Brett Keller says:

    I used to work at M+R Strategic Services — an email fundraising and advocacy firm — and we did split tests like these (quick and dirty randomized trials more or less) all the time, and sometimes longer-term experiments. M+R publishes some of the studies at http://labs.mrss.com/ (while others are just shared with the clients) and the results can be pretty interesting. Generally we would try and test something across several organization’s email lists to get more generalizable results.

    Thumb up 1 Thumb down 0
  5. Aaron Miller says:

    I’m grateful you thought my comment was worth repeating, but I’m sad I was wrong on my guesses. (I wasn’t far off with the large donors, but I got the direction wrong with the small donors.)

    It all works out anyway, because I’d already ordered a copy of the book.

    Thumb up 1 Thumb down 0
  6. Steve Bennett says:

    I got the signs right – but I’m surprised that so few others did. It just seemed logical to me that the people making small donations are more swayed by emotion than reason. Probably the people making bigger donations are richer to begin with, and more “economically rational” or something.

    One issue I have with this kind of research is that it amounts to working out which buttons are best to push to get people to respond in a certain way. Wikipedia recently went through this: collecting a huge amount of data to see which of various slogans were the most effective. It could be a very slippery slope to presenting a public profile quite out of step with the behind the scenes reality.

    On the other hand, maybe that’s the only way to attract funding for “unappealing” charities, like cancers of unmentionable places…

    Thumb up 0 Thumb down 0