Heart + Mind? Or Just Heart? Experiments in Aid Effectiveness (And a Contest!)

(iStockphoto)

When signing our book, More Than Good Intentions, Jacob Appel and I often sign “Heart + Mind = Good Giving.” Nobody argues with the premise that we should act with compassion, but be smart about it. And of course nobody would ever say they do not care about the effectiveness of the charity they support.

But in practice, does evidence about charitable effectiveness impact donations? Or does the presentation of dorky evidence turn off the emotions that cause us to donate in the first place? If people donate for the stated purpose of the charity (reduced poverty for example), then of course they will strongly prefer charities that give them more bang for their buck. On the other hand, if it’s the giving itself – for the social recognition, the warm glow of having donated, or something else – then charitable effectiveness may not be so important. Naturally we may say we give in order to further the mission of the charity. But we all say lots of things; what is interesting (to us empirical economists) is figuring out what people really do, and to which incentives they really respond.

Here I had an opportunity for a bit of a meta-study. I did an experiment with Freedom From Hunger, a nonprofit that works to improve microfinance around the world, to measure its effectiveness incorporating business training into a microcredit program in Peru. The paper is here and was written about in the New York Times (here). The program showed positive impacts on income in bad months for recipients of the training, compared to those who did not get offered training (as part of a randomized control trial).

So stage two is now to go to Freedom From Hunger donors and see if this information impacts fundraising.

What do you think, will it help or hurt? I’ve often thought it would be interesting to have a market for results from randomized trials. Not only could people be forced to put their money where their mouth is regarding their predicted results, but we could also generate market predictions about likely efficacy of different ideas. Sort of like the beloved Intrade. (R.I.P John Delaney) Thinking about such a market gave me a simple idea for a contest: I’d like to see the collective opinions on what we will find when we present evidence on effectiveness, rather than mere human interest stories about recipients of the aid.

To make it interesting, I will offer an incentive: two free (signed “Heart + Mind”!) copies of my book, More Than Good Intentions, to the two people who guess the closest to the two estimates. Keep reading for full details on the study and how to enter.

Study details:

Freedom From Hunger (FFH) sends solicitations to people who have donated in the past, as part of its regular fundraising efforts. In this study, previous donors were assigned, at random, to either receive a standard mailing or a special mailing that emphasized the results of a randomized trial showing that FFH was effective in helping poor microcredit borrowers. We then tracked whether each person who received a mailing donated to FFH again, and if so, how much money they gave. FFH sent different mailers to their large prior donors, and we thought this may generate important heterogeneities too. We also thought frequency of prior gifts may influence the treatment effect. We hypothesized that people who donated more frequently or in larger amounts in the past might be more likely to actually read and pay attention to mailings from FFH, or may put more thought into their contribution.

Here are the exact treatment and control scripts (the rest of the solicitations were identical):

Treatment Mailer

In order to know that our programs work for people like Rita, we look for more than anecdotal evidence. That is why we have coordinated with independent researchers to conduct scientifically rigorous impact studies of our programs. In Peru they found that women who were offered our Credit with Education program had 16% higher profits in their businesses than those who were not, and they increased profits in bad months by 27%!  This is particularly important because it means our program helped women generate more stable incomes throughout the year.

These independent researchers used a randomized evaluation, the methodology routinely used in medicine, to measure the impact of our programs on things like business growth, children’s health, investment in education, and women’s empowerment.

Postscript at bottom of letter:

Rita is one of more than a million women Freedom from Hunger serves. We work hard to deliver services that make a difference – and we employ rigorous research to keep us on track, to maximize our impact on women and their children all over the world. Your prior gifts have made these measurable impacts possible. Please continue to help us and make a gift today!

Control Mailer

Many people would have met Rita and decided she was too poor to repay a loan. Five hungry children and a small plot of mango trees don’t count as collateral. But Freedom from Hunger knows that women like Rita are ready to end hunger in their own families and in their communities.

Postscript at bottom of letter:

Rita is one of more than a million women Freedom from Hunger serves – women who continue to prove the power of credit and education in the hands of a determined mother. Your prior support has been an essential ingredient in this worldwide recipe for financial security. Please continue to help us and make a gift today!

How to enter:

In the comments section, make two guesses:

1)     The percentage point change for treatment relative to control (make a positive number mean treatment did better than control…. Eg a 41.0% response rate for treatment and a 38.5% response rate for control would generate a “guess” of 2.5 percentage points) for those who gave less than $100 in the past.

2)     The percentage point change for treatment relative to control (make a positive number mean treatment did better than control…. Eg a 41.0% response rate for treatment and a 38.5% response rate for control would generate a “guess” of 2.5 percentage points) for those who gave more than $100 in the past.


Whoever comes closest on each estimate will get a free book.  Ties will be decided randomly.

Thanks for participating everybody. See here for the winners.

 


AaronS

Some time back, I determined to "follow" my compassion. Now, compassion is not just the standard "I feel bad about that" that we all may feel when we see well-done commercials by charitable organizations. No, compassion, I believe, is something far deeper. It grips you to the point of tears, prays, and wholehearted giving.

For most of my lifetime, I gave to projects because they seemed to be worthwhile. But I found that if I will give when my compassion is triggered, and if you will give when your compassion is triggered, everything will be taken care of. You may have a thing for stray cats, while I may have a heart for children in Ecuador. But if EVERYONE will simply act on their true compassion--and not just on the carnival barking of some charities--I am convinced that the work that needs to be done WILL be done.

That being said, I think in those cases UNDER $100, you will see the Control Mailer be must more effective, while in those giving OVER $100 (and perhaps because bigger givers likely also calculate the effectiveness of the charity), you will see that the Treatment Mailer will be more effective.

Let compassion be your compass.

Read more...

Quin

1. 3%
2. 12%

Ben

1. 25%
2. -1%

leo Piccioli

12.01%
0.01%

Aaron Miller

This is a cool idea. I teach my students that it boils down to warm glow for most donors, and that even a preference for impact ultimately translates into warm glow that motivates the donation. (Prestige utility doesn't usually come into play with a typical mailer.)

My guess:
1. 4
2. 2

I do wonder how insightful the results will be for most nonprofits. FFH has a strong reputation for research-backed development, a reputation that I suspect extends to its donor base. (They also operate in an industry that draws impact-oriented donors: microcredit.) Because these are prior donors, their participation may not change much because they already presume that FFH has rigorously demonstrated impact. A similar appeal might not make much difference for a typical nonprofit, but for different reasons: their donors don't really care about rigorously demonstrated impact.

I also wonder how the length of the treatment mailer's message affects the participation. It's more than twice as long as the control mailer's message. The graphic design that goes into fitting the longer message into the same space could negatively affect it. Also, are the "16%" and "24%" going to be called out in the text? I'd argue that you could send out two versions of the treatment mailer, with even slight variations in text design, and see significantly different results among those who might be persuaded by the message in the treatment mailer.

Finally, I think donors that give over $100 in response to mailers are more likely to participate out of loyalty than donors who give less than $100, so the treatment mailer will make less of a difference for them.

Of course, it's more likely than not that I haven't thought of something else and I'm way off. :)

Read more...

Rick Yagodich

1. +0.3%
2. -12.8%

Dylan

1) 10%
2)-20%

I think the economic slow down would impact the giving of larger donations. But I'm glad as you say "Not only could people be forced to put their money where their mouth is regarding their predicted results, but we could also generate market predictions about likely efficacy of different ideas"

Eric M. Jones.

Forgive my skepticism, but can any charities show that their work has produced measurable and long-term results? Now, I know this sounds cynical, but the unexpected consequences are so large in the developing world that this must be addressed.

My guess is that colonialism was the best thing for most countries. When they left, the whole thing usually went to hell.

DanV

My guesses
1) -3.3%
2) 14.5%

Dave

1) 10%
2) 3%

Meredith T

1) - 2.5%
2)5%

Katie B

My guesses:
1. -15%
2.+8%

My experience working with a children's charity is that most small donors are more moved by anecdotes than data. However this may be very different for microfinance donors!

Ben New

1. 0.2%
2. 0.2%

Andy

1) -31.6%
2) 14.8%

MarioH

I think that there needs some sort of surprise in the numbers or Dean wouldn't be incentivised to such an unique poll. As such I'll say:

1) -2.5%
2) +0.02%

I figure, as I expect most of us do, that small donors are more motivated by that "warm, fuzzy feeling" rather than researched statistics. More, unemotional text will reduce their likelihood of donating.

Similarly, expect large donors to be largely unaffected given the expectation that their donation is based on other factors beyond emotional response to donation plea. Chose 0.02% so I could be on the high side of Leo. Sorry Leo.

Worth noting that I only thought it worthwhile to bother to enter because I could see that there were only 7 entries so far. Odds sound good to me.

Amanda

My guesses:

1. 6.5%
2. -10%

Steve Bennett

1) -8.2%
2) +6.6%

Huzaifa Saeed

For #1 (-1.5%)
For #2 (+ 6%)

Meaghan

1. 2%
2.17%

Barbara

1. -4
2. +1.5