Why Ranking Charities by Administrative Expenses is a Bad Idea

How does one know whether a charitable donation will make an impact? For this we need a simple formula (easy to write, hard to apply):

Idea  X Implementation = Bang for your buck

When I give talks about aid effectiveness, people often comment that they too think this is important. And to make sure they are supporting good charities, they always hone in on the charities’ finances to see how much goes to administrative and fundraising expenses. Charity Navigator, for example, scrubs these numbers and doles out stars to charities that don’t spend “too” much on operations.

Given the title of my book with Jacob Appel, More Than Good Intentions, many assume that they are speaking my language, and that I admire such focus on those numbers too.

But I do not. Those numbers do not tell you what is really happening.

(Hemera)

For assessments of organizations, I like a website called Givewell.org. I may not agree with every assessment they make, but they are the best I’m aware of at doing this. And I’m not sure I could do any better. It is always easier to criticize than to improve. Givewell scrubs for all sorts of information, both publicly available and also through direct communications with charities. They look to understand what the charity does, and how much room they have for growth. Last but not least, they assess whether the idea of what the organization does has been shown to be effective, i.e., the first part of the above formula. (Disclosure: Givewell uses research from Innovations for Poverty Action extensively to gather such evidence, so naturally I agree with many of their conclusions, but we have no formal relationship. Givewell is considering Innovations for Poverty Action for a recommendation.)

So for ratings of organizations—not just ideas—as far as I’m aware, Givewell comes closest to assessing the above formula. When people ask me where to send money to maximize its impact—aside from my beloved IPA of course—this is where I usually send them.

This past week we did a simple exercise. We looked up the fundraising and administrative expenses of each of the U.S. 501(c)3 charities that Givewell rated. We calculated whether those rated better by Givewell had lower administrative and fundraising expenses. They did not. In fact, the opposite is true. The higher rated charities by Givewell have higher administrative and fundraising expenses.

The mean of the fundraising expense ratio for charities (the money spent on fundraising divided by the total expenses of the entire organization) that earned a gold, silver, or notable rating by Givewell (41 charities) is 0.073, while that of the charities reviewed by Givewell but not ranked well (253 charities) is 0.054 (p-value of 0.05, i.e., statistically significant at the 95% level). The mean of the administrative expense ratio for charities that earned a gold, silver, or notable rating by Givewell is 0.102, while that of the charities reviewed by Givewell but not ranked well is 0.092 (p-value of 0.35). Adding administrative and fundraising together, the mean ratio for charities that earned a gold, silver, or notable rating by Givewell is 0.174, while that of the charities reviewed by Givewell but not ranked well is 0.147 (p-value of 0.11).

See below for three charts which show the distributions (the red line (“recommended” in some way) is consistently to the right of the blue line (“reviewed but not recommended”).

Does this mean ignore expenses entirely? No, I’d say that huge outliers—charities whose expenses are much higher than their peers’ doing similar work—should be approached with caution. But within the reasonable expense band that most charities are in, I simply ignore these numbers. There are many reasons why. To touch on one: they are really easy to manipulate. Just Google “Feed the Children” and see the accounting shenanigans behind gifts-in-kind (amongst other nasty stuff). In short, overvalue the gifts-in-kind you receive and voilà, your program services just got bigger, holding everything else constant (and the corporation gets a bigger tax write-off too). Garbage in, garbage out. For more on this, check out Uncharitable by Dan Pallotta.

 

What sources do Freakonomics readers like to use to guide their decisions?

 

Leave A Comment

Comments are moderated and generally will be posted if they are on-topic and not abusive.

 

COMMENTS: 9

View All Comments »
  1. Mike B says:

    Hidden due to low comment rating. Click here to see.

    Disliked! Like or Dislike: Thumb up 28 Thumb down 37
  2. John B says:

    One basis for deciding which charity to support is public vs. private funds.

    If two charities claim to do similar work and one is primarily private donations and the other primarily governmental grants or contracts, we go with the private funds.

    Thumb up 3 Thumb down 1
  3. CMT says:

    Many not-for-profits and charities simply do pay well.

    I’d venture a guess that this stems at least partly from not wanting to be perceived as “wasting” money on admin expenses, in addition to other factors.

    I think that the charities that have higher expenses are rated higher by Givewell because they are offering salaries comparative enough to attract talented people.

    Well-loved. Like or Dislike: Thumb up 9 Thumb down 1
  4. Nancy Bacon says:

    I greatly appreciate the subject here, something that I wrote about on my own blog: http://socialchangecollaboratory.org/2011/05/03/give-better/ (where I mention your book and GiveWell.org.) I have worked with small, effective grassroots NGOs for 10 years, and I have reached a different conclusion that makes me less keen on GiveWell: It is just as bad to rank international charities based on “first world” tools developed without understanding poor people’s realities. The NGO leaders I work with in Brazil and Mexico are working on a range of initiatives aimed at changing the society that keeps their people in poverty. This is long-term work that must be done on their terms because they know their culture. Of course we provide and build capacity around financial monitoring and evaluation tools to bridge their work with the needs of funders, but judging a social change organization based on evaluations posted on their websites is unreasonable. The requirement for external evaluations knocks out at least 90% of small to mid-size NGOs. GiveWell proves this. I work in the field of poverty alleviation through education. Indeed, the only education project anywhere in the world that Givewell endorses is Pratham in India. Its external evaluation was conducted by MIT’s Poverty Action Lab, for which a Givewell board member worked in 2007. No other program passes its tests? Really? We need to know that the programs we fund are effective, but the systems we use to determine this need to be inclusive of the experience of the people we are trying to serve.

    Well-loved. Like or Dislike: Thumb up 13 Thumb down 0
    • Mitch Hinz, WWF says:

      Dean, I could not agree with Nancy more.

      As a 35-year veteran of the NGO sector, I’ve worked in groups of all sizes, and it is FAR easier for the larger groups to “slide their numbers around” than the small ones, when in fact it is (as you conclude, though I didn’t see any overall budget mean on your “n”) small groups that usually have higher overhead, often because they don’t have large budgets for fundraising (which is what I do).

      For the 25 years I was US-based, I used to fight with Charity Navigator on using cost-of-fundraising as a PRIMARY indicator of how good a charity was, for the same reason: the fact that many US groups use direct mail (which costs a lot, but also has “educational value”) makes that a better indicator of one’s accounting firm acuity than one’s fundraising practices.

      Now that I work internationally, Nancy’s point is even clearer: there are literally hundreds of thousands of nonprofits operating under the radar of groups like GiveWell. And I”m sorry, but I was completely offended by their “take” on the KONY 2012 issue – and I posted that on their site.

      Finally, speaking as someone who has never even been able to afford to own his own house, do GiveWell and other “measurement groups” take into account how much overhead is SAVED, year after year, by having people at upper management who could make 2 or 3 times the salary, but instead work at such low wages because they care about the issues? Because I can tell you, working at a small NGO can often barely put a roof over one’s head, albeit a rented one.

      Respectfully (I have an Econ and Soc degree, and love your work),
      Mitch

      Thumb up 4 Thumb down 0
      • Julia Wise says:

        >do GiveWell and other “measurement groups” take into account how much overhead is SAVED

        Yes.

        One thing GiveWell measures is cost-effectiveness. They look at the amount of donations going into an organiation and the effects accomplished with that money, and they use that to calculate the cost to save a life (or a disability-adjusted life year, or other metrics).

        A charity that uses money well in any way (whether it’s by having great leadership or spending little on staff salaries or whatever) is going to look better by that metric. The metric cares what goes into the charity and what comes out, and doesn’t rate what goes on in the middle. That said, GiveWell is not wild about cost-effectiveness estimates as the best way to evaluate charities.

        As for size, GiveWell’s top-rated charity for several years was Village Reach, which was on the small side – they had a budget of just over $1 million when GiveWell first recommended them. So it’s not impossible for small, excellent charities to get noticed, provided they provide good data on what they’re accomplishing.

        Thumb up 1 Thumb down 0
    • Julia Wise says:

      I love the idea of targeting the problems local people identify as important to them. I used to work for an organization that did a lot of that. But when I wanted to know how effective these programs were, there wasn’t much to show for them.

      I hope for a day when more strategic anti-poverty methods have good evidence showing they’re cost-effective. But until I see that, I’m going to donate where I know my money will help people most. And I think GiveWell’s methods are good at finding those proven interventions.

      Thumb up 0 Thumb down 0
  5. rick davies says:

    This issue has been covered before by others, and needs continued coverage.

    “Economy does not equal effectiveness” is one message that many people still need to hear and understand.

    For more on the same issue see “The Worst Question to Ask About a Charity” at http://evaluatingkatine.wordpress.com/2009/09/21/the-worst-question-to-ask-about-charity/

    Thumb up 0 Thumb down 0
  6. Joey Boots says:

    So what you’re saying is maybe that year Marine Toys For Tots raised millions of $$$ and didn’t purchase a single toy might have been a GOOD THING?

    Interesting.

    Thumb up 2 Thumb down 3