How can we measure the impact of development programmes?

Buzzwords tell you a lot about an organization’s culture. Two that we hear a lot at the Gates Foundation are “impact” and “innovation.” In fact, Bill Gates recently wrote a paperabout the future of development titled “Innovation with Impact.”

There’s just one catch. Getting from innovation to impact, especially in global health, can be a very long process. It can take literally decades to turn a big idea into a prototype that works and finally into an actual product that’s available to all the people who need it.

So, if you define impact as the number of lives saved, how do you gauge success when the prospect of lives saved could be 20 years away? How do you avoid declaring failure too soon? Does lots of success mean you’re tackling too many easy problems?  Are you taking too much credit for things that might have happened anyway?

Our Grand Challenges team at the Gates Foundation, along with colleagues who run affiliated Grand Challenges programs all over the world, spends an inordinate amount of time puzzling over these questions.

A couple weeks ago, 100 of us spent two days in Washington, DC trying to answer them. Some of us work for foundations. Many of us work for governments. For every single one of us, though, the question of impact is a matter of urgency. Whether our money comes from philanthropists or taxpayers, our bosses want to know if it’s being spent wisely.

Our conversation raised just as many questions as it answered, but we’re making progress toward a clear way of expressing the return on investment in innovation in health and development fields.

The problem of success when failure is expected: When we started Grand Challenges 10 years ago, we knew most of the projects we funded would fail, if by fail you mean not lead directly to a life-saving product. The idea was that the successes would provide huge returns and “pay” for the failures many times over. If it works the way we expect it to, the Eliminate Dengue project will by itself save enough money to pay for every single Grand Challenge grant we have ever made. In a way, we’re like a venture capital firm for saving lives. If you read the literature about venture capital, the hit rates cited vary from 10 percent to 33 percent. How people feel about those numbers has a lot to do with their innate risk-tolerance as an individual. If I say that 38 percent of our first batch of Grand Challenges investments are either in line to lead to a new product or have generated scientific insights that are key to a new product, some people will perceive that as success and some will perceive it as failure. In short, being in a failure-oriented enterprise makes it more challenging to persuade people that you’re succeeding.

The challenge of attribution: One of the most interesting conversations we had during our meeting was about how much credit the Saving Lives at Birth Grand Challenge deserves for the fact that Nepal is now using the antiseptic chlorhexidine to clean babies’ umbilical cords—and saving thousands of lives in the process. On the one hand, chlorhexidine had already been shown to be effective and feasible to deliver through public health services in Nepal.  On the other hand, Grand Challenges “transition to scale” funding was critical to its nationwide roll-out as part of a newborn care package in Nepal.  Does that mean nobody would have initiated this scale-up, ever, if not for Grand Challenges? By how many years did the Saving Lives at Birth program accelerate the nationwide use of chlorhexidine in Nepal? How fast will other poor countries start using chlorhexidine, and how much of that impact should be associated with the grant in Nepal? We are not the only innovation programs in health and development, and innovation is not the only way to do health and development. When you include counterfactuals and hypotheticals in your calculations, the number of variables gets very large very quickly, but the important news is that no matter how you distribute credit, important progress is clearly being made.  This challenge of attribution is not unique to social innovation, but is even more acute for projects that make important scientific advances, but do not directly lead to a specific product.

Our Evolving Return on Investment Model: The primary return we’re looking for—that scientists use our money to turn their idea into lives saved—typically reveals itself over the course of years, if not decades. However, while we’re being patient (and rigorous about making sure projects are meeting interim milestones), we can also measure a number of secondary returns on top of the launch of a new product or service. Last year, I wrote a blog post explaining some of these secondary returns. You can read it here, but one of the key points was that we’ve engaged thousands of scientists from all over the world in solving the most pressing problems of the poorest people in the world. Some receive funding from our programs. Others get funding from other places. Even those who aren’t funded in the short term have engaged in thinking about global health and development, and this broader perspective will influence their work over the long term.

At this year’s meeting, we tried to keep improving our return on investment model by focusing on two key benefits missing from the earlier version. First, the original model did not account for the value of advancing knowledge that does not yet directly result in a product or intervention.  Science is a cumulative enterprise, and perhaps some future Grand Challenges success story will benefit from knowledge created by one of the projects we ranked as unsuccessful last year.  Second, we did not account for the value of the research and development capacity being developed by a global innovation program that has now made over 500 grants in 55 low- and middle- income countries.

We may not yet have a silver bullet when it comes to measuring impact. But it was a pleasure to spend two days with innovators whose lives are dedicated to having an impact.

One presenter, Shasha Jumbe, a brilliant colleague of mine from the Gates Foundation who is applying sophisticated mathematical methods to better understand the factors that contribute to growth and development of children, began his presentation by showing a photograph of himself when he was a boy in Rhodesia, now Zimbabwe, and talking about the impact of innovation on his life. We need to make sure that more boys and girls in Zimbabwe and countries like it have the opportunity to survive, to thrive, and to lead the lives they want, and innovation is the most compelling, if the most complex, way to do that.

That’s why this conversation about measuring impact is so important. Because the better we can measure it, the easier it will be for us to prove that innovation changes the world.

This article is published in collaboration with The Bill & Melinda Gates Foundation. Publication does not imply endorsement of views by the World Economic Forum.

To keep up with the Agenda subscribe to our weekly newsletter.

Author: Dr. Buchsbaum received a Ph.D. in Physics and a Master of Pacific International Affairs from the Graduate School of International Relations and Pacific Studies at the University of California, San Diego. 

Image: A passenger plays with his child while waiting to enter a crowded railway station. REUTERS/Anindito Mukherjee.