Ellen van de Poel
On September 18th- 23rd, the World Bank held its Results and Impact Evaluation workshop for Results-Based Financing in Harare, Zimbabwe. Ellen Van de Poel from the Institute of Health Policy and Management, Erasmus University Rotterdam summarizes some of her take home messages.
Spending a week in Harare for the Annual Results and Impact Evaluation workshop for Results-Based Financing turned out to be quite refreshing and stimulating. I was impressed by the enormous amount of work that has been done, both from the implementation and research side. Teams from 23 countries have moved forward with implementing some form of RBF and really interact with and learn from each other.
Better incentivizing quality of care, expanding RBF to community health workers and reducing the cost of verification were some of the hot topics of discussion during the week. I was even more impressed by the genuine interest from policy makers in the impact evaluation (IE) results. These studies, and the wealth of data that has been and will be collected, will undoubtedly contribute to the evidence base on whether, how and why RBF works and what are the most important components of such programs. Having so many relatively similar impact evaluations offers possibilities for generalization that are much more difficult to obtain from a literature review. We should make the most of this opportunity.
Feedback from an afternoon parallel session
On Wednesday 21st, together with a group of researchers, we debated on some of the issues IE researchers are currently struggling with. Here is a list of key points of interest.
Way forward
Perhaps the most important thing I learned at this workshop is that the relevant debate is really not around whether we should continue with RBF or not. The core ideas of RBF (a provider payment reform coupled with improved accountability and autonomy) are indeed essential to health systems strengthening everywhere. The relevant question is whether the RBF vehicle as it is currently used is the most efficient one, and to which extent it needs to be updated or supplemented with demand side incentives. With this energetic group of people on board in all of these projects, I’m hopeful we’ll get answers to many of these questions soon.
On September 18th- 23rd, the World Bank held its Results and Impact Evaluation workshop for Results-Based Financing in Harare, Zimbabwe. Ellen Van de Poel from the Institute of Health Policy and Management, Erasmus University Rotterdam summarizes some of her take home messages.
Spending a week in Harare for the Annual Results and Impact Evaluation workshop for Results-Based Financing turned out to be quite refreshing and stimulating. I was impressed by the enormous amount of work that has been done, both from the implementation and research side. Teams from 23 countries have moved forward with implementing some form of RBF and really interact with and learn from each other.
Better incentivizing quality of care, expanding RBF to community health workers and reducing the cost of verification were some of the hot topics of discussion during the week. I was even more impressed by the genuine interest from policy makers in the impact evaluation (IE) results. These studies, and the wealth of data that has been and will be collected, will undoubtedly contribute to the evidence base on whether, how and why RBF works and what are the most important components of such programs. Having so many relatively similar impact evaluations offers possibilities for generalization that are much more difficult to obtain from a literature review. We should make the most of this opportunity.
Feedback from an afternoon parallel session
On Wednesday 21st, together with a group of researchers, we debated on some of the issues IE researchers are currently struggling with. Here is a list of key points of interest.
- When do we consider a pilot to be successful? Most schemes incentivize many health services, and therefore evaluation studies look at impact on many indicators. Some find effects on ANC, others on institutional deliveries or on vaccinations. Given the variation in scheme design and policy context, it’s difficult to get to a standard evaluation framework. But it might be useful to formulate some clear main hypotheses, perhaps based on the relative pricing of services, the coverage rate at the baseline and on policy priorities, and have these guide the research, rather than looking at all service indicators available in the data and getting enthusiastic when at least one of such effects is statistically significant.
- How to generalize findings across countries and schemes? We learned that the World Bank is planning to conduct a meta-analysis of all the IE results. It will help drawing some overall conclusions on the barriers and enablers for successful implementation of RBF. Levels of integration, credibility of the link between payment and results, power of the incentives, autonomy of the provider, capacity to hire and fire, and baseline levels are likely important aspects to consider in such a meta-analysis.
- What time horizon should we consider in RBF impact evaluations? Most of the IE studies have a two-three years’ time horizon. Participants wondered whether this is not too short to do justice to the real impact as pilots typically take about a year to be fully operational.
- How to explain some of the effects? In some countries, the research design allowed to test different components of the theory of change of PBF (transfer of funds directly to facilities, autonomy, the incentive contract…). In Zambia, and to some extent in Cameroon, IE results suggest that providing additional budget with autonomy to facilities can be equally effective, and therefore more cost-effective, than the RBF programs with incentives. We’re not quite sure yet why this is the case. Are we really measuring the effect of additional budget, or are control areas also indirectly affected by the incentive structure in the treatment areas?
- How to increase the policy relevance of further IE evidence? Robust Randomized Controlled Trials (RCT) designs are important to establish causal impact on main outcomes. This evidence was/is urgently called for, especially given the relatively high implementation costs of RBF programs. But as we move forward, the implementation questions that countries are struggling with do not easily lend themselves for RCT type of designs. So we’ll need to think more about how to use operational or administrative data to provide (quick) answers to these questions.
Way forward
Perhaps the most important thing I learned at this workshop is that the relevant debate is really not around whether we should continue with RBF or not. The core ideas of RBF (a provider payment reform coupled with improved accountability and autonomy) are indeed essential to health systems strengthening everywhere. The relevant question is whether the RBF vehicle as it is currently used is the most efficient one, and to which extent it needs to be updated or supplemented with demand side incentives. With this energetic group of people on board in all of these projects, I’m hopeful we’ll get answers to many of these questions soon.
