One of the two editorials of the August issue of the WHO Bulletin was dedicated to Performance-Based Financing (PBF). Fretheim et al. reported the main findings of their Cochrane review published a few months earlier. In the weeks that followed – despite many experts being on holidays and the summer “torpor” – a discussion developed on the online discussion forum of the PBF Community of Practice (CoP). The discussion benefited from contributions by two authors of the review (Atle Fretheim and Sophie Witter). This blog post summarizes the main points of the discussion. It tries to be as objective as possible; you can access the whole exchange of emails on the Resources page of this blog.
Collection, editing, titling and introduction: Bruno Meessen, September 12th 2012.
Launch of the debate
On August 9th, Emmanuel Ngabire (School of Public Health, Kigali), co-facilitator of the CoP, shared the article on the Google group. The first response came two hours later, from Stefaan Van Bastelaere (Belgium Technical Cooperation, Brussels). He expressed his concern on the possible impact of the editorial and his frustration: “The authors reduce PBF to a strategy that generates ‘quantities’, which in my opinion is unfair.”
Bruno Meessen (Institute of Tropical Medicine, Antwerp), lead facilitator of the CoP, then invited members of the discussion group to read the paper and share their views.
Longin Gashubije (Ministry of Health of Burundi, Bujumbura) questioned the narrow definition of PBF used by Fretheim et al. in the editorial: “I think PBF is more than the transfer of money; when well implemented, it allows to transform the whole health system”. He also explained why no randomized study was done in Burundi: the mere observation, through the monitoring system, of improvement of quantitative and qualitative indicators (that had never improved before the implementation of the PBF) turned out to be sufficient to persuade decision-makers to scale up the pilot experience.
Authors of the review are willing to engage in dialogue
On August 10th, Atle Fretheim (Norwegian Knowledge Center for the Health Services, Oslo) joined the group. After introducing himself, he recommended members of the discussion group to read the Cochrane review as well, not only the editorial. He expressed his willingness to engage in a dialogue on how PBF schemes might be evaluated in a practical and feasible, but robust way.
On the same day, Bruno Meessen reacted. The accumulated frustration was palpable – much of his long mail concerns the researchers’ lack of interest (so far) in setting up a real dialogue with PBF implementers. As far as the systematic review is concerned, a better connection with the field and the implementers would have prevented some misunderstandings.
Start of a discussion on the Cochrane review technique applied to health system reforms and interventions
In the same mail, Bruno also shared his personal view (as a researcher) on the limits of systematic reviews. However, he acknowledged that the discussion will be beneficial to strengthening the impact evaluation program on PBF.
Still on the 10th of August, Sophie Witter (University of Aberdeen), first author of the Cochrane Review, answered Bruno. “As a health systems person, I can fully understand your frustrations with the Cochrane process. It has its strengths and limitations, like all methods.” In her mail, she reminded where systematic reviews fit in the body of knowledge. She acknowledged that “it may be a bit early for the evidence on PBF, but these reviews are updated periodically, so that should not be a problem in the longer term” and that “PBF is a particularly tricky topic for systematic reviews as it has been interpreted and implemented in very different ways and very different contexts. She added that “these are all issues we highlight in the review. We call for more research on systemic effects and the relationship to different contexts.”
On August 11th, Por Ir (National Institute of Public Health, Cambodia ) – who just finalized a (non-systematic) review of evidence on RBF in maternal and neonatal health on behalf of the German Development Cooperation – shared his surprise “to see many reactions to this editorial, but not to the Cochrane review itself when it was published. Por did read the Cochrane review, and found it “well written with very transparent Cochrane recommended methods and well balanced findings and conclusions. So, if we want to challenge the findings, we better challenge the Cochrane review methods (mainly for health system research), but not the authors.”
A Cochrane review too keen on including enough studies?
The debate quieted for nearly a week, but was then re-launched by Joanne Harnmeijer (ETC Crystal, Netherlands) on August 17th. She came back on one of the points raised by Bruno, his assessment that the Cochrane review came too early. The discussion which followed provided the CoP members with some insight into the tensions internal to the systematic review. To Joanne, the review was unfair: the reviewers included data, reports or studies which did not necessarily pursue Cochrane standards and then in the next step of the assessment process, they correctly gave these studies a very low score of rigor.
On the same day, Sophie Witter reacted, providing more information on the approach taken and how it complied with Cochrane standards. Joanne responded still on the same day, reiterating her point. She emphasized Atle’s recommendation to have “a debate on how PBF-schemes might be evaluated – in a practical and feasible, but robust way” – is an important one. Her own contributions fit in this agenda.
Joanne’s mail also triggered a response by Atle. He commented on the selection criteria. He reminded us of a few Cochrane rules that need to be respected if the review concerns a health systems intervention. He also wrote: “We do not criticize authors of the original studies. It is not our intention, anyway. On the contrary! They may have conducted the best study possible given the circumstances. I would like to add that we are very thankful to many of the original authors. Several of them responded promptly to our e-mails and even sent us their full datasets. I think Cochrane reviewers in general struggle much more than we did when trying to engage study authors and getting access to unpublished data. So, we are very thankful, indeed!”
Still on the same day, Joanne answered Atle. The disagreement remains.
How to deal with the contextual factors in the assessment of PBF schemes?
On August 21st, Eric Bigirimana (AEDES and BREGMANS Consulting, Cameroon) came back on the importance of the context for the design and effectiveness of PBF schemes. He illustrated that with incisive observations made by participants of a study tour in three African Great Lakes countries. Eric is also a researcher. He reckoned an alternative research approach – the realist evaluation one – would take contextual factors better into account. In his long mail, he argued why.
Sophie, who is very familiar with the realist evaluation approach (she is currently the lead coordinator of a research project on selective free maternal health care which partly relies on this approach), responded:
“I think that the realist evaluation approach is very interesting and agree that it would be well applied to PBF. For the Cochrane methods, if there are enough robust studies, then you can look for context patterns. Unfortunately, if you only have a few (as was the case for the PBF review), then that is ruled out. But I would just note that the methodology in itself is not unable to take into account contextual differences.”
Contribution by Robert Soeters: a synthesis, some other criticisms and a way forward in terms of research
On August 22nd, Robert Soeters (SINA Health, Netherlands) sent his contribution. In his long mail, Robert provided some background on how he had been involved in the systematic review process. He felt that his commitment to transparency had not been rewarded by a similar commitment on the side of the reviewers. If field workers had been given the opportunity to give early feedback on the findings of the review, that would have been a valuable validation process for him. Robert also explained how current PBF knowledge has developed gradually over time through accumulation of experience and hinted that adoption of PBF best practices is another kind of validation process, one in terms of relevance. His assessment of the review was that “the result is a biased set of recommendations and some conclusions, which are communicated out of context.” He then developed this point by providing some more information on a few countries he has been working in over the past years.
In the rest of his mail, Robert put forward a few ideas for a different approach on how to conduct PBF research. He raised different concerns, which matter for this future agenda. One is to find research strategies which do not harm the policy process, for instance randomization across districts (and not within) and gradual roll out of PBF “whereby at first a number of districts are included in the PBF intervention and control districts that are not (yet) included in PBF.”
As for systematic reviews, he insisted on focusing on homogenous interventions. “ There is a quickly growing consensus in low- and mid –income countries on the definition of PBF and it should be avoided that studies are included of projects that do not qualify as PBF. It is unhelpful to study a project that is not PBF and then draw conclusions as if it is PBF. Cases in point were studies on Uganda (Palmer et al) and Zambia (Cochrane Review).” Similarly to others, he stressed the importance of context.
He concluded by endorsing the common objective of working on research methodologies. “We invite the academic world to constructively engage on the PBF health reforms and thereby to improve the research agenda.”
A rebuttal letter by Atle and a shared commitment to a strong research program on PBF
On the same day, Atle responded to Robert, point by point. However, for an unknown reason, the mail never reached the Google group. In his email, Atle reminded CoP members of some of the rules of a Cochrane review. He also provided guidance on how authors of primary studies could help systematic reviewers (e.g. provide more background information on context). In a private email, four days later, Robert thanked Atle and Sophie for their feedback in the CoP and concluded: "We are very happy that a serious discussion about how to accompany the promising PBF developments in many parts of the world with serious research is now well underway. Of course, you understand that I still beg to differ on a variety of points with you, but at least a dialogue has been opened which we really welcome. (...) We are looking forward to further debates with you and thank you once again for at least having recommended further research in this field. There, we fully agree!"
On August 25th, after the announcement by a member of the approbation by the Burundian parliament of an IDA grant of US$ 14,8 million for PBF, the Google group discussion veered off into another direction – on the sustainability of PBF, in French first.
Obviously, though, the discussion on the best research strategy for PBF interventions will go on. Feel free to contribute via the comment section.
Collection, editing, titling and introduction: Bruno Meessen, September 12th 2012.
Launch of the debate
On August 9th, Emmanuel Ngabire (School of Public Health, Kigali), co-facilitator of the CoP, shared the article on the Google group. The first response came two hours later, from Stefaan Van Bastelaere (Belgium Technical Cooperation, Brussels). He expressed his concern on the possible impact of the editorial and his frustration: “The authors reduce PBF to a strategy that generates ‘quantities’, which in my opinion is unfair.”
Bruno Meessen (Institute of Tropical Medicine, Antwerp), lead facilitator of the CoP, then invited members of the discussion group to read the paper and share their views.
Longin Gashubije (Ministry of Health of Burundi, Bujumbura) questioned the narrow definition of PBF used by Fretheim et al. in the editorial: “I think PBF is more than the transfer of money; when well implemented, it allows to transform the whole health system”. He also explained why no randomized study was done in Burundi: the mere observation, through the monitoring system, of improvement of quantitative and qualitative indicators (that had never improved before the implementation of the PBF) turned out to be sufficient to persuade decision-makers to scale up the pilot experience.
Authors of the review are willing to engage in dialogue
On August 10th, Atle Fretheim (Norwegian Knowledge Center for the Health Services, Oslo) joined the group. After introducing himself, he recommended members of the discussion group to read the Cochrane review as well, not only the editorial. He expressed his willingness to engage in a dialogue on how PBF schemes might be evaluated in a practical and feasible, but robust way.
On the same day, Bruno Meessen reacted. The accumulated frustration was palpable – much of his long mail concerns the researchers’ lack of interest (so far) in setting up a real dialogue with PBF implementers. As far as the systematic review is concerned, a better connection with the field and the implementers would have prevented some misunderstandings.
Start of a discussion on the Cochrane review technique applied to health system reforms and interventions
In the same mail, Bruno also shared his personal view (as a researcher) on the limits of systematic reviews. However, he acknowledged that the discussion will be beneficial to strengthening the impact evaluation program on PBF.
Still on the 10th of August, Sophie Witter (University of Aberdeen), first author of the Cochrane Review, answered Bruno. “As a health systems person, I can fully understand your frustrations with the Cochrane process. It has its strengths and limitations, like all methods.” In her mail, she reminded where systematic reviews fit in the body of knowledge. She acknowledged that “it may be a bit early for the evidence on PBF, but these reviews are updated periodically, so that should not be a problem in the longer term” and that “PBF is a particularly tricky topic for systematic reviews as it has been interpreted and implemented in very different ways and very different contexts. She added that “these are all issues we highlight in the review. We call for more research on systemic effects and the relationship to different contexts.”
On August 11th, Por Ir (National Institute of Public Health, Cambodia ) – who just finalized a (non-systematic) review of evidence on RBF in maternal and neonatal health on behalf of the German Development Cooperation – shared his surprise “to see many reactions to this editorial, but not to the Cochrane review itself when it was published. Por did read the Cochrane review, and found it “well written with very transparent Cochrane recommended methods and well balanced findings and conclusions. So, if we want to challenge the findings, we better challenge the Cochrane review methods (mainly for health system research), but not the authors.”
A Cochrane review too keen on including enough studies?
The debate quieted for nearly a week, but was then re-launched by Joanne Harnmeijer (ETC Crystal, Netherlands) on August 17th. She came back on one of the points raised by Bruno, his assessment that the Cochrane review came too early. The discussion which followed provided the CoP members with some insight into the tensions internal to the systematic review. To Joanne, the review was unfair: the reviewers included data, reports or studies which did not necessarily pursue Cochrane standards and then in the next step of the assessment process, they correctly gave these studies a very low score of rigor.
On the same day, Sophie Witter reacted, providing more information on the approach taken and how it complied with Cochrane standards. Joanne responded still on the same day, reiterating her point. She emphasized Atle’s recommendation to have “a debate on how PBF-schemes might be evaluated – in a practical and feasible, but robust way” – is an important one. Her own contributions fit in this agenda.
Joanne’s mail also triggered a response by Atle. He commented on the selection criteria. He reminded us of a few Cochrane rules that need to be respected if the review concerns a health systems intervention. He also wrote: “We do not criticize authors of the original studies. It is not our intention, anyway. On the contrary! They may have conducted the best study possible given the circumstances. I would like to add that we are very thankful to many of the original authors. Several of them responded promptly to our e-mails and even sent us their full datasets. I think Cochrane reviewers in general struggle much more than we did when trying to engage study authors and getting access to unpublished data. So, we are very thankful, indeed!”
Still on the same day, Joanne answered Atle. The disagreement remains.
How to deal with the contextual factors in the assessment of PBF schemes?
On August 21st, Eric Bigirimana (AEDES and BREGMANS Consulting, Cameroon) came back on the importance of the context for the design and effectiveness of PBF schemes. He illustrated that with incisive observations made by participants of a study tour in three African Great Lakes countries. Eric is also a researcher. He reckoned an alternative research approach – the realist evaluation one – would take contextual factors better into account. In his long mail, he argued why.
Sophie, who is very familiar with the realist evaluation approach (she is currently the lead coordinator of a research project on selective free maternal health care which partly relies on this approach), responded:
“I think that the realist evaluation approach is very interesting and agree that it would be well applied to PBF. For the Cochrane methods, if there are enough robust studies, then you can look for context patterns. Unfortunately, if you only have a few (as was the case for the PBF review), then that is ruled out. But I would just note that the methodology in itself is not unable to take into account contextual differences.”
Contribution by Robert Soeters: a synthesis, some other criticisms and a way forward in terms of research
On August 22nd, Robert Soeters (SINA Health, Netherlands) sent his contribution. In his long mail, Robert provided some background on how he had been involved in the systematic review process. He felt that his commitment to transparency had not been rewarded by a similar commitment on the side of the reviewers. If field workers had been given the opportunity to give early feedback on the findings of the review, that would have been a valuable validation process for him. Robert also explained how current PBF knowledge has developed gradually over time through accumulation of experience and hinted that adoption of PBF best practices is another kind of validation process, one in terms of relevance. His assessment of the review was that “the result is a biased set of recommendations and some conclusions, which are communicated out of context.” He then developed this point by providing some more information on a few countries he has been working in over the past years.
In the rest of his mail, Robert put forward a few ideas for a different approach on how to conduct PBF research. He raised different concerns, which matter for this future agenda. One is to find research strategies which do not harm the policy process, for instance randomization across districts (and not within) and gradual roll out of PBF “whereby at first a number of districts are included in the PBF intervention and control districts that are not (yet) included in PBF.”
As for systematic reviews, he insisted on focusing on homogenous interventions. “ There is a quickly growing consensus in low- and mid –income countries on the definition of PBF and it should be avoided that studies are included of projects that do not qualify as PBF. It is unhelpful to study a project that is not PBF and then draw conclusions as if it is PBF. Cases in point were studies on Uganda (Palmer et al) and Zambia (Cochrane Review).” Similarly to others, he stressed the importance of context.
He concluded by endorsing the common objective of working on research methodologies. “We invite the academic world to constructively engage on the PBF health reforms and thereby to improve the research agenda.”
A rebuttal letter by Atle and a shared commitment to a strong research program on PBF
On the same day, Atle responded to Robert, point by point. However, for an unknown reason, the mail never reached the Google group. In his email, Atle reminded CoP members of some of the rules of a Cochrane review. He also provided guidance on how authors of primary studies could help systematic reviewers (e.g. provide more background information on context). In a private email, four days later, Robert thanked Atle and Sophie for their feedback in the CoP and concluded: "We are very happy that a serious discussion about how to accompany the promising PBF developments in many parts of the world with serious research is now well underway. Of course, you understand that I still beg to differ on a variety of points with you, but at least a dialogue has been opened which we really welcome. (...) We are looking forward to further debates with you and thank you once again for at least having recommended further research in this field. There, we fully agree!"
On August 25th, after the announcement by a member of the approbation by the Burundian parliament of an IDA grant of US$ 14,8 million for PBF, the Google group discussion veered off into another direction – on the sustainability of PBF, in French first.
Obviously, though, the discussion on the best research strategy for PBF interventions will go on. Feel free to contribute via the comment section.
