Verifying results before payment is a core principle of Results-Based Financing (RBF). But verification is also costly. In a recent blog post, Matthieu Antony called for less intensive approaches. Some countries are moving in this direction. Erik Josephon reports on the approach adopted in Zimbabwe.
Results-based financing (RBF) mechanisms have been growing very quickly since Cambodia first initiated one in 1999. They are now commonplace across Africa, in several Latin American countries and several OECD countries also have versions of RBF mechanisms. One of the core principles of RBF is to verify results prior to payment. In many settings the methodology put in place to achieve this has been to verify the results of all quantity indicators, and to conduct an assessment of quality, at every health facility at least once per quarter. However with increasing scrutiny of the non-incentive operating cost of RBF mechanisms, the intensity of verification is becoming the topic of discussion across the RBF community, and in health policy-making circles.
Towards a less costly verification
Specifically questions are being asked as to how to construct a verification methodology which is less onerous, and less costly, but which still provides close to the same level of trust that the services being paid for were actually provided.
In certain settings, such as Afghanistan, sampling has been used to reduce the intensity of verification each quarter. This is certainly an avenue worth pursuing, albeit that non-risk-based sampling does come with disadvantages, such as the possibility that some facilities won’t be visited as often as they need to be. Where the sampling is random, the methodology isn’t designed to focus where the problems may be. After all low data management capacity is not universal – some facilities will have stronger staff than others – and the desire to commit fraud will only be present in a select few facilities. Random sampling does not take these factors into consideration.
A different way of looking at verification is to design the methodology such that it specifically takes into account that error in declaration, resulting from whatever motivation, will not be uniformly distributed across health facilities. That is to say, errors will be larger and more persistent over time in some facilities. With this in mind some examples of what is being called risk-based verification are springing up. This has been happening in Argentina for some years, and is based on the volume of patients at each facility. Another such example is in Zimbabwe.
The evolving verification in Zimbabwe
During a visit to Harare in September 2016, I sat with Dr. Tafadzwa Goverwa who is a Provincial Medical Director, Chenjerai Sisimayi of the World Bank and Arjanne Rietsema of Cordaid. Dr Goverwa and Arjanne Riestema helpfully tag-teamed to set the context of verification in the Zimbabwean RBF.
Dr Goverwa started by explaining how verification was originally conducted in Zimbabwe. Verification was on a monthly basis, with all indicators verified. It was a heavy verification – every facility, every indicator, every month. We applied sanctions to errors in declaration since the start of the program. Arjanne Rietsema clarified that if there were differences between declared and verified of more than 5%, that specific indicator would not be paid at all. This verification was carried out by Cordaid. Dr Goverwa then explained that the reason for changing from the old methodology was largely financial. The sustainability of the program was on their minds. Going to all facilities every month entails a significant cost.
Arjanne pointed out that during the monthly, all facility verification, they weren’t finding that many errors, and started to wonder at the utility of this intensity of verification. The needs changed. At the beginning the verifiers would also build capacity on operational plans, and the workings of the health center committee. Over a period of time, this capacity building became less necessary as facility staff took on their responsibilities.
This last point is important when thinking about the evolution of verification, and particularly its frequency. Verifiers have from the start of RBF also in some cases been supervisors. In order to encourage district or county supervisors to carry out their role, and often to have the resources to do so, the first order verification role has been given to district or county supervisors. With time this dual role can sometimes be seen as conflicted.
So initially in Zimbabwe they conducted monthly verification both because of the capacity building needs, and also because they suspected the level of error between declared and verified results would warrant it. About two years into the program, they noted the level of error wasn’t as large any more, and so they started thinking about a lighter verification methodology. However rather than simply lessen the frequency, they had other ideas.
The adoption of risk-based verification
As part of the preparation for this interview, I also spoke with Petra Vergeer of the World Bank. She explained that around the time that Zimbabwe was considering changing verification approaches, she and three colleagues (including the author of this blog) were in the midst of writing a cross-case analysis of six country case studies on verification in RBF. Petra shared the findings and recommendations from the cross-case analysis with the team in Zimbabwe, and one of the recommendations was to introduce risk-based verification. Petra helped the team think through what triggers to use, based on lessons learned from, for example, Argentina and the United Kingdom.
Dr Goverwa explained that in the risk-based methodology the Zimbabwe team settled on, the review of differences between declared and verified data occurs every six months (in June and December according to the Zimbabwe verification manual). If the average of errors has been under 5% between declared and verified for a given health facility, they are classified as green. Error means the verified result is different positively or negatively from the declared result. Green faciliites are verified once per quarter, and during that visit results are checked in full for one of the months in the foregoing quarter, and a lighter verification is carried out for the other two months in the quarter. The lighter verification involves spot checks of high-volume indicators, or indicators where the volume has moved by more than 20% from the previous month. Where a facility has had an average error rate of between 5% and 10%, they are classified as amber. Amber facilities are visited twice per quarter. Facilities with an average error rate above 10% are classified as red, as are new health facilities, and hospitals due to their high volumes. Red facilities are visited every month.
I remarked that the analytical aspect of this, e.g. identifying which facilities had had errors in each category, looking for high-volume indicators and checking trends between months would be much simplified with a dedicated IT solution. Chenjerai Sisimayi explained that for now, the data is downloaded from the DHIS2 software which is used as a health information system in Zimbabwe, and the analysis is done in Excel. There is a desire, but currently no plans, to create software which would do much of the analytical heavy lifting.
RBF currently covers 394 health facilities in Zimbabwe. As of September 2016, 319 facilities were in the green category, 51 were in the amber, and 34 were in the red, meaning that only 10% of facilities are in the red category. Arjanne Rietsema noted that only a few rural health centers are in that category, and that most of the red facilities are hospitals, which stay red by design, given the financial risk resulting from them receiving large RBF payments. She also noted that they have wanted to include in the verification process an activity to pick one non-incentivized indicator at random and include that in the assessment of error, however this has not yet been implemented. Quality verification is carried out quarterly in all cases, with no risk categorization.
Real benefits
I asked how the revised verification methodology had impacted on cost. Arjanne Rietsema said that a limited assessment had been done, and it had shown an 80% reduction in vehicle costs. And as Chenjerai Sisimayi remarked, the improvement in the quality of data entry has continued in the context of the risk-based verification.
Some within the RBF community would posit that over time, the intensity of verification can reduce because the mindsets of health facility managers changes and they realize that correctly declared data has multiple advantages. Dr Goverwa noted that health facility managers in Zimbabwe now know that incorrectly entered data will result in having to correct it down the line. As a result registers have become more complete, where entire columns used to be empty, they are now filled in. Arjanne Rietsema added that there is in some districts an intentional competition created between health facilities, as their color status is made known publicly. So moving from red to amber, and amber to green can be the source of pride. And the reverse is also true.
Another important element is the sanction policy. Any indicator with a difference between declared and verified of more than 5% is not paid. I mentioned that this is very strict compared to most other RBF implementations where there may be sanctions which are rarely implemented, and others where there are no sanctions at all, in policy or in practice. I asked whether there had been resistance to the sanction policy at the start. Arjanne’s response was that since the sanction policy came in at the start of RBF, no-one questioned it. From the point of view of facility managers, this was part and parcel of the RBF approach. And they very quickly understood its implications on their RBF payments.
It’s worth noting that although Zimbabwe has moved to a risk-based methodology, they have effectively moved from a very heavy initial verification methodology, to one which would still be considered heavy in some settings. Even green facilities are visited once per quarter, which in other places is the starting point from which regulators and purchasers are trying to lighten the burden.
Zimbabwe is blazing a trail for verification in RBF in LICs and LMICs. There will very likely be a number of countries looking to follow suit, with Benin already known to be among them. Verification is a key part of RBF and particularly the theory of change of RBF. Thinking through how to make verification, and with it RBF in general, more cost-effective is an important challenge, and is particularly important for quality of care, which is far more open to gaming than is quantity of services delivered.
Towards a less costly verification
Specifically questions are being asked as to how to construct a verification methodology which is less onerous, and less costly, but which still provides close to the same level of trust that the services being paid for were actually provided.
In certain settings, such as Afghanistan, sampling has been used to reduce the intensity of verification each quarter. This is certainly an avenue worth pursuing, albeit that non-risk-based sampling does come with disadvantages, such as the possibility that some facilities won’t be visited as often as they need to be. Where the sampling is random, the methodology isn’t designed to focus where the problems may be. After all low data management capacity is not universal – some facilities will have stronger staff than others – and the desire to commit fraud will only be present in a select few facilities. Random sampling does not take these factors into consideration.
A different way of looking at verification is to design the methodology such that it specifically takes into account that error in declaration, resulting from whatever motivation, will not be uniformly distributed across health facilities. That is to say, errors will be larger and more persistent over time in some facilities. With this in mind some examples of what is being called risk-based verification are springing up. This has been happening in Argentina for some years, and is based on the volume of patients at each facility. Another such example is in Zimbabwe.
The evolving verification in Zimbabwe
During a visit to Harare in September 2016, I sat with Dr. Tafadzwa Goverwa who is a Provincial Medical Director, Chenjerai Sisimayi of the World Bank and Arjanne Rietsema of Cordaid. Dr Goverwa and Arjanne Riestema helpfully tag-teamed to set the context of verification in the Zimbabwean RBF.
Dr Goverwa started by explaining how verification was originally conducted in Zimbabwe. Verification was on a monthly basis, with all indicators verified. It was a heavy verification – every facility, every indicator, every month. We applied sanctions to errors in declaration since the start of the program. Arjanne Rietsema clarified that if there were differences between declared and verified of more than 5%, that specific indicator would not be paid at all. This verification was carried out by Cordaid. Dr Goverwa then explained that the reason for changing from the old methodology was largely financial. The sustainability of the program was on their minds. Going to all facilities every month entails a significant cost.
Arjanne pointed out that during the monthly, all facility verification, they weren’t finding that many errors, and started to wonder at the utility of this intensity of verification. The needs changed. At the beginning the verifiers would also build capacity on operational plans, and the workings of the health center committee. Over a period of time, this capacity building became less necessary as facility staff took on their responsibilities.
This last point is important when thinking about the evolution of verification, and particularly its frequency. Verifiers have from the start of RBF also in some cases been supervisors. In order to encourage district or county supervisors to carry out their role, and often to have the resources to do so, the first order verification role has been given to district or county supervisors. With time this dual role can sometimes be seen as conflicted.
So initially in Zimbabwe they conducted monthly verification both because of the capacity building needs, and also because they suspected the level of error between declared and verified results would warrant it. About two years into the program, they noted the level of error wasn’t as large any more, and so they started thinking about a lighter verification methodology. However rather than simply lessen the frequency, they had other ideas.
The adoption of risk-based verification
As part of the preparation for this interview, I also spoke with Petra Vergeer of the World Bank. She explained that around the time that Zimbabwe was considering changing verification approaches, she and three colleagues (including the author of this blog) were in the midst of writing a cross-case analysis of six country case studies on verification in RBF. Petra shared the findings and recommendations from the cross-case analysis with the team in Zimbabwe, and one of the recommendations was to introduce risk-based verification. Petra helped the team think through what triggers to use, based on lessons learned from, for example, Argentina and the United Kingdom.
Dr Goverwa explained that in the risk-based methodology the Zimbabwe team settled on, the review of differences between declared and verified data occurs every six months (in June and December according to the Zimbabwe verification manual). If the average of errors has been under 5% between declared and verified for a given health facility, they are classified as green. Error means the verified result is different positively or negatively from the declared result. Green faciliites are verified once per quarter, and during that visit results are checked in full for one of the months in the foregoing quarter, and a lighter verification is carried out for the other two months in the quarter. The lighter verification involves spot checks of high-volume indicators, or indicators where the volume has moved by more than 20% from the previous month. Where a facility has had an average error rate of between 5% and 10%, they are classified as amber. Amber facilities are visited twice per quarter. Facilities with an average error rate above 10% are classified as red, as are new health facilities, and hospitals due to their high volumes. Red facilities are visited every month.
I remarked that the analytical aspect of this, e.g. identifying which facilities had had errors in each category, looking for high-volume indicators and checking trends between months would be much simplified with a dedicated IT solution. Chenjerai Sisimayi explained that for now, the data is downloaded from the DHIS2 software which is used as a health information system in Zimbabwe, and the analysis is done in Excel. There is a desire, but currently no plans, to create software which would do much of the analytical heavy lifting.
RBF currently covers 394 health facilities in Zimbabwe. As of September 2016, 319 facilities were in the green category, 51 were in the amber, and 34 were in the red, meaning that only 10% of facilities are in the red category. Arjanne Rietsema noted that only a few rural health centers are in that category, and that most of the red facilities are hospitals, which stay red by design, given the financial risk resulting from them receiving large RBF payments. She also noted that they have wanted to include in the verification process an activity to pick one non-incentivized indicator at random and include that in the assessment of error, however this has not yet been implemented. Quality verification is carried out quarterly in all cases, with no risk categorization.
Real benefits
I asked how the revised verification methodology had impacted on cost. Arjanne Rietsema said that a limited assessment had been done, and it had shown an 80% reduction in vehicle costs. And as Chenjerai Sisimayi remarked, the improvement in the quality of data entry has continued in the context of the risk-based verification.
Some within the RBF community would posit that over time, the intensity of verification can reduce because the mindsets of health facility managers changes and they realize that correctly declared data has multiple advantages. Dr Goverwa noted that health facility managers in Zimbabwe now know that incorrectly entered data will result in having to correct it down the line. As a result registers have become more complete, where entire columns used to be empty, they are now filled in. Arjanne Rietsema added that there is in some districts an intentional competition created between health facilities, as their color status is made known publicly. So moving from red to amber, and amber to green can be the source of pride. And the reverse is also true.
Another important element is the sanction policy. Any indicator with a difference between declared and verified of more than 5% is not paid. I mentioned that this is very strict compared to most other RBF implementations where there may be sanctions which are rarely implemented, and others where there are no sanctions at all, in policy or in practice. I asked whether there had been resistance to the sanction policy at the start. Arjanne’s response was that since the sanction policy came in at the start of RBF, no-one questioned it. From the point of view of facility managers, this was part and parcel of the RBF approach. And they very quickly understood its implications on their RBF payments.
It’s worth noting that although Zimbabwe has moved to a risk-based methodology, they have effectively moved from a very heavy initial verification methodology, to one which would still be considered heavy in some settings. Even green facilities are visited once per quarter, which in other places is the starting point from which regulators and purchasers are trying to lighten the burden.
Zimbabwe is blazing a trail for verification in RBF in LICs and LMICs. There will very likely be a number of countries looking to follow suit, with Benin already known to be among them. Verification is a key part of RBF and particularly the theory of change of RBF. Thinking through how to make verification, and with it RBF in general, more cost-effective is an important challenge, and is particularly important for quality of care, which is far more open to gaming than is quantity of services delivered.
