Effect of performance-based financing on health service delivery: a case study from Adamawa state, Nigeria

Abstract The Nigeria State Health Investment Project (NSHIP) was implemented in three Nigerian states between 2013 and 2018. Under the NSHIP, some local government areas were randomly assigned to Performance-Based Financing (PBF) intervention while others received decentralized facility financing (DFF) for comparison. This article evaluates the effect of PBF compared with DFF on health service delivery indicators in Adamawa state, under this quasi-experimental design, using the difference-in-differences technique. The analysis used health facility monthly data collected by the Health Management Information System through the District Health Information Software 2 (DHIS2). The PBF intervention group significantly increased the quantity of most of its service delivery indicators, such as antenatal care visits and deliveries by skilled personnel compared with the comparison group (DFF) after the introduction of NSHIP, although the baseline level of service delivery between PBF and DFF health facilities was statistically identical prior to the introduction of the intervention. We also conducted robustness check analysis to confirm the effect of PBF. Overall, we found a significant positive effect of PBF on most service delivery outcomes, except full vaccinations and post-natal care. One important policy implication is that we should carefully use PBF for targeted indicators.


Introduction
Although Nigeria is an economic giant in sub-Saharan Africa, 1 it lags behind many other African countries in terms of health outcome indicators. For example, the mortality rate in 2017 of children <5 y of age was 100 per 1000 live births, which is one of the highest in the world. 2 Low health outcome indicators can be attributed to low health service utilization. For example, the percentage of births assisted by a skilled birth attendant is only 43% and full immunization coverage is 23%. 3 While demand-side barriers are found to be contributing factors to low health service utilization, 4 supply-side barriers such as unqualified health workers, staff absenteeism, inadequate staff, limited operating hours and informal payments also have been found to contribute significantly to the low level of health service utilization. 5,6 Strengthening the health system could potentially improve service delivery mechanisms, which could lead to improved health outcomes. One popular approach is called performance-based financing (PBF). This approach intends to improve the service delivery mechanism in the health sector by providing financial incentives to health service providers based on their performance. 7 Many developing countries have been applying this innovative financing method to strengthen their health systems. 8 PBF is considered an innovative way to improve service quality. Traditionally health facilities are financed independent of their performance, which can lead to low quality of service delivery. 9 More than 50 academic papers have evaluated the effectiveness of PBF. 10 Evidence of its effectiveness is mixed. 11 For example, Basinga et al. 12 studied one of the early PBF programs in Africa using a randomized controlled trial. They found that PBF improved both the quantity and quality of maternal and child health service provision in Rwanda. On the other hand, de Walque et al. 13 found that there was no difference in many indicators between the PBF group and the control group, among which the incentive was not linked to performance.
This article evaluates the effect of the experimental PBF program implemented in Nigeria on various indicators around maternal and child health service utilization, with a rigorous econometrics tool. We particularly focus on Adamawa state, located in the northeastern zone of Nigeria. Broadly, health outcome and utilization indicators are in general poorer in northern Nigeria than in southern Nigeria. For example, facility deliveries are much lower in the northeast (25%) than in the south (82%). Only 25% of births in the northeast are attended by a skilled provider, compared with 85% in the south. 14 One of the reasons for the poorer health indicators in the north is poor healthcare financing. 15 However, because public health facilities are the main providers of healthcare for people in the north, strengthening the health system is one of the priorities in northern Nigeria.

Methodology Intervention: Nigeria State Health Investment Project (NSHIP)
With sponsorship from the World Bank, the Nigerian government initiated the NSHIP in 2012, with the general objective of improving the quantity and quality of services delivered in primary healthcare. The NSHIP was implemented in three states initially: Adamawa, Nasarawa and Ondo. In this study, we focus on Adamawa.
Under the NSHIP, two different payment schemes were implemented to evaluate their effectiveness on service delivery. One was the performance-based financing (PBF) scheme and the other was the decentralized facility financing (DFF) scheme. Under PBF, health facilities (HFs) received a quarterly payment based on the quantity of services delivered. The type of service, which was the basis of the payment, was predetermined. Examples of the types of services included outpatient consultation, complete vaccination cases, tetanus-toxoid vaccination of pregnant women, post-natal care (PNC) consultation, antenatal care (ANC) consultation, family planning services and institutional delivery. Each health facility was paid a certain price for a given service (unit price). The monetary incentives were based on the quantity of services provided, multiplied by the unit price. These funds were used for operational costs and bonuses for health workers of 50% each. Operational costs included the costs for maintenance and repair, as well as drugs and consumables. For detailed information on the incentives for each service, see Kandpal et al. 16 In contrast, under DFF, health facilities received a certain payment, regardless of the quantity of services delivered. The payments under DFF were calculated to be equal to the average amount of funds for only operational costs earned by the PBF HFs, as DFF HFs were not allowed to use funds to pay health staff bonuses. Thus, by design, the amount DFF HFs received was half the amount earned by PBF HFs.
In Adamawa state, the treatment was randomly assigned to each local government area (LGA). Adamawa has 21 LGAs, 11 of which received PBF and the remaining 10 received DFF. The total number of health facilities that received either PBF or DFF was 445. There were a total of 947 health facilities in the state, which were recorded in the data portal called District Health Information Software 2 (DHIS2). About half of the health facilities in Adamawa state did not receive either PBF or DFF and operated as 'business as usual', thus they constituted the control group. The health facilities in the control group were those that did not have a sufficient level of functionality in terms of infrastructure and personnel. Figure 1 presents the distribution of health facilities by treatment status.

Data
We used the health facility-level data collected by the Health Management Information System through the DHIS2. The DHIS2 records monthly information on the quantity of various health services provided by all the health facilities in Adamawa state before and after the introduction of the NSHIP. To identify health facilities in each category (PBF, DFF and control) we compared two sets of health facilities. One was the census list of all the health facilities in Adamawa state, which we obtained from DHIS2 data. The other data source was the restricted list of health facilities, either PBF or DFF, which is publicly available online. 17 If a health facility in the DHIS2 data was not listed in the restricted data from the PBF portal, which meant that the health facility did not receive PBF or DFF, then it was a control HF.

Outcomes
We focused on seven outcomes for health services. All the outcomes were measured as quantities, i.e. how many times each health facility provided a particular health service per month. Our seven health service indicators included ANC visit, normal delivery administered, delivery services administered by skilled personnel, full vaccinations in children, outpatient care visit, PNC visit and the third dose of tetanus-toxoid vaccination (TT3) administered. For each outcome we had the information for each health facility for a particular month of the year. We selected these seven indicators because they were important maternal and child health outcomes.

Statistical analysis
Using the DHIS2 data for health facilities, we evaluated the effect of the PBF intervention compared with DFF on the quantity of various health services provided at health facilities. To do so, we employed the difference-in-differences (DiD) approach. First, we compared the change in the outcome variables between PBF HFs and DFF HFs before and after the introduction of the NSHIP using the following regression framework: where y it is an outcome variable, service provision at health facility i at time t. PBF it is a dummy variable that indicates whether the health facility was assigned to PBF treatment (PBF HFs). The comparison group DFF HFs. After it is a dummy variable that indicates whether the NSHIP had been introduced to health LGAs started to enrol in the NSHIP in July 2014 or January 2015, although introduction of the NSHIP differed by LGA. After it is 0 before introduction of the NSHIP and 1 after introduction. The difference in timing of the introduction of the NSHIP was accounted for in this dummy variable. PBF After it is an interaction term between PBF and After. We used the LGA fixed effect, v, to control for LGA-specific characteristics. Because assignment of the treatment (PBF or DFF) was at the LGA level, once we introduced the LGA-level fixed effect, the variable PBF it was dropped from the analysis due to perfect multicollinearity. The main analysis compared the effectiveness of PBF intervention to that of DFF on various outcomes, restricting the sample to health facilities that were assigned to either PBF or DFF, eliminating control health facilities, which were not assigned to either group. β 1 identifies differences between PBF HFs and DFF HFs before introduction of the NSHIP. However, as mentioned above, with the LGA fixed effect, β 1 is dropped automatically. β 2 identifies the time trend after initiation of the NSHIP among DFF HFs, as compared with before introduction of the NSHIP. β 3 is the coefficient of interest. It captures the difference-in-differences estimator of the effect of PBF. This DiD estimation strategy is valid only under the assumption that the time trend of the outcome would have been the same between PBF HFs and DFF HFs in the absence of the NSHIP intervention. This parallel trend assumption prior to the introduction of NSHIP seems to hold ( Figure 2).
We hypothesize that the PBF program improved service delivery more than the DFF program. In other words, we expect β 3 > 0.
Second, we evaluated the effectiveness of the PBF intervention by comparing PBF HFs and control HFs. Because assignment of the treatment, either PBF or DFF, was done at the LGA level, we focused only on PBF LGAs. We used the same identification strategy as in equation (1) with the LGA fixed effect, with control HFs being the comparison group. In this analysis, β 1 does not drop, because there is a variation in the treatment assignment, either PBF or control, within each health facility.
Finally, to confirm the rigorous effect of the PBF intervention, we conducted a placebo test. Using equation (1), we evaluated the difference in the provision of health services between health facilities in the control group (control HFs) within PBF LGAs and health facilities in the control group (control HFs) within DFF LGAs. In the absence of the NSHIP, heath facilities in PBF LGAs should have similar characteristics to ones in DFF LGAs, unless they were systematically different. If the randomization at the LGA level worked well, we expect that β 3 is not different from 0.

Results
Of 947 health facilities (HFs) in Adamawa state, we used 932 facilities for the analysis. The remaining 15 health facilities did not have sufficient data to be included in the analysis. Table 1 presents the baseline level of health services provision according to the intervention status-PBF HFs (column 1), DFF HFs (column 2) and control HFs (column 3)-before introduction of the NSHIP. For example, the average number of ANC service provisions per month in PBF HFs is 59.45, while it is 62.60 and 35.00 under DFF HFs and control HFs, respectively.
All seven outcome indicators are balanced between PBF and DFF HFs. None of the outcome variables at baseline is significantly different between the PBF and DFF HFs (column 4). In contrast, we found a significant difference in outcome indicators between PBF and control HFs (column 5). The baseline health service provision is generally higher among PBF HFs than among control HFs, because control HFs were those that did not have sufficient functionality in terms of infrastructure and personnel. Figure 2 presents the trend in the quantity of services provided over time, separately among PBF, DFF and control HFs. The horizontal axis presents the time. At time 0, the NSHIP was introduced in each LGA. The timing of the NSHIP's introduction differs by LGA. Any time before time 0 is the pre-intervention period for each LGA. The time trend of the quantity of service provision among PBF HFs and DFF HFs before introduction of the NSHIP was mostly balanced (Table 1 column 4), while the quantity was significantly lower among control HFs (Table 1 column 5).
Immediately after introduction of the NSHIP, the quantity of the provision of most services under PBF increased more than under DFF or controls ( Figure 2). The exception is the full vaccination cases. Although the number of full vaccination cases seems much higher under PBF than controls, it is similar to the number under DFF. Table 2 presents the main result of the effect of the PBF intervention on various health services provision by comparing PBF HFs and DFF HFs using a DiD approach. For five main outcomes out of seven, the PBF intervention significantly increased the quantity of service provision as compared with DFF after introduction of the NSHIP ('PBF After' in Table 2). The number of ANC cases increased by about 30 under PBF relative to DFF (column 1), the number of normal deliveries by 12 cases (column 2), the number of deliveries by skilled personnel by 18 cases (column 3), the number of outpatients by 106 cases (column 5) and the number of third-dose administrations of tetanus-toxoid by 8 cases (column 7). Among DFF HFs, the quantity of provision of some services, such as ANC (column 1), normal delivery (column 2), full vaccination (column 4) and outpatients (column 5), also significantly increased after introduction of the NSHIP ('After' in Table 2). Table 3 compares PBF and control HFs ('business as usual') in PBF LGAs as a robustness check analysis. The PBF intervention produced an increase in the quantity of services provided for five indicators out of seven in PBF HFs as compared with the control HFs. For example, after introduction of the PBF program, the number of ANC cases increased by 33 in PBF HFs relative to control HFs (column1), the number of deliveries assisted by skilled personnel by 9 (column 3), the number of full vaccination cases by 5 (column 4), outpatients by 80 (column 5) and the number of third doses of tetanus-toxoid by 3 (column 7). Table 4 compares DFF and control HFs in DFF LGAs as another robustness check analysis. The DFF scheme produced an increase in the quantity of services provided for two indicators out of seven in DFF HFs as compared with the control HFs. After introduction of the DFF program, the number of deliveries assisted by skilled personnel increased by six (column 3) and the number of full vaccination cases by two (column 4). Table 5 presents the result of a placebo test. We tested whether health facilities in PBF LGAs and DFF LGAs would have had different characteristics over time in the absence of the NSHIP by comparing control HFs in PBF LGAs and control HFs in DFF LGAs. The effect of the NSHIP program on all of the indicators of service provision was not significantly different between control HFs in PBF LGAs and control HFs in DFF LGAs ('PBF After' in Table 4).

Discussion
The effect of PBF on the quantity of services delivered was evaluated in the northeastern Nigerian state of Adamawa.  The sample includes HFs that are in a PBF LGA, either PBF HFs or control HFs. The comparison group is control HFs. The analysis controls for LGA fixed effect, with LGA clustered standard error. *Significant at 10%, **significant at 5%, ***significant at 1%.
Overall, PBF was found to be highly effective in increasing the quantity of health service delivery such as ANC, child delivery, PNC and vaccination as compared with DFF. The PBF intervention significantly increased the quantity of most of the service delivery indicators more than DFF after introduction of the NSHIP, while the baseline level of service delivery between PBF and DFF HFs was statistically identical prior to introduction of the intervention. Our robustness check analysis confirms the positive effect of PBF on service delivery by comparing outcomes between PBF HFs and control HFs within PBF LGAs. We also found that DFF HFs are better at increasing service utilization than control HFs within DFF LGAs, but not as much as the increase observed among PBF HFs. Our placebo test reassures us that the increase in the quantity of service provision is due to the introduction of PBF. Among seven indicators, PBF interventions did not induce a significant increase in the number of cases of full vaccination or PNC as compared with the DFF intervention. The insignificant result of PBF on these two outcomes is consistent with findings from other studies. 12,18 Because the service utilization of both full immunization and PNC are extremely limited in northern Nigeria, yet are considered critical to reduce maternal and child health burdens, 18,19 it is important to investigate further the potential reasons why the PBF intervention did not influence the full vaccination cases and PNC. LGA clustered standard error. *Significant at 10%, **significant at 5%, ***significant at 1%.  20 The rate of PNC visits in northeastern Nigeria is 34.3%, while the southern regions achieve >65%. 21 Basinga et al. 12 explained that the low unit price for service delivery causes the insignificant effect of the PBF intervention. However, the average unit price for full vaccination is about 1454 naira (US$1=360 naira as of June 2019) and the price for PNC is 394 naira (Table 6). These unit prices are not as low as that of ANC (294 naira) or the third dose of tetanus-toxoid (197 naira), for which we found a strong positive effect of PBF. Thus the low unit price is unlikely to be causing the insignificant effects of PBF. Basinga et al. 12 also mentioned that if the baseline immunization rate is already high, then there is little room for improvement. However, as explained above, both full immunization and PNC visits are extremely limited in northern Nigeria. Thus the high-level of baseline health service utilization is unlikely to be causing the insignificant effect of PBF.

International Health
Future studies should explore reasons why the PBF intervention did not impact the full immunization rate and PNC service. One potential reason why PBF had no effect on full immunization might have been because it involved repeated actions of caregivers' clinic visits, which made it difficult to monitor. Despite strong evidence of the effectiveness of the PBF intervention, there is some concern about its sustainability. Although we observed that the quantity of services provided increased right after the introduction of PBF, the quantity of many services delivered, such as the number of child deliveries, outpatients, PNC and third doses of tetanus-toxoid decreased toward the end of the intervention (Figure 2). This inverted-U relationship between the time since the intervention's introduction and the quantity of services provided poses some questions on the sustainability of the PBF intervention. Future work should explore the reasons for this inverted-U relationship. To the authors' knowledge, there is no study explaining the inverted-U relationship of PBF and health indicators. One potential explanation of the inverted-U relationship is evaluation fatigue.
Our results have an important policy implication. Because PBF does not improve the health service utilization of some indicators in a short-term or sustainable way, we should carefully use the PBF scheme for targeted indicators.

Limitations
This study has several limitations. First, DHIS2 is an aggregated database that does not contain any patient information. Second, the data also do not contain the location information of each health facility, except the name of the LGA it belongs to. This limited availability of information makes it difficult for the study to analyse the differential effect of PBF on different segments of the population, for example, by wealth level and by access to a health facility. Third, the data incompleteness of the DHIS2, such as missing monthly information on main health indicators as well as missing key information on the characteristics of health facilities, can prevent us from determining a rigorous estimate of the PBF effect. Fourth, the comparison between PBF HFs and control HFs does not rigorously reveal the effect of PBF because the baseline characteristics of control HFs are fundamentally different from those of PBF HFs. Thus the robustness check analysis should be interpreted with caution. Fifth, the mechanisms of PBF's superiority over DFF were not identified under this current research design. PBF was presumably successful either because the total amount of funds available to PBF was double that available to DFF or the incentive structure of PBF worked. Extant studies in other African countries have also pointed out the complexity of the potential mechanisms of PBF. 24 Sixth, this study only focuses on service utilization, not health outcomes. Although some of the health service indicators we used in the analysis, such as vaccination, are proven to lead to improved health, we should exercise caution in translating our results to policy implications, because the increase in the number of service deliveries might not necessary lead to improvement in the quality of service provision.

Conclusions
PBF is highly effective in increasing the quantity of health services delivered in Adamawa state, Nigeria. However, the number of full immunizations and PNC cases did not increase due to PBF. Future work should explore why PBF was effective in increasing the delivery of some services but not others.