Monitoring Results in Routine Immunization: Development of Routine Immunization Dashboard in Selected African Countries in the Context of the Polio Eradication Endgame Strategic Plan

Background To monitor immunization-system strengthening in the Polio Eradication Endgame Strategic Plan 2013–2018 (PEESP), the Global Polio Eradication Initiative identified 1 indicator: 10% annual improvement in third dose of diphtheria-tetanus-pertussis–containing vaccine (DTP3) coverage in polio high-risk districts of 10 polio focus countries. Methods A multiagency team, including staff from the African Region, developed a comprehensive list of outcome and process indicators measuring various aspects of the performance of an immunization system. Results The development and implementation of the dashboard to assess immunization system performance allowed national program managers to monitor the key immunization indicators and stratify by high-risk and non–high-risk districts. Discussion Although only a single outcome indicator goal (at least 10% annual increase in DTP3 coverage achieved in 80% of high-risk districts) initially existed in the endgame strategy, we successfully added additional outcome indicators (eg, decreasing the number of DTP3-unvaccinated children) as well as program process indicators focusing on cold chain, stock availability, and vaccination sessions to better describe progress on the pathway to raising immunization coverage. Conclusion When measuring progress toward improving immunization systems, it is helpful to use a comprehensive approach that allows for measuring multiple dimensions of the system.


INTRODUCTION
The Polio Eradication Endgame Strategic Plan (PEESP) [1] is guided by 4 strategic objectives. The second strategic objective states that to eradicate polio, immunization systems must be strengthened in the 10 focus countries (Afghanistan, Angola, Chad, Democratic Republic of Congo, Ethiopia, India, Nigeria, Pakistan, Somalia, and South Sudan) of the Global Polio Eradication Initiative (GPEI); the 10 countries, of which 6 are in the World Health Organization (WHO) African Region, were selected because they contain significant polio assets and large numbers of unvaccinated and undervaccinated children. In carrying out this objective, the original GPEI authors of PEESP set a goal to achieve at least 10% annual increase in the third dose of diphtheria-tetanus-pertussis-containing vaccine (DTP3) coverage rates in 80% of the polio high-risk districts of the 10 focus countries from 2014 to 2018.
Shortly after development of the PEESP, the Immunization Systems Management Group (IMG) was created to monitor the second strategic objective of the PEESP implementation, and the monitoring of progress toward the 10% goal became the responsibility of the IMG Routine Immunization (IMG-RI) subgroup. This subgroup was made up of experts from multiple agencies, and they questioned the feasibility and validity of the 10% goal, particularly because historical evidence indicated that, on average, countries had increased coverage annually by 2.7% during the 1980-2009 period [2]. Additionally, considering the short timeframe (5 years) for PEESP, the subgroup members believed that a focus on efforts to strengthen immunization-system processes would be best; therefore, including process indicators alongside ≥1 coverage indicators would be most pertinent.
Regarding the monitoring of vaccination coverage, the subgroup did consider conducting coverage surveys; however, the cost of conducting surveys at a district level on a yearly basis was estimated in the millions when totaled across all 10 countries, and as such, was considered a very cost-ineffective approach to obtaining outcome data. The group of RI experts who comprised the IMG-RI subgroup proposed the development of a dashboard of RI indicators using existing immunization data to monitor progress toward coverage and system improvement in high-risk districts; the proposal was accepted by the IMG.
Each country in the WHO African Region has an established process of RI data collection and indicator monitoring using country-specific information systems, including Health Information Management System (HIMS), RI District Data management Module (RIM), District Vaccine Data Management Tool (DVDMT), and other local systems. The data catalogued by these systems are often referred to as country administrative data, as opposed to information obtained from vaccination coverage surveys, which generally make use of home-based records to determine vaccination status. All countries collect and submit district-level data to the WHO Regional Office on a monthly basis. These data are used to produce a regional monthly database comprised of RI coverage and numbers of children vaccinated.
Because each country in the African Region already had established processes for RI data collection, reporting, and monitoring using country-specific information systems, the IMG-RI subgroup requested that the 6 focus countries in the African Region (Angola, Chad, Democratic Republic of Congo, Ethiopia, Nigeria, and South Sudan) pilot the monitoring of a standard set of agreed-upon indicators through an RI dashboard. This paper describes the process, opportunities, and challenges for monitoring progress in RI through the establishment of a dashboard using agreed standard indicators that reflect steps in the pathway of RI strengthening with a focus on polio high-risk districts.

METHODS
The process of developing the dashboard and the associated monitoring framework evolved with time and included the participation and input from different teams, including immunization and polio programs from the respective countries' Ministries of Health as well as the various partners who comprised the IMG-RI subgroup (Bill and Melinda Gates Foundation; Gavi, the Vaccine Alliance; Task Force for Global Health; United Nations Children's Fund [UNICEF]; United States Centers for Disease Control and Prevention [CDC]; and WHO).
The IMG-RI subgroup started by identifying a list of commonly collected data elements and indicators [3,4]. This list of 15 indicators was organized around 4 performance categories: (1) the annual integrated plan for the Expanded Programme on Immunization (EPI), to assess the extent of country-level planning; (2) RI coverage improvement; (3) RI system processes, to assess how well the system is functioning and help explain the changes in coverage; and (4) RI data quality, to monitor the reliability of the data being collected. Table  1 presents the dashboard with categories, indicators, status, interpretation, data sources, reporting frequency, and reporting lag. To avoid creation of new data, the team recommended that countries use only data already being collected systematically to calculate the indicators, understanding that not all countries were collecting all exact data needed, and as such some countries would not report on some of the IMG-RI indicators.
Indicators were looked at between high-risk and non-high-risk districts. Non-high-risk districts were the ones with relatively high immunization coverage, high surveillance performance, and absence of recent circulation of polio virus, whereas the high-risk were districts with low immunity, both looking at immunization coverage and immune status of cases. Countries with support from partners developed improvement plans targeting the identified high-risk districts to increase the coverage. This is why a comparison of magnitude of changes between the 2 groups would give an idea of the impact of the intervention.

Planning
The Annual EPI Plan was assessed in each country through measurement of 1 overall indicator, which included the following 5 components:

1.
Contains SMART (specific, measurable, assignable, realistic, time-related) objectives with a harmonized calendar of supplemental immunization activities (SIAs), data quality improvement, and surveillance activities;

2.
Highlights critical activities to reach all districts and communities with a focus on high-risk districts;

3.
Defines clearly the roles and contributions of polio-funded assets;

4.
Contains a fully costed budget with identification of funding gaps; and

5.
Is endorsed by the government and the Immunization Inter-agency Coordination Committee [4].
Based on the components above, every year the IMG-RI sub-group members rated the country Annual EPI Plans after review of available operational planning materials, including Annual EPI Plans and Coverage Improvement Plans (CIPs) [5].

Coverage Improvement
Coverage improvement was assessed in each country through 5 indicators: 1 national indicator and 4 indicators stratified by high-and non-high-risk districts. On a yearly basis, using the WHO/UNICEF Estimates of National Immunization Coverage (WUENIC), national level percentage decrease in the number of children unimmunized with DTP3 was calculated (compared with prior year). On a quarterly basis, using country administrative data and stratified by high-and non-high-risk districts, the following indicators were also calculated:

1.
Percentage of districts that have achieved ≥10% relative increase in DTP3 coverage compared with prior year, which is the original PEESP systems strengthening indicator (Table 1);

2.
Percentage of districts that have achieved ≥10% relative decrease in the percentage of children unimmunized with DTP3 compared with the prior year; 3.

1
Percentage of districts that have achieved ≥80% scheduled fixed RI sessions conducted;

3
Percentage of districts with stock outs or supply chain interruptions. (This indicator is being collected as such in the existing information management system in countries in the African region, which why the indicate is has been phrased as it is, when one would expect it to be "percentage of districts without stock outs," which would need additional data treatment with risk of error at country level.
The following 2 indicators were calculated on an annual basis:

4
Percentage of districts with updated immunization micro plans; and

5
Percentage of districts receiving supervision from the next higher level.

Data Quality
To assess the quality of RI data, the team selected the following 4 indicators, stratified by high-and non-high-risk districts and calculated on a quarterly basis using the countries' administrative data:

1.
Percentage of districts with complete immunization data from all health facilities for the period [6-8];

2.
Percentage of districts reporting complete immunization data for the period;

3.
Percentage of districts with a negative DTP1-3 dropout (DTP1-3 dropout was calculated by subtracting the number of DTP3 doses from the number of DTP1 doses), and
High-Risk Area Identification-Several methods may be used to identify polio high-risk districts in a given country [9][10][11]. The most common information used has included population immunity, sensitivity of disease surveillance, and insecurity. To ensure comparability of the data across countries, it was recommended that countries should use the same method and data parameters to identify the high-risk districts.
In 2011, the WHO Regional Office for Africa (AFRO) polio team developed a risk assessment tool [12] that calculates a risk level for an area based on 20 immunization coverage, surveillance, and other population and program indicators, taking into consideration the indicators' trends during the previous 4 years ( Table 2). The indicators used in the tool were identified at a global meeting on polio risk assessment held in Atlanta in July 2010 with the participation of polio staff from all WHO regions. Developed in Microsoft Excel, this user-friendly tool has been used on a quarterly basis since 2011 by >40 African Region countries to assess the risk level of their districts (Table 2). This tool was used to identify high-risk polio districts in the current project.

Dashboard Development and Capacity-Building Workshop
AFRO, in collaboration with the CDC, developed an Excelbased dashboard to facilitate and streamline data entry by country-level data managers. Within months of using this tool, it became apparent that countries used different methods to collect data and calculate the dashboard indicators. The IMG-RI partners responded by holding a regional workshop in June 2015, inviting data managers from AFRO regional and country offices, as well as representatives from Ministry of Health EPI programs. The goal of this workshop was to understand how the data management process used by each country varied, discuss reasons for these variations, and adopt a standardized approach to improve data comparability across all 6 African Region countries.
Among the outcomes of this workshop was a job aid designed to assist data managers with completing the dashboard data entry. Workshop participants agreed to complete the dashboard on a quarterly timeline, using the newly defined data management protocols and job aid.
This paper focuses on data collected in 2014 and 2015 because district data were not available for 2016 when the paper was drafted.

RESULTS
The capacity building workshop and job aids development facilitated the dashboard reporting process, clarifying both indicator definitions and reporting time lines (Table 3). Despite capacity challenges and staff turnover, all 6 African countries monitored the set indicators on a quarterly basis (Tables 4 and 5).
Data availability and quality improved between the first quarter of 2014 and fourth quarter of 2015, especially for the process indicators driven by the impact of the country capacity strengthening (Tables 4 and 5). As indicated in Tables 4 and 5  Four countries-Chad, Democratic Republic of Congo, Ethiopia, and Nigeria-experienced national decreases of 15%, 3%, 38%, and 12%, respectively, in the number of children unimmunized with DTP3 in 2015 compared with 2014 using the latest WHO-UNICEF estimates [2] (Table 5), thought to be due in part to the improvement of the planning process in these countries and adequate implementation of the coverage improvement plan, which resulted in better vaccine availability at the operational level and better system monitoring. However, the number of children unimmunized with the third dose of DTP-containing vaccine rose by 2% and 16% in Angola and South Sudan, respectively (Table 5 and Figure  2). Chad, Ethiopia, and Nigeria have made a big jump in reduction of unimmunized children. These countries, together with the Democratic Republic of Congo constitute countries where the essential of polio asset is also located in the African Region. Polio staff is also working on RI activities in these countries now with focus on supporting the improvement plan may have had impacted these changes. Angola and South Sudan also had polio asset but did not manage to increase the coverage within the selected period.
Despite the attempt to implement the improvement plan, security challenges in South Sudan restricted access to much of the population, resulting in low coverage. South Sudan's poor performance among the process indicators, including percentage of districts with updated immunization microplans and "percentage of districts receiving supervision from next higher level," as well as its inability to reach 100% for planning (Table 5), helps to explain the low coverage. According to the dashboard, comparison of vaccination coverage between high-risk districts and non-high-risk districts does not indicate definitive trends. This might be due to implementation of the both the reach every district (REC) and reach every community approaches in high and non-high risk districts in the African Region. These approaches are supposed to be implemented in the entire country. In the priority countries this is done with a special focus, and activities are undertaken to maintain and improve the coverage. Countries need to continue to explore innovative strategy, including correct implementation of the REC approach and use of new technology such as Geographic Information System (GIS) for microplanning, to make sure the last unreached are reached.
As shown in Tables 4 and 5 and Figure 2, 100% of districts in Chad and Democratic Republic of Congo had updated microplans for both 2014 and 2015 in both high-risk and non-high-risk districts, whereas in Nigeria there were updated microplans in >82% of both high-risk and non-high-risk districts in 2014 compared with 77% and 73% in 2015 in highrisk and non-high-risk districts, respectively.
Tables 4 and 5 and Figure 3 further reveal that ≥90% of both high-risk and non-high-risk districts reported complete immunization data in the first quarter of 2014 and the fourth quarter of 2015 in Angola, Chad, and Democratic Republic of Congo. Nigeria achieved a district completeness of ≥90% for the first quarter of 2014 in both high-risk and non-highrisk districts; however, in the fourth quarter of 2015, it reported 81% and 78% in high-risk and non-high-risk districts, respectively. South Sudan's district completeness in non-highrisk areas increased from 33% to 91% from the first quarter of 2014 to the fourth quarter of 2015; however, in high-risk districts, completeness remained similar at 43% and 41%, respectively ( Figure 3).

DISCUSSION
We have documented our process of creating and implementing a multidimensional monitoring tool (ie, dashboard) designed to provide information on immunization system performance. Originally developed to monitor performance of several countries during the polio eradication endgame period, the tool may also support immunization system strengthening through use to monitor progress and compare across programs. Although only a single outcome indicator goal (>10% annual increase in DTP3 coverage achieved in 80% of high-risk districts [1]) initially existed in the endgame strategy, we successfully added additional outcome indicators (eg, decreasing the number of DTP3-unvaccinated children) as well as program process indicators focusing on cold chain, stock availability, and vaccination sessions to better describe progress on the pathway to raising immunization coverage. The process of implementing the monitoring dashboard in 6 African countries provided an opportunity to improve process monitoring at a district level and highlighted the need for improvements in the availability of routinely collected process information to help inform program decision making. It appears likely that these countries will continue maintaining this process for their immunization program even beyond the IMG-RI monitoring timeframe.
Program managers may be best able to effect change when their immunization system monitoring activities include indicators for both vaccination outcomes and processes. Focusing solely on vaccination coverage will indicate too late that there is a problem; monitoring process indicators at the health facility level will be an early warning system that coverage improvements are not on track and will help program managers to determine the system components where bottlenecks exist. The system components included as process indicators (eg, working cold chain, vaccine stock management, and immunization session implementation) are contributors to coverage. Process indicators are critical to ensure that comprehensive program strategies such as RED are being implementing effectively, although routine process monitoring is not currently widely practiced in many low-and middle-income countries.
In Chad, a RED monitoring system was set up with a set of process indicators collected at health facility supervisory visits, and data from these indicators have been used in hub and district management meetings and in the annual EPI planning to analyze and address problems (Chad 2016 Annual EPI plan); coverage has improved in this country, from 48% in 2013 to 55% in 2015 (http://apps.who.int/immunization_monitoring/globalsummary/ estimates?c=TCD), with immunization monitoring thought to have played a key role.
The exact formulation of process indicators will vary by country; potential indicators may include stock availability, cold chain status, and number of planned versus conducted vaccination sessions by facility and month. Compiling information on these process indicators at a higher level provides evidence on where to take action; for example, in India, polio-funded staff routinely collect data on similar indicators and discuss this information in district taskforces. Although AFRO has successfully compiled process indicator data for 5 countries (Figure 2), the use of these data for action has not yet been fully optimized. In Chad and Nigeria, process indicators are collected and analyzed locally, but the extent of systematic use in national decision-making processes can be further improved. Further efforts are required to better align the indicators to country decision-making processes and data systems. The development of this RI dashboard to monitor progress during the polio endgame period is a step in the right direction, particularly with the focus on stratifying information by high-risk and non-high-risk districts. The main perceived benefit is the systematic monitoring of the performances within and across countries using standard parameters and set periodicity.
Important limitations include the quality of the administrative data and staff turnover affecting the capacity of the country to maintain the system. To address coverage based on administrative data, we used WHO/UNICEF estimates for national coverage and supported countries to conduct regular data quality assessments and self-assessment to identify gaps and implement corrective actions or data quality improvement plans. These latter activities were reported to be useful and hopefully will continue. Additionally, countries should increase the number of staff who are trained to use the dashboard to ensure its sustainability and benefits.
In comparison, malaria programs currently use a scorecard under the African Leaders Malaria Alliance (ALMA) initiative (http://alma2030.org/) that can provide lessons to immunization programs on the usefulness of visual aids on program performance. AFRO and the UNICEF regional offices in Africa and other immunization partners are exploring these lessons and their use in advocacy among African heads of state, as a follow-up of the Ministerial Conference on Immunization in February 2016 (http:// immunizationinafrica2016.org/). The RI system is complex and should be monitored using a multidimensional approach because this allows for program staff to identify bottlenecks that may be inhibiting program outcomes. Additionally, outcome goals should be realistic and reflect the input of those most knowledgeable about the potential achievements possible during short timeframes. Dashboard monitoring has the potential for providing many benefits to those countries that continue to use it beyond polio eradication but will only prove useful if it is included in local and national decision-making processes and not only used at a global level.