Abstract

Administrative coverage data are commonly used to assess coverage of mass vaccination campaigns. These estimates are obtained by dividing the number of doses administered by the number of children of eligible age, usually at the health district level. This study used data from a cluster survey conducted in each of the 53 Burkina Faso health districts immediately after 1999 the National Immunization Days to assess whether administrative estimates correlated with those obtained through survey and whether the former identified districts that achieved suboptimal coverage as measured by cluster survey. During the first round of the campaign there was no significant correlation between data obtained by either method. The correlation was only marginally better during the second round. Although useful to help plan the logistics of a campaign, administrative coverage data should be used with other evaluation techniques in order to determine the number of eligible children vaccinated during a mass campaign.

In Africa, beginning in the mid‐1990s, mass poliomyelitis vaccination campaigns were progressively implemented in all countries. As countries gained experience in conducting these campaigns many added administration of vitamin A supplements and a few included measles vaccination in all or part of their territories [1]. According to the World Health Organization (WHO), the assessment of campaign vaccination coverage should rely on a simple formula that divides the number of children who belong to the age group covered by the campaign and who were vaccinated by the number of children of the same age group who reside in the area [2]. This method, also called “administrative estimates of immunization coverage,” was believed to provide a good measure of the relative performance by geographic area based on WHO experience [3].

Burkina Faso has conducted two annual rounds of poliomyelitis National Immunization Days (NIDs) since 1996. Administrative coverage estimates for each round exceeded 95% nationally. In 1998, coverage estimates from the 11 health regions ranged between 87%–113% for the first round and 100%–111% for the second [4]. Although some of these data are clearly overestimates, it remains unclear whether administrative coverage data actually identified areas that achieved suboptimal coverage.

In 1999, NIDs in Burkina Faso included vitamin A supplementation and measles vaccination in addition to poliomyelitis vaccination. To obtain a detailed validation of the coverage achieved by each intervention, a survey was conducted in each of the country’s 53 health districts. Here we compare the coverage estimates obtained at district level by both methods, describe the distribution of differences in estimates, and verify whether a correlation exists between estimates obtained by both methods. We also assess how these relationships evolve when estimates are aggregated at a level larger than that of the health district.

Methods

1999 campaign design. The first round of the campaign was 4–6 November. Oral poliovirus vaccine (OPV) was offered to children aged 0–59 months and vitamin A was offered to those aged 6–59 months. During the second round, 4–8 December, a second dose of OPV was offered to children in the same age group while measles vaccine was offered to children aged 9–59 months.

Determination of administrative coverage estimates. During the NIDs, teams of health care workers and volunteers administered vaccine and supplements. Each dose administered was recorded on a standard tally sheet. Each team used one new tally sheet each day. Team supervisors collected tally sheets at the end of the day or, for teams reaching remote populations, as they supplied those teams with new material. To aggregate data from the teams, supervisors used a standard summary sheet. Summary data were communicated to health post managers on a daily basis, except for supervisors working in remote areas, and health posts subsequently communicated their data to district managers. District managers collected tally sheets and summary forms at the end of each round and computed the final counts. District teams communicated denominator population data during microplanning for the campaign.

Methods to estimate numbers of children who should receive the vaccine or supplements varied by health district. Usually, one of three methods was used: first, by using the 1996 national census, the fraction of the district population corresponding to the age‐group retained for each intervention was calculated and then adjusted for population growth; second, was summing of the age‐specific population served by each health center when those data were available for the whole district; or third, by taking the 1998 average number of OPV doses administered during each round and correcting these by a multiplication factor corresponding to the average annual population increase.

Coverage survey. In each of the 53 health districts, a coverage survey modeled after the WHO Expanded Program on Immunization method [5] was conducted within 12 days of the end of the second round of the campaign (described in [6]). In brief, 30 clusters of 7 children each were identified per district by use of a two‐stage sampling scheme. Survey agents interviewed the principal caregiver of each child to record the type of vaccine or supplement received during each round. An indirect questioning technique was used according to a method experimented with on a smaller scale in 1998 [7]. Because no written records were used for the campaign, caregivers were asked to describe what the child received during each round of the campaign instead of having to answer by “yes” or “no” to questions that would have explicitly described all antigens and supplements offered during the campaign. Point estimates of vaccine or supplement coverage were computed among children in the corresponding age group. Confidence intervals (CIs) were obtained with the CSAMPLE module of EPI INFO 6.04d software [8]. Because the measles vaccine was not offered to all children surveyed, the number of children included in the analysis of measles vaccine coverage was lower than the usual 210 included in standard cluster surveys. Therefore, some CIs can be broader than ±10% SD.

Comparison of administrative and survey data.  We computed statistics of central tendency for coverage estimates obtained from both methods as well as for the differences between administrative and survey estimates. To verify if administrative data predicted measurements obtained from the survey at the district level, we performed linear regression by using the inverse of the survey measurement variance as a weight. A perfect correlation would yield a regression coefficient of 1. This would mean that measures obtained with the administrative coverage method would correspond to the same value measured by the survey. To verify if administrative estimates better reflect aggregate coverage on a larger scale, we also compared estimates obtained for each of the 11 health regions in the country. Each health region comprises 2–11 health districts.

Results

By design we attempted to survey 11,130 children aged 0–59 months. Complete survey data were available for 11,026 (99%) and were retained for subsequent analysis of OPV coverage. Vitamin A coverage was analyzed for 9833 and measles coverage for 8551 children eligible for the respective supplement or vaccination. For each district, there were 192–210 observations on OPV, 170–201 on vitamin A, and 59–188 on measles vaccine. National coverage estimates from the administrative method and the survey were, respectively, 102% and 88% (95% CI, 87%–89%) for OPV during the first round, 100% and 89% (95% CI, 88%–90%) for vitamin A, 109% and 90% (95% CI, 89%–91%) for OPV during the second round, and 111% and 88% (95% CI, 87%–89%) for measles vaccine.

Survey results identified up to 13 districts with coverage estimates for 1 dose of vaccine or supplement lower than 85% and up to 5 districts with an upper value of the 95% CI lower than 90% (table 1). All districts that had an upper bound of less than 90% also had coverage point estimates below 85% for the same dose of vaccine or supplement. Administrative estimates were higher than survey estimates by a median of 12 percentage points for vitamin A during the first round to 24 percentage points for measles vaccine during the second. The number of districts with higher survey point estimates than administrative estimates was 5 (9%) of 53 for OPV during the first round, 12 (23%) for vitamin A and 2 (4%) for both OPV and measles vaccine during the second round.

Table 1. 

Characteristics of survey and administrative data from 53 districts, Burkina Faso, 1999.

Table 1. 

Characteristics of survey and administrative data from 53 districts, Burkina Faso, 1999.

Figure 1 shows the relationship between survey and administrative estimates for measles vaccination. Except for 2 districts with administrative estimates below 70%, all others were higher than survey estimates. Linear regression coefficients (table 2) indicate that administrative estimates do not predict data obtained from the survey during the first round. A slightly better relationship was found in the second round, as an increase of 100% in administrative estimates corresponded to an average percentage point increase of 17% (95% CI, 9%–25%) for OPV coverage and 14% (95% CI, 8%–21%) for measles coverage estimates.

Figure 1. 

District level measles vaccine coverage measured through the survey as compared to administrative estimates in Burkina Faso, 1999. Points and attached lines indicate the survey estimate and 95% confidence intervals (CI) for each district; diagonal line indicates 100% agreement between survey and administrative data.

Figure 1. 

District level measles vaccine coverage measured through the survey as compared to administrative estimates in Burkina Faso, 1999. Points and attached lines indicate the survey estimate and 95% confidence intervals (CI) for each district; diagonal line indicates 100% agreement between survey and administrative data.

Table 2. 

Regression coefficients for administrative estimate prediction of survey findings, Burkina Faso, 1999.

Table 2. 

Regression coefficients for administrative estimate prediction of survey findings, Burkina Faso, 1999.

When data were aggregated at the level of health regions, CIs around the survey estimate for measles vaccination coverage became narrower, especially for regions that comprise 5 or more districts (table 3). All administrative estimates of regional coverage were higher than those obtained from the survey. The ability of administrative data to identify areas with lowest coverage was only marginally better when those were aggregated at a larger scale (health regions). For measles vaccination (table 3), administrative coverage estimates were higher than 100% for all 11 regions, although 3 (Bobo‐Dioulasso, Ouagadougou, and Tenkodogo) had upper bound values of survey estimates of 90% or less. The linear regression coefficient applied to aggregated data did not indicate a better correlation than with district level data; an increase of 100% in administrative coverage estimates corresponded to a 24% increase (95% CI, −3% to 50%) in survey estimates.

Table 3. 

Comparison of coverage estimates obtained at the regional level for measles vaccination, Burkina Faso, 1999.

Table 3. 

Comparison of coverage estimates obtained at the regional level for measles vaccination, Burkina Faso, 1999.

Discussion

Our data suggest that during 1999 NIDs in Burkina Faso, where all districts achieved moderate‐to‐high coverage for all antigens and supplements offered, administrative coverage estimates did not allow districts with moderate coverage to be distinguished from those with high coverage. Imprecise denominator population estimates available at district level, vaccination of children older than 59 months, and mobility of populations across district borders to reach vaccination sites are the three most likely reasons for the differences observed. Because of concerns with the quality of administrative data, several countries conducted surveys after poliomyelitis NIDs [9, 10].

Comparison of vaccination coverage estimates obtained from administrative data and cluster surveys have been reported from the routine program in Cameroon [11] and Zimbabwe [12]. In Cameroon, administrative data tended to overestimate vaccination coverage by 1%–29% and in Zimbabwe they tended to underestimate coverage by 4%–10%. Comparisons of routine vaccination coverage are complicated by the fact that the age distribution of children included in the calculations usually differs from administrative data (children <1 year old) and surveys (children aged 12–23 months). For campaign evaluations, provided that the survey occurs shortly after the campaign, the population that should have received vaccinations during the campaign is virtually the same as that included in the survey.

All evaluation methods have advantages and limitations. Administrative data are valuable because they are comprehensive, easy to obtain, and can potentially identify small areas or even villages where very few vaccine doses were administered, thereby allowing for prompt corrections. They also provide useful logistical information for planning of future immunization activities, in particular when accurate population data are missing. Cluster surveys, on the other hand, can provide more accurate measurements of actual coverage for the area surveyed as a whole. In our example, these measurements identified districts where population data or management of administrative data was most problematic. However, because of the limited sampling scheme, they do not distinguish which part of the area studied might require additional attention [13]. Because they are also more costly and time consuming and require a higher level of staff training, they are difficult to implement as a continuous evaluation tool.

Administrative data and cluster surveys are not the only methods used to assess mass vaccination campaign coverage. In the Americas, a 20‐house survey method conducted by supervisors in each community proved useful in identifying insufficiently vaccinated neighborhoods [14]. This method, implemented as an intensive supervision tool, also allows day‐to‐day correction of vaccination coverage. Another method that was very useful during polio NIDs enumerates children who never received a vaccine dose (“zero‐dose children”). This method is particularly useful for vaccinations that require more than one dose, because as the quality of campaigns improves, areas that were previously imperfectly vaccinated still include large proportions of zero‐dose children. In areas with higher coverage, most children identified as receiving zero doses are in the youngest age group. Another approach, lot quality assurance sampling (a method successfully applied to the assessment of routine vaccination [15]), could also be considered in campaign settings. Finally, the WHO is currently implementing a new tool to audit data quality in vaccination programs. Better reporting procedures might result from these audits and possibly could also apply to campaign assessments.

Data from this study confirm that administrative coverage must be interpreted cautiously, even when the number of vaccine doses administered exceeds the number of children who should have been vaccinated. Imprecision of both numerators and denominators can provide false reassurance that the coverage objective has been met. Accurate coverage data are also important to assess the impact of different vaccination strategies. Adequate quantitative evaluation of mass vaccination campaigns requires the combination of several approaches. When a country conducts a one‐time large‐scale campaign, such as a mass measles vaccination campaign, the additional effort and expense incurred by conducting cluster surveys at the province or national level can provide a more accurate validation of the effective aggregate coverage. The inclusion of such surveys in the strategic design of vaccination campaigns also indicates the commitment of health authorities to reach the largest possible number of eligible children.

References

1.
Otten
MW
Jr
Okwo‐Bele
J‐M
Kezaala
R
Biellik
R
Eggers
R
Nshimirimana
D
Impact of alternative approaches to accelerated measles control: experience in the African Region, 1996–2002
J Infect Dis
 , 
2003
, vol. 
187
 
Suppl 1
(pg. 
S36
-
43
(in this issue)
2.
World Health Organization
Field guide for supplementary activities aimed at achieving polio eradication
 , 
1997
Geneva
WHO
 
(WHO/EPI/GEN/95.01 REV.1)
3.
Birmingham
ME
Aylward
RB
Cochi
SL
Hull
HF
National Immunization Days: state of the art
J Infect Dis
 , 
1997
, vol. 
175
 
Suppl 1
(pg. 
S183
-
8
)
4.
Rapport d’execution des journées nationales de vaccination 1999 (report on 1999 National Immunization Days)
Ouagadougou, Burkina Faso: Ministère de la Santé du Burkina Faso
 , 
1999
5.
Henderson
RH
Sundaresan
T
Cluster sampling to assess immunization coverage: a review of experience with a simplified sampling method
Bull World Health Organ
 , 
1982
, vol. 
60
 (pg. 
253
-
60
)
6.
Yaméogo
KR
Yaméogo
A
Nacoulma
SD
Zuber
PLF
Measles vaccination coverage during poliomyelitis National Immunization Days in Burkina Faso, 1999
J Infect Dis
 , 
2003
, vol. 
187
 
Suppl 1
(pg. 
S74
-
9
(in this issue)
7.
Zuber
PLF
Conombo
KSG
Dembélé Traoré
A
, et al.  . 
Mass measles vaccination in urban Burkina Faso, 1998
Bull World Health Organ
 , 
2001
, vol. 
79
 (pg. 
296
-
300
)
8.
Dean
AG
Dean
JA
Coulombier
D
, et al.  . 
EpiInfo version 6: a word processing, database, and statistics program for public health on IBM‐compatible microcomputers
 , 
1996
Atlanta
CDC
9.
Reichler
MR
Aslanian
R
Lodhi
ZH
, et al.  . 
Evaluation of oral poliovirus vaccine delivery during the 1994 National Immunization Days in Pakistan
J Infect Dis
 , 
1997
, vol. 
175
 
Suppl 1
(pg. 
S205
-
9
)
10.
Reichler
MR
Darwish
A
Stroh
G
, et al.  . 
Cluster survey evaluation of coverage and risk factors for failure to be immunized during the 1995 National Immunization Days in Egypt
Int J Epidemiol
 , 
1998
, vol. 
27
 (pg. 
1083
-
9
)
11.
Guyer
B
Atangana
S
A programme of multiple‐antigen childhood immunization in Yaounde, Cameroon: first year evaluation 1975–76
Bull World Health Organ
 , 
1977
, vol. 
55
 (pg. 
633
-
42
)
12.
Borgdorff
MW
Walker
GJ
Estimating vaccination coverage: routine information or sample survey?
J Trop Med Hyg
 , 
1988
, vol. 
91
 (pg. 
35
-
42
)
13.
Lemeshow
S
Robinson
D
Survey to measure programme coverage and impact: a review of the methodology used by EPI
World Health Stat Q
 , 
1985
, vol. 
38
 (pg. 
65
-
75
)
14.
Izurieta
H
Venczel
L
Dietz
V
, et al.  . 
Monitoring measles eradication in the Region of the Americas: critical activities and tools
J Infect Dis
 , 
2003
, vol. 
187
 
Suppl 1
(pg. 
S133
-
9
(in this issue)
15.
Tawfik
Y
Hoque
S
Siddiqi
M
Using lot quality assurance sampling to improve immunization coverage in Bangladesh
Bull World Health Organ
 , 
2001
, vol. 
79
 (pg. 
501
-
5
)
Financial support: UNICEF.