Improving adherence to family planning guidelines in Kenya: an experiment

Abstract

Quality problem

Research in Kenya in the mid-1990s suggested poor quality family planning services and limited access to services. Clinical guidelines for family planning and reproductive health were published in 1991 and updated in 1997, but never widely distributed.

Choice of solution

Managers and trainers chose intensive, district-level training workshops to disseminate guidelines and update health workers on guideline content and best practices.

Intervention

Training workshops were held in 41 districts in 1999. Trainees were instructed to update their untrained co-workers afterwards. As a reinforcement, providers in randomly selected areas received a ‘cascade training package’ of instructional materials and training tips. Providers in 15 randomly selected clinics also received ‘supportive supervision’ visits as a second reinforcement.

Evaluation methodology

A cluster-randomized experiment in 72 clinics assessed the overall impact of the training and the marginal benefits of the two reinforcing activities. Researchers and trainers created several dozen indicators of provider knowledge, attitudes, beliefs and practices. Binomial and multivariate analyses were used to compare changes over time in indicators and in aggregated summary scores. Data from patient interviews were analysed to corroborate provider practice self-reports. Cost data were collected for an economic evaluation.

Results

Post-test data collected in 2000 showed that quality of care and access increased after the intervention. The cascade training package showed less impact than supportive supervision, but the former was more cost-effective.

Lessons learned

Service delivery guidelines, when properly disseminated, can improve family planning practices in sub-Saharan Africa.

family planning, clinical guidelines, training, Kenya, Africa

Quality problem

Clinical practice guidelines have come to sub-Saharan Africa and the rapid increase in their popularity has mirrored that in richer countries. As elsewhere, adherence is a problem, but in cash-strapped Africa, even the physical distribution of guidelines is a barrier to good compliance. Though guidelines and clinical protocols for family planning and reproductive health services are now becoming ubiquitous, little research has been conducted in Africa to assess their impact on quality of care.

Kenya is an East African country with a population of about 30 million and per capita GNP less than $1000. After a long period of growth, contraceptive prevalence in Kenya plateaued in the late 1990s at around 33% [1]. In the mid-1990s, research suggested that family planning services in Kenya were not always accessible or of high quality [2, 3]. Furthermore, the growing HIV problem in the region necessitated that family planning clinics make special efforts to offer services to their clients, which would help to protect them not only from pregnancy but also from sexually transmitted infections. These challenges were exacerbated by the fact that many health providers lacked even basic training in modern contraception.

Clinical guidelines had been developed and published in 1991 [4] as one tool to help standardize and improve the quality of family planning and reproductive health services in Kenya. However, most providers were never been trained in their use or even saw them. An updated version of the guidelines was published in 1997 [5], but the subsequent distribution also had a limited reach.

The goal of this intervention was to improve family planning services by intensively disseminating updated clinical guidelines. Trainers and evaluators agreed that key measures for improvement were those that directly addressed access and quality, particularly in the context of the HIV epidemic and previous failures at disseminating guidelines through simple distribution. The evaluation component assessed changes over time in several dozen indicators and two summary scores of provider knowledge, attitudes and practices. The marginal impact of two training enhancements was also evaluated. Finally, we assessed costs of interventions and client reports corroborating information reported by providers.

Choice of solution

To update nurses and midwives on revisions to Kenya's guidelines, trainers from the Ministry of Health's (MOH) Division of Primary Health Care and Johns Hopkins University chose an approach — intensive, multifaceted dissemination close to the level of actual service delivery — based on the literature showing that such a strategy was more likely to improve compliance with new guidelines [6–8]. A notable characteristic of the approach was its reliance on a cascade training model in which trainees who attend workshops were to update their co-workers upon return to their clinics. In some cases, ‘support supervision’, which has been shown to improve performance [9], was used to reinforce the training. The approach also emphasized new clinical recommendations such as those for emergency contraception and for the routine offer of condoms to clients. Another emphasis was on new, liberalized eligibility criteria meant to reduce barriers to access.

Besides producing standard training materials, staff from the MOH and Johns Hopkins also created a ‘cascade training package’ to help trainees update their colleagues. The package consisted of ‘training of trainers’ instruction to enhance the trainees' teaching skills as well as extra printed materials that could be presented to co-workers.

The authors developed an evaluation plan to rigorously assess the impact of this intervention. Because we also wanted to compare the standard training with that enhanced by the cascade training package, we requested and received permission from the MOH to randomize the intervention by regions. We also examined the incremental benefit of extra supervisory visits as well as the cost-effectiveness of both these enhancements to the standard training.

Intervention

The intervention ran from October 1999 through April 2000 as rural-based trainers led 2-day trainings in 41 districts. At the district level, one family planning provider from each facility attended the training and was instructed to update co-workers back at his or her clinic. This was the standard, or control, intervention. In randomly selected training areas, the nurses and midwives being trained were also provided with and coached in the use of the cascade training package. This was the enhanced intervention.

As a further training reinforcement, a team also conducted ‘support supervision’ visits in 15 of the sites receiving the cascade training package enhancement. Taking place 1–3 months after the main intervention, these visits enabled trainers to assess provider knowledge and skills, provide feedback and help solve problems.

Evaluation methodology

We used a two-stage cluster-randomized experiment in 72 clinics to assess the impact of the intervention and to compare standard versus enhanced dissemination strategies. In the first stage, 6 of Kenya's 13 training areas were selected and randomly assigned (using the random number generator in Epi Info software) to control and experimental states. In the second stage, we selected all 25 hospitals and 47 of 166 health centres in the areas (with probability of selection of health centres proportional to the number of centres within each district). Finally, 15 clinics were randomly selected from within experimental districts to receive a supervisory visit to reinforce training and guideline messages.

We developed pre- and post-test provider questionnaires with a combination of open- and close-ended questions. They were to be administered to all family planning providers present during 1-day visits by study staff, most of whom had experience in similar data collection exercises. A shorter client exit questionnaire was also developed to be administered to as many clients as possible, particularly new clients, on the data of the clinic visit.

In the 72 clinics selected for the evaluation, we collected baseline data from 177 providers and 482 clients during September and October 1999, after which the intervention took place over several months. Follow-up data collection from 176 providers and 451 clients took place in the same 72 clinics in July 2000. One hundred and forty-eight providers were administered both the pre-test and post-test questionnaires.

We used binomial and multivariate analysis to compare changes over time in provider knowledge, attitudes and reported practices among the three exposure groups (standard training, standard plus cascade training package and standard plus cascade package plus support supervision). We also analysed data from patient interviews to corroborate provider practice self-reports. Finally, we collected cost data for use in our economic evaluation. To correct for the clustered nature of the sample, we used SUDAAN software that accounted for intraclass correlation at the level of the clinic and the district.

We created two continuous scores to provide summary indicators of impact. The first, a 34-element score, measured provider reports of their own practices, particularly in regard to known barriers to access. Examples of elements in the first score include whether the provider said he/she provided the recommended number of pill cycles and whether the provider was willing to provide various contraceptive methods when a client was not menstruating. The second score, which had 19 elements, assessed whether providers had appropriate knowledge and attitudes for family planning service provision. Examples of indicators in the second score include knowledge of the duration of effectiveness of different methods and opinions regarding emergency contraception and provision of contraception to adolescents. Both scores were standardized to a 100-point scale.

Results

Of the 177 providers interviewed at baseline, 72% were nurses and 21% were midwives. The remaining 7% were mostly nurse auxiliaries. (Physicians do not normally provide family planning in Kenya's public sector.) The mean age was 38 and, on average, these health workers had been providing family planning services for 9 years. At follow-up, 91% of providers reported having seen the guidelines that were the subject of the intervention (up from 45% at baseline) and 74% reported having read the guidelines (up from 29%). Twenty-seven percent of providers interviewed at follow-up reported having attended the training sessions.

Table 1 shows changes between baseline and follow-up in the proportion of providers giving correct responses for 10 typical quality indicators. The first five indicators pertain to barriers to access and the remaining five reflect knowledge of contraceptive technology covered in the new guidelines. Each indicator is also broken down by the intensity of intervention. For 7 out of 10 indicators, the magnitude of the change increased when the cascade training package was added to the intervention. The magnitude of change increased for all 10 indicators when both the training package and supervision were added. The P-values reported in the table show the significance of the differences in the intervention arms over time, compared to the standard training arm. They are from logistic regressions run on each indicator that controlled for urban/rural status, type of clinic, attendance at training and the clustered nature of the sample.

Table 1

Proportion of providers with correct responses for selected quality indicators, by intensity of intervention

	Standard training		Standard + cascade training package			Standard + package + supervision
Indicator	Pre n = 70(%)	Post n = 71(%)	Pre n = 53(%)	Post n = 51(%)	p-value*	Pre n = 54(%)	Post n = 54(%)	p-value*
Know IUD can be inserted at any time during menstrual cycle	34	31	23	73	0.00	22	76	0.60
Disagree with ‘Depo Provera can cause permanent sterility’	54	68	59	69	0.76	65	87	0.26
Disagree with ‘Women must have a child before using IUD’	23	54	30	59	0.83	30	70	0.24
Disagree with ‘Adolescents should not be given contraceptives’	70	85	64	86	0.55	67	83	0.62
Disagree with ‘The Pill should not be prescribed to women over 35’	69	76	64	80	0.38	57	67	0.42
Know three criteria for using ‘lactational amenorrhea method’	7	11	2	24	0.07	6	33	0.65
Know emergency contraception must be used within 3 days after sex	29	54	30	75	0.09	26	74	0.76
Know IUD effectiveness lasts at least 10 years	36	58	55	63	0.13	52	76	0.02
Know two goals of dual method use	74	51	74	84	0.00	80	80	0.30
Know oral contraceptives do not increase risk of endometrial cancer	54	75	53	80	0.92	63	89	0.50

*P-value for test of differences between standard training and each of the enhanced intervention arms, accounting for time, in a logistic regression that also controlled for urban/rural status, type of clinic and attendance at training.

As noted above, we created two continuous scores summarizing provider knowledge/attitudes and practices. Table 2 shows the significant (P < 0.001) increases in the scores of the 148 providers who were interviewed both at baseline and follow-up. The increases were more marked among providers in clinics with reinforcing interventions. We also conducted multivariable analysis to estimate the marginal improvement attributable to the type of reinforcement (Table 3). Controlling for potential confounders such as type of clinic and whether or not the provider personally attended a training, as well as for the clustered nature of the sample, we found that the cascade training package was associated with three-point improvements in both the knowledge/attitude and practice scores over the standard training, but neither increase was significant. Receiving both the cascade training package and supervision enhancements was associated with significant eight- and nine-point improvements over the standard training, respectively, in the knowledge/attitude and practice scores. However, the largest increase, accounting for 11 and 12 point increases in the two scores, was associated with having personally attended the training.

Table 2

Changes in raw provider ‘Knowledge/Attitude’ and ‘Practice’ scores, by intensity of intervention

Scores % (n)	Standard training		Standard + cascade training package		Standard + package + supervision
	Pre	Post*	Pre	Post*	Pre	Post*
Knowledge/attitude — all	54.4 (56)	66.3 (56)	59.9 (44)	72.5 (44)	59.7 (48)	79.6 (48)
Practice — all	43.7 (56)	59.0 (56)	47.5 (44)	63.6 (44)	48.8 (48)	71.6 (48)

*All pre–post differences were significant at P < 0.001.

Table 3

Multivariate regression of ‘Knowledge/Attitude’ and ‘Practice’ scores, n = 147

Parameters	Knowledge/attitude			Practice
	Estimate*	95% CI	P-value	Estimate*	95% CI	P-value
Intercept	34.5	29.5 – 39.4	<0.001	37.1	32.2 – 42.1	<0.001
Score at time 1	0.5	0.4 – 0.6	<0.001	0.3	0.2 – 0.4	<0.001
Urban setting	5.7	3.2 – 8.2	0.03	4.4	1.2 – 7.7	0.18
Hospital	−0.3	−2.8 – 2.2	0.91	1.7	−1.4 – 4.8	0.59
Provider attended training	10.7	8.4 – 13.0	<0.001	13.5	11.2 – 15.8	<0.001
Received cascade package	3.3	0 – 6.6	0.33	2.9	−0.1 – 5.9	0.34
Received support supervision	8.6	5.0 – 12.2	0.02	9.5	6.3 – 12.7	0.004

*Estimate is the number of points contributed to the provider scores shown in Table 2.

Client interview data generally corroborated what the providers reported about their own practices. Table 4 shows the improvements in several of the indicators we measured. For example, the proportion of continuing clients who said that providers recommended condoms in addition to their regular method increased from 9 to 23%. And after the introduction of a new job aid that helps providers rule out pregnancy among non-menstruating clients, the proportion of new, non-menstruating clients receiving services increased from 53% to 71%.

Table 4

Changes in proportion of clients reporting quality practices

Indicator	All clinics
	Pre (%) (n)	Post (%) (n)
Continuing pill clients receiving ≥3 cycles	69 (102)	72 (85)
New pill clients receiving ≥3 cycles	17 (29)	22 (27)
New, non-menstruating clients receiving services	53 (58)	71* (56)
Condom use recommended to continuing clients	9 (391)	23** (363)
Condom use recommended to new clients	40 (91)	57* (88)

*P < 0.05, **P < 0.0001.

Cost-effectiveness analysis

We conducted a cost-effectiveness analysis using two cost scenarios and a 100-point score of knowledge and practice as our measure of effectiveness. For both the cascade training package and supportive supervision, we calculated the incremental impact per clinic, as well as the incremental cost-effectiveness ratio.

Improvement scores for each clinic were calculated by averaging the differences between providers' baseline and follow-up scores. The Z-score for each clinic's average improvement score was calculated by subtracting the clinics' mean score from each clinic score and dividing by the standard deviation. The distribution of Z-scores was shifted, so the minimum value was 0 (as opposed to a negative number) and scaled so the largest value was 100. The score was regressed upon dummy variables for the two training enhancements, urban/rural status of the facility and type of facility. Clustering of clinics within districts was controlled for in the analysis.

The first scenario uses actual costs measured during the intervention (Table 5). The cascade training package costs a total of $24, 601 to develop and produce for trainings in an estimated 933 clinics, at an average cost of $26.37 per clinic. (Implementation costs were assumed to be zero, since the cascade training package was ‘piggybacked’ onto the standard training.) Clinics exposed to the package had an average increase in 6.5 points in our scale, resulting in an incremental cost-effectiveness ratio (cost per one-point change in score) of $4.06.

Table 5

Cost-effectiveness analysis

	Cascade training package		Supportive supervision
	Scenario 1 (933 clinics)	Scenario 2 (1866 clinics)	Scenario 1 (15 clinics)	Scenario 2 (100 clinics)
Development costs	$24601	$24601	$7186	$7186
Implementation costs	0	0	$13171	$34421
Total costs	$24601	$24601	$20357	$41607
Incremental cost per clinic	$26.37	$13.18	$1357	$416.07
Incremental impact per clinic (change in impact score)	+6.5	+6.5	+20	+20
Incremental cost-effectiveness ratio (cost per one-point change in score)	$4.06	$2.03	$67.86	$20.80

Since this initial pilot, the cascade training package has been scaled up and adapted in Kenya and in the region not only for use in family planning, but also for antenatal care, prevention of malaria in pregnancy and voluntary counseling and testing (VCT) for HIV. To account for the increased cost-effectiveness implied by this scaling up of the package, we created a more realistic second scenario in which the development costs were spread over twice as many clinic-based trainings. As a result, the incremental cost-effectiveness ratio achieved under the second scenario was halved.

Development and delivery costs of the supportive supervision intervention were, respectively, $7186 and $13 171, and consisted mainly of development of tools, supervisor time, travel and per diem. In the pilot program, however, supportive supervision was delivered to only 15 clinics, resulting in an average cost of $1357 per clinic. Clinics that received extra supervision had an average increase of 20 points in the scale (over and above the increase attributed to the cascade training package alone), resulting in an incremental cost-effectiveness ratio of $67.86 per one-point increase in the scale.

Again, however, we created a second scenario that spreads the high initial fixed costs of developing the supervision program over a larger number of clinics. It also reflects the more modest variable costs of a routine program with less centralized support (by reducing the implementation cost per supervision visit to $250 after the initial 15 visits). In this second scenario, the cost per one-point change in score decreased by two-thirds, from $68 to $21.

Lessons learned

This quality improvement intervention for family planning services in Kenya supports evidence that more intensive disseminations of guidelines yield better outcomes. Our evaluation had several limitations, such as the lack of a real control group to control for maturation and testing effects. Nonetheless, it provides the most clear-cut evidence to date in Africa that family planning service delivery guidelines, when properly disseminated, can improve practice. Not only did the key indicators in this evaluation improve, but also their improvements seemed to be sustained, as they were measured after a period averaging 8 months.

This good news was tempered by some of our other results. Although adding reinforcing supervisory visits to the cascade training package was much more effective than cascade learning alone, it was also quite expensive. In sub-Saharan Africa, there is little likelihood that many family planning programs will have the means to reinforce guidelines disseminations with new supportive supervision programs. However, some supervision, often of poor quality [10], already takes place, so our results suggest that improving supervisors' skills could pay handsome dividends in improved quality and access. Another disappointment was that the cascade training package did not produce significant improvements in our (perhaps underpowered) multivariable analyses. However, providers in these clinics generally performed better on our range of measures than those in our ‘standard’ treatment group. Since program managers have little to lose and much to gain by adding simple materials and instructions to ongoing trainings, we recommend this practice despite our mixed findings. Since the intervention, this same cascade training package has been used several other times by the MOH and by various national and international organisations, further increasing its cost-effectiveness.

Acknowledgements

The authors thank the United States Agency for International Development for its financial support of this project and the Population Council for other resources. We thank the Kenyan Ministry of Health's Division of Primary Health Care and its trainers, as well as all the providers and clients who participated in this evaluation. We gratefully acknowledge the contributions of Nancy Toroitich, Noni Gachuhi, Tamara Smith, Barbara Janowitz, Carmen Cuthbertson, Davy Chikamata, Jane Gitonga, Laura Johnson, Maureen Kuyoh, John Bratt, Julius Munyao, Suellen Miller and Zahida Qureshi.