Abstract

Health facility supervisors are in a position to increase motivation, manage resources, facilitate communication, increase accountability and conduct outreach. This study evaluated the effectiveness of a training intervention for on-site, in-charge reproductive health supervisors in Kenya using an experimental design with pre- and post-test measures in 60 health facilities. Cost information and data from supervisors, providers, clients and facilities were collected. Regression models with the generalized estimating equation approach were used to test differences between study groups and over time, accounting for clustering and matching. Total accounting costs per person trained were calculated. The intervention resulted in significant improvements in quality of care at the supervisor, provider and client–provider interaction levels. Indicators of improvements in the facility environment and client satisfaction were not apparent. The costs of delivering the supervision training intervention totalled US$2113 per supervisor trained. In making decisions about whether to expand the intervention, the costs of this intervention should be compared with other interventions designed to improve quality. KEY MESSAGES • A training intervention for on-site, in-charge reproductive health supervisors in low resource settings has the potential to improve the quality of care offered by providers. • Solutions are needed to minimize the negative impact of transfers by allowing for continuity if the supervisor is transferred or is away. • Supervisor training techniques are not specific to reproductive health services; thus, the intervention is probably appropriate for other types of health care supervisors and supervisors at different levels. • Cost information is helpful to guide decisions to allocate limited resources; the training can improve the quality of care at a cost of US$2113 per supervisor or even lower if implemented locally.

Introduction

Supervisors work at different levels within health systems, and their proximity to health services can influence their effect on quality of care. In traditional models of supervision the supervisor is often external to the facility and his or her job is to visit a health care facility on a regular basis to ‘inspect’ or ‘control’ performance (Garrison et al.2004). Under the supportive supervision model, supervision is not limited to the external, off-site supervisor but can occur throughout all levels of health services and include on-site supervisors (Marquez and Kean 2002). On-site, in-charge supervisors are physically closer to the provider and the client, have more regular contact with them than off-site supervisors, and some may perform their role as a supervisor often while also providing services (Garrison et al.2004). On-site, in-charge supervisors may be better placed to improve quality of care than external supervisors who visit facilities from time to time.

Supervisors at the various levels, on-site or off-site, have an important role to play in ensuring quality reproductive health services. They can affect quality by effectively managing existing resources, facilitating communication and feedback within a facility, and reaching out to the clients and communities to understand their perceptions of the programme quality. Supervisors can affect provider performance by improving provider motivation, promoting training opportunities, and holding providers accountable for the quality of their services.

Supportive supervision, a particular approach to supervision (and also synonymous with ‘facilitative supervision’), is an approach in which supervisors and providers engage in two-way communication and seek joint solutions to problems (Marquez and Kean 2002). This participatory approach draws on other sources of information, such as that from peers and the community, to improve provider performance. Staff members are accountable for quality as a team, so responsibility shifts from the individual to the group, and staff are encouraged to monitor their own quality through self- and peer-assessments.

Despite a strong theoretical backing and widespread reliance on supervision as a key strategy to improve family planning and health service quality of care in developing countries (Marquez and Kean 2002), there is a paucity of published literature from developing countries on the effects of supervisor training on client outcomes. The empirical link between enhanced supervision and improved client outcomes is intuitive but only one study has attempted to test this link. In Bangladesh, a study that focused on improving family planning officers’ competence found an average increase in the contraceptive prevalence rate of 9.1 points from baseline to follow-up (Jain et al.1999). Although reasonable evidence was provided to exclude any threats to internal validity, there was no control group comparison.

Other studies in family planning and primary health care services have documented the process by which enhanced supervision may influence client outcomes. Studies from Ghana and Zimbabwe suggest providers benefit directly from improved supervision in terms of communication, knowledge, skills and services provided (Combary et al.1999; Trap et al.2001). Other research from the Philippines, Indonesia and Mexico has shown that providers’ ability to provide high quality care is linked to increases in client knowledge and satisfaction (Kols and Sherman 1998; Kim et al.2000; Costello et al.2001; Kim et al.2002). However, documenting the effect of supervision on client outcomes, such as satisfaction and contraceptive uptake and continuation, is still lacking.

No known information exists about quality of on-site supervision and no published studies have examined the effectiveness of training the on-site, in-charge supervisor, although all health facilities have at least one such supervisor.

In the past decade in Kenya, there has been substantial effort to improve the quality of reproductive health services. The coverage of external, or traditional, supervision visits appears to have increased over time. According to the 2004 Service Provision Assessment,1 91% of government facilities had received a supervisory visit during the last 6 months compared with 68% in the previous 3 months in 1999 (MOH et al.2000; NCAPD et al.2005). However, in 2004, a smaller proportion of facilities (67%) reported the package of ‘supportive management practices’. The package included the supervision visit to the facility in the last 6 months, but also counted whether at least half of all providers had received in-service training during the past year and whether the interviewed provider was personally supervised in the last 6 months (NCAPD et al.2005).

In Kenya, the person on-site who is the immediate supervisor of reproductive health services is also known as the ‘in-charge’. Results from a study in Kenya strengthen the rationale for focusing on the on-site, in-charge supervisor. The study investigated the qualities associated with clinics that consistently exceeded expectations and that come highly recommended as a source of reproductive health care. In the study, JHPIEGO, an affiliate of Johns Hopkins University that works globally to improve the quality of health care for women and families, in collaboration with the Division of Reproductive Health (DRH) of the Kenya Ministry of Health (MOH) found that two of the most important characteristics of these ‘high performing sites’ were the presence of well-trained and motivated staff, and capable and dynamic leadership and management (Rawlins et al.2001). In response to the study findings, JHPIEGO developed a supportive supervision training package for performance improvement to be used with on-site, in-charge supervisors in Kenya.

The purpose of the current study is to evaluate the effect of the JHPIEGO training programme for on-site, in-charge supervisors on several aspects of reproductive health services at the supervisor, facility, provider and client levels. The conceptual model in Figure 1 suggests that supportive supervision at the on-site, in-charge level can influence client satisfaction by working through improved client–provider interactions, facility functioning and provider work environment (Figure 1). In turn, improved quality of services and client satisfaction can ultimately influence improved behaviours and health, although measuring changes in behaviour and health was outside the scope of the study.

Figure 1

Conceptual model

Figure 1

Conceptual model

Methods

Intervention

To achieve supportive supervision, JHPIEGO developed a Performance Quality Improvement (PQI) training package that encourages participants to identify the root causes limiting performance, and then participants identify unique and creative solutions to the obstacles limiting performance (JHPIEGO 2000; Measure Evaluation Project 2000; Garrison et al.2004). Under this approach, supervisors learn skills that help them to create a shared vision with stakeholders, to define desired performance for their health facilities, to assess their site's performance, to identify the causes of gaps in performance, to identify and implement interventions to improve performance, and to monitor and evaluate performance.

The training package was pre-tested in one district of Kenya, and lessons learned were incorporated into a final package which included a reference manual, trainer's guide and trainee's notebook. In addition, during the training supervisors developed action plans to utilize their updated knowledge, skills and materials.

The training was implemented by JHPIEGO with the MOH in a 1-week performance-improvement training carried out with two groups of 15 supervisors over 2 weeks (8–19 April 2002). During the initial training workshop, supervisors were to be equipped with the necessary knowledge, attitudes and skills required of a supervisor using the PQI process. The components of the training included how to define desired performance (e.g. set performance standards or expectations for confidentiality), assess performance (e.g. conduct a peer assessment of client–provider interactions), find root causes (e.g. identify factors such as provider knowledge that affect performance), select and implement interventions (e.g. match solutions such as on-the-job training to identified root causes), and monitoring and evaluating performance (e.g. use methods to assess performance and find root causes to evaluate whether intended changes have been achieved) (Garrison et al.2004). Supervisors also acquired some tools which included a checklist for preparing a meeting, a job description example, ‘do's and don'ts’ of active listening, and sample client exit interview questions.

Rather than relying on a long training programme that would keep supervisors away from their jobs, JHPIEGO subsequently mailed articles and job aids on the following: leadership, how to give feedback, the roles of a supervisor, and infection prevention. They also conducted a 1-day follow up meeting in August 2002 with supervisors to discuss implementation of the action plan, difficulties, challenges and problem solving. In February/March 2003, the MOH and JHPIEGO conducted site visits with the trainees. The purpose of these visits was to support supervisors in the implementation of their action plans, to observe supervisors, to give feedback, and to answer any questions and fill in gaps.

Design

To evaluate the effectiveness of the training intervention, this study used an experimental design with pre- and post-test measures and randomly assigned intervention and control groups. For this study and for the training, the on-site in-charge supervisor was defined as the in-charge of maternal and child health (MCH) and family planning in a district or sub-district hospital or the in-charge in a health centre. Providers included in the study were supervised by the in-charge. The representative sample of clinics included 72 health facilities used in a previous FHI/JHPIEGO/Population Council collaborative study where providers were trained in the use of updated reproductive health/family planning guidelines (Stanback et al.2001). In the previous study, the training was implemented in all facilities in the country, although 36 of the 72 study facilities received a less intensive training. This combined with the fact that the previous training did not involve improving supervision should not affect the representativeness of the facilities in the current study.

Power calculations, using a formula suggested by Feuer and Kessler (1989), determined that 60 facilities were sufficient for this study. To arrive at a sample size of 60 from the previous study sample of 72, five small clinics were purposively dropped from the original sample because there were fewer than three providers (in other words, one of the two would have been the on-site supervisor). We dropped another seven at random. For group assignment, facilities were matched on type of facility (hospital, sub-district hospital, heath centre or dispensary), urban or rural location, type of reproductive health services available (family planning, sexually transmitted infections [STIs] and/or MCH), and district. Matched pairs were randomly assigned into training and control groups. The sample of 60 facilities represented 20 districts in the provinces of Nyanza, Eastern, Rift Valley, Central, Western and Coast.

Data collection

Baseline data were collected during April 2002 and follow-up data were collected during March/April 2003. At each facility, there was an interview with the on-site in-charge supervisor, a facility assessment, provider interviews, client–provider interaction observations, and client exit interviews. Instruments contained mainly close-ended questions (i.e. quantitative), though some were open-ended. Most questions required a spontaneous answer from respondents with no prompting from research assistants. Clients eligible for the observation or interview had to be family planning, MCH (antenatal or postnatal clients were given priority), or STI/HIV clients.

While the number of facilities was justified using power calculations, all other units of observation (supervisor, provider and client) were determined logistically. Data collection teams stayed in each facility for 1 day. Research assistants interviewed all supervisors and on-duty providers. The number of client interviews and client–provider observations depended on the client load at that facility.

Indicators for the study were drawn mainly from two sources. Elements of quality of care were selected from a Maximizing Access and Quality (MAQ) document that serves as a comprehensive listing of all components of quality and access relevant to family planning services based on consensus from experts in the field of family planning and reproductive health quality (MAQ 1998). ‘High priority’ items from this document include: clients select and receive their chosen family planning method, providers address contraceptive side effects, providers have received good refresher training recently, facilities have adequate utilities, facilities have commodities available, and rooms have visual and auditory privacy. We also added other elements of quality such as adequate space and infection prevention procedures. To improve measurement, the proposed indicators were operationalized specifically for this study.

Other study indicators were drawn from elements that were included in the supervisor training and would indicate whether the supervisor implemented performance improvement techniques they had learned (Garrison et al.2004). For example, this included techniques to assess provider performance, to motivate staff, to plan meetings, etc. Supervisors learned a number of techniques for assessing facility performance; we focused on supervisory and peer assessment of providers' interactions with clients.

While we tried to capture the same indicators at baseline and post-intervention, this was not always possible. In some cases, measures were so specific to the training that they were not realistic to measure at baseline (e.g. the 10 elements for building teams). Other measures were added after the baseline assessment had already occurred because of changes to the intervention that were made after it was pre-tested, for example, the emphasis on confidentiality.

Data analysis

Facilities and the supervisors, providers and clients in them were randomized to either one of the study arms. Data collection methods included in the analysis were supervisor interviews, facility functionality inventory, provider interviews or client–provider interaction (CPI) observations, or client exit interviews. The analysis of facility, supervisor and provider-based outcomes was conducted under a cohort design that accounts for clustering due to repeated measurements at pre- and post-test, while the analysis of client-based outcomes was based on a cross-sectional design that accounts for the clustering due to facility. All of the analyses were based on an intent to treat, meaning those facilities assigned to the intervention and control groups, and hence the supervisors, providers and clients associated with those facilities, were treated as such in the analysis. This is an accepted way of analysing randomly assigned data and prevents bias caused by loss of participants.

Key variables were summarized by group and study time. Regression models with the generalized estimating equation (GEE) approach with identity link for quantitative variables or with logit link for binary variables, where appropriate, were conducted to assess the training effect from pre-test to post-test, accounting for clustering and matching. The matching of facilities on type of facility, urban or rural location, type of reproductive health services available, and district was accounted by allowing the association of the matched facility pairs in the correlation matrix of the model. The models also include variables for time (pre- or post-test) and an interaction term of time and group. The interaction term allows us to understand if the training group had a significantly larger and positive increase from pre-test to post-test compared with the control group. In some cases where measures are available at post-test only, the analyses were based on the model that excludes the time and the interaction term of time and group.

Data were analysed using SAS version 9.1 (Cary, NC). Proc GENMOD of SAS was used for fitting the models. P-values less than 0.05 were considered significant. Though power calculation was done for a certain variable only, the training intervention effect was also assessed for other key variables. Accordingly, P-values corresponding to these assessments are considered as additional information.

Cost data collection and analysis

The economic component of this study assessed the costs of the supervisor training intervention by measuring the costs of the four primary intervention components: training preparation activities and curriculum development, the 1-week intensive training session, the brief follow-up training activity, and the supportive supervision visits conducted in the field.

An Excel-based costing instrument was developed by Family Health International (FHI) to track the use of personnel and non-personnel resources used during all four components of the intervention. Cost data were collected and recorded by JHPIEGO staff using this Excel-based instrument. During an initial data cleaning, FHI staff followed up with JHPIEGO staff to clarify data inconsistencies. Subsequently, FHI staff further analysed the collected data for reliability and made adjustments as necessary.

The analysis of costs was conducted primarily from an accounting perspective to identify the ‘additional outlays’ necessary to implement the four phases of the intervention. Resources include personnel (salary of persons that participated in various activities and per diems) as well as non-personnel items (including transportation, venue, mailings, materials and supplies). Although potentially important in terms of overall management decision-making within a local resource scenario (i.e. MOH conducting training scale-up activities), the economic costs of participant time were not calculated. In this analysis, total accounting costs, total accounting cost per intervention component, and the total cost per person trained were calculated.

Under the expectation that the training would be conducted locally on scale up, we conducted an analysis of costs assuming that local JHPIEGO staff would conduct the entire training and all support supervision visits in the field. Furthermore, since the performance improvement training package had already been developed by JHPIEGO and was expected to be replicated in its existing form, costs associated with the development of the training package were not included in this scenario. Some limited personnel time for becoming acquainted with the curriculum was included. Lastly, research costs were excluded from these analyses.

Ethical approval for the study was obtained in the United States and Kenya, and verbal informed consent was obtained from all participants.

Results

In both training and control groups, the majority of supervisors were supervisors of health centres; in the training group 17 of the 30 were from health centres, and 16 of 30 in the control group (Table 1). The remaining supervisors were supervisors in hospitals, sub-district hospitals, and sub-heath centres. The majority of supervisors reported their clinical job title as ‘nurse’. Providers were mainly nurses or midwives.

Table 1

Number of types of health facilities by group; supervisors’, providers’ and clients’ background characteristics by group at baseline; and sample sizes for each data collection activity by timing

Training Control
Type of health facility
Health centre 17 16
District hospital
University hospital
Sub-district hospital
Sub-health centre
Baseline comparison of supervisors’, providers’ and clients’ characteristics
Supervisors (n = 30) (n = 30)
Mean age (in years) 36.8 38.4
Mean number of living children 2.3 2.6
Female 50.0% 60.0%
Marital status
Married, living together or apart 87% 73%
Single/divorced/separated/widowed 13% 27%
Professional qualifications
Clinical officer 27% 27%
Nurse/midwife 73% 73%
Providers (n = 96) (n = 100)
Average age (in years) 39.1 39.7
Mean no. of living children 3.2 2.8
Female 89% 77%
Marital status
Married, living together or apart 79% 83%
Single/divorced/separated/widowed 21% 17%
Clients (n = 132) (n = 124)
Mean age (in years) 25.9 25.8
Mean no. of living children 2.4 2.4
Female 98% 98%
Marital status
Married, living together or apart 79% 83%
Single/divorced/separated/widowed 31% 17%
Highest level of education completed
No schooling 3% 5%
Primary 69% 68%
Secondary or higher 28% 27%
Main occupation
Housewife 43% 40%
Farmer 27% 31%
Salaried worker or other 7% 13%
Services client sought on day of interviewa
Family planning 43% 44%
Maternal and child health 59% 59%
STI/HIV/AIDS 8% 9%
Sample sizes for each data collection activity Baseline Post-test
On-site supervisor interviews and facility inventories 60 60
Provider interviews 196 178
Client interviews 256 274
CPI observations 256 274
Training Control
Type of health facility
Health centre 17 16
District hospital
University hospital
Sub-district hospital
Sub-health centre
Baseline comparison of supervisors’, providers’ and clients’ characteristics
Supervisors (n = 30) (n = 30)
Mean age (in years) 36.8 38.4
Mean number of living children 2.3 2.6
Female 50.0% 60.0%
Marital status
Married, living together or apart 87% 73%
Single/divorced/separated/widowed 13% 27%
Professional qualifications
Clinical officer 27% 27%
Nurse/midwife 73% 73%
Providers (n = 96) (n = 100)
Average age (in years) 39.1 39.7
Mean no. of living children 3.2 2.8
Female 89% 77%
Marital status
Married, living together or apart 79% 83%
Single/divorced/separated/widowed 21% 17%
Clients (n = 132) (n = 124)
Mean age (in years) 25.9 25.8
Mean no. of living children 2.4 2.4
Female 98% 98%
Marital status
Married, living together or apart 79% 83%
Single/divorced/separated/widowed 31% 17%
Highest level of education completed
No schooling 3% 5%
Primary 69% 68%
Secondary or higher 28% 27%
Main occupation
Housewife 43% 40%
Farmer 27% 31%
Salaried worker or other 7% 13%
Services client sought on day of interviewa
Family planning 43% 44%
Maternal and child health 59% 59%
STI/HIV/AIDS 8% 9%
Sample sizes for each data collection activity Baseline Post-test
On-site supervisor interviews and facility inventories 60 60
Provider interviews 196 178
Client interviews 256 274
CPI observations 256 274

a 28 clients sought more than one service (multiple responses allowed).

Few differences between supervisors, providers and clients in training and control facilities at baseline were noted (Table 1). Supervisors were more likely to be married and living with their partner in the training group. Providers in the training group were more likely to be female. No other noticeable differences between training and control groups at baseline were noted including among client background characteristics.

A summary of sample sizes for each data collection activity by timing is given in Table 1. Although 60 supervisors, one per facility, were interviewed at baseline and post-test, these were not necessarily the same supervisors each time. Six supervisors in the training group were transferred from their facility before post-training data were collected. In the control group, 10 of the supervisors had been transferred and two were on leave at post-test.

The presentation of results is organized according to the conceptual framework (Figure 1). Thus, the section outlines the effect of the intervention on data collected on the following groups: on-site, in-charge supervisor, facility and provider work environment, client–provider interaction, and client satisfaction.

On-site, in-charge supervisors’ techniques and knowledge

The objective for this section of the study was to determine whether supervisors’ techniques in motivation, supervision, and communication and knowledge improved as a result of the intervention. Supervisors either learned these techniques as part of the training or had access to resources with suggested techniques. All responses were obtained without relying on prompts to supervisors. Supervisor reports suggest that supervisors in the training group knew significantly more techniques than supervisors in the control group to assess provider performance, to motivate staff, and to communicate their expectations to staff (Table 2).

Table 2

Summary of reported supervisory techniques and job-related knowledge scores by group and study time, and corresponding P-values for testing the training intervention effect obtained from regression with GEE

Training Control
Prec (n = 30) Post (n = 30) Prec (n = 30) Post (n = 30) P-value for assessing the intervention effecta,b
Techniques used to assess individual provider performance – 2.6 – 1.5 0.0007
Techniques used to motivate staff 1.6 3.0 2.2 2.6 0.0272
Techniques used to communicate expectations – 2.6 – 1.6 0.0016
Knowledge of team building elementsd – 3.6 – 2.5 0.0009
Knowledge of meeting planning elementsd – 3.8 – 2.5 <0.0001
Knowledge of conducting meetings elementsd – 3.8 – 2.1 0.0001
Training Control
Prec (n = 30) Post (n = 30) Prec (n = 30) Post (n = 30) P-value for assessing the intervention effecta,b
Techniques used to assess individual provider performance – 2.6 – 1.5 0.0007
Techniques used to motivate staff 1.6 3.0 2.2 2.6 0.0272
Techniques used to communicate expectations – 2.6 – 1.6 0.0016
Knowledge of team building elementsd – 3.6 – 2.5 0.0009
Knowledge of meeting planning elementsd – 3.8 – 2.5 <0.0001
Knowledge of conducting meetings elementsd – 3.8 – 2.1 0.0001

a For pre-post, P-value is associated with the interaction term of time and group fitted from a model that included group, time and the interaction term of time and group as the explanatory variables and accounted for clustering and matching.

b For post only, P-value is associated with the group from a model that included group as the explanatory variable and accounted for clustering and matching.

c For post-test observation only measures, this will be missing.

d For job knowledge, a score of 10 was possible.

Supervisors gave examples of the techniques they used to assess performance. These included obtaining client feedback, obtaining client satisfaction data, seeking general impressions from staff, observing skills, relying on service statistics. To motivate staff, popular techniques included tea breaks or staff parties, time off, financial benefits, and awards or certificates. To communicate expectations, supervisors demonstrated their expectations, held group meetings, had one-on-one conversations, allocated duties or wrote memos.

In terms of job knowledge, supervisors in the training groups were significantly more likely to be able to name elements that are important to team building, planning meetings or conducting meetings, although measures were post-training only (Table 2). This is not particularly surprising since the specific topics and elements were covered during the training, and these scores do not necessarily mean they implemented team building or other elements. However, the overall scores were low (3.6–3.8 out of 10 for the intervention group) despite statistically significant differences between groups.

Examples of team building elements included knowing the importance of limiting the size of the team, developing goals, providing feedback and developing an action plan. For meeting planning, supervisors knew such elements as determining the need for the meeting, developing objectives, gathering information prior to the meeting and preparing an agenda. Meeting elements included starting on time, welcoming attendees, facilitating discussion and summarizing the content.

Improvements in facility functioning and provider work environment

According to the conceptual model, improved supervision should lead to improvements in performance. In terms of facility functioning, the proportion of supervisors in both groups that report identifying performance problems increased over time (77% to 93% in the training group vs. 53% to 90% in the control group) (results not shown). These findings were confirmed by providers who reported that the identification of major facility performance problems had increased over time for both groups (40% to 54% in the training group vs. 41% to 51% in the control group) (results not shown). Because both groups improved by similar amounts over time, there were no statistically significant differences.

Little change was detected from pre-test to post-test and few differences between training and control facilities were found with respect to facility amenities (i.e. electricity and waiting room) (Table 3). Baseline measures of clean water and working toilets were almost universal in both groups at baseline; thus, those results are excluded. During the observation of examination area conditions, one statistically significant difference was found from baseline to post-training between training and control group facilities: almost all training group facilities had adequate water in the examination area during post-test measures. Both groups increased in the proportion where sharp objects and needles were incinerated or buried, although this was of borderline statistical significance (P = 0.07) in favour of the training group facilities. Although few statistical differences were found, the training group improved over time across all but one of the facility amenities and examination area indicators, while the control group facilities’ auditory privacy, visual privacy, clean linen and cleanliness appeared to decrease between pre and post test.

Table 3

Summary of observed facility amenities, examination area conditions and equipment availability by study group and time, and corresponding P-values for testing the training intervention effect obtained from regression with GEE

Training Control
Pre (n = 30) % Post (n = 30) % Pre (n = 30) % Post (n = 30) % P-value for assessing the intervention effecta,b
Facility amenities
Electricity 70 63 57 60 0.3145
Waiting room/area for clients 83 97 80 97 0.8801
Examination area conditions
Auditory privacy 67 70 60 50 0.3605
Visual privacy 77 80 63 53 0.4126
Cleanlinessc 83 100 87 80 –
Adequate lightc 83 100 77 90 –
Adequate waterc 80 97 83 77 0.0258
Clean linen 70 80 67 53 0.1506
Sharp objects and needles incinerated or buried 57 97 63 83 0.0681
Training Control
Pre (n = 30) % Post (n = 30) % Pre (n = 30) % Post (n = 30) % P-value for assessing the intervention effecta,b
Facility amenities
Electricity 70 63 57 60 0.3145
Waiting room/area for clients 83 97 80 97 0.8801
Examination area conditions
Auditory privacy 67 70 60 50 0.3605
Visual privacy 77 80 63 53 0.4126
Cleanlinessc 83 100 87 80 –
Adequate lightc 83 100 77 90 –
Adequate waterc 80 97 83 77 0.0258
Clean linen 70 80 67 53 0.1506
Sharp objects and needles incinerated or buried 57 97 63 83 0.0681

a For pre-post, P-value is associated with the interaction term of time and group fitted from a model that included group, time and the interaction term of time and group as the explanatory variables and accounted for clustering and matching.

b Since the model did not converge, P-value is missing.

cCleanliness’ means at the start of the day, floors swept and mopped, no dust on windowsills and tables. ‘Adequate light’ means functioning electric light or sufficient natural light. 'Adequate water' means a sufficient quantity of clean water for washing hands and equipment.

One of the most striking results was that after the training, providers in the group in which supervisors were trained were significantly more likely to report being observed during interactions with their clients compared with providers in the control group (Table 4). The majority of these observations were carried out by the on-site supervisor, but the main contributor to the increase was more observations by a colleague. In addition, providers in the training group were more likely to get feedback.

Table 4

Summary of provider reports of the likelihood of being observed during interactions with clients and of receiving feedback on performance if observed by study group and time, and corresponding P-values for testing the training intervention effect obtained from regression with GEE

Training Control
Pre (n = 96) % Post (n = 89) % Pre (n = 100) % Post (n = 89) % P-value for assessing the intervention effecta
Reported observation of performance with clients at least once in last 6 months 31 53 33 29 0.0036
If observed …
Given feedback 81 92 73 77 n.a.
Observed by on-site supervisor 52 56 42 46 n.a.
Observed by colleague 23 15 n.a.
Training Control
Pre (n = 96) % Post (n = 89) % Pre (n = 100) % Post (n = 89) % P-value for assessing the intervention effecta
Reported observation of performance with clients at least once in last 6 months 31 53 33 29 0.0036
If observed …
Given feedback 81 92 73 77 n.a.
Observed by on-site supervisor 52 56 42 46 n.a.
Observed by colleague 23 15 n.a.

a For pre-post, P-value is associated with the interaction term of time and group fitted from a model that included group, time and the interaction term of time and group as the explanatory variables and accounted for clustering and matching.

Quality of client–provider interactions

Research assistants observed providers’ interaction with clients to assess how quality may have changed (not to be confused with providers’ reports of observations). The majority of observations occurred in the maternal/child health care setting (usually antenatal care clients) or family planning services. Less than 10% of the observations observed were STI/HIV sessions.

Overall, the client–provider interaction observations suggest that providers in the training group improved their performance more than the control group (Table 5). In this study, infection prevention was an important indicator of quality health care regardless of the service being provided. For two of the five observed infection prevention activities in Table 5, significant differences were found; providers in the training group were more likely to wash their hands before and after a procedure than the control group.

Table 5

Summary of observed providers’ infection prevention, communication and confidentiality procedures by study group and time, and corresponding P-values for testing the training intervention effect obtained from regression with GEE

Training Control
Prec (n = 132) % Post (n = 135) % Prec (n = 126) % Post (n = 139) % P-value for assessing the intervention effecta,b
Infection prevention
Uses gloves (if needed) 21 17 15 11 0.7933
Uses sharps disposal containers (if sharp objects used) 96 95 86 90 0.5263
Washes hands with soap and water before procedure 15 38 15 13 0.0014
Washes hands with soap and water after procedure 30 48 27 13 0.0001
Hazardous waste disposed of in leak-proof containers (if hazardous waste) 55 84 54 73 0.3629
Communication
Greets the client in a friendly/respectful manner – 93 – 76 0.0002
Refers to client by name – 59 – 29 <0.0001
Asks questions to ensure her/his understanding of what client is telling – 83 – 67 0.0085
Paraphrases client's response and asks the client if that was correct – 58 – 36 0.0001
Of clients given condoms, contraceptive method or drugs, % told of dose to take/when to take 46 83 68 75 0.0033
Of clients given condoms, contraceptive method or drugs, % told of side effects 25 48 34 39 0.0404
Confidentiality
No other person enters or leaves the room during exam – 66 – 49 0.0048
Consultation cannot be observed or overheard by others – 77 – 63 0.0072
During consultation, the door to the exam room is closed – 80 – 64 0.0033
Clients records are not left lying about – 97 – 84 0.0014
Provider informs client that whatever is discussed with client will not be discussed with anyone else – 27 – 17 0.0061
Training Control
Prec (n = 132) % Post (n = 135) % Prec (n = 126) % Post (n = 139) % P-value for assessing the intervention effecta,b
Infection prevention
Uses gloves (if needed) 21 17 15 11 0.7933
Uses sharps disposal containers (if sharp objects used) 96 95 86 90 0.5263
Washes hands with soap and water before procedure 15 38 15 13 0.0014
Washes hands with soap and water after procedure 30 48 27 13 0.0001
Hazardous waste disposed of in leak-proof containers (if hazardous waste) 55 84 54 73 0.3629
Communication
Greets the client in a friendly/respectful manner – 93 – 76 0.0002
Refers to client by name – 59 – 29 <0.0001
Asks questions to ensure her/his understanding of what client is telling – 83 – 67 0.0085
Paraphrases client's response and asks the client if that was correct – 58 – 36 0.0001
Of clients given condoms, contraceptive method or drugs, % told of dose to take/when to take 46 83 68 75 0.0033
Of clients given condoms, contraceptive method or drugs, % told of side effects 25 48 34 39 0.0404
Confidentiality
No other person enters or leaves the room during exam – 66 – 49 0.0048
Consultation cannot be observed or overheard by others – 77 – 63 0.0072
During consultation, the door to the exam room is closed – 80 – 64 0.0033
Clients records are not left lying about – 97 – 84 0.0014
Provider informs client that whatever is discussed with client will not be discussed with anyone else – 27 – 17 0.0061

a For pre-post, P-value is associated with the interaction term of time and group fitted from a model that included group, time and the interaction term of time and group as the explanatory variables and accounted for clustering and matching.

b For post only, P-value is associated with the group from a model that included group as the explanatory variable and accounted for clustering and matching.

c For post-test observation only measures, this will be missing.

Observers documented providers’ communication techniques and confidentiality. On all measures, providers in the training group were significantly more likely to use better communication techniques and to respect confidentiality, although these were post-test measures only (Table 5).

Providers in the training group were significantly more likely than the control group to improve communications with the client about his/her treatment or family planning method (Table 5). Providers in the training group were more likely to tell clients about what dose of the drug to take or when to take the family planning method (of those clients given a drug or family planning method). Providers in the training group were significantly more likely to tell clients about side effects (of those clients given a drug or a family planning method), compared with the control group and over time.

Client's reports of various service aspects and satisfaction

Given the higher level of quality of care in sites with trained supervisors documented in the previous sections, we turn to assessing clients’ reports. We measured whether the supervisor training led to improved content of services received, facility environment, confidentiality, waiting time and client satisfaction.

Exit interviews with clients reveal that almost all clients were female, and the typical client was married and living with her partner, had primary schooling, and worked as a housewife, farmer or small trader. There were no changes from pre-test to post-test or between training and control groups on clients’ reports of the content of the particular services they received. Clients in training and control groups reported similar levels of contraceptive method provision (if they were family planning clients), scheduling of follow-up appointments, referrals, and education about side effects of the method of contraception or medication received (results not shown). There were no differences in clients’ reports of facility cleanliness and privacy (Table 6). Unfortunately, in both groups from pre-test to post-test there was a large decline in the proportion of clients reporting that they thought their information will remain confidential.

Table 6

Summary of clients’ reports of facility environment, confidentiality, waiting time and satisfaction with services received by study group and time and corresponding P-values for testing the training intervention effect obtained from regression with GEE

Training Control
Prec (n = 132) % Post (n = 135) % Prec (n = 126) % Post (n = 139) % P-value for assessing the intervention effecta,b
Environment and confidentiality
Very satisfied with cleanliness of facility – 64 – 56 0.3564
Believes information will remain confidential 89 79 94 76 0.1417
Waiting time
Reported waiting time 30 minutes or less 67 56 55 62 0.0324
Thought waiting time was reasonable 87 75 79 79 0.0782
Satisfaction
Very satisfied with visit 67 67 74 60 0.1654
Treated well by provider 65 70 74 60 0.0628
Very satisfied with counselling and information received from provider – 76 – 67 0.1640
Training Control
Prec (n = 132) % Post (n = 135) % Prec (n = 126) % Post (n = 139) % P-value for assessing the intervention effecta,b
Environment and confidentiality
Very satisfied with cleanliness of facility – 64 – 56 0.3564
Believes information will remain confidential 89 79 94 76 0.1417
Waiting time
Reported waiting time 30 minutes or less 67 56 55 62 0.0324
Thought waiting time was reasonable 87 75 79 79 0.0782
Satisfaction
Very satisfied with visit 67 67 74 60 0.1654
Treated well by provider 65 70 74 60 0.0628
Very satisfied with counselling and information received from provider – 76 – 67 0.1640

a For pre-post, P-value is associated with the interaction term of time and group fitted from a model that included group, time and the interaction term of time and group as the explanatory variables and accounted for clustering and matching.

b For post only, P-value is associated with the group from a model that included group as the explanatory variable and accounted for clustering and matching.

c For post-test observation only measures, this will be missing.

The results of clients’ reports of their waiting time to see the provider were mixed (Table 6). Clients in the training group actually reported a significant increase in waiting time compared with the control group. Moreover, clients in the training group were less likely (although not to the point of statistical significance) to report that the wait was reasonable. Conversely, waiting time appeared to decrease over time for control group clients.

Clients’ reports of satisfaction with the visit held constant over time in the training group and declined in the control group (Table 6). Clients’ reports that they were treated well by the provider increased slightly over time in the training group, yet declined in the control group. This indicator approached statistical significance (P = 0.06). In the one post-test only measure, clients in the training group were more likely to report being very satisfied with the counselling and information received from the provider.

Costs

The costs of delivering the supervision training intervention totalled US$63 318,2 or US$2113 per supervisor trained. The largest proportion of costs was associated with the 1-week training (45%) and preparation costs accounted for 18% of total costs. Follow-up training and support supervision activities accounted for 15% and 22% of total costs, respectively (Table 7).

Table 7

Costs of the original supervision training intervention by category

Costs (US$)a Preparation Training Follow-up training Support supervision % of total costs 18% 45% 15% 22% Personnel: Trainers$9999 $9446$1831 $7183b Support personnel$1165 $2774$1103 –
Per-diems – $3339$1137 $1971 Venue –$8754 $3500 – Accommodation – – –$1618
Transportation – $925$979 $1061 Other direct costs: Materials and supplies$371 $3022$1293 $1915 Total$11 535 $28 260$9844 $13 746 Total cost$63 385
Costs (US$)a Preparation Training Follow-up training Support supervision % of total costs 18% 45% 15% 22% Personnel: Trainers$9999 $9446$1831 $7183b Support personnel$1165 $2774$1103 –
Per-diems – $3339$1137 $1971 Venue –$8754 $3500 – Accommodation – – –$1618
Transportation – $925$979 $1061 Other direct costs: Materials and supplies$371 $3022$1293 $1915 Total$11 535 $28 260$9844 $13 746 Total cost$63 385

a 1 US$= 76 KSh. b Supportive supervision team. The primary resources used in the preparation phase included the time of international trainers and support personnel to develop the curriculum. In the main training and the follow-up training phases, the main resources included personnel time in supporting and delivering the training, per diem costs of attendees, and venue costs. Lastly, the primary resource used in the support supervision phase included three local personnel and one international staff member who conducted the support supervision over a 2-week period. Costs of a ‘Local JHPIEGO Scenario’ Under the ‘Local JHPIEGO Scenario’, the local JHPEIGO staff would conduct the entire training and all support supervision activities. As the curriculum would already be available to local JHPIEGO staff, the preparation costs under this scenario would decrease by 70% compared with the original study. The three other phases utilize approximately the same level of resources as under the original study, but the salaries are slightly lower. These factors contribute to an overall lower cost of delivering the intervention under this scenario. The total cost of this scenario is US$50 030, or US$1669 per person trained (assumed 30 trainees), yielding an overall savings of US$13 302 or 20% of costs when compared with the original study. Figure 2 shows the cost differences between the original study and local JHPIEGO scenarios.

Figure 2

Cost comparison (original and local JHPIEGO scenarios)

Figure 2

Cost comparison (original and local JHPIEGO scenarios)

It is realistic to expect that the MOH will take responsibility for future trainings given the relatively recent implementation of the Decentralised Reproductive Health Training and Supervision (DRHTS) Teams. By relying on this resource, this training intervention could be implemented locally with substantial cost savings.

Discussion

The intervention appeared to result in significant improvements at the supervisor, provider and client–provider interaction levels, but not at the client level. A primary benefit of the intervention appears to be that providers are more likely to be observed and provided feedback about their interaction with their clients. Another important finding is that the performance of providers in the training group significantly improved with respect to communication with clients, infection prevention practices and confidentiality, important aspects of quality health care regardless of the service being provided. The documented improvements in the training group providers’ performance suggest that training supervisors can indirectly lead to improved quality of care offered by providers.

Results were not optimistic in terms of the success of the intervention when gauged against facility improvements. The on-site, in-charge supervisors may not be able to remedy broader systems-related problems such as lack of electricity, waiting room conditions or working toilets. However, results associated with examination area conditions in training groups were more promising, particularly with aspects such as having adequate water.

Findings pertaining to client satisfaction were promising, but not statically significant. The tendency for more satisfied clients in the training group suggests that trained supervisors may be able to affect client satisfaction. The reports of greater satisfaction occurred despite longer waiting times for clients in the training group. It is difficult to attribute the increased waiting times to the intervention, because longer waiting times may be due to an increase in client volume or a decrease in staffing. They could be due to the increase in time that providers spent with clients in order to improve their quality of care, although more research is needed to confirm this attribution. As for the noted decline in clients’ trust in confidentiality in both groups, unfortunately, we did not explore this finding in more depth. While this study did not find differences in whether improved supervision lead to improved client satisfaction, it is possible that better measures of the client's experience than just measures of client satisfaction are needed or that more time and/or bigger sample sizes are needed to detect changes. Further, clients’ reports do not necessarily represent the full picture; they are less likely to make note of some aspects of improved quality (such as hand washing).

The intervention was probably less effective in study facilities than it could have been because trained supervisors were transferred out of the intervention group and untrained supervisors were transferred into it, although transferred supervisors could have improved services in their new facilities. For example, three supervisors were transferred before post-test data collection and three did not receive the training. The practice of transferring in-charges will reduce the effectiveness of any training intervention unless the in-charges take their skills to their new facilities and mechanisms are in place to train the new supervisors. Solutions need to be identified, such as training the deputies of in-charge supervisors, to minimize the negative impact of transfers by allowing for continuity if the supervisor is transferred or is away. Further, the control group might have done better than expected. There were anecdotal reports of supervisors sharing materials with colleagues in other facilities, which could have affected supervisors in the control group.

The information on costs can inform policy decision-making at the MOH. Although the cost analyses were not specifically applied to a scenario where the MOH conducts the supervision training independently, several lessons can be observed from the original study and JHPIEGO scenarios. First, inputs for delivering the intervention would be similar to those accounted for in the local JHPIEGO scenario. The only foreseeable additional cost pertains to the training of trainers that would be necessary in a scale-up effort. However, under a scenario where the MOH takes the lead, a cost analysis from a broader economic perspective would be valuable for the MOH and the Kenyan Government. This analysis would incorporate the value of time that trainers spend preparing for and conducting trainings as well as the value of trainee participant time. Furthermore, to explore broader economic efficiencies within the MOH, an analysis of potential economic gains of training supervisors to conduct managerial activities across a variety of health domains, including reproductive health, should be considered.

Most of the techniques that the supervisor training addresses are not specific to reproductive health services. Thus, the intervention is probably appropriate for other types of supervisors including higher level on-site supervisors, such as medical officers and hospital matrons, and off-site supervisors. Participants in the dissemination meeting made recommendations to this effect, and JHPIEGO has already applied the training with different types of supervisors in Kenya, including different types of on-site supervisors such as hospital matrons as well as off-site supervisors such as health inspectors and members of district health management teams (public heath nurses) (Lynam 2003).

References

Combary
P
Newman
C
Glover
K
, et al.  .
Technical Report 7: Study of the effects of technical supervision training on CBD supervisor's performance in seven regions of Ghana
1999
Chapel Hill, NC
IntrahHealth
Costello
M
Lacuesta
M
RamaRao
S
Jain
A
A client-centered approach to family planning: the Davao project
Studies in Family Planning
,
2001
, vol.
32
(pg.
302
-
14
)
Feuer
EJ
Kessler
LG
Test statistic and sample size for a two-sample McNemar test
Biometrics
,
1989
, vol.
45
(pg.
629
-
36
)
Garrison
K
Caiola
N
Sullivan
R
Lynam
P
Supervising healthcare services: improving the performance of people
,
2004
Baltimore, MD
JHPIEGO Corporation
Jain
SC
Barkat
A
Lundeen
K
, et al.  .
Improving family planning program performance through management training: the 3Cs paradigm
Journal of Health & Population in Developing Countries
,
1999
, vol.
2
(pg.
1
-
25
)
JHPIEGO Corporation
Performance improvement for quality reproductive health services
,
2000
Baltimore, MD
JHPIEGO Corporation
Kim
YM
Putjuk
F
Basuki
E
Kols
A
Self-assessment and peer review: improving Indonesia service providers’ communication with clients
International Family Planning Perspectives
,
2000
, vol.
26
(pg.
4
-
12
)
Kim
YM
Figueroa
ME
Martin
A
, et al.  .
Impact of supervision and self-assessment on doctor-patient communication in rural Mexico
International Journal of Quality of Health Care
,
2002
, vol.
14
(pg.
359
-
67
)
Kols
AJ
Sherman
JE
Family planning programs: improving quality
Population Reports
,
1998
Baltimore, MD
Johns Hopkins University School of Public Health, Population Information Program
Lynam
P
Supervision training in Kenya
2003
Presentation at a results dissemination meeting
11 November
Nairobi, Kenya
Maximizing Access and Quality (MAQ) Initiative
Checklist for family planning service delivery, with linkages to reproductive health
,
1998
Washington, DC
MAQ
Marquez
L
Kean
L
Making supervision supportive and sustainable: new approaches to old problems
MAQ Paper, No. 4
,
2002
Bethesda, MD
University Research Co., LLC
Measure Evaluation Project & Monitoring and Evaluation Committee of the MAQ Initiative
Quick Investigation of Quality (QIQ): A user's guide for monitoring quality of care
,
2000
Chapel Hill, NC
Measure Evaluation Project, Carolina Population Center
Ministry of Health [Kenya], National Council for Population and Development [Kenya], ORC Macro
Kenya Service Provision Assessment Survey 1999
,
2000
Calverton, MD
Ministry of Health [Kenya], National Council for Population and Development [Kenya], and ORC Macro
National Coordinating Agency for Population and Development (NCAPD) [Kenya], Ministry of Health [Kenya], Central Bureau of Statistics, ORC Macro
Kenya Service Provision Assessment Survey 2004
2005
Nairobi, Kenya
National Coordinating Agency for Population and Development, Ministry of Health, Central Bureau of Statistics, ORC Macro
Rawlins
B
Garrison
K
Lynam
P
, et al.  .
Study of high-performing healthcare facilities in Kenya
,
2001
Baltimore, MD
JHPIEGO Corporation
Stanback
J
Griffey Brechin
SJ
, et al.  .
The effectiveness of national dissemination of updated reproductive health / family planning guidelines in Kenya. Final report
2001
Research Triangle Park, NC
Family Health International
Trap
B
Todd
CH
Moore
H
Laing
R
The impact of supervision on stock management and adherence to treatment guidelines: a randomized controlled trial
Health Policy and Planning
,
2001
, vol.
16
(pg.
273
-
80
)