Abstract

Purpose

To review the literature concerning strategies for implementing quality indicators in hospital care, and their effectiveness in improving the quality of care.

Data sources

A systematic literature study was carried out using MEDLINE and the Cochrane Library (January 1994 to January 2008).

Study selection

Hospital-based trials studying the effects of using quality indicators as a tool to improve quality of care.

Data extraction

Two reviewers independently assessed studies for inclusion, and extracted information from the studies included regarding the health care setting, type of implementation strategy and their effectiveness as a tool to improve quality of hospital care.

Results

A total of 21 studies were included. The most frequently used implementation strategies were audit and feedback. The majority of these studies focused on care processes rather than patient outcomes. Six studies evaluated the effects of the implementation of quality indicators on patient outcomes. In four studies, quality indicator implementation was found to be ineffective, in one partially effective and in one it was found to be effective. Twenty studies focused on care processes, and most reported significant improvement with respect to part of the measured process indicators. The implementation of quality indicators in hospitals is most effective if feedback reports are given in combination with an educational implementation strategy and/or the development of a quality improvement plan.

Conclusion

Effective strategies to implement quality indicators in daily practice in order to improve hospital care do exist, but there is considerable variation in the methods used and the level of change achieved. Feedback reports combined with another implementation strategy seem to be most effective.

Introduction

With increasing frequency, hospitals in various countries report and monitor indicator data in order to improve the quality of care [1–4]. Quality indicators aim to detect sub-optimal care either in structure, process or outcome, and can be used as a tool to guide the process of quality improvement in health care [5]. Monitoring the health care quality makes hospital care more transparent for physicians, hospitals and patients. Furthermore, it provides information to target quality improvement initiatives. However, collection of indicator data also implies an administrative burden for physicians and hospitals; therefore, the use of this information should be optimized. It is unclear which implementation strategy for quality indicators is optimal, and what effects can be achieved when quality improvement is guided by indicator information.

The implementation of quality indicators as a tool to assist quality improvement requires effective communication strategies and the removal of hindrances [6]. Evidence suggests that audit and feedback based on indicator data can be effective in changing health care professional practice [7, 8]. Monitoring the indicator data may also help to target specific quality improvement initiatives such as educational programs and development of protocols.

The effect of monitoring indicator data to promote quality improvement, and ultimately patient care, has been demonstrated in specific situations. For example, in the Bradford Teaching Hospital in the United Kingdom, feedback of mortality rates resulted in the reduction of the standardized mortality rate from 0.95 to 0.75 [9].

At present, no clear overview is available about strategies for implementing indicators and the effects on quality of care in hospitals. Some reviews do address the issue of implementation of indicators, but do not focus on hospital care [10, 11]. Another review of the literature has a limited focus on audit and feedback as implementation strategies [8]. With respect to the effectiveness of using indicators to promote quality improvement, previous reviews have focused on specific diseases or medical disciplines, e.g. pneumonia or cardiac surgery [12, 13]. In our review, we focus on hospital care in general, and take into account all possible implementation strategies described in the literature. The purpose of our study is firstly to review the literature concerning strategies for implementing quality indicators, and secondly to examine their effectiveness in improving the quality of hospital care.

Methods

Data source

A systematic literature search was conducted in MEDLINE and the Cochrane Library for the period from January 1994 to January 2008. We searched all articles published in the English and Dutch languages. The search was limited to randomized controlled trials (RCTs), controlled clinical trials (CCTs) and controlled before–after studies (CBAs), as categorized in MEDLINE. A RCT is the most robust study design to show the effect of quality improvement strategies [14]. However, as some strategies are not amenable to randomization, we also included non-randomized trials.

The search strategy in MEDLINE combined a truncated search for ‘quality indi*’ with the text words ‘hospital care’ or ‘quality improvement’. In addition, we searched the Cochrane Library, based on the Medical Subject Heading: quality indicators, health care’. The reference lists of all retrieved articles were searched for additional relevant references.

Two reviewers independently assessed the studies for inclusion. In case of disagreement between the two researchers, a third researcher was consulted.

Study selection

Firstly, we selected studies based on the relevance of the focus of the study. Studies reporting the use of quality indicators as a tool to improve hospital care were included. Studies that measured care processes or patient outcomes were also included, if the focus was on inpatient care at the hospital level, ward, or individual specialist. Studies concerned with primary care, e.g. general practitioners, chronic health, mental health and dental care were excluded because the delivery of care may differ considerably in these care settings from the hospital care setting.

Secondly, we selected studies based on study design and quality of the study. Studies had to report a baseline and follow-up measurement, and include a control and an intervention group. The effects of the implementation strategy had to be quantified, and studies had to be carried out in two or more hospitals because of generalization of the results.

For those studies that met the inclusion criteria, we classified the implementation strategies in which the information on quality indicators was used directly into the following categories (see Table 1): (1) educational meeting, (2) educational outreach, (3) audit and feedback, (4) development of a quality improvement plan and (5) financial incentives.

Table 1

Classification of implementation strategies

Implementation strategies in which the information on quality indicators was directly used 
 Educational meeting Participation in conferences, seminars, lectures, workshops or training sessions. During these meetings, feedback of quality indicators was presented, and study participants discussed how to improve performance. 
 Educational outreach A trained independent person or investigator who met with health professionals or managers in their practice setting to provide information (e.g. feedback of quality indicators). 
 Audit and feedback Report including a summary of clinical performance over a specified period of time had to be given. 
 Development of a quality improvement plan A plan based on indicator data was used to improve the quality of care. 
 Financial incentive Rewarding individual health professionals or institutions with higher payments when they improve performance. 

 
Supporting activities 

 
 Distribution of educational material Distribution of educational material was used if published or printed recommendations for clinical care or quality improvement were used. 
 Local opinion leader Professionals named by their colleagues as influential with emphasis on acting as authority locally. 
 Quality improvement facilities An implementation process organized by a quality improvement team or organizations to improve the quality of care and implementing system support methods (support by phone or e-mail for quality improvement). 
Implementation strategies in which the information on quality indicators was directly used 
 Educational meeting Participation in conferences, seminars, lectures, workshops or training sessions. During these meetings, feedback of quality indicators was presented, and study participants discussed how to improve performance. 
 Educational outreach A trained independent person or investigator who met with health professionals or managers in their practice setting to provide information (e.g. feedback of quality indicators). 
 Audit and feedback Report including a summary of clinical performance over a specified period of time had to be given. 
 Development of a quality improvement plan A plan based on indicator data was used to improve the quality of care. 
 Financial incentive Rewarding individual health professionals or institutions with higher payments when they improve performance. 

 
Supporting activities 

 
 Distribution of educational material Distribution of educational material was used if published or printed recommendations for clinical care or quality improvement were used. 
 Local opinion leader Professionals named by their colleagues as influential with emphasis on acting as authority locally. 
 Quality improvement facilities An implementation process organized by a quality improvement team or organizations to improve the quality of care and implementing system support methods (support by phone or e-mail for quality improvement). 

Implementation strategies that did not directly use the information on quality indicators, but support the implementation, were categorized in ‘distribution of educational material’, ‘local opinion leaders’ and ‘quality improvement facilities’ (see Table 1). Educational meeting was regarded as a supporting activity if the meeting focused on quality improvement techniques instead of presenting feedback on quality indicators.

Common to all studies on which we focus in this review is the use of key information of structure, process and outcome of care, and the systematic use of this information to improve quality of care. Central to the use of quality indicators is the feedback of information. Therefore, in order to summarize the implementation strategies that were used, we categorized the contribution of feedback to the implementation strategy in ‘receiving no feedback report’, ‘receiving a feedback report only’ and ‘receiving a feedback report combined with another strategy, which also used quality indicators as part of the implementation strategy.’

For the studies included, information was collected concerning the healthcare setting, methods used to implement quality indicators in hospitals, and their effectiveness in improving the quality of hospital care. The effectiveness of these strategies may be explained by the fact that they are capable of dealing with different barriers simultaneously [15]. We have summarized the barriers reported in some of the studies.

Results

Selection of articles

As a result of the search, 516 studies were identified (see Fig. 1). Of these, 465 were excluded, because these studies did not aim to measure the effect of the use of quality indicators. Four additional new articles were obtained from the reference lists. A total of 55 articles was evaluated by two reviewers, based on the quality of the studies. Finally, 21 studies were included.

Figure 1

Flow chart of study selection process.

Figure 1

Flow chart of study selection process.

Study characteristics

We included nine RCTs [16–24], two CCTs [25, 26] and ten CBAs [27–36].

The majority of the trials were conducted in the United States (17 studies); the others were carried out in Canada [17], Australia [33], Sweden [16] and Laos [21]. Furthermore, quality indicators were in a wide range of medical disciplines within hospital care. The majority of studies focused on the use of quality indicators in cardiovascular care (67%) [16, 17, 19, 20, 22–24, 27, 29, 32–36]. Most studies (81%) aimed at improving quality of care in one specific medical discipline. The sample size showed great variation, from one to 379 hospitals in the intervention group (see Table 2).

Table 2

Characteristics and results of the studies included

First author, year Study design Clinical area Methods to implement quality indicators Effects on care processes Effects on patient outcome 
Pandey et al., 2006 [27CBA (CG = 6, IG = 7) Cardiovascular care Educ. outreach and educ. meeting vs. chart audit only No sign. improvement in 6 out of 7 process indicators, except for lipid screening (adj. OR 19.93; 90% CI 2.99–36.86) Not measured 
Carlhed et al., 2006 [16RCT (CG = 19, IG = 19) Cardiovascular care Real-time feedback report, educ. meetings and QI plan vs. no intervention Supporting activities (QI facilities, incl. QI team and ongoing support by phone and on request site visits) Sign. improvement in 4 out of 5 process indicators: use of ACE inhibitor 1.4% vs. 12.6% (P = 0.002), use of lip. low. 2.3% vs. 7.2% (P = 0.065), use of heparin 5.3% vs. 16.3% (P = 0.010) and use of Cor-Angio 6.2% vs. 18.8% (P = 0.027) Not measured 
Grossbart, 2006 [28CBA (CG = 6, IG = 4) Cardiovascular care, pneumonia, hip/ knee Feedback report and rewarding hospitals with an incentive bonus vs. no intervention Sign. improvement in composite process indicator scores of 6.7 % vs. 9.3% (P < 0.001) Not measured 
Moscucci et al., 2006 [29CBA (CG = 7, IG = 5) Cardiovascular care Quarterly + annual feedback reports, educ. outreach, educ. meeting, distribution educ. material and QI plan vs. no intervention Sign. improvement in all 6 process indicators Sign. improvement in 4 out of 6 outcome indicators; contrast nephropathy (adj. OR 0.59; 95% CI 0.44–0.77), emergency CABG (adj. OR 0.54; 95% CI 0.32–0.90), stroke (adj. OR 0.33; 95% CI 0.16–0.65) and death (adj. OR 0.57; 95% CI 0.40–0.82). 
Rosenthal et al., 2005 [30CBA (CG = 31, IG = 134) Cancer screening, mammography, hemoglobin testing Rewarding health care professionals with an incentive bonus vs. no intervention No sign. improvement in 2 out of 3 process indicators, except for cervical cancer screening (3.6% improvement, P = 0.02) Not measured 
Beck et al., 2005 [17RCT (CG = 38, IG = 38) Cardiovascular care Rapid feedback report vs. delayed feedback report No sign. improvement in any of the 12 process indicators No sign. improvement in mortality at 30 days after discharge (adj. OR 0.6; 95% CI –0.70–1.8) 
Snyder and Anderson 2005 [31CBA (CG = 142, IG = 199) Cardiovascular care, pneumonia Feedback report vs. no intervention Supporting activities (distribution educ. material and QI facilities, incl. assisting implementing system change) No sign. improvement in 14 out of 15 process indicators, except for pneumonia immunization (P = 0.005). Not measured 
Landon et al., 2004 [25CCT (CG = 25, IG = 44) HIV infection Monthly feedback reports, educ. meetings and QI plan vs. no intervention Supporting activities (QI facilities, incl. QI team and monthly conference calls) No sign. improvement in 7 out of 8 process indicators, except for screening and prophylaxis papanicolaou smear (P = 0.06) Not measured 
Horbar et al., 2004 [18RCT (CG = 57, IG = 57) Surfactant preterm infants Feedback report vs. no intervention Supporting activities (educ. meeting for QI techniques and QI facilities incl. ongoing support by quarterly conference calls and mail discussion list) Sign. improvement in all 3 process indicators: proportion receiving surfactant in delivery room (adj. OR 5.38; 95% CI 2.84–10.20), proportion receiving first surfactant >2 h after birth (adj. OR 0.35; 95% CI 0.24–0.53) and median time from birth to first dose surfactant (P < 0.0001) No sign. improvement in rate of death before discharge 
Berner et al., 2003 [19RCT (CG = 6, IG1 = 8, IG2 = 7) Cardiovascular care Educ. meeting and QI plan (IG1) vs. no intervention Supporting activities (distribution educ. material and IG2 added opinion leader) No sign. improvement in 4 out of 5 process indicators, except for antiplatet medication within 24 h for IG2 vs. CG (adj. OR 1.92; 95% CI 1.19–3.12) and antiplatet medication within 24 h for IG1 vs. IG2 (adj. OR 1.79; 95% CI 1.09–2.94) Not measured 
Chu et al., 2003 [26CCT (CG = 16 (crossover) IG = 20) Pneumonia Feedback report, educ. outreach and QI plan Supporting activities (distribution educ. material and QI facilities incl. support QI training and site visits) Sign. improvement in 2 out of 4 process indicators: antibiotics given emergency department (adj. OR 10.72; 95% CI 3.56–32.30) and blood culture obtained in 4 h (adj. OR 2.48; 95% CI 1.17–5.25) No sign. improvement in LOS and unadjusted mortality 
Ferguson et al., 2003 [20RCT (CG = 115 IG1 = 101 IG2 = 107) Cardiovascular care Feedback report, QI plan and distribution educ. material (one arm received educ. beta-blockade (IG1), other arm received educ. IMA grafting (IG2)) vs. no intervention Supporting activities (local opinion leader) Sign. improvement in 1 out of 2 process indicators: use of preoperative beta-blockade (IG1 group vs. CG (P < 0.001)) and (IG2 vs. CG (P = 0.02)) Not measured 
Wahlström et al., 2003 [21RCT (CG = 12, IG = 12) Malaria, diarrhea and pneumonia Educ. meetings vs. no intervention Supporting activities (educ. meeting incl. QI and QI facilities, incl. QI team) Sign. improvement in overall mean process indicator scores for malaria, diarrhea and pneumonia together (OR 0.63; 95% CI 0.16–1.112) Not measured 
Hayes et al., 2002 [22RCT (CG = 16, IG = 16) Cardiovascular care Educ. outreach vs. feedback report and educ. material Supporting activities (opinion leader, educ. meeting) No sign. improvement in 4 out of 5 process indicators, except for discharge counseling for daily weights (OR 2.63; 95% CI 1.14–6.07) Not measured 
Mehta et al., 2002 [32CBA (CG = 11, IG = 10) Cardiovascular care Educ. outreach, feedback report and QI plan vs. no intervention Supporting activities (opinion leader (outside hospital), distribution educ. material) Sign. improvement in 4 out of 8 process indicators: use of aspirin on admission (81% vs. 87%; P= 0.02), use of beta-blockers on admission (65% vs. 74%; P= 0.04), use of aspirin after discharge (84% vs. 92%; P= 0.002) and smoking counseling at discharge (53% vs. 65%; P= 0.02) Not measured 
Scott et al., 2001 [33CBA (CG = 112, IG = 1) Cardiovascular care Feedback reports and educ. meeting vs. no intervention Supporting activities (distribution educ. material) Not measured Sign. improvement in inpatient mortality (adj. OR, 0.59; 95% CI 0.45–0.77) 
Hayes et al., 2001 [23RCT (CG = 15, IG= 14) Cardiovascular care Educ. meeting and QI plan vs. feedback report Supporting activities (opinion leader and distribution educ. material) No sign. improvement in any of the 5 process indicators Not measured 
Sauaia et al., 2000 [34CBA (CG = 9, IG = 9) Cardiovascular care Educ. outreach and QI plan vs. mailed feedback report Supporting activities (opinion leader) No sign. improvement in any of the 7 process indicators Not measured 
Ellerbeck et al., 2000 [35CBA (CG = 73, IG = 44) Cardiovascular care QI plan based on feedback vs. no QI plan based on feedback Sign. improvement in 3 out of 8 process indicators: aspirin during hospitalization (6% vs. 13%) and at discharge (6% vs. 15%) and use of beta-blockers (14% vs. 22%). Not measured 
Marciniak et al., 1998 [36CBA (CG = not given, IG= 379) Cardiovascular care Feedback report and QI plan vs. no intervention Sign. improvement in 3 out of 7 process indicators: use of aspirin at discharge (OR, 5.6; 95% CI 2.6–8.7), use of beta-blockers (OR, 8.0; 95% CI 1.4–14.6) and smoking counseling (OR, 8.5; 95% CI 1.6–15.5) No sign. improvement in hospital mortality 
Soumerai et al., 1998 [24RCT (CG = 17, IG = 20) Cardiovascular care Educ. meeting vs. mailed feedback report Supporting activities (local opinion leader and distribution educ. material) Sign. improvement in 2 out of 4 process indicators: use of oral aspirin (P < 0.04) and use of beta-blockers (P < 0.02) Not measured 
First author, year Study design Clinical area Methods to implement quality indicators Effects on care processes Effects on patient outcome 
Pandey et al., 2006 [27CBA (CG = 6, IG = 7) Cardiovascular care Educ. outreach and educ. meeting vs. chart audit only No sign. improvement in 6 out of 7 process indicators, except for lipid screening (adj. OR 19.93; 90% CI 2.99–36.86) Not measured 
Carlhed et al., 2006 [16RCT (CG = 19, IG = 19) Cardiovascular care Real-time feedback report, educ. meetings and QI plan vs. no intervention Supporting activities (QI facilities, incl. QI team and ongoing support by phone and on request site visits) Sign. improvement in 4 out of 5 process indicators: use of ACE inhibitor 1.4% vs. 12.6% (P = 0.002), use of lip. low. 2.3% vs. 7.2% (P = 0.065), use of heparin 5.3% vs. 16.3% (P = 0.010) and use of Cor-Angio 6.2% vs. 18.8% (P = 0.027) Not measured 
Grossbart, 2006 [28CBA (CG = 6, IG = 4) Cardiovascular care, pneumonia, hip/ knee Feedback report and rewarding hospitals with an incentive bonus vs. no intervention Sign. improvement in composite process indicator scores of 6.7 % vs. 9.3% (P < 0.001) Not measured 
Moscucci et al., 2006 [29CBA (CG = 7, IG = 5) Cardiovascular care Quarterly + annual feedback reports, educ. outreach, educ. meeting, distribution educ. material and QI plan vs. no intervention Sign. improvement in all 6 process indicators Sign. improvement in 4 out of 6 outcome indicators; contrast nephropathy (adj. OR 0.59; 95% CI 0.44–0.77), emergency CABG (adj. OR 0.54; 95% CI 0.32–0.90), stroke (adj. OR 0.33; 95% CI 0.16–0.65) and death (adj. OR 0.57; 95% CI 0.40–0.82). 
Rosenthal et al., 2005 [30CBA (CG = 31, IG = 134) Cancer screening, mammography, hemoglobin testing Rewarding health care professionals with an incentive bonus vs. no intervention No sign. improvement in 2 out of 3 process indicators, except for cervical cancer screening (3.6% improvement, P = 0.02) Not measured 
Beck et al., 2005 [17RCT (CG = 38, IG = 38) Cardiovascular care Rapid feedback report vs. delayed feedback report No sign. improvement in any of the 12 process indicators No sign. improvement in mortality at 30 days after discharge (adj. OR 0.6; 95% CI –0.70–1.8) 
Snyder and Anderson 2005 [31CBA (CG = 142, IG = 199) Cardiovascular care, pneumonia Feedback report vs. no intervention Supporting activities (distribution educ. material and QI facilities, incl. assisting implementing system change) No sign. improvement in 14 out of 15 process indicators, except for pneumonia immunization (P = 0.005). Not measured 
Landon et al., 2004 [25CCT (CG = 25, IG = 44) HIV infection Monthly feedback reports, educ. meetings and QI plan vs. no intervention Supporting activities (QI facilities, incl. QI team and monthly conference calls) No sign. improvement in 7 out of 8 process indicators, except for screening and prophylaxis papanicolaou smear (P = 0.06) Not measured 
Horbar et al., 2004 [18RCT (CG = 57, IG = 57) Surfactant preterm infants Feedback report vs. no intervention Supporting activities (educ. meeting for QI techniques and QI facilities incl. ongoing support by quarterly conference calls and mail discussion list) Sign. improvement in all 3 process indicators: proportion receiving surfactant in delivery room (adj. OR 5.38; 95% CI 2.84–10.20), proportion receiving first surfactant >2 h after birth (adj. OR 0.35; 95% CI 0.24–0.53) and median time from birth to first dose surfactant (P < 0.0001) No sign. improvement in rate of death before discharge 
Berner et al., 2003 [19RCT (CG = 6, IG1 = 8, IG2 = 7) Cardiovascular care Educ. meeting and QI plan (IG1) vs. no intervention Supporting activities (distribution educ. material and IG2 added opinion leader) No sign. improvement in 4 out of 5 process indicators, except for antiplatet medication within 24 h for IG2 vs. CG (adj. OR 1.92; 95% CI 1.19–3.12) and antiplatet medication within 24 h for IG1 vs. IG2 (adj. OR 1.79; 95% CI 1.09–2.94) Not measured 
Chu et al., 2003 [26CCT (CG = 16 (crossover) IG = 20) Pneumonia Feedback report, educ. outreach and QI plan Supporting activities (distribution educ. material and QI facilities incl. support QI training and site visits) Sign. improvement in 2 out of 4 process indicators: antibiotics given emergency department (adj. OR 10.72; 95% CI 3.56–32.30) and blood culture obtained in 4 h (adj. OR 2.48; 95% CI 1.17–5.25) No sign. improvement in LOS and unadjusted mortality 
Ferguson et al., 2003 [20RCT (CG = 115 IG1 = 101 IG2 = 107) Cardiovascular care Feedback report, QI plan and distribution educ. material (one arm received educ. beta-blockade (IG1), other arm received educ. IMA grafting (IG2)) vs. no intervention Supporting activities (local opinion leader) Sign. improvement in 1 out of 2 process indicators: use of preoperative beta-blockade (IG1 group vs. CG (P < 0.001)) and (IG2 vs. CG (P = 0.02)) Not measured 
Wahlström et al., 2003 [21RCT (CG = 12, IG = 12) Malaria, diarrhea and pneumonia Educ. meetings vs. no intervention Supporting activities (educ. meeting incl. QI and QI facilities, incl. QI team) Sign. improvement in overall mean process indicator scores for malaria, diarrhea and pneumonia together (OR 0.63; 95% CI 0.16–1.112) Not measured 
Hayes et al., 2002 [22RCT (CG = 16, IG = 16) Cardiovascular care Educ. outreach vs. feedback report and educ. material Supporting activities (opinion leader, educ. meeting) No sign. improvement in 4 out of 5 process indicators, except for discharge counseling for daily weights (OR 2.63; 95% CI 1.14–6.07) Not measured 
Mehta et al., 2002 [32CBA (CG = 11, IG = 10) Cardiovascular care Educ. outreach, feedback report and QI plan vs. no intervention Supporting activities (opinion leader (outside hospital), distribution educ. material) Sign. improvement in 4 out of 8 process indicators: use of aspirin on admission (81% vs. 87%; P= 0.02), use of beta-blockers on admission (65% vs. 74%; P= 0.04), use of aspirin after discharge (84% vs. 92%; P= 0.002) and smoking counseling at discharge (53% vs. 65%; P= 0.02) Not measured 
Scott et al., 2001 [33CBA (CG = 112, IG = 1) Cardiovascular care Feedback reports and educ. meeting vs. no intervention Supporting activities (distribution educ. material) Not measured Sign. improvement in inpatient mortality (adj. OR, 0.59; 95% CI 0.45–0.77) 
Hayes et al., 2001 [23RCT (CG = 15, IG= 14) Cardiovascular care Educ. meeting and QI plan vs. feedback report Supporting activities (opinion leader and distribution educ. material) No sign. improvement in any of the 5 process indicators Not measured 
Sauaia et al., 2000 [34CBA (CG = 9, IG = 9) Cardiovascular care Educ. outreach and QI plan vs. mailed feedback report Supporting activities (opinion leader) No sign. improvement in any of the 7 process indicators Not measured 
Ellerbeck et al., 2000 [35CBA (CG = 73, IG = 44) Cardiovascular care QI plan based on feedback vs. no QI plan based on feedback Sign. improvement in 3 out of 8 process indicators: aspirin during hospitalization (6% vs. 13%) and at discharge (6% vs. 15%) and use of beta-blockers (14% vs. 22%). Not measured 
Marciniak et al., 1998 [36CBA (CG = not given, IG= 379) Cardiovascular care Feedback report and QI plan vs. no intervention Sign. improvement in 3 out of 7 process indicators: use of aspirin at discharge (OR, 5.6; 95% CI 2.6–8.7), use of beta-blockers (OR, 8.0; 95% CI 1.4–14.6) and smoking counseling (OR, 8.5; 95% CI 1.6–15.5) No sign. improvement in hospital mortality 
Soumerai et al., 1998 [24RCT (CG = 17, IG = 20) Cardiovascular care Educ. meeting vs. mailed feedback report Supporting activities (local opinion leader and distribution educ. material) Sign. improvement in 2 out of 4 process indicators: use of oral aspirin (P < 0.04) and use of beta-blockers (P < 0.02) Not measured 

CG, number of hospitals in control group; IG, number of hospitals intervention group; QI, quality improvement; OR, odds ratio; CI, confidence interval; sign., significant; adj., adjusted; incl., including; educ.,educational; lip.low, lipid-lowering therapy; Cor-Angio, coronary angiography; IMA, internal mammary artery; LOS, length of stay.

Types of implementation strategies

The methods used to implement quality indicators were classified into implementation strategies in which the information on quality indicators was used directly, or that did not use the information on quality indicators directly, but only supported the implementation, such as the involvement of a quality improvement team.

Table 2 shows the implementation strategies used. The most frequently used implementation strategies in which the information on quality indicators was used directly were audit and feedback (12 studies), followed by the development of a quality improvement plan based on quality indicator data (10 studies), 57% and 48%, respectively. The combination of these strategies was used in seven studies, and often supplemented by educational meetings and/or educational outreach [16, 25, 26, 29, 32]. The most frequently used supporting activity was distribution of educational material (9 studies). Other supporting activities were the use of a local opinion leader and the development of a quality improvement team.

In most studies (86%), multiple implementation strategies were used. In 14 studies, implementation strategies that related directly to quality indicators were combined with supporting activities. In four studies, strategies that related directly to quality indicators alone were used [27–29, 36].

Three studies reported a single implementation strategy in which the information on quality indicators was used directly. The single implementation strategies described were as follows: providing external feedback with an incentive bonus [30], providing immediate feedback [17] and using a quality improvement plan [35].

Most follow-up measurements of process and outcome indicators were performed 6 months after the strategy was implemented [16–18, 20, 21, 23, 25, 32].

Effects of quality indicator use

Different designs, implementation strategies and outcome measurements to measure the effect of quality indicators were described. In Table 2, the results are summarized as per study.

Most studies measured several outcomes, e.g. the change in several process indicators. In an attempt to summarize the results of the studies, we divided them into three categories: effective, partly effective and ineffective. We categorized the studies as ‘effective’ if more than half of all the outcome measures improved significantly. Studies were considered ‘partly effective’ if approximately half of the outcomes improved significantly, and ‘ineffective’ if there was significant improvement in less than half of all the outcomes.

Nine RCTs, two CCTs and ten CBAs were included. There was no clear relationship between the study design used and the effectiveness. Four out of nine RCTs showed the implementation to be ineffective [17, 19, 22, 23], one CCT was ineffective [25] and four out of ten CBAs did not show clear positive effects [27, 30, 31, 34]. The results of the studies included are reported in three different types of outcomes: overall composite score, patient outcomes and care processes, e.g. hospital mortality and prescribing medication. In two studies, an overall indicator score was measured. These studies showed a statistically significant improvement in the composite process indicator score [21, 28]. Five studies reported patient outcomes as well as care processes [17, 18, 26, 29, 36]. One study measured patient outcomes only [33]. In total, six studies evaluated whether or not quality indicator implementation improved patient outcomes. Four studies were found to be ineffective, one was partly effective and one was categorized as effective (see Table 2). Five studies reported inpatient mortality as endpoint. Two studies found significant improvements in patient outcomes: reduction in inpatient mortality [29, 33], stroke or transient ischemic attack [29], emergency coronary artery bypass graft (CABG) [29] and contrast nephropathy [29].

In 20 studies, process indicators were used to measure care processes (see Table 2). In each of these studies, more than one process indicator was measured. In three studies, there was no significant improvement in all the process indicators measured [17, 23, 34]. Two studies reported significant improvements in all process indicators [18, 29]. Most studies reported significant improvements in part of the measured process indicators. Of these studies, seven studies seemed to be effective or partly effective. These studies reported mostly on higher rates of prescribing drugs: inhibitors of angiotensin-converting enzyme (ACE) [16], heparin [16], antibiotics at emergency department [26], beta-blockers [20, 24, 32] and aspirin [24, 32]. In addition, these studies also reported on treatments given: lipid-lowering therapy [16], coronary angiography [16], blood culture obtained in 4 h [26] and higher rates of smoking counseling [32].

Not all studies adjusted the analyses for differences in distributions of other determinants when comparing the effect in the intervention group with that of the control group. Fourteen of the studies reported adjusted outcome measurements at patient level (age, co-morbidity) and/or hospital level (teaching status/ volume). Of these studies eight were found to be ineffective [17, 19, 22, 23, 25, 27, 31, 34], four were partly effective [18, 20, 26, 32] and only two were categorized as effective [29, 33]. Looking at the studies using unadjusted outcome measurements, three studies were found to be effective [16, 21, 28], three were partly effective [24, 35, 36] and one was ineffective [30].

The follow-up measurement period varied from 4 months [27, 35, 36] to 4 years [33]. Studies with a follow-up measurement period of less than 6 months showed less significant improvement and functioning of the hospital outcome measures [27, 34–36].

Types of implementation strategies and their effects

In order to summarize the prevailing implementation strategies, we divided them into three categories: receiving no feedback report, a feedback report only and receiving a feedback report combined with another implementation strategy (see Table 3). There seemed to be a relation between implementation strategies used and the effectiveness of the study (Kruskal–Wallis χ2 = 6,720, P = 0.035).

Table 3

Types of implementation strategies and their effects

Implementation strategies in which indicator scores were used directly Effectivea Partly effectiveb Ineffectivec 
No feedback report 
Feedback report only 
Feedback report combined with another implementation strategy 
Implementation strategies in which indicator scores were used directly Effectivea Partly effectiveb Ineffectivec 
No feedback report 
Feedback report only 
Feedback report combined with another implementation strategy 

aEffective, if more than half of all outcomes improved significantly.

bPartly effective, if approximately half of the outcomes improved significantly.

cIneffective, if less than half of all outcomes improved significantly.

Effective or partly effective studies appear to use feedback reports combined with other implementation strategies. For example, feedback reports in combination with education and the use of a quality improvement plan seemed to be effective [16, 20, 26, 29, 32]. Studies that did not use feedback reports systematically seemed to be less effective [19, 22–24, 27, 30, 34, 35]. Studies using a feedback report only also seemed to be less effective [17, 31].

From the studies describing an implementation strategy in which the information on quality indicators was used directly, eight studies reported a single implementation strategy, with additional supporting activities in five of these. Only one out of eight studies was effective [21]. This study used monthly educational meetings, including feedback and discussion on performance improvement. Thirteen studies used multifaceted implementation strategies, and of these four were effective [16, 28, 29, 33].

Reported barriers

Analyses of barriers to changing practice, such as a review of 76 studies on doctors, have shown that obstacles to change can arise at different levels at the health care system, such as at the level of the patient, the individual professional, the health care team, the health care organization, or the wider environment [37].

In our study, we identified reported barriers to implementation. In seven of the studies included, perceived barriers to change were reported. In these studies, we also identified barriers at different levels of the health care system (see Table 4): knowledge and cognitions (not convinced of the evidence) of the individual health care professional; interaction within the team (no mutual accountability and control, no leadership) and functioning of the hospital (facilities).

Table 4

Studies addressing the perceived barriers

Barriers at different levels Focus of factors Barriers Study 
Professional Knowledge Unawareness [19
Cognitions Lack of credible data [17, 21
Team or unit Social influence and leadership No support management/physicians [17, 22
Hospital Resources Lack of resources [20, 22, 23, 30
Barriers at different levels Focus of factors Barriers Study 
Professional Knowledge Unawareness [19
Cognitions Lack of credible data [17, 21
Team or unit Social influence and leadership No support management/physicians [17, 22
Hospital Resources Lack of resources [20, 22, 23, 30

Four studies reported a lack of resources, e.g. time investment and lack of administrative support (see Table 4). Several facilitating factors were reported, such as the availability of supportive/collaborative management, administration support, using detailed and credible data feedback to evaluate effects and the ability of persons receiving feedback to act on it.

Discussion

The two main objectives of this review were to explore the best implementation strategy for quality indicators, and to quantify the effectiveness of using quality indicators as a tool to improve quality of hospital care. Our results show that the majority of the studies included reported combinations of implementation strategies in which audit and feedback were most frequently used. Few studies showed significant improvements in the outcomes measured. Most of them focused on process measures, and reported significant improvements in part of the measured process indicators. Only few studies focused on the improvement of patient outcomes.

We recognize that significant improvements in patient outcomes are difficult to achieve. In our review, studies with a follow-up measurement period of less than 6 months showed less significant improvements in outcome measures. Short follow up on the effects of the implementation strategies may have contributed to the lack of effectiveness in some studies.

Looking at the type of implementation strategies used and their effects, there does seem to be a link between how quality indicators are used and the effectiveness of the study. Although this relationship was statistically significant (Kruskal–Wallis χ2 = 6,720, P = 0.035), we should be cautious in interpreting this partly arbitrary data. Effective or partly effective studies appeared to use feedback reports combined with other implementation strategies. Receiving a feedback report combined with education and the use of a quality improvement plan seemed to be effective. Less effective were those implementation strategies in which health care providers or managers did not receive a feedback report of quality indicator data. To improve patient outcomes or provider performance, health care providers should receive feedback on their performance in order to change practice and improve patient outcomes.

It has been suggested that multifaceted implementation strategies are more effective than single implementation strategies [10, 38]. In this review, we also found combinations of implementation strategies to be most effective. However, we could not really confirm these results because only few studies involved single implementation strategies.

The prevailing view on implementation of strategies to improve quality of care is that they should be tailored to potential barriers [39]. Ideally, possible barriers should be analyzed before the quality improvement implementation strategies are developed in order to influence both type and content of the implementation strategy [39]. Remarkably, none of the studies included reported translation of a priori identified barriers into tailor-made implementation strategies, but only reported barriers after the strategy was implemented. This may have affected the effectiveness.

The studies included in this review showed a great diversity in outcomes measured. Therefore, in an attempt to summarize the results of these studies, we categorized studies as ‘effective’, ‘partly effective’ and ‘ineffective’. However, the results of this aggregation have to be interpreted with caution. All outcomes were included on an equal basis, but outcomes may be valued differently for their relevance for quality of care. For example, one measure for patient outcome may be of more value than a process measure.

The implications of the findings reported in the present review must be considered within the context of the limits of the study. Firstly, we used strict selection criteria and as a result, the studies included are limited. Studies without a control group were excluded and, consequently, interrupted time series were excluded as well.

Due to our inclusion criteria, we report only on studies with primary quantitative outcome measurements. As a result, insights from qualitative studies fall outside the scope of this paper.

We noted the relatively narrow range of clinical areas studied. While cardiovascular care is an important clinical topic, other important areas, such as intensive care and obstetrics, were not covered in the studies. As a result, it is difficult to make generalized conclusions about hospital care as a whole.

Secondly, there is a great variation in quality of the studies. The availability of well-designed studies on this topic is limited. In the results section, we reported that adjustments for differences in distributions of other determinants varied when comparing the effects in the intervention group with the control group. Studies using unadjusted outcome measurements seemed to be more effective than studies using adjusted outcomes. In addition, most studies describe a combination of implementation strategies, which hampers a quantification of the effects of separate implementation strategies. Finally, implementation strategies used in the studies were often poorly described; therefore, we checked them against a standardized list of strategies.

In conclusion, there are many different implementation strategies in which the information on quality indicators was used directly, focusing on feedback, education, etc. Often, these strategies were combined with supporting activities. Receiving a feedback report, combined with education, and the development of a quality improvement plan seemed to be most effective. Effective strategies to implement quality indicators in daily practice in order to improve hospital care do exist, but there is considerable variation in methods used and the level of change achieved. Based on the present review, receiving a feedback report combined with another implementation strategy is recommended. There is a need for thoroughly designed studies on the implementation of quality indicators to further guide future implementation.

References

Agency for Healthcare Research and Quality, United States
 
http://www.qualityindicators.ahrq.gov/introduction.htm (11 February 2008, date last accessed)
Chiu
W-T
Yang
C-M
Lin
H-W
, et al.  . 
Development and implementation of a nationwide health care quality indicator system in Taiwan
Int J Qual Health Care
 , 
2007
, vol. 
19
 (pg. 
21
-
8
)
Collopy
BT
Clinical indicators in accreditation: an effective stimulus to improve patient care
Int J Qual Health Care
 , 
2000
, vol. 
12
 (pg. 
211
-
6
)
Commission for Health Improvement, United Kingdom
 
http://www.chi.nhs.uk/ratings/ (18 February 2008, date last accessed)
Donabedian
A
Explorations in Quality Assessment and Monitoring: The Definition of Quality and Approaches to Its Assessment.
 , 
1980
Michigan
Health Administration Press
Davis
DA
Taylor-Vaisey
A
Translating guidelines into practice. A systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines
CMAJ
 , 
1997
, vol. 
157
 (pg. 
408
-
16
)
Foy
R
Eccles
MP
Jamtvedt
G
, et al.  . 
What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review
BMC Health Serv Res
 , 
2005
, vol. 
5
 pg. 
50
 
Jamtvedt
G
Young
JM
Kristoffersen
DT
, et al.  . 
Audit and feedback: effects on professional practice and health care outcomes
Cochrane Database Syst Rev
 , 
2006
 
CD000259
Wright
J
Dugdale
B
Hammond
I
, et al.  . 
Learning from death: a hospital mortality reduction programme
J R Soc Med
 , 
2006
, vol. 
99
 (pg. 
303
-
8
)
Oxman
AD
Thomson
MA
Davis
DA
, et al.  . 
No magic bullets: a systematic review of 102 trials of interventions to improve professional practice
CMAJ
 , 
1995
, vol. 
153
 pg. 
1423
 
Grimshaw
JM
Shirran
L
Thomas
R
, et al.  . 
Changing provider behavior: an overview of systematic reviews of interventions
Med Care
 , 
2001
, vol. 
39
 (pg. 
II2
-
45
)
Bratzler
DW
Nsa
W
Houck
PM
Performance measures for pneumonia: are they valuable, and are process measures adequate?
Curr Opin Infect Dis
 , 
2007
, vol. 
20
 (pg. 
182
-
9
)
Epstein
AJ
Do cardiac surgery report cards reduce mortality? Assessing the evidence
Med Care Res Rev
 , 
2006
, vol. 
63
 (pg. 
403
-
26
)
Thorsen
TMM
Changing Professional Practice. Theory and Practice of Clinical Guidelines Implementation
 , 
1999
Copenhagen
Danish Institute for Health Services Research and Development
Wensing
M
Grol
R
Single and combined strategies for implementing changes in primary care: a literature review
Int J Qual Health Care
 , 
1994
, vol. 
6
 (pg. 
115
-
32
)
Carlhed
R
Bojestig
M
Wallentin
L
, et al.  . 
Improved adherence to Swedish national guidelines for acute myocardial infarction: the Quality Improvement in Coronary Care (QUICC) study
Am Heart J
 , 
2006
, vol. 
152
 (pg. 
1175
-
81
)
Beck
CA
Richard
H
Tu
JV
, et al.  . 
Administrative Data Feedback for Effective Cardiac Treatment: AFFECT, a cluster randomized trial
JAMA
 , 
2005
, vol. 
294
 (pg. 
309
-
17
)
Horbar
JD
Carpenter
JH
Buzas
J
, et al.  . 
Collaborative quality improvement to promote evidence based surfactant for preterm infants: a cluster randomised trial
BMJ
 , 
2004
, vol. 
329
 pg. 
1004
 
Berner
ES
Baker
CS
Funkhouser
E
, et al.  . 
Do local opinion leaders augment hospital quality improvement efforts? A randomized trial to promote adherence to unstable angina guidelines
Med Care
 , 
2003
, vol. 
41
 (pg. 
420
-
31
)
Ferguson
TB
Peterson
ED
Coombs
LP
, et al.  . 
Use of continuous quality improvement to increase use of process measures in patients undergoing coronary artery bypass graft surgery: a randomized controlled trial
JAMA
 , 
2003
, vol. 
290
 (pg. 
49
-
56
)
Wahlström
R
Kounnavong
S
Sisounthone
B
, et al.  . 
Effectiveness of feedback for improving case management of malaria, diarrhoea and pneumonia—a randomized controlled trial at provincial hospitals in Lao PDR
Trop Med Int Health
 , 
2003
, vol. 
8
 (pg. 
901
-
9
)
Hayes
RP
Baker
DW
Luthi
J-C
, et al.  . 
The effect of external feedback on the management of medicare inpatients with congestive heart failure
Am J Med Qual
 , 
2002
, vol. 
17
 (pg. 
225
-
35
)
Hayes
R
Bratzler
D
Armour
B
, et al.  . 
Comparison of an enhanced versus a written feedback model on the management of Medicare inpatients with venous thrombosis
Jt Comm J Qual Improv
 , 
2001
, vol. 
27
 (pg. 
155
-
68
)
Soumerai
SB
McLaughlin
TJ
Gurwitz
JH
, et al.  . 
Effect of local medical opinion leaders on quality of care for acute myocardial infarction: a randomized controlled trial
JAMA
 , 
1998
, vol. 
279
 (pg. 
1358
-
63
)
Landon
BE
Wilson
IB
McInnes
K
, et al.  . 
Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: the EQHIV study
Ann Intern Med
 , 
2004
, vol. 
140
 (pg. 
887
-
96
)
Chu
LA
Bratzler
DW
Lewis
RJ
, et al.  . 
Improving the quality of care for patients with pneumonia in very small hospitals
Arch Intern Med
 , 
2003
, vol. 
163
 (pg. 
326
-
32
)
Pandey
DK
Cursio
JF
Data feedback for quality improvement of stroke care: CAPTURE Stroke experience
Am J Prev Med
 , 
2006
, vol. 
31
 (pg. 
S224
-
9
)
Grossbart
SR
What's the return? Assessing the effect of ‘pay-for-performance’ initiatives on the quality of care delivery
Med Care Res Rev
 , 
2006
, vol. 
63
 (pg. 
29S
-
48S
)
Moscucci
M
Rogers
EK
Montoye
C
, et al.  . 
Association of a continuous quality improvement initiative with practice and outcome variations of contemporary percutaneous coronary interventions
Circulation
 , 
2006
, vol. 
113
 (pg. 
814
-
22
)
Rosenthal
MB
Frank
RG
Li
Z
, et al.  . 
Early experience with pay-for-performance: from concept to practice
JAMA
 , 
2005
, vol. 
294
 (pg. 
1788
-
93
)
Snyder
C
Anderson
G
Do quality improvement organizations improve the quality of hospital care for Medicare beneficiaries?
JAMA
 , 
2005
, vol. 
293
 (pg. 
2900
-
7
)
Mehta
RH
Montoye
CK
Gallogly
M
, et al.  . 
Improving quality of care for acute myocardial infarction: The Guidelines Applied in Practice (GAP) Initiative
JAMA
 , 
2002
, vol. 
287
 (pg. 
1269
-
76
)
Scott
IA
Coory
MD
Harper
CM
The effects of quality improvement interventions on inhospital mortality after acute myocardial infarction
Med J Aust
 , 
2001
, vol. 
175
 (pg. 
465
-
70
)
Sauaia
A
Ralston
D
Schluter
WW
, et al.  . 
Influencing care in acute myocardial infarction: a randomized trial comparing 2 types of intervention
Am J Med Qual
 , 
2000
, vol. 
15
 (pg. 
197
-
206
)
Ellerbeck
EF
Kresowik
TF
Hemann
RA
, et al.  . 
Impact of quality improvement activities on care for acute myocardial infarction
Int J Qual Health Care
 , 
2000
, vol. 
12
 (pg. 
305
-
10
)
Marciniak
TA
Ellerbeck
EF
Radford
MJ
, et al.  . 
Improving the quality of care for Medicare patients with acute myocardial infarction: results from the Cooperative Cardiovascular Project
JAMA
 , 
1998
, vol. 
279
 (pg. 
1351
-
7
)
Cabana
MD
Rand
CS
Powe
NR
, et al.  . 
Why don't physicians follow clinical practice guidelines? A framework for improvement
JAMA
 , 
1999
, vol. 
282
 (pg. 
1458
-
65
)
Hulscher
ME
Wensing
M
Grol
RP
, et al.  . 
Interventions to improve the delivery of preventive services in primary care
Am J Public Health
 , 
1999
, vol. 
89
 (pg. 
737
-
46
)
Bosch
M
van der Weijden
T
Wensing
M
Grol
R
Tailoring quality improvement interventions to identified barriers: a multiple case analysis
J Eval Clin Pract
 , 
2007
, vol. 
13
 (pg. 
161
-
8
)

Supplementary data