Abstract

To improve health outcomes, effective and systematic mechanisms to foster the adoption of evidence-based guideline recommendations into routine practice need to be identified. A cyclical process for achieving this objective involving three key phases is suggested.

Phase 1

Writing actionable best-evidence guidelines that prioritize key recommendations while indicating the levels of adoption needed for population health benefits to be accomplished.

Phase 2

Developing implementation plans for the priority guideline recommendations. These should systematically consider skills training and accreditation; social influences including opinion leaders and patient influences; environmental factors; monitoring and feedback; and incentives for clinical change.

Phase 3

Pilot testing the effectiveness of proposed approaches in producing the desired clinical changes. If implementation requires system changes and evaluation at an organizational level, the use of alternative research designs to the randomized controlled trial could be considered. The purpose evaluation would be to enable refinement of the implementation plans before widespread dissemination.

Introduction

Closing the gap between best evidence and existing clinical practice has the potential to improve health outcomes [1]. Evidence-based clinical guidelines aim to do this; however guidelines alone may not result in the required change in clinical practice [2]. We propose a three-phase process for achieving these objectives. The first involves producing evidence-based guidelines that maximize the likelihood of implementation. The second relates to developing implementation plans that give guidance on improving uptake of priority recommendations. The third component involves testing of the implementation plan to enable refinement before widespread dissemination. Although the model is resource intensive, such investment is warranted in order to translate evidence development efforts into clinical benefit in priority health areas. An overview of key steps in the model is shown in Box 1.

The need for an organizing framework to aid the application of theoretical concepts to the development of evidence-implementation interventions has been identified [3]. The current model suggests strategies targeting factors associated with behavioural change in a logical sequence. Our focus is on improving uptake of guideline recommendations that require changes in individual behaviour, while recognizing that system changes may also be needed to support these changes. A summary of the strategies suggested, underlying theoretical constructs and empirical references is shown in Table 1.

Box 1
Key points for the implementation of the three-phase model
  • Carefully consider how the guideline is presented and how recommendations are phrased

  • Identify priority recommendations within the guideline

  • Produce guideline in conjunction with an implementation plan for priority recommendations

  • Involve key stakeholders in production of implementation plan

  • Pilot test both the guideline and implementation plan

  • Refine guideline and implementation plan before widespread dissemination

Table 1

Strategies to promote guideline implementation: theoretical constructs and examples of application

Strategy Relevant constructs Key illustrative examples 
Phase 1 
 Concrete and specific recommendations Knowledge, executability, decidability Concrete and specific recommendations were more likely to be adopted by GPs than vague non-specific recommendations. Observational study. Grol et al. [19
 Identify priorities Goal-setting, action planning Of 228 primary care patients with cardiovascular disease risk factors who made an action plan to identify behavioural change goals, 53% also reported making behavioural change related to their action plan. Descriptive study. Handley et al. [11
 Set targets for implementation Goal-setting 
 Present a rationale Beliefs, attitudes, perceived relative advantage Recommendations compatible with current values were more likely to be adopted by GPs than those perceived as controversial or incompatible with values. Observational study. Grol et al. [19
 Highlight clinical norms Normative beliefs, attitudes, modelling/verbal persuasion An intervention to improve myocardial infarction care that involved using local medical opinion leaders to influence peers through small group discussions, informal consultation and revisions of clinical protocols was compared with performance feedback alone. Hospitals in both groups improved from baseline to follow up on indicators of quality, however the improvement was greatest for those allocated to the peer intervention. RCT Soumerai et al. [21
 Orient to the need of the end user Complexity Among the guideline characteristics most commonly endorsed to promote use by GPs was ‘clarity, simplicity and availability of a short format’. Descriptive study of 391 GPs. Watkins et al. [20
Phase 2 
 Skills training Skills, knowledge, self-efficacy CME improves knowledge, skills, attitudes and patient outcomes. CME that is interactive, uses multimedia, live media, and involves multiple exposures is more effective than other types. Systematic review. Marinopoulos et al. [23
 Social influences Normative beliefs, attitudes, modelling, verbal persuasion The use of local opinion leaders in hospital settings can be effective in promoting evidence-based practice. Systematic review of 12 studies. Doumitt et al. [24
 Environmental influences Cues to action, environmental triggers Guideline adherence improved due to the implementation of a computerized clinical decision aid that gave clinicians real time recommendations for venous thromboembolism prophylaxis. Time series study. Durieux et al. [26
 Patient-mediated Knowledge, skills and attitudes of patients Patient request for a new drug and patient acceptability were cited as contributing to decisions to prescribe a new drug in ∼20% of cases. Descriptive study. Prosser et al. [28
 Feedback Positive/negative reinforcement; goal-setting; skill development Audit and feedback are effective strategies for improving care, particularly when baseline adherence to the recommended practice is low. Systematic review of 118 studies. Jamtvedt et al. [30
 Incentives Positive/negative reinforcement Five out of six studies examining physician level incentives, and seven out of nine studies examining provider group-level incentives demonstrated partial or positive effects on quality indicators. Systematic review. Petersen et al. [36
Phase 3 
 Pilot testing with iterative refinement of implementation strategies Perceived advantages; beliefs; trialability Breakthrough collaborative model intervention that involved a series of iterative plan, do, study, act cycles was found to be effective in improving care for chronic heart failure. Quasi experimental, controlled study. Asch et al. [44
Strategy Relevant constructs Key illustrative examples 
Phase 1 
 Concrete and specific recommendations Knowledge, executability, decidability Concrete and specific recommendations were more likely to be adopted by GPs than vague non-specific recommendations. Observational study. Grol et al. [19
 Identify priorities Goal-setting, action planning Of 228 primary care patients with cardiovascular disease risk factors who made an action plan to identify behavioural change goals, 53% also reported making behavioural change related to their action plan. Descriptive study. Handley et al. [11
 Set targets for implementation Goal-setting 
 Present a rationale Beliefs, attitudes, perceived relative advantage Recommendations compatible with current values were more likely to be adopted by GPs than those perceived as controversial or incompatible with values. Observational study. Grol et al. [19
 Highlight clinical norms Normative beliefs, attitudes, modelling/verbal persuasion An intervention to improve myocardial infarction care that involved using local medical opinion leaders to influence peers through small group discussions, informal consultation and revisions of clinical protocols was compared with performance feedback alone. Hospitals in both groups improved from baseline to follow up on indicators of quality, however the improvement was greatest for those allocated to the peer intervention. RCT Soumerai et al. [21
 Orient to the need of the end user Complexity Among the guideline characteristics most commonly endorsed to promote use by GPs was ‘clarity, simplicity and availability of a short format’. Descriptive study of 391 GPs. Watkins et al. [20
Phase 2 
 Skills training Skills, knowledge, self-efficacy CME improves knowledge, skills, attitudes and patient outcomes. CME that is interactive, uses multimedia, live media, and involves multiple exposures is more effective than other types. Systematic review. Marinopoulos et al. [23
 Social influences Normative beliefs, attitudes, modelling, verbal persuasion The use of local opinion leaders in hospital settings can be effective in promoting evidence-based practice. Systematic review of 12 studies. Doumitt et al. [24
 Environmental influences Cues to action, environmental triggers Guideline adherence improved due to the implementation of a computerized clinical decision aid that gave clinicians real time recommendations for venous thromboembolism prophylaxis. Time series study. Durieux et al. [26
 Patient-mediated Knowledge, skills and attitudes of patients Patient request for a new drug and patient acceptability were cited as contributing to decisions to prescribe a new drug in ∼20% of cases. Descriptive study. Prosser et al. [28
 Feedback Positive/negative reinforcement; goal-setting; skill development Audit and feedback are effective strategies for improving care, particularly when baseline adherence to the recommended practice is low. Systematic review of 118 studies. Jamtvedt et al. [30
 Incentives Positive/negative reinforcement Five out of six studies examining physician level incentives, and seven out of nine studies examining provider group-level incentives demonstrated partial or positive effects on quality indicators. Systematic review. Petersen et al. [36
Phase 3 
 Pilot testing with iterative refinement of implementation strategies Perceived advantages; beliefs; trialability Breakthrough collaborative model intervention that involved a series of iterative plan, do, study, act cycles was found to be effective in improving care for chronic heart failure. Quasi experimental, controlled study. Asch et al. [44

Phase 1: Producing actionable guidelines

Although guidelines are a way of helping clinicians keep up-to-date with best evidence, their utility rests upon their capacity to effect implementation [4]. While much effort has been invested in developing rigorous processes for appraising the evidence used to construct evidence-based guideline recommendations [5], until recently there has been less focus on other attributes of guidelines which may affect clinical behaviour. A study of 51 lung cancer guidelines appraised using the AGREE (Appraisal of Guidelines Research and Evaluation) instrument found that implementation issues were inadequately addressed [6]. Leading international guideline developers are now exploring mechanisms that may improve the implementability of guidelines. Strategies have included use of the GLIA (GuideLine Implementability Appraisal) instrument [11] and the use of questionnaires and consultation processes that seek feedback on local applicability [8].

Some guideline recommendations may be more difficult to implement because of the intrinsic nature of the required practice changes [9]. The evidence-based behaviour may be unconventional, require acquisition of new skills or equipment, or require systems to change in ways that are expensive or difficult. Although some of these matters may not be able to be modified, guideline developers should be cognizant of how recommendations are phrased and presented. The following mechanisms, which might optimize the actionability of the guidelines, should be considered [10].

Recommendations should specifically describe the desired clinical behaviour

The guideline should describe the desired behaviour in concrete and specific terms. This would include details about the specific recommended clinical action, who should undertake the action, to which group of patients it should be applied, and under what circumstances [5, 7]. The GLIA tool [7] refers to concepts of ‘decidability’ (i.e. the precise circumstances under which something should be done), and ‘executability’ (i.e. exactly what should be done under the circumstances defined).

Guidelines should identify priorities

Prioritization of actions is an important feature of action plans, a tool commonly used to attain goals [11]. To increase guideline effectiveness, developers should prioritize those recommendations that exhibit important divergence from best practice; which are sufficiently potent to produce significant health benefits if broadly adopted; and for which there are accurate outcome measures of provider adherence. Currently there is limited research on how these priority recommendations should be presented within the guideline format. Educational research suggests that presenting priority recommendations first in any document, emphasizing their importance, and re-iterating them in a summary may increase recall and hence potentially implementation [12].

Targets for adoption should be defined

Goal-setting is a key component of many behavioural change theories [13, 14]. Combinations of short-term and longer-term goals may be more effective than long-term goals alone [13]. Therefore, guidelines should specify a series of incremental population-level targets for clinically significant change in uptake of priority recommendations, and indicate a reasonable time frame in which this level of change could be expected. Advice on specific indicators that may be used to measure whether targets have been achieved should be included [7].

Recommendations should specify benefits associated with change

A number of theories including Operant [15], Social Cognitive [14] and Diffusion of Innovations [16] postulate that the way people behave is influenced by their beliefs about the likely outcomes and the perceived value attached to those outcomes. Therefore, guidelines should include rationales for adoption in order to increase the likelihood of implementation. An evaluation of the implementation of guidelines issued by the UK National Institute for Clinical Excellence concluded that practice had changed faster in areas where practitioners and managers saw clear advantages to adoption [17].

Compatibility with existing clinical norms should be highlighted

Social influences play an important role in the adoption of new practices [16] and behaviours that are perceived to be normative are more likely to be adopted [18]. If the practice is one that is readily endorsed by a majority of the target group, emphasis of this may be beneficial. A study of Dutch general practitioners found that guideline recommendations perceived as controversial and incompatible with current values were less likely to be adopted than those without this attribute [19].

Guideline presentation should be oriented to the needs of end users

New practices that are difficult to understand or use may be less likely to be adopted [16]. Consequently guidelines that are oriented to the needs of the end user in terms of their presentation and by provision of tools to facilitate use may be more easily adopted. General practitioner (GPs) prefer guidelines that are presented in a short format or flowcharts [20]. Tools for application such as patient education materials and guideline summaries may aid implementation; while presentation and formatting can be used to make priority recommendations easy to identify [5, 7]. As a result it is recommended that careful attention is paid to the way in which guidelines are presented, in particular, the provision of a compelling and comprehensive rationale for practice change. Guideline developers should also consider the identification of priorities and targets for implementation, as well as the resources for guideline users that could accompany the guideline document.

Phase 2: Designing implementation plans

Providing strategies for implementation in guidelines [7] and suggestions about overcoming organizational barriers to implementation may also be helpful [5]. These may require a mix of individual, organizational and general system strategies. Because different guideline recommendations focus on different aspects of clinical behaviour or system performance, each priority recommendation should be accompanied by a detailed plan that provides step-by-step suggestions for implementation. These plans are most likely to be successful if there is early identification and involvement of key stakeholders from the area of practice where the need for change is identified (such as clinicians, managers and policy makers and representatives from professional associations involved in accreditation and professional development). Specialists such as behavioural scientists, epidemiologists, health economists and legal experts may also be needed to help design implementation strategies. Plans should consider six key areas: requirement for new skills; social influences such as opinion leaders; environmental factors that may promote adoption; patient-mediated influences; monitoring and feedback; and incentives. Organizations that have a commitment to implementing the guidelines should then identify a leadership team who can take responsibility for assessing how the plan can be used to inform quality improvement initiatives at the local level. The strategy of engaging local leaders has been used successfully to promote improvements in care across a variety of areas, e.g. for acute myocardial infarction [21].

Assessing whether new skills are required

Like knowledge, appropriate level of skill is considered a necessary pre-condition for behaviour change [14]. If a recommendation requires that clinicians learn new skills or improve existing skills, then skills assessment, training and accreditation may be required. This is likely to be required for specialty procedural skills, such as robot-assisted surgery [22]. Consultation with clinical experts and professional bodies may enable implementation plans to provide appropriate strategies for assessment of existing skills in the workforce. Depending on the complexity of the skill, strategies may include self-assessment, peer assessment, external assessment, or credentialing by a professional body.

If formal training and accreditation are required, the plans should provide information on types of training likely to be effective and means of accreditation. A recent review has found evidence that continuing medical education (CME) improves knowledge, skills, attitudes, clinical behaviour, and patient outcomes [23]. Providing opportunities for learners to see a new skill performed (modelled), giving opportunities for practice, and providing constructive performance feedback builds both skills and confidence to perform a new behaviour [14]. Reviews of CME effectiveness have found that educational approaches which are interactive, use multimedia and live media and involve multiple exposures appear more effective in changing clinical performance [23]. Collaboration between implementation plan developers and professional associations would facilitate the development of appropriate training packages.

Using social influences to enhance implementation

Behavioural change theories [14, 16, 18] recognize the role of perceptions of the social environment in influencing behaviour. A Cochrane review indicates that opinion leaders can effectively promote evidence implementation [24], but there are still uncertainties about how best to maximize effectiveness. A recent systematic review on innovation in health [25] similarly found evidence to support the role of opinion leaders and champions (those committed to the adoption of a new practice) in effecting change. Individuals who have strong social ties both within and between organizations may be particularly influential in promoting adoption of a new practice [25]. Adoption is greater when those who are the focus of the intervention are similar in terms of background to the change agent [25]. Drawing on best evidence, implementation plans should provide practical strategies for identifying, engaging, and using opinion leaders or other change agents to promote implementation.

Using environmental factors to promote adoption

The environment plays an important role in influencing behaviour [14, 15]. Systems can support or deter a clinical behaviour. Automated reminders have been used to improve adherence to venous thromboembolism prophylaxis guidelines [26]. Similarly, implementation of organizational procedures and protocols can change practice and reduce adverse events [27]. Implementation plans should assess how systems can operate to maximize the probability of clinical change at the organizational level (e.g. by development of policies and procedures) or at the regional level (e.g. diabetes registers). Plans should outline methods for effectively adapting to existing systems or setting up new systems to promote implementation.

Using patient-mediated influences to increase adoption

Patient preferences and expectations are a form of social influence that can have a powerful effect on care. For example, where GPs made a decision to prescribe a new drug, the decision was reported to be influenced by patient requests in 22% of cases, and by patient acceptability in 20% of cases [28]. Strategies to improve patient health literacy, involvement in treatment decision-making, and self-management of health conditions may assist patients to have a positive impact on their own health care [29]. Consequently implementation plans should identify effective strategies to influence patient attitudes and knowledge about best evidence practice.

Role of audit and feedback in adoption

Feedback on performance is an important feature of behavioural change theories that encompass goal-setting and skill development [13, 14]. Early feedback allows refinement of goals and detection of knowledge or skill deficits that may hinder goal attainment, as well as enhance motivation [13]. Systematic reviews have shown that performance feedback is an effective means of changing provider practice [30]. Therefore feedback on the extent of implementation of priority recommendations should be provided to clinicians and to health care organizations.

If given non-punitively, feedback may be most effective given at the individual rather than group level [31]. The impact of feedback is illustrated by Tibballs' study [32] of handwashing among intensive care staff. Staff washed their hands both before and after patient contact 4.3% of the time when covertly observed. This rose to 22.6% during a period of overt observation and to 55.2% when feedback was added to the overt observation.

Feedback may provide comparisons with past performance, peers, benchmarks, or gold standards. The type of comparative data provided with feedback should be considered within the implementation plan. This may depend, in part, on the level at which feedback is provided, e.g. to individual clinicians, wards, practices, hospitals, or regional health services. Feedback is likely to be more effective when given close to the time of patient contact [33] and on a frequent basis [30]. Therefore, the feedback schedule, as well as practical issues such as cost, should be considered when examining the feasibility of different data collection methods.

Using incentives to increase adoption

Positive consequences of behaviour encourage adoption of specific behaviours, while negative consequences discourage adoption [15]. Desire to improve patient care, professional pride and peer influences may act as powerful intangible motivators [34]. Intangible incentives such as these may be more effective where the type of clinical performance required is difficult to accurately quantify [35]. These should be the first choice for trialling with an implementation plan. Over reliance on external tangible incentives such as financial rewards may damage internal motivators such as professionalism [35].

There is some evidence that pay for performance strategies for the individual clinician may change the care provided [36]. A recent randomized study comparing interventions to improve dentists' use of sealant treatments to prevent cavities in newly erupted molars found that providing a fee for service increased sealant use by ∼10%. Education alone had no significant effect on practice [37]. Not all studies, however, have found evidence for the effectiveness of financial incentives [38]. Several factors have been identified that may influence the effectiveness of such approaches: the size of the incentive, who it is directed at, and how the targets for incentive payments are set [39]. Using fixed performance targets might reward those institutions with the highest baseline performance rather than those who demonstrated the greatest improvement [38].

Given this literature, the implementation plan should identify ways that both intangible and tangible incentives may promote implementation. If internal motivators are found to be ineffective then external incentives should be considered. Careful consideration should be given to the way in which both the targets and the incentives are formulated. Punitive strategies should be considered only when the relevant clinical behaviour may result in serious patient harm.

Phase 3: Evaluating the effectiveness of guideline dissemination through implementation plans

Stakeholder opinions about implementation approaches are commonly used to indicate the likely effectiveness of implementation strategies. However in spite of positive reviews by providers this information does not always predict appropriate practice change [40, 41]. Preferably evaluation should measure if the guideline produces change in actual clinical performance rather than reported guidelines intention to change [3]. Allowance should be made for the time required for change to occur. If the desired change in clinical performance does not occur, there is an opportunity to refine either the format of the guidelines or the proposed implementation strategy.

Evaluations should be done in a cost-effective and rigorous manner. As implementation often requires systems change, the effectiveness of the combined guidelines and implementation plans may be tested on large units such as hospitals, rather than on individual clinicians. If so, the use of randomized controlled designs to test effectiveness is likely to present difficulties in terms of preventing contamination between experimental conditions, cost and logistics [42]. The multiple baseline design is a feasible alternative for evaluating system-level interventions. This design requires a minimum of two organizations (or other units) to which the intervention (in this case, the implementation plan) is applied at different time points [43]. Adherence to the clinical recommendation is monitored for at least three time points before the implementation guide is released in each organization so that baseline performance can be determined. Monitoring continues after the release of the implementation plan and guideline to determine their effect on performance. Steps in the implementation plan may be added sequentially, allowing the impact of the addition of each further strategy to be assessed.

To enable timely evaluation and dissemination of implementation plans, initial monitoring of the priority recommendations in one or two organizations could commence before finalizing the plans for testing. The results of the evaluation should enable detection of any problems with the implementation strategies, or aspects of the guidelines themselves that need clarification. Hence, data from the evaluation can be used iteratively to improve the guidelines and the implementation plans.

Conclusion

Guideline production and dissemination do not necessarily result in the widespread adoption of best-evidence clinical care. Guideline developers should give more attention to the coordinated and systematic approaches required for implementation. We suggest that three overlapping phases are required. Firstly, the guidelines should be presented in a way that identifies priorities and maximizes the likelihood of implementation. Secondly, mechanisms designed to maximize implementation should be identified concurrently with the guideline development. Such implementation plans should involve testable strategies that systematically address knowledge, skills, motivational and environmental factors that influence clinical performance. Thirdly, pilot testing to evaluate the effectiveness of implementation plans should routinely occur. This would allow for improvement in the plans as well as the accumulation of evidence about how to implement best-evidence care successfully.

References

National Institute of Clinical Studies
Evidence-practice Gaps Report
 , 
2005
, vol. 
Vol. 2
 
Melbourne, Australia
 
Grimshaw
JM
Thomas
RE
MacLennan
G
Effectiveness and efficiency of guideline dissemination and implementation strategies
Health Technol Assess
 , 
2004
, vol. 
8
 
Eccles
M
Grimshaw
J
Walker
A
, et al.  . 
Changing the behaviour of healthcare professionals: the use of theory in promoting the uptake of research findings
J Clin Epidemiol
 , 
2005
, vol. 
58
 (pg. 
107
-
12
)
Shaneyfelt
TM
Building bridges to quality
JAMA
 , 
2001
, vol. 
286
 (pg. 
2600
-
01
)
Cluzeau
FA
Burgers
JS
Brouwers
M
, et al.  . 
Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project
Qual Saf Health Care
 , 
2003
, vol. 
12
 (pg. 
18
-
23
)
Harpole
LH
Kelley
MJ
Schreiber
G
, et al.  . 
Assessment of the scope and quality of clinical practice guidelines in lung cancer
Chest
 , 
2003
, vol. 
123
 
Suppl. 1
Shiffman
RN
Dixon
J
Brandt
C
, et al.  . 
The GuideLine Implementability Appraisal (GLIA): development of an instrument to identify obstacles to guideline implementation
BMC Med Inform Decis Mak
 , 
2005
, vol. 
5
 pg. 
23
  
doi:10.1186/1472-6947-5-23
Marshall
C
Summary report: assessing the implementability of guidelines
Report to the National Institute of Clinical Studies
  
Eccles
MP
Grimshaw
JM
Selecting, presenting and delivering clinical practice guidelines: are there any ‘magic bullets?
Med J Aust
 , 
2004
, vol. 
180
 
Suppl. 6
(pg. 
S52
-
S54
)
Davis
D
Goldman
J
Palda
VA
Handbook on Clinical Practice Guidelines.
 , 
2007
Canadian Medical Association
Handley
M
MacGregor
K
Schillinger
D
, et al.  . 
Using action plans to help primary care patients adopt healthy behaviours: a descriptive study
J Am Board Fam Med
 , 
2006
, vol. 
19
 (pg. 
224
-
31
)
Ley
P
Memory for medical information
Br J Soc Clin Psychol
 , 
1979
, vol. 
18
 (pg. 
245
-
55
)
Locke
EA
Latham
GP
Building a practically useful theory of goal setting and task motivation
Am Psychol
 , 
2002
, vol. 
57
 (pg. 
705
-
17
)
Bandura
A
Social Foundations of Thought and Action.
 , 
1986
Englewood Cliffs, NJ, USA
Prentice-Hall
Gleitman
H
Psychology
 , 
1991
3rd edn
NY
W.W. Norton & Company, Inc
Rogers
EM
Diffusion of Innovations
 , 
2003
5th edn
New York, NY
Free Press
Sheldon
T
Cullum
N
Dawson
D
, et al.  . 
What's the evidence that NICE guidance has been implemented? Results from a national evaluation using time series analysis, audit of patient's notes, and interviews
Br Med J
 , 
2004
, vol. 
329
 pg. 
999
 
Azjen
I
Kuhl
J
Beckman
J
From intentions to actions: a theory of planned behaviour
Action-Control: From Cognition to Behaviour
 , 
1985
Heidelberg
Springer
(pg. 
11
-
39
)
Grol
R
Dalhuijsen
J
Thomas
S
, et al.  . 
Attributes of clinical guidelines that influence use of guidelines in general practice: observational study
Br Med J
 , 
1998
, vol. 
317
 (pg. 
858
-
61
)
Watkins
C
Harvey
I
Langley
C
, et al.  . 
General practitioners' use of guidelines in the consultation and their attitudes to them
Br J Gen Pract
 , 
1999
, vol. 
49
 (pg. 
11
-
5
)
Soumerai
SB
McLaughlin
TJ
Gurwitz
JH
, et al.  . 
Effect of local medical opinion leaders on quality of care for acute myocardial infarction
JAMA
 , 
1998
, vol. 
279
 (pg. 
1358
-
63
)
Kypson
AP
Nifong
LW
Chitwood
WR
Jr.
Robot-assisted surgery: training and re-training surgeons
Int J Med Robot
 , 
2004
, vol. 
1
 (pg. 
70
-
6
)
Marinopoulos
SS
Dorman
T
Ratanawongsa
N
, et al.  . 
Effectiveness of continuing medical education
Evid Rep Technol Assess
 , 
2007
, vol. 
14
 (pg. 
1
-
69
)
Doumitt
G
Gattellari
M
Grimshaw
J
, et al.  . 
Local opinion leaders: effects on professional practice and health care outcomes
Cochrane Database Syst Rev
 , 
2007
Issue 1
 
Art No.: CD000125. doi: 10.1002/14651858.CD000125.pub3
Greenlagh
T
Robert
G
Bate
P
How to spread good ideas. A systematic review of the literature on diffusion, dissemination and sustainability of innovations in health service delivery and organization
Report for the National Co-ordinating Centre for NHS Service Delivery and Organisation R&D (NCCSDO)
  
April 2004
Durieux
P
Nizard
R
Ravaud
P
, et al.  . 
A clinical decision support system for prevention of venous thromboembolism: effect on physician behaviour
JAMA
 , 
2000
, vol. 
283
 (pg. 
2816
-
21
)
Couzigou
C
Lamory
J
Salmon-Ceron
D
, et al.  . 
Short peripheral venous catheters: effect of evidence-based guidelines on insertion, maintenance and outcomes at a university hospital
J Hosp Infect
 , 
2005
, vol. 
59
 (pg. 
197
-
204
)
Prosser
H
Almond
S
Walley
T
Influences on GPs' decision to prescribe new drugs: the importance of who says what
Fam Pract
 , 
2003
, vol. 
20
 (pg. 
61
-
8
)
Coulter
A
Ellins
J
Effectiveness of strategies for informing, educating, and involving patients
BMJ
 , 
2007
, vol. 
335
 (pg. 
24
-
7
)
Jamtvedt
G
Young
JM
Kristofferson
DT
, et al.  . 
Audit and feedback: effects on professional practice and health care outcomes
Cochrane Database Syst Rev
 , 
2006
, vol. 
2
  
CD000259
Hysong
SJ
Best
RG
Pugh
JA
Audit and feedback and clinical practice guideline adherence: making feedback actionable
Implement Sci
 , 
2006
, vol. 
1
 pg. 
9
 
Tibballs
J
Teaching hospital medical staff to handwash
Med J Aust
 , 
1996
, vol. 
164
 (pg. 
395
-
98
)
Davis
D
Taylor-Vaisey
A
Translating guidelines into practice: a systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines
Can Med Assoc J
 , 
1997
, vol. 
157
 (pg. 
408
-
16
)
Spooner
A
Chapple
A
Roland
M
What makes British general practitioners take part in a quality improvement scheme?
J Health Serv Res Policy
 , 
2001
, vol. 
6
 (pg. 
145
-
50
)
Marshall
M
Harrison
S
Its about more than money: financial incentives and internal motivation
Qual Saf Health Care
 , 
2005
, vol. 
14
 (pg. 
4
-
5
)
Petersen
LA
Woodward
LD
Urech
T
, et al.  . 
Does pay-for-performance improve the quality of health care?
Ann Intern Med
 , 
2006
, vol. 
145
 (pg. 
265
-
72
)
Clarkson
JE
Turner
S
Grimshaw
JM
, et al.  . 
Changing clinicians' behavior: a randomized controlled trial of fees and education
J Dent Res
 , 
2008
, vol. 
87
 (pg. 
640
-
44
)
Rosenthal
MB
Frank
RG
Zhonghe
L
, et al.  . 
Early experience with pay-for-performance
JAMA
 , 
2005
, vol. 
294
 (pg. 
1788
-
92
)
Rowe
JW
Pay-for-performance and accountability: related themes in improving health care
Ann Intern Med
 , 
2006
(pg. 
695
-
699
)
Fretheim
A
Oxman
D
Flottorp
S
Improving prescribing of antihypertensive and cholesterol-lowering drugs: a method for identifying and addressing barriers to change
BMC Health Serv Res
 , 
2004
, vol. 
4
 pg. 
23
  
doi: 10.1186/1472-6963-4-23
Fretheim
A
Oxman
AD
Havelsrud
K
, et al.  . 
Rational prescribing in primary care (RaPP): a cluster randomized trial or a tailored intervention
PLoS Med
 , 
2006
, vol. 
3
  
doi: 10.1371/journal.pmed.0030134
Sanson-Fisher
RW
Bonevski
B
Green
L
, et al.  . 
Limitations of the randomized controlled trial in evaluating population-based health interventions
Am J Prev Med
 , 
2007
, vol. 
33
 (pg. 
155
-
61
)
Hawkins
NG
Sanson-Fisher
RW
Shakeshaft
A
, et al.  . 
The multiple baseline design for evaluating population-based research
Am J Prev Med
 , 
2007
, vol. 
33
 (pg. 
162
-
68
)
Asch
SM
Baker
DW
Keesey
J
, et al.  . 
Does the collaborative model improve care for chronic heart failure
Med Care
 , 
2005
, vol. 
43
 (pg. 
667
-
75
)