Abstract

Objective To synthesize the literature on clinical decision-support systems' (CDSS) impact on healthcare practitioner performance and patient outcomes.

Design Literature search on Medline, Embase, Inspec, Cinahl, Cochrane/Dare and analysis of high-quality systematic reviews (SRs) on CDSS in hospital settings. Two-stage inclusion procedure: (1) selection of publications on predefined inclusion criteria; (2) independent methodological assessment of preincluded SRs by the 11-item measurement tool, AMSTAR. Inclusion of SRs with AMSTAR score 9 or above. SRs were thereafter rated on level of evidence. Each stage was performed by two independent reviewers.

Results 17 out of 35 preincluded SRs were of high methodological quality and further analyzed. Evidence that CDSS significantly impacted practitioner performance was found in 52 out of 91 unique studies of the 16 SRs examining this effect (57%). Only 25 out of 82 unique studies of the 16 SRs reported evidence that CDSS positively impacted patient outcomes (30%).

Conclusions Few studies have found any benefits on patient outcomes, though many of these have been too small in sample size or too short in time to reveal clinically important effects. There is significant evidence that CDSS can positively impact healthcare providers' performance with drug ordering and preventive care reminder systems as most clear examples. These outcomes may be explained by the fact that these types of CDSS require a minimum of patient data that are largely available before the advice is (to be) generated: at the time clinicians make the decisions.

Introduction

In ‘Crossing the quality chasm,’ the Institute of Medicine pointed out the wide variations in healthcare practice, and the inefficiencies, dangers, and inequalities that have resulted from nonoptimal patient care.1 Evidence-based medicine (EBM) aims to reduce practice variation and improve quality of care. It does so by combining the clinical skills and experience of the healthcare professional and preferences of the patient with the best external clinical evidence available in order to make balanced decisions about medical care.2 EBM has, since its introduction in the 1980s, become widespread and has been adopted by international healthcare organizations such as the WHO and the Institute of Medicine. EBM seems a fairly common-sense solution, but it has proved to be far from simple to implement. A report from Grol et al provides a brief overview of strategies for the effective implementation of change in patient care.3 One of the interventions discussed is the use of reminders and computers for the implementation of evidence in daily practice. It is concluded that, among other interventions on the organizational and team level, professional development needs to be built into daily patient care as much as possible. This preferably should take place at the point of care with clinical decision-support tools and real-time patient-specific reminders to help doctors make the best decisions. Clinical decision support is defined as: ‘providing clinicians or patients with computer-generated clinical knowledge and patient-related information, intelligently filtered or presented at appropriate times, to enhance patient care.’4 Clinical knowledge incorporated in clinical decision-support systems (CDSS), for instance, can be based on available best evidence which is represented in guideline recommendations.

There are many different types of clinical tasks that can be supported by CDSS. A well-known and frequently applied CDSS is the patient-monitoring device (eg, an ECG or pulse oximeter) that warns of changes in a patient's condition. CDSS integrated in Electronic Medical Record systems (EMRs) and computerized physician order entry systems (CPOEs) can send reminders or warnings for deviating laboratory test results, check for drug–drug interactions, dosage errors, and other prescribing contraindications such as a patient's allergies, and generate lists of patients eligible for a particular intervention (eg, immunizations or follow-up visits). When a patient's case is complex or rare, or the healthcare practitioner making the diagnosis is inexperienced, a CDSS can help in formulating likely diagnoses based on (a) patient data and (b) the system's knowledge base of diseases. Subsequently, the CDSS can formulate treatment suggestions based upon treatment guidelines.

Research into the impact of CDSS on healthcare practitioner performance and patient outcomes in hospital settings has increased, and evidence of the effectiveness of CDSS has been synthesized into several systematic reviews (SRs). However, an overview of this evidence based on a critical appraisal of SRs focusing on CDSS impact is not available. Therefore, we set out to provide a synthesis of high-quality SRs examining CDSS interventions in hospital settings. The objective is (a) to summarize their effects on practitioner performance and patient outcome, and (b) to highlight areas where more research is needed.

Methods

Search strategy

To find relevant SRs, we developed a search strategy in cooperation with an experienced clinical Librarian. The search strategy was first developed for Medline to be adapted later to search Embase, Inspec, Cinahl, and Cochrane/Dare. To identify SRs we used a multiple-term search strategy as proposed by Montori et al5 and multiple keywords and Medical Subject Headings (MeSH) terms for the interventions CDSS or CPOE (table 1). No time period or language limitation was applied. Box 1 displays the search strategy for PubMed.

Table 1

Groups of keywords and Medical Subject Headings (MeSH) terms (in bold) used in the search strategy

Medical order entry systems Decision support systems, clinical 
  Computerized order entry Decision-support systems 
  Computerized prescriber order entry Clinical decision-support systems 
  Computerized provider order entry Reminder systems 
  Electronic order entry Computer assisted decision-making 
Electronic prescribing Diagnosis, computer-assisted 
  Electronic physician order entry Therapy, computer-assisted 
  Computerized physician order entry Expert systems 
Drug-therapy, computer-assisted  
Medical order entry systems Decision support systems, clinical 
  Computerized order entry Decision-support systems 
  Computerized prescriber order entry Clinical decision-support systems 
  Computerized provider order entry Reminder systems 
  Electronic order entry Computer assisted decision-making 
Electronic prescribing Diagnosis, computer-assisted 
  Electronic physician order entry Therapy, computer-assisted 
  Computerized physician order entry Expert systems 
Drug-therapy, computer-assisted  
Box 1

Search strategy in PubMed

(literature review[tiab] OR critical appraisal[tiab] OR meta analysis[pt] OR systematic review[tw] OR medline[tw]) AND (medical order entry systems[mh] OR medical order entry system*[tiab] OR computerized order entry[tiab] OR computerized prescriber order entry[tiab] OR computerized provider order entry[tiab] OR computerized physician order entry[tiab] OR electronic order entry[tiab] OR electronic prescribing[mh] OR electronic prescribing[tiab] OR cpoe[tiab] OR drug-therapy, computer assisted[mh] OR computer assisted drug therapy[tiab] OR decision support systems, clinical[mh] OR decision support system*[tiab] OR reminder system*[tiab] OR decision-making, computer assisted[mh] OR computer assisted decision making[tiab] OR diagnosis, computer assisted[mh] OR computer assisted diagnosis[tiab] OR therapy, computer assisted[mh] OR computer assisted therapy[tiab] OR expert systems[mh] OR expert system*[tiab] OR *CDS*[tiab]).

Inclusion of relevant studies

To assess whether publications that were found were relevant, we applied the following inclusion criteria: types of studies, intervention, target groups, and outcome measures.

Studies

Only SRs were eligible for inclusion. To determine wether a publication was a SR, we used the checklist for assessment of systematic reviews of the Dutch Cochrane Centre.6 For the initial screening of titles and abstracts, we considered a review to be systematic if at least (a) Medline had been searched, and (b) the methodological quality of the included studies had been assessed by the reviewer(s). For the screening based on full text, we added the AMSTAR criteria7,8 as described hereafter.

Intervention

CDSS combining clinical knowledge with patient characteristics, including CPOE systems decision-support functionality and CDSS for diagnostic performance.

Target groups

The CDSS interventions studied should be aimed at healthcare professionals such as: physicians, nurses, and other practitioners who are directly responsible for patient care in the hospital setting (in- and outpatient). We excluded CDSS interventions aimed at healthcare professionals who are indirectly involved in patient care at ancillary clinical departments such as laboratories, radiology, pathology, and physiological function departments.

Outcome measures

Within the SRs, either practitioner performance or patient outcomes should be measured.

A two-stage inclusion process was applied. In the first stage, titles and abstracts of articles identified by the search strategy were screened by two reviewers independently to assess whether these publications met the inclusion criteria. When the title and/or abstract provided insufficient information to determine relevance, full paper copies of the articles were retrieved in order to determine wether they fulfilled the inclusion criteria. Additionally, a manual search of the reference list of the selected full text papers was performed to identify SRs which our search could have missed. Any disagreements between the two reviewers were resolved by discussion and consultation of a third reviewer to reach consensus.

The second stage of inclusion relates to the methodological assessment of the reviews. All reviews that remained after the first stage were assessed with the Assessment of Multiple Systematic Reviews (AMSTAR) tool.7,8 AMSTAR is an 11-item measurement tool for the assessment of multiple systematic reviews that have good reliability and validity.8,9 The AMSTAR items are scored as ‘Yes,’ ‘No,’ ‘Can't answer,’ or ‘Not applicable.’ The AMSTAR criteria comprise: (1) ‘A priori’ design provided; (2) duplicate study selection/data extraction; (3) comprehensive literature search; (4) status of publication used as inclusion criterion; (5) list of studies (included/excluded) provided; (6) characteristics of included studies documented; (7) scientific quality assessed and documented; (8) appropriate formulation of conclusions; (9) appropriate methods of combining studies; (10) assessment of publication bias; and (11) conflict of interest statement.

The maximum score on AMSTAR is 11 and scores of 0–4 indicate that the review is of low quality; 5–8 that the review is of moderate quality; and 9–11 that the review is of high quality. Data were extracted only from high-quality reviews (=with scores of 9 and above) because low-quality reviews may reach different conclusions than high-quality reviews, and also to avoid false conclusions that are based on low-quality evidence.7

Two reviewers undertook independent critical appraisal. Discussion among the two reviewers and a third independent reviewer occurred on all dual-appraised articles to verify appraisal processes, and to resolve disagreements on individual item score allocation.

Data extraction and management

Data extraction was independently performed by two reviewers and verified by a third reviewer. The data abstracted were categorized in outcome measures: (a) practitioner performance and (b) patient outcome. The following data were extracted from each included SR using a structured data collection form: data related to clinical settings and target groups of CDSS implementation, number and type of trials included, and main outcomes and conclusions. Separate summaries were made for practitioner performance and patient outcomes. Within these summaries, a distinction was made between general SRs, disease/therapy-specific SRs and setting/patient population-specific SRs. Results on practitioner performance and patient outcomes were assessed by grading them on the strength of evidence for improvement. The evidence strength is based on the included randomized controlled trials (RCT) in the SRs. RCT was defined as an experimental design used for testing the effectiveness of a clinical decision-support tool in which individuals are assigned randomly to the intervention and a control group (standard procedure) and for which the outcomes are compared:

  • strong evidence: results based on RCTs and effect in 50% of more of the studies;

  • limited evidence: results based on RCTs and effect in 40–50% of the studies;

  • insufficient evidence: results based on non-randomized studies or effects in less than 40% of the studies.

Results

Selection of studies

After duplicates had been removed, the searches in the different databases resulted in an initial set of 849 references of potential interest. Initial sifting based on title and abstract resulted in exclusion of 703 articles with full agreement between the two reviewers, reducing the first set to 146 references. An additional manual search of the reference list of the selected studies resulted in another 12 potentially relevant references. Full texts of the remaining potential relevant articles (n=158) were assessed independently by two reviewers against the inclusion criteria. Agreement between reviewers in this phase was 94% with subsequent exclusion of 115 articles. Discussion among the three reviewers was needed for eight references, and agreement was subsequently reached (all eight articles excluded). A set of 35 references finally proved to fulfil the inclusion criteria for type and content of study.

In the following stage, two reviewers independently assessed the 35 included reviews on their methodological quality, using the AMSTAR tool.7,8 Initial reviewer agreement on 354 of the 385 (35×11) individual AMSTAR item scores was reached in this phase (92%); disagreements on score allocations were resolved through discussion with a third reviewer.

Further study excluded two SRs as preliminary reports in conference proceedings for which a full SR article was included.10,11 Of the remaining set of 33 SRs, 15 had a mean quality score lower than 9 and were excluded,1026 and 18 (51%) were high-quality SRs and advanced to the stage of data-extraction and analysis. One of these 18 SRs was excluded because it did not provide results on practitioner performance or patient outcomes. The flow diagram of the inclusion process is shown in figure 1. Table 2 provides the critical appraisal results of the SRs included.

Figure 1

Search flow for systematic review literature on clinical decision-support systems. Cochrane/Dare results were all duplicates of the other searches.

Figure 1

Search flow for systematic review literature on clinical decision-support systems. Cochrane/Dare results were all duplicates of the other searches.

Table 2

Systematic review critical appraisal based on Assessment of Multiple Systematic Reviews (AMSTAR) 11-item measurement tool7,8

First author, year AMSTAR criteria Total 
10 11 
Included 
  Johnston, 1994 
  Hunt, 1998 
  Montgomery, 1998 10 
  Chatellier, 1998 
  Walton, 1999 11 
  Kaushal, 2003 
  Balas, 2004 
  Garg, 2005 10 
  Liu, 2006 10 
  Sanders, 2006 
  Randell, 2007 
  Shamliyan, 2007 10 
  Wolfstadt, 2008 
  Yourman, 2008 
  Mollon, 2009 
  Durieux, 2009 10 
  Tan, 2009 10 
Excluded 
  Shiffman, 1999 
  Kaplan, 2001 
  Montani, 2001 
  Jackson, 2005 
  Chaudhry, 2006 
  Conroy, 2007 
  Eslami, 2007 
  Georgiou, 2007 
  Rommers, 2007 
  Shebl, 2007 
  Sintchenko, 2007 
  Ammenwerth, 2008 
  Mack, 2009 
  Van Rosse, 2009 
  Schedlbauer, 2009 
First author, year AMSTAR criteria Total 
10 11 
Included 
  Johnston, 1994 
  Hunt, 1998 
  Montgomery, 1998 10 
  Chatellier, 1998 
  Walton, 1999 11 
  Kaushal, 2003 
  Balas, 2004 
  Garg, 2005 10 
  Liu, 2006 10 
  Sanders, 2006 
  Randell, 2007 
  Shamliyan, 2007 10 
  Wolfstadt, 2008 
  Yourman, 2008 
  Mollon, 2009 
  Durieux, 2009 10 
  Tan, 2009 10 
Excluded 
  Shiffman, 1999 
  Kaplan, 2001 
  Montani, 2001 
  Jackson, 2005 
  Chaudhry, 2006 
  Conroy, 2007 
  Eslami, 2007 
  Georgiou, 2007 
  Rommers, 2007 
  Shebl, 2007 
  Sintchenko, 2007 
  Ammenwerth, 2008 
  Mack, 2009 
  Van Rosse, 2009 
  Schedlbauer, 2009 

Scale of item score: C, can't answer; N, no; NA, not applicable; Y, yes.

The AMSTAR criteria are (1) a priori design; (2) duplicate study selection and data extraction; (3) comprehensive literature search; (4) inclusive publication status; (5) included studies provided; (6) characteristics of included studies provided; (7) quality assessment of studies; (8) study quality used appropriately in formulating conclusions; (9) appropriate methods used to combine studies; (10) publication bias assessed; and (11) conflict of interest stated.

Synthesis of evidence

The results of the 17 included SRs are summarized in tables 3, 4.2743 The SRs were published between 1994 and 2009. In total these SRs included 411 references, 229 of which represented unique studies, with 188 RCTs. Of these unique 188 RCTs, 108 RCTs studied practitioner performance or patient outcome. No increase in SR quality, with regard to fulfilllment of the AMSTAR criteria, was visible over the years. Sixteen of the 17 SRs examined the influence of CDSS on practitioner performance: nine general SRs,2733,40,41 five disease- or therapy-specific SRs3437,42 and two setting-or patient-population-specific SRs.38,39 Evidence that CDSS significantly impacted practitioner performance was found in 52 out of 91 unique studies of the 16 SRs that examined this effect (57%). Twelve of these 16 SRs found strong evidence that CDSS improved practitioner performance: six general SRs,2732 four disease-or therapy specific SRs 3437 and two setting- or patient-population-specific SRs.38,39 Two general SRs found limited evidence,40,41 and one general33 and one disease- or therapy-specific SRs42 found insufficient evidence that CDSS impacted practitioner performance. Findings were mainly positive for computer reminder systems for preventive care and computer-assisted drug ordering and dosing systems. Preventive care CDDS led to improvements in management of high blood pressure, diabetes care, and asthma care. Drug-prescribing CDDS resulted in improvements in clinicians' ordering patterns of drug dosages and frequencies, decreases in serious medication ordering errors, and adequate drug concentrations in patients. There was insufficient evidence that CDSS improved anticoagulant prescription by physicians.

Table 3

Levels of evidence for clinical decision-support systems (CDSSs) impacting practitioner performance and patient outcomes of general systematic reviews

First author, year CDSS focus No of trials and subjects Conclusions 
Johnston, 1994 Healthcare practitioners 28 RCTs 41 924 patients 
  • Practitioner performance: improvement in 15 of 24 studies (63%)—SE

  • Drug dosing: improvement in three of four studies.

  • Diagnostic aids: improvement in one of five studies

  • Preventive care reminder: improvement in four of six studies

  • Studies evaluating CDSSs for other medical care: improvement in seven of nine studies

  • Patient outcome: improvement in three of 10 studies (30%)—IE

  • Drug dosing: three studies, no evidence of improvement

  • Diagnostic aid: Improvement in one study

  • Preventive care reminder: one study, no evidence of improvement

  • Studies evaluating CDSSs for other medical care: improvement in two (40%) of five studies

 
Hunt, 1998 Healthcare practitioners 68 RCTs ∼65 000 patients 
  • Practitioner performance: improvement in 43 of 65 studies (66%)—SE

  • Drug dosing: improvement in nine (60%) of 15 studies

  • Diagnostic aids: improvement in one (20%) of five studies

  • Preventive care reminder: improvement in 14 (74%) of 19 studies

  • Studies evaluating CDSSs for other medical care: Improvement in 19 (73%) of 26 studies

  • Patient outcome: Improvement in five of 13 studies (39%)—IE

  • Drug-dosing systems, preventive care vaccination

 
Walton, 1999 Healthcare practitioners Six RCTs, one non-randomized 671 patients 
  • Practitioner performance: improvement in seven of 11 studies (4%)—SE

  • Drug dosing: improvement in seven of 11 studies

  • Patient outcome: improvement in seven of 13 studies (75%)—SE

  • Drug dosing: improvement in four of six studies (66%)

 
Kaushall, 2003  Seven RCTs, three pre–post, two non-randomized; ∼19 000 patients 
  • Practitioner performance: Improvement in six of seven studies (86 %)—SE

  • Drug prescribing: improvement in six of seven studies.

  • Patient outcome: Improvement in three of seven studies (43%)—LE

  • Drug dosing systems: improvement in four studies

  • Preventive care reminder systems: improvement in one study

 
Garg, 2005 Healthcare practitioners 88 RCTs, 12 controlled non-randomized trials ∼92 895 patients 3826 practitioners 
  • Practitioner performance: improvement in 52 of 86 studies (59%)—SE

  • Drug dosing: improvement in 19 (66%) of 29 studies

  • Diagnostic aids: improvement in four (40%) of 10 studies

  • Preventive care reminder: improvement in 16 (76%) of 21 studies

  • Studies evaluating CDSSs for other medical care: improvement in 23 (62%) of 37 studies

  • Patient outcome: improvement in nine of 50 studies (18%)—IE

  • Drug-dosing systems: improvement in two of 18 studies (11%).

  • Disease management: improvement in five of 27 RCTs (19%)

 
Randell, 2007 Clinical nurses Eight RCTs ∼24 000 patients, ∼100 nurses 
  • Practitioner performance: improvement in one of two studies (50%), poorer performance in one (50%) of two studies—LE

  • Preventive care reminder: four studies

  • Patient outcome: Improvement in 0 of three studies (0%)—IE

 
Shamliyan, 2007 Healthcare practitioners One RCT, eight prospective non-randomized, three retrospective studies ∼200 000 patients 
  • Practitioner performance: improvement in nine (100%) of nine studies, one RCT—IE

  • Drug prescribing: improvement in nine of nine studies

  • Patient outcome: improvement in two (50%) of four studies, one RCT—IE

 
Wolfstadt, 2008 Healthcare practitioners 10 prospective non-randomized, ∼300 000 patients 
  • Practitioner performance: not studied

  • Patient outcome: improvement in five (50%) of 10 studies, all 10 non-randomized—IE

 
Mollon, 2009 Healthcare practitioners 41 RCTs, 612 556 patients 2963 providers 
  • Practitioner performance: improvement in 25 (61%) of 41 studies—SE

  • Drug prescribing: improvement in 25 of 41 studies

  • Patient outcome: improvement in five of 23 studies (22%)—IE

 
Durieux, 2009 Healthcare practitioners 20 RCTs, three controlled trials ∼3000 patients 
  • Practitioner performance: pooling of results, limited evidence—LE

  • Drug prescribing and dosing

  • Patient outcome: pooling of results, limited evidence—LE

 
First author, year CDSS focus No of trials and subjects Conclusions 
Johnston, 1994 Healthcare practitioners 28 RCTs 41 924 patients 
  • Practitioner performance: improvement in 15 of 24 studies (63%)—SE

  • Drug dosing: improvement in three of four studies.

  • Diagnostic aids: improvement in one of five studies

  • Preventive care reminder: improvement in four of six studies

  • Studies evaluating CDSSs for other medical care: improvement in seven of nine studies

  • Patient outcome: improvement in three of 10 studies (30%)—IE

  • Drug dosing: three studies, no evidence of improvement

  • Diagnostic aid: Improvement in one study

  • Preventive care reminder: one study, no evidence of improvement

  • Studies evaluating CDSSs for other medical care: improvement in two (40%) of five studies

 
Hunt, 1998 Healthcare practitioners 68 RCTs ∼65 000 patients 
  • Practitioner performance: improvement in 43 of 65 studies (66%)—SE

  • Drug dosing: improvement in nine (60%) of 15 studies

  • Diagnostic aids: improvement in one (20%) of five studies

  • Preventive care reminder: improvement in 14 (74%) of 19 studies

  • Studies evaluating CDSSs for other medical care: Improvement in 19 (73%) of 26 studies

  • Patient outcome: Improvement in five of 13 studies (39%)—IE

  • Drug-dosing systems, preventive care vaccination

 
Walton, 1999 Healthcare practitioners Six RCTs, one non-randomized 671 patients 
  • Practitioner performance: improvement in seven of 11 studies (4%)—SE

  • Drug dosing: improvement in seven of 11 studies

  • Patient outcome: improvement in seven of 13 studies (75%)—SE

  • Drug dosing: improvement in four of six studies (66%)

 
Kaushall, 2003  Seven RCTs, three pre–post, two non-randomized; ∼19 000 patients 
  • Practitioner performance: Improvement in six of seven studies (86 %)—SE

  • Drug prescribing: improvement in six of seven studies.

  • Patient outcome: Improvement in three of seven studies (43%)—LE

  • Drug dosing systems: improvement in four studies

  • Preventive care reminder systems: improvement in one study

 
Garg, 2005 Healthcare practitioners 88 RCTs, 12 controlled non-randomized trials ∼92 895 patients 3826 practitioners 
  • Practitioner performance: improvement in 52 of 86 studies (59%)—SE

  • Drug dosing: improvement in 19 (66%) of 29 studies

  • Diagnostic aids: improvement in four (40%) of 10 studies

  • Preventive care reminder: improvement in 16 (76%) of 21 studies

  • Studies evaluating CDSSs for other medical care: improvement in 23 (62%) of 37 studies

  • Patient outcome: improvement in nine of 50 studies (18%)—IE

  • Drug-dosing systems: improvement in two of 18 studies (11%).

  • Disease management: improvement in five of 27 RCTs (19%)

 
Randell, 2007 Clinical nurses Eight RCTs ∼24 000 patients, ∼100 nurses 
  • Practitioner performance: improvement in one of two studies (50%), poorer performance in one (50%) of two studies—LE

  • Preventive care reminder: four studies

  • Patient outcome: Improvement in 0 of three studies (0%)—IE

 
Shamliyan, 2007 Healthcare practitioners One RCT, eight prospective non-randomized, three retrospective studies ∼200 000 patients 
  • Practitioner performance: improvement in nine (100%) of nine studies, one RCT—IE

  • Drug prescribing: improvement in nine of nine studies

  • Patient outcome: improvement in two (50%) of four studies, one RCT—IE

 
Wolfstadt, 2008 Healthcare practitioners 10 prospective non-randomized, ∼300 000 patients 
  • Practitioner performance: not studied

  • Patient outcome: improvement in five (50%) of 10 studies, all 10 non-randomized—IE

 
Mollon, 2009 Healthcare practitioners 41 RCTs, 612 556 patients 2963 providers 
  • Practitioner performance: improvement in 25 (61%) of 41 studies—SE

  • Drug prescribing: improvement in 25 of 41 studies

  • Patient outcome: improvement in five of 23 studies (22%)—IE

 
Durieux, 2009 Healthcare practitioners 20 RCTs, three controlled trials ∼3000 patients 
  • Practitioner performance: pooling of results, limited evidence—LE

  • Drug prescribing and dosing

  • Patient outcome: pooling of results, limited evidence—LE

 

IE, insufficient evidence; LE, limited evidence; RCT, randomized controlled trial; SE, strong evidence.

Table 4

Levels of evidence for clinical decision-support systems (CDSSs) impacting practitioner performance and patient outcomes of disease-/therapy-specific and setting-/patient-population-specific systematic reviews

First author, year CDSS focus No of trials and subjects Conclusions 
Disease-/therapy-specific systematic reviews 
  Montgomery, 1998 Hypertension Seven RCTs 11 962 patients, 91 practices Note: three unique studies 
  • Practitioner performance: improvement in four of five studies (80%)—SE

  • Preventive care reminder: improvement in four of five studies

  • Patient outcome: improvement in two of six studies (33%)—IE

 
  Chatellier, 1998 Anticoagulant therapy Nine RCTs 1336 patients 
  • Practitioner performance: improvement in two of seven studies (29%)—IE

  • Drug prescribing: improvement in two of seven studies

  • Patient outcome: insufficient power of studies for analysis of this outcome

 
  Balas, 2004 Diabetes care 44 RCTs, 31 CDSS focused 6109 patients 
  • Practitioner performance: improvement in five of six studies (83%)—SE

  • Preventive care reminder: improvement in five of six studies

  • Patient outcome: improvement in 13 of 24 studies (54%)—SE

 
  Liu, 2006 Acute abdominal pain One RCT 5193 patients 
  • Practitioner performance: improvement in one of one studies (100%)—SE

  • Care management: improvement in one of one studies

  • Patient outcome: no evidence of improvement—IE

 
  Sanders, 2006 Asthma care 18 RCTs, three uncontrolled trials nine CDSS focused 5757 patients 
  • Practitioner performance: improvement in two of four studies (50%)—SE

  • Preventive care reminder: improvement in two of four studies

  • Patient outcome: improvement in three of the five studies (60%)—SE

  • Preventive care reminder: improvement in two of two studies (100%)

 
Setting/patient-population-specific systematic reviews 
  Yourman, 2008 Medication prescribing adults ≥60 years Five RCTs, one pre-/poststudy, one cohort study, three interrupted time series 
  • Practitioner performance: improvement in five of five studies (100%)—SE

  • Drug prescribing: improvement in five of five studies

  • Patient outcome: no evidence of improvement—IE

 
  Tan, 2009 Effects of CDSS on neonatal care Three RCTs 282 patients, 27 health professionals 
  • Practitioner performance: improvement in one of two studies (50%)—SE

  • Drug prescribing: improvement in one of two studies

  • Patient outcome: one study, no evidence of improvement—IE

 
First author, year CDSS focus No of trials and subjects Conclusions 
Disease-/therapy-specific systematic reviews 
  Montgomery, 1998 Hypertension Seven RCTs 11 962 patients, 91 practices Note: three unique studies 
  • Practitioner performance: improvement in four of five studies (80%)—SE

  • Preventive care reminder: improvement in four of five studies

  • Patient outcome: improvement in two of six studies (33%)—IE

 
  Chatellier, 1998 Anticoagulant therapy Nine RCTs 1336 patients 
  • Practitioner performance: improvement in two of seven studies (29%)—IE

  • Drug prescribing: improvement in two of seven studies

  • Patient outcome: insufficient power of studies for analysis of this outcome

 
  Balas, 2004 Diabetes care 44 RCTs, 31 CDSS focused 6109 patients 
  • Practitioner performance: improvement in five of six studies (83%)—SE

  • Preventive care reminder: improvement in five of six studies

  • Patient outcome: improvement in 13 of 24 studies (54%)—SE

 
  Liu, 2006 Acute abdominal pain One RCT 5193 patients 
  • Practitioner performance: improvement in one of one studies (100%)—SE

  • Care management: improvement in one of one studies

  • Patient outcome: no evidence of improvement—IE

 
  Sanders, 2006 Asthma care 18 RCTs, three uncontrolled trials nine CDSS focused 5757 patients 
  • Practitioner performance: improvement in two of four studies (50%)—SE

  • Preventive care reminder: improvement in two of four studies

  • Patient outcome: improvement in three of the five studies (60%)—SE

  • Preventive care reminder: improvement in two of two studies (100%)

 
Setting/patient-population-specific systematic reviews 
  Yourman, 2008 Medication prescribing adults ≥60 years Five RCTs, one pre-/poststudy, one cohort study, three interrupted time series 
  • Practitioner performance: improvement in five of five studies (100%)—SE

  • Drug prescribing: improvement in five of five studies

  • Patient outcome: no evidence of improvement—IE

 
  Tan, 2009 Effects of CDSS on neonatal care Three RCTs 282 patients, 27 health professionals 
  • Practitioner performance: improvement in one of two studies (50%)—SE

  • Drug prescribing: improvement in one of two studies

  • Patient outcome: one study, no evidence of improvement—IE

 

IE, insufficient evidence; LE, limited evidence; RCT, randomized controlled trial; SE, strong evidence.

Sixteen out of the 17 SRs studied the impact of CDSS on patient outcomes. Evidence that CDSS significantly impacted patient outcomes was found in 25 out of 81 unique studies of the 16 SRs that examined this effect (30%). These effects were related to drug ordering and dosing systems and CDSS for preventive care and disease management. Only three of these 16 SRs29,35,37 found strong evidence that CDSS impacted patient outcomes: one general SR29 and two disease-/therapy-specific SRs.35,37 Two general SRs found limited evidence,30,41 and the remaining 11 SRs found insufficient evidence: seven general,27,28,3133,40,41 two disease-/therapy-specific34,36 and the two setting-/patient-population-specific SRs.38,39

Discussion

Quality of studies included in the systematic reviews

Although we merely synthesized the findings of SRs with an AMSTAR score of 9–11, the methodology quality of the studies included in these SRs was a major discussion point. The SRs included a total of 229 unique studies with 188 RCTs (82%). Non-randomized uncontrolled interventions may provide biased overestimated effects of CDSS. None of the 10 studies in the SR by Wolfstadt et al,43 for example, were RCTs, which highly reduced the significance of their findings. Risk of contamination of results was another concern in the RCT studies included in at least three SRs.32,37,38 Finally, Liu et al36 reported that the uncontrolled before–after study and interrupted-time series, both taking limited account of known but not at all for unknown confounding factors, have been the most popular to evaluate CDSS for acute abdominal pain. Walton et al29 concluded that a common bias in many of the studies of their SR was that the same clinicians treated patients allocated to the CDSS intervention and control condition. As a result, the effects of the CDSS may spill over into the control group. Contamination of the control group in this manner would tend to make it more difficult to show a beneficial effect from CDSS.

Regarding the lack of positive findings on patient outcomes, many authors discussed that this may be due to the small sample sizes in the original studies that consequently were underpowered.9,28,38 Furthermore, follow-up periods in most studies were often not long enough to assess long-term differences on patient outcomes related to the computerized interventions (eg, see Shamliyan et al33). Studies with too small sample sizes or too short follow-up periods are at risk of overinterpretation of non-significant results. Fortunately, Hunt et al28 and Garg et al31 indicate that the number and methodological quality of trials have improved over time. An increase in RCTs including a power analysis to calculate the minimum sample size required to show an impact of CDSS was also reported.

Synthesis of the systematic reviews results

It is clear from our synthesis that few SRs have found benefits on patient outcomes, though many SRs have been too small in sample size or too short in time to reveal clinically important effects related to patients. There is, however, significant evidence that CDSS can positively impact healthcare providers' performance with preventive care reminder systems and drug prescription systems as most clear examples. Exceptions are anticoagulant prescription systems for which the findings thus far are inconclusive. The studies of diagnostic CDSS are likewise less positive.

An explanation for the findings related to diagnostic CDSS can be found in evidence that suggests that clinicians, based on their clinical experience, are better able to rule out alternative diagnoses than diagnostic CDSS. This may lower the impact of these CDSS in clinical practice. Moreover, the level of specificity of diagnostic advice varies considerably among diagnostic CDSS.36 The specificity level of computer-generated advice is also known to highly influence the chance that physicians adhere to the advice, with low specificity resulting in computer-advice fatigue and in situations where physicians ignore the advice.12 Second, the diagnostic reasoning models of these CDSS often require input of a large number of patient data (demographic data, data on complaints, symptoms, previous history, physical examination, laboratory, and other tests) to deliver the decision support. As long as these data are not electronically available, for example in an electronic patient record (EMR), clinicians are to enter all these data items manually. The burden of data entry may make them give up and not use the CDSS: as a consequence, they may perform no better than unaided clinicians. There is evidence that arduous data-entry facilities adversely affect clinicians' satisfaction with CDSS that makes them abandon the CDSS.12 When data entry is incomplete, the diagnoses generated by CDSS will be less accurate, and this also may reduce their impact in practice. Part of the data that diagnostic CDSS need becomes available during the clinical process. This prolongs the time that these CDSS can deliver their advice. Advice that shows up too late in the work flow of CDSS users increases the likelihood that they over-ride it. There is indeed evidence that the impact of diagnostic CDSS is lowered if the time that their output becomes available mismatches clinicians' workflow and does not reach them in time.36,44

Unlike diagnostic CDSS, most preventive care systems and CDSS drug-prescribing systems require a limited number of patient data items for input to the decision-support facility. Preventive care reminder systems are for the most applied in routine tasks (blood-pressure tests, Pap smears, vaccinations), prompt doctors to call patients in for a procedure, or alert them that a procedure is due when the patient is at the physician's office. Most CDSS for drug prescription warn clinicians when there is a drug interaction or an allergy listed in the patient's data file, or when they have ordered an unusual dose or frequency of a certain drug. Only few drug prescription CDSSs can also perform drug-disease and/or drug–lab interaction checking and have the advanced feature of patient-specific dose calculation. So, the majority of these CDSS require patient data that are largely available before the advice is (to be) generated: at the time and place clinicians make the decisions. It has been shown that computer advice improves providers' right drug choices and reduces the likelihood of adverse drug events when it is delivered at the time when it is most needed.33,45 So, usage patterns of diagnostic CDSS seem to depend on their complexity, the number of additional data items to collect, the ease of data entry, and the extra time needed to work with the CDSS.36 Several studies on diagnostic aids indeed suggest that CDSS was inefficient because it required more time and effort from the user compared to the paper-based situation.31,36 In contrast, preventive care and drug-prescribing CDSS require minimal to no extra data input from the user except the data they already produced in the context of a patient visit or ordering task. As a result, these types of CDSS minimize the interruption of the user's workflow.

Anticoagulant prescribing CDSS share certain features with diagnostic CDSS and are therefore more complex than other drug-prescribing CDSS. Anticoagulant therapy is a course of drug therapy that a clinician must supervise carefully because it carries a number of risks. For example, many drugs can interact dangerously with anticoagulants, and the patient needs to be monitored continuously for complications. Anticoagulation therapy guidelines thus vary by patient and situation, and the clinician must take care to confirm that the course of therapy is appropriate.46 Management of this therapy by CDSS therefore requires a full patient history dataset to learn about the patient's lifestyle and to identify any risk factors which could complicate the therapy. These factors likewise complicate the entry of patient data: data which are needed to deliver the computer advice for support of clinicians in their decision-making and in managing the therapy during the course of its administration. All these aspects may explain the low impact of anticoagulant prescribing CDSS in practice.

The efficacy of CDSS can be improved when the specificity and sensitivity levels of their advice increase, the need for manual input of (extra) patient data is minimized, and the computer advice is given at the time the clinicians make decisions. The future impact of CDSS will therefore depend on (1) the progress in the biomedical-informatics research domain related to knowledge discovery and reasoning, and (2) the development of integrated environments with a merging of EMRs and CDSS. Research progress in knowledge discovery and reasoning has been impressive over the past decade with advanced machine learning, data-mining techniques, and temporal reasoning as a few examples. Even more sophisticated knowledge discovery techniques that permit the integration of clinical expertise with machine-learning methods are under way.47 Less advancement has, however, been achieved in the integration and sharing of this knowledge in EMRs. The current lack of commonly accepted terminologies, ontologies, and standards for intelligent interfacing make the electronic exchange and interoperation of healthcare data and knowledge hard to achieve. To fully realize the potential of EMRs, future challenges are in defining and reaching agreement on these communication and data-sharing standards, and in realizing complete datasets that are coded according to agreed upon terminological systems.47 Future CDSS should be integrated with these EMRS and provide their decision support at the right time with minimal interruption of clinicians' work flow. Physicians seem to perform better in circumstances where CDSS automatically prompt them than when they have to initiate the interaction themselves.45 This suggests that these systems should work in the background and continuously monitor and check whether the care (to be) delivered to individual patients is in accordance with applicable guidelines. The CDSS should then only deliver its advice in situations where clinicians do not follow these guideline recommendations or when unforeseen patient outcomes occur.

Advantages and limitations of selection process

This synthesis of high-quality SRs on CDSS has several strengths and weaknesses. First, the literature search was thorough: we did not limit our search to a certain time period, and we screened 849 SRs for relevance. Second, we critically appraised the quality of the preincluded SRs based on the standardized AMSTAR 11-item measurement tool. Third, we used two independent reviewers for preselection of SRs based on predefined inclusion criteria, for the assessment of SRs' quality and for the final data extraction. Fourth, we manually searched the reference lists of the selected SRs to identify SRs that we could have missed in our literature search.

One limitation of the study is that we excluded SRs with an AMSTAR score below 9.

One could argue that not all item scores of the AMSTAR measurement tool should have an equal weight in critical appraisal of the SRs. Furthermore, in certain clinical domains, the quality of candidate studies may be systematically poorer than in other domains. As a consequence, SRs of CDSS studies in these domains would never achieve a score of 9 or above while these same studies may be the benchmark by which clinical guidelines are set.

Another limitation of the study is that 14 of the 17 SRs with an AMSTAR score of 9 or higher did not assess the likelihood of publication bias. As explained earlier, publication bias against studies that failed to show an effect which were not included in the SRs limits the results of this synthesis of evidence on CDSS that impact practitioner performance and patient outcomes.

On the one hand, we defined effect in 50% or more of the RCTs as strong evidence, in 40–50% of the RCTs as limited evidence, and in less than 40% of the RCTs or non-randomized studies as insufficient evidence respectively. These ranges, along with the strict inclusion criteria of this synthesis may have underestimated CDSS success rates and their impact on practitioners' performance and patient outcomes. On the other hand, there was a large overlap in studies included in the SRs which may have led to an overestimation of CDSS impact rates. We therefore analyzed the number of unique studies from the total number included in all SRs and provided overall estimates of the evidence that CDSS significantly impacted practitioner performance and patient outcomes. Finally, the conclusions from this synthesis are probably limited in so far as the SRs included in this synthesis described CDSS, some of which were developed more than a decade ago. As discussed, CDSS are constantly evolving; newer generations of CDSS probably have greater capability and usability, and will therefore have a different impact on practitioners' performance and patient outcomes.

Acknowledgments

We thank A Leenders, clinical librarian, for the development of the search strategy.

Competing interests

None.

Provenance and peer review

Not commissioned; externally peer reviewed

References

1
Institute of Medicine
.
Crossing the Quality Chasm: A New Health System for the 21st Century
 .
Washington DC
:
National Academy of Sciences
,
2001
.
2
Sackett
DL
Rosenberg
WM
Gray
JA
et al
.
Evidence based medicine: what it is and what it isn't
.
BMJ
 
1996
;
312
:
71
2
.
3
Grol
R
Grimshaw
J
.
From best evidence to best practice: effective implementation of change in patients' care
.
Lancet
 
2003
;
362
:
1225
30
.
4
Osheroff
JA
Pifer
EA
Teich
JM
et al
.
Improving Outcomes with Clinical Decision Support: An Implementer's Guide
 .
Chicago
:
Healthcare Information and Management Systems Society
,
2005
.
5
Montori
VM
Wilczynski
NL
Morgan
D
et al
.
Optimal search strategies for retrieving systematic reviews from Medline: analytical survey
.
BMJ
 
2005
;
330
:
68
.
7
Shea
BJ
Grimshaw
JM
Wells
GA
et al
.
Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews
.
BMC Med Res Methodol
 
2007
;
7
:
10
.
8
Shea
BJ
Bouter
LM
Peterson
J
et al
.
External validation of a measurement tool to assess systematic reviews (AMSTAR)
.
PLoS One
 
2007
;
2
:
e1350
.
9
Shea
BJ
Hamel
C
Wells
GA
et al
.
AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews
.
J Clin Epidemiol
 
2009
;
62
:
1013
20
.
10
Niès
J
Colombet
I
Degoulet
P
et al
.
Determinants of success for computerized clinical decision support systems integrated in CPOE systems: a systematic review
.
AMIA Annu Symp Proc
 
2006
:
594
8
.
11
Langton
KB
Johnston
ME
Haynes
RB
et al
.
A critical appraisal of the literature on the effects of computer-based clinical decision support systems on clinician performance and patient outcomes
.
Proc Annu Symp Comput Appl Med Care
 
1992
:
626
30
.
12
Shiffman
RN
Liaw
Y
Brandt
CA
et al
.
Computer-based guideline implementation systems: a systematic review of functionality and effectiveness
.
J Am Med Inform Assoc
 
1999
;
6
:
104
14
.
13
Kaplan
B
.
Evaluating informatics applications—clinical decision support systems literature review
.
Int J Med Inform
 
2001
;
64
:
15
37
.
14
Montani
S
Bellazzi
R
Quaglini
S
et al
.
Meta-analysis of the effect of the use of computer-based systems on the metabolic control of patients with diabetes mellitus
.
Diabetes Technol Ther
 
2001
;
3
:
347
56
.
15
Jackson
CL
Bolen
S
Brancati
FL
et al
.
A systematic review of interactive computer-assisted technology in diabetes care. Interactive information technology in diabetes care
.
J Gen Intern Med
 
2006
;
21
:
105
10
.
16
Chaudhry
B
Wang
J
Wu
S
et al
.
Systematic review: impact of health information technology on quality, efficiency, and costs of medical care
.
Ann Intern Med
 
2006
;
144
:
742
52
.
17
Conroy
S
Sweis
D
Planner
C
et al
.
Interventions to reduce dosing errors in children: a systematic review of the literature
.
Drug Saf
 
2007
;
30
:
1111
25
.
18
Eslami
S
de Keizer
NF
Abu-Hanna
A
.
The impact of computerized physician medication order entry in hospitalized patients—a systematic review
.
Int J Med Inform
 
2008
;
77
:
365
76
.
19
Georgiou
A
Williamson
M
Westbrook
JI
et al
.
The impact of computerised physician order entry systems on pathology services: a systematic review
.
Int J Med Inform
 
2007
;
76
:
514
29
.
20
Rommers
MK
Teepe-Twiss
IM
Guchelaar
HJ
.
Preventing adverse drug events in hospital practice: an overview
.
Pharmacoepidemiol Drug Saf
 
2007
;
16
:
1129
35
.
21
Shebl
NA
Franklin
BD
Barber
N
.
Clinical decision support systems and antibiotic use
.
Pharm World Sci
 
2007
;
29
:
342
9
.
22
Sintchenko
V
Magrabi
F
Tipper
S
.
Are we measuring the right end-points? Variables that affect the impact of computerised decision support on patient outcomes: a systematic review
.
Med Inform Internet Med
 
2007
;
32
:
225
40
.
23
Ammenwerth
E
Schnell-Inderst
P
Machan
C
et al
.
The effect of electronic prescribing on medication errors and adverse drug events: a systematic review
.
J Am Med Inform Assoc
 
2008
;
15
:
585
600
.
24
Mack
EH
Wheeler
DS
Embi
PJ
.
Clinical decision support systems in the pediatric intensive care unit
.
Pediatr Crit Care Med
 
2009
;
10
:
23
8
.
25
Van Rosse
F
Maat
B
Rademaker
CM
et al
.
The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review
.
Pediatrics
 
2009
;
123
:
1184
90
.
26
Schedlbauer
A
Prasad
V
Mulvaney
C
et al
.
What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior?
J Am Med Inform Assoc
 
2009
;
16
:
531
8
.
27
Johnston
ME
Langton
KB
Haynes
RB
et al
.
Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research
.
Ann Intern Med
 
1994
;
120
:
135
42
.
28
Hunt
DL
Haynes
RB
Hanna
E
et al
.
Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review
.
JAMA
 
1998
;
280
:
1339
46
.
29
Walton
R
Dovey
S
Harvey
E
et al
.
Information in practice. Computer support for determining drug dose: systematic review and meta-analysis
.
BMJ
 
1999
;
318
:
984
90
.
30
Kaushal
R
Shojania
KG
Bates
DW
.
Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review
.
Arch Intern Med
 
2003
;
163
:
1409
16
.
31
Garg
AX
Adhikari
NK
McDonald
H
et al
.
Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review
.
JAMA
 
2005
;
293
:
1223
38
.
32
Mollon
B
Chong
J
Holbrook
AM
et al
.
Features predicting the success of computerized decision support for prescribing: a systematic review of RCTs
.
BMC Med Inform Decis Mak
 
2009
;
9
:
11
20
.
33
Shamliyan
TA
Duval
S
Du
J
et al
.
Just what the doctor ordered. Review of the evidence of impact of compurized physician order entry systems on medication errors
.
Health Serv Res
 
2008
;
43
:
32
53
.
34
Montgomery
AA
Fahey
T
.
A systematic review of the use of computers in the management of hypertension
.
J Epidemiol Community Health
 
1998
;
52
:
520
5
.
35
Balas
EA
Krishna
S
Kretschmer
RA
et al
.
Computerized knowledge management in diabetes care
.
Med Care
 
2004
;
42
:
610
21
.
36
Liu
LY
Wyatt
JC
Deeks
JJ
et al
.
Systematic reviews of clinical decision tools for acute abdominal pain
.
Health Technol Assess
 
2006
;
10
:
1
167
. iii–iv.
37
Sanders
DL
Aronsky
D
.
Biomedical informatics applications for asthma care: a systematic review
.
J Am Med Inform Assoc
 
2006
;
13
:
418
27
.
38
Yourman
L
Concato
J
Agostini
JV
.
Use of computer decision support interventions to improve medication prescribing in older adults: a systematic review
.
Am J Geriatr Pharmacother
 
2008
;
6
:
119
29
.
39
Tan
K
Dear
PR
Newell
SJ
.
Clinical decision support systems for neonatal care
.
Cochrane Database Syst Rev
 
2005
;(
2
):
CD004211
.
40
Randell
R
Mitchell
N
Dowding
D
et al
.
Effects of computerized decision support systems on nursing performance and patient outcomes: a systematic review
.
J Health Serv Res Policy
 
2007
;
12
:
242
9
.
41
Durieux
P
Trinquart
L
Colombet
LI
et al
.
Computerized advice on drug dosage to improve prescribing practice
.
Cochrane Database Syst Rev
 
2008
;(
3
):
CD002894
.
42
Chatellier
G
Colombet
I
Degoulet
P
.
An overview of the effect of computer-assisted anticoagulant therapy on the quality of anticoagulation
.
Int J Med Inform
 
1998
;
49
:
311
20
.
43
Wolfstadt
JI
Gurwitz
JH
Field
TS
et al
.
The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review
.
J Gen Intern Med
 
2008
;
23
:
451
8
.
44
Khajouei
R
Jaspers
MW
.
The impact of CPOE medication systems' design aspects on USABILITY, Workflow and medication orders
.
Methods Inf Med
 
2010
;
49
:
3
19
.
45
Kawamoto
K
Houlihan
CA
Balas
EA
et al
.
Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success
.
BMJ
 
2005
;
330
:
765
.
46
American Heart Association/American College of Cardiology Foundation Guide to Anticoagulation
.
Therapy
. http://www.heart.org/HEARTORG/Conditions/CongenitalHeartDefects/TheImpactofCongenitalHeartDefects/Anticoagulation_UCM_307110_Article.jsp (accessed 2 Dec 2010).
47
Patel
VL
Shortliffe
EH
Stefanelli
M
et al
.
The coming of age of artificial medicine
.
Artif Intell Med
 
2009
;
46
:
5
17
.

Comments

0 Comments