Abstract

Objective To systematically review studies assessing the effects of health information technology (health IT) on patient safety outcomes.

Materials and Methods The authors employed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement methods. MEDLINE, Cumulative Index to Nursing Allied Health (CINAHL), EMBASE, and Cochrane Library databases, from 2001 to June 2012, were searched. Descriptive and comparative studies were included that involved use of health IT in a clinical setting and measured effects on patient safety outcomes.

Results Data on setting, subjects, information technology implemented, and type of patient safety outcomes were all abstracted. The quality of the studies was evaluated by 2 independent reviewers (scored from 0 to 10). A total of 69 studies met inclusion criteria. Quality scores ranged from 1 to 9. There were 25 (36%) studies that found benefit of health IT on direct patient safety outcomes for the primary outcome measured, 43 (62%) studies that either had non-significant or mixed findings, and 1 (1%) study for which health IT had a detrimental effect. Neither the quality of the studies nor the rate of randomized control trials performed changed over time. Most studies that demonstrated a positive benefit of health IT on direct patient safety outcomes were inpatient, single-center, and either cohort or observational trials studying clinical decision support or computerized provider order entry.

Discussion and Conclusion Many areas of health IT application remain understudied and the majority of studies have non-significant or mixed findings. Our study suggests that larger, higher quality studies need to be conducted, particularly in the long-term care and ambulatory care settings.

Effectively harnessing the potential of health information technology (health IT) to improve patient safety, reduce harm, and improve patient outcomes remains a unifying national goal among healthcare providers, patients, and regulators. For several decades, the use of computer systems has been considered a potential mechanism to support and improve clinical care. 1 Through the Medicare and Medicaid Electronic Health Record (EHR) Incentive Program, known as the meaningful use program, the federal government is investing billions of dollars to promote the adoption of health IT in order to improve patient outcomes. 2 Rates of health IT adoption in the inpatient and outpatient settings are increasing, and the range of available technology remains vast and varied. 3 An important barrier to health IT adoption has been the uncertain effect on patient outcomes, particularly given the costliness of implementation of computerized infrastructures. 4 In order to evaluate the current state of the literature, we conducted a systematic review to determine the effect of multiple health IT tools on patient safety outcomes.

While 31 systematic reviews have been conducted with a focus on health IT interventions and patient safety outcomes, this systematic review is different for several reasons. First, many of the systematic reviews focused upon one specific health IT, 5–10 most commonly clinical decision support (CDS). 11–26 Second, prior reviews often focused upon one area of clinical care such as outpatient, 13,15,27,28 inpatient, 6,16,29 intensive care, 30 pediatrics, 5,30 or geriatrics. 23 Other papers targeted very specific outcomes, such as the effects of health IT as it relates to antibiotic medications, 22 anticoagulant therapy, 20 lab testing, 7 or treatment of hypertension. 13 Finally, many prior reviews looked specifically at effects of health IT on one safety outcome—adverse drug events (ADEs). 5,6,19,24,26,29,31–33

Prior studies generally included both non-randomized and randomized trials. 4–9,15–19,21–23,27–29,32–34 Eleven of the prior systematic reviews included only the highest level of evidence studies, randomized controlled trials (RCT). 10–14,20,24–26,31,35 Findings from these systematic reviews were mixed. Three of the previously mentioned 11 studies conducted a meta-analysis: 1 found improvement in patient safety outcomes, 24 1 stated insufficient studies to conclude, 26 and the final paper was equivocal. 10

Therefore, this systematic review serves to provide a cumulative picture of the effects of multiple types of health IT on an array of direct patient safety outcomes in all clinical areas. This is an important time to be studying health IT as adoption rates continue to rise, policymakers continue to support and promote its use, and the determination of how and when to begin regulation of health IT remains under debate. To our knowledge, no prior systematic review has evaluated a comprehensive set of health IT tools while also exclusively focusing on determining the effects of those technologies on direct patient safety outcomes.

MATERIALS AND METHODS

Study Identification and Selection

Health IT was broadly defined as any automated or computerized system implemented to aid in the management of health information. We focused on the following health information technologies: computerized physician order entry (CPOE), e-prescribing, CDS, order entry alerts, EHR, health information exchange (HIE), patient portals, automated error detection software to detect medication errors (AED), electronic medication administration records (eMAR), medication administration barcodes, electronic medication reconciliation software (eMedRec), automated medication dispensing systems (AutoDisp), and electronic clinical pathways. Medication administration barcodes included barcode systems that dispense medication from an automated machine, as well as barcode systems that are used to ensure correct patient identification during the process of medication administration. Automated error detection systems referred to systems that look back to find the orders that may have led to an ADE or a pADE, in contrast to CPOE, which is designed to help aid the provider in correct prescribing at the point of care. We chose these tools through a combination of a priori knowledge of the literature, as well as health IT tools identified as part of the systematic review search process. Other patient-centered interventions such as health IT phone applications or home automated blood pressure cuff monitoring were not actively excluded; however, we did not identify any studies that assessed the impact of these technologies on direct patient outcomes. In cases in which authors did not identify the type of health IT employed using commonly known acronyms or terminology, reviewers used the description of the intervention to determine which type of health IT was being employed.

The authors also identified the clinicians under study. For cases in which the clinicians employing a particular health IT intervention were not identified, the authors reported “NR,” not reported. In cases where a clinician type was not applicable—for example, patient centered tools—those studies were denoted as N/A.

The patient outcomes chosen were identified from the studies included in the review, as well as from author knowledge of outcomes likely to be affected by health IT. After the analysis was completed, outcomes were then grouped on the basis of similar types of outcomes. In the articles for which more than one patient safety outcome was studied, reviewers included in the summary table only the primary outcome numerical effect size. However, for all outcomes, whether or not statistical significance was reached, the positive, negative, or non-significant effect on patient outcomes was considered and recorded ( Table 1 ).

Table 1:

Published Articles Studying Health IT Effects on Patient Outcomes.

Authors (year) Study Period (months) Study Design (Prosp or Retrosp) Quality Score (0–10)  Country Study Site ( n )  Setting Clinician Affected No. Clinicians Health IT Intervention No. Patients  Patient Outcome a Outcome  Effect Size b P -value  Key Findings 
Alvis et al. (1985). 36 NR RCT (Prosp) US Hosp (1) OR MDs – Smart Pump 30 Hemodynamic Stability (hypo- and hypertension) ARR 14% <.05 Patients who had computer-assisted, smartpump-mediated fentanyl infusion had greater hemodynamic stability. Fewer adjuvant drugs were used with smartpump technology than with manual bolus of fentanyl approach. 
Barenfanger, Short, and Groesch (2001). 37 1998–1999 (5) Cohort (Prosp) US Hosp (1) All wards MDs – CDS 378  Mortality a LOS  ARR 1.4% – Decreased mortality and LOS were observed when adjusted for severity of cases when using CDS surrounding antibiotic usage. Costs were also decreased. 
Buising et al. (2008). 38 2000–2007 (84) Time Series (Prosp) Australia Hosp (1) All wards MDs – CDS 189  GN bacteremia a In-hosp mortality LOS  NS – 30-day mortality, rates of GN bacteremia, and LOS remained unchanged when CDS was employed to encourage appropriate antibiotic usage. 
Chapuis et al. (2010). 39  Cohort (Prosp) France Hosp (1) Medical ICU RNs – AutoDisp 115 ADE NS – While automated dispensing system caused fewer medication errors as compared to baseline and concurrent control units, it did not reduce errors causing patient harm. Subjectively, nurses reported improved working conditions with automated dispensing system. 
Chertow et al. (2001). 40 1997–1998 (8) Cohort (Prosp) US Hosp (1) All adult wards MDs – CDS CPOE 17 828 admissions  Renal Impairment a LOS  NS – There was no significant decrease in their primary outcome of the number of patients who experienced a decline in function during hospitalization. However, there was a significant reduction in length of stay during intervention periods with physician alerts to adjust medications based on renal function. 
Cho et al. (2012). 41 2009–2010 (11) Observ (Prosp) Korea Hosp (1) All adult wards MDs – CDS 285 Contrast-induced AKI ARR 7% – Rates of contrast-induced AKI dropped after implementation of a computer alert program reminding providers to order prophylaxis for patients with an elevated GFR undergoing a CT. 
Cook et al. (2011). 42 2005–2009 (60) Observ (Retro) US Hosp (1) All adult wards MDs – CDS EHR CPOE 3612 charts  Nosocomial C. diff a Nosocomial MRSA  NS – Rates of nosocomial C. difficile infections did not change, but rates of MRSA infections decreased significantly after implementation of a EHR with CDS. 
Colpaert et al. (2006). 43 2004 (1.25) RCT (Prosp) Belgium Hosp (1) Surgical ICU IntensivistsResidents 5 3 8 Total CPOE CDS 90  Non-intercept potential ADEs a Rate of Non-intercepted potential ADEs Total ADEs Rate of ADEs  ARR 1.4 rates of Non-intercepted ADEs <.001 The potential ADEs and actual ADEs were reduced in both absolute number and rate in units which CPOE with CDS was used as opposed to paper-based units. CPOE with CDS was found to have the most impact on medication prescribing errors that had the least potential negative patient impact. 
Del Beccaro et al. (2006). 44 2002–2004 (26) Observ (Prosp) US Hosp (1) Pediatric ICU MDs – CPOE 2533 Mortality NS – No significant change in mortality post CPOE implementation in a PICU was found when morality was tracked for 13 months pre- and post- implementation. 
Dowding, Turley, and Garrido. (2012). 45 2003–2009 (84) Time series (Prosp) US Hosp (29) All wards RNs – EHReMARCPOECDS NR  Hospital-acquired pressure ulcers a Fall injuries  NS – EHR implementation with risk assessment tools was associated with decreased rates in HAPU, but not in fall rates, but finding was not significant when controlled for over time. 
Dykes et al. (2010). 46 2009 (6) RCT (Prosp) US Hosp (4) Medicine wards RNs – CDS 10 264 Fall Rates ARR 1.03 falls per 1000 patient days – Use of CDS fall prevention tool was associated with decreased fall rate in intervention units. Patients over 65 years old had greatest decrease in fall rates. There was no significant change in falls resulting in patient injury. 
Etchells et al. (2011). 47 2008 (6) Cohort (Prosp) Canada Hosp (2) Medicine wards MDs – CDS AED 271 Rates AEs NS – Real-time alerting system with simultaneous decision support, but had no significant impact on rates of adverse events. 
Evans et al. (1998). 48 1992–1995 (36) Observ (Prosp) US Hosp (1) ICU MDs – CDS 1681  ADEs a LOS  70% reduction .018 Significant reduction in the rate of ADE from antibiotics was reduced by an anti-infective CDS tool. LOS was also decreased. 
Evans et al. (1995). 49 1994–1995 (19) Observ (Retro) US Hosp (1) Adult ICU MDs – CDS 962  ADEs a NS – Using a CDS system for antibiotic selection did not decrease rates of ADEs. 
Evans et al. (1993). 50 1989–1992 (48) Observ (Retro) US Hosp (1) All adult wards MDs – AED CDS 92 649 ADEs ARR 13.6% <.001 Computerized surveillance of ADEs reduced the number of severe ADEs. Additionally, their surveillance system was used to create computer alerts to pharmacists when physicians prescribed medications to patients with previously known drug allergies, which also significantly reduced the rate of ADEs. 
Fitzmaurice et al. (1996). 51 1993–1994 (12) RCT (Prosp) US Amb (2) Primary Care MDs – CDS 49  Mortality a Thrombotic events Hemorrhagic events  NS – There was no difference in adverse event rates for patients who were randomized to CDS dosing of their warfarin. INR was better controlled in intervention group. 
Frances et al. (2001). 52 1997 (4) RCT (Prosp) US Amb (2) Primary Care MDs 63 CDS 730  Mortality a MI  NS – Rates of MI and mortality were not significantly different for patients whose physicians were randomized to have CDS reminding them to order appropriate post-MI medical management. 
Fiumara et al. (2010). 53 2006–2008 (23) Cohort (Prosp) US Hosp (1) Medical and Surgical wards MDs 425 CDS 880 90-day incidence of VTE NS – CDS to encourage DVT prophylaxis did not decrease rates of VTE at 90-days. 
Fleming et al. (2009). 54 2006–2008 (30) Cohort (Prosp) US Hosp (8) Medical wards MDs – CDS Alg 4454  Inpatient Mortality a 30-day mortality  2.9% ARR <.01 Unadjusted results showed reduction in both in-hospital mortality and 30-day mortality when order set was used. Results adjusted for covariates were not found to be significant with use of CDS. Adjusted results were of borderline significance. 
Gandhi et al. (2005). 55 1999–2000 Cohort (Prosp) US Amb (4) Internal Medicine MDs 34 CPOE 661 ADEs NS – Rates of ADE were the same in the CPOE vs. handwritten medication orders in the ambulatory care setting. Non-significant trend towards increase in ADE was found. 
Glassman et al. (2007). 56 2001–2002 (8) Cohort (Prosp) US Amb (NR) Internal Medicine MDs – AED CPOE 913 ADEs NS – Rates of ADEs were the same in the retrospective error detection system for a CPOE system. The AED focused primarily on drug-drug interactions and drug-disease interactions. 
Graumlich et al. (2009). 57 2004–2007 (39) RCT (Prosp) US Hosp (1) Medicine wards MDs 69 eMedRecCPOE 631  Readmission rate a ED visit rate ADEs post-discharge  NS – Discharge software did not affect readmission rates, ED visit rates or adverse events post-discharge. 
Gurwitz et al. (2008). 58 (12) RCT (Prosp) US and Canada LTC (2) All wards MDs PA NPs 37 CPOE CDS 1118  ADE rate a Preventable ADE rate  NS – CPOE with CDS did not reduce adverse drug event rate or preventable adverse drug event rate. 
Han et al. (2005). 59 2001–2003 (18) Observ (Retro) US Hosp (NR) Pediatric ICU MDs – CPOE 1942 Mortality Neg OR 3.71 – There was an unexpected increase in mortality coincident with CPOE implementation from 13 months before implementation and 5 months after CPOE implementation. 
Holdsworth et al. (2007). 60 2000–2001, 2004 (9+7) Cohort (Prosp) US Hosp (1) Pediatric ICU NR – CPOE CDS Baseline: 1210 post-CPOE: 1197  Rates of ADE a Total ADEs Rates of potential and preventable ADEs  OR 0.76 – There was significant reduction in the total ADEs, preventable ADEs, and potential ADEs after implementation of CPOE system. Sub-group analyses found that there were significant reductions in adverse events associated with certain antibiotic drug classes. LOS remained nearly identical between the ADE and potential ADE groups both before and after CPOE implementation. 
Jani, Barber, and Wong. (2010). 61 2005–2006 (13) Cohort (Prosp) UK Hosp (1) Pediatric wards MDs – CPOE 1590 Med dosing errors varying severity ARR 1% <.001 Electronic prescribing can reduce dosing errors without CDS. 
Jha et al. (2008). 62 2004–2005 (24) Cohort (Prosp) US Hosp (1,672) Medicine wards N/A – CPOE NR  Adjusted 30-day Mortality Rates for: (1) AMI a (2) CHF (3) PNA  ARR 1.5% <.002 Hospitals that had implemented CPOE and participated in reporting were found to have lower rates of adjusted 30-day mortality in both AMI and PNA patients. There was no difference in mortality in CHF patients. 
Jimenez-Munoz et al. (2011). 63 2006–2007 (4) Cohort (Prosp) Spain Hosp (1) Medicine wards MDs RNs – AutoDisp CPOE NR AEs ARR 2.73% <.003 CPOE and automated dispensing systems reduced medication errors affecting patient in two phases drug administering phases: transcription and administration when compared with the paper-based group. Error prevalence rates requiring monitoring or not, were not affected in the prescription phase. 
Keene et al. (2007). 64 1995–1997 (39) Observ (Prosp) US Hosp (1) Pediatric and Neonatal ICUs MDs – CPOE 1291 Mortality NS – Rates of mortality did not change with CPOE implementation. 
King et al. (2003). 65 1993–1996 1997–1999 (72) Cohort (Retro) Canada Hosp (1) Pediatric: Medical and Surgical wards MDs Residents – CPOE 36 103 AEs NS – There was no effect of rate ratios of ADEs in the pre- and post-CPOE implementation periods. There was a significantly decreased rate of potential ADEs in the control wards (with handwritten orders). By contrast, medication error rates were also significantly reduced in the CPOE wards. 
Kucher et al. (2005). 66 2000–2004 (41) RCT (Prosp) US Hosp (1) Medical and Surgical wards MDs 120 CDS 2506  VTE at 90 days a Mortality at 30 and 90 days  3.3% ARR <.001 For patients for whom thromboembolic prophylaxis had not been ordered, a computer generated alert to physicians reduced VTE rates at 90 days with no difference in mortality between groups. 
Kuperman et al. (1999). 67 1994–1995 (4) RCT (Prosp) US Hosp (1) Adult Medical and Surgical wards MDs – CDS NR  Mortality a Arrest Transfer to ICU MI Delirium Stroke Renal Insufficiency AKI Dialysis Return to OR All outcomes  NS – There was no difference in adverse event rates for patients who were randomized to having their physicians receive alerts regarding concerning laboratory values in their medical records. The intervention group had shorter median time intervals before an appropriate treatment was ordered. 
Lecumberri et al. (2008). 68 2005–2007 (18) Observ (Prosp) Spain Hosp (1) Medical and Surgical Wards MDs – CDS 19 338 VTE OR 0.53 – Rates of VTE were not significantly reduced by alerts to physicians overall in the post-intervention period, but in sub-group analysis of surgical patients, a significant reduction in VTE events was identified, which was stable over time. 
Lesourd et al. (2002). 69 2001 (NR) RCT (both Retro and Prosp) France Hosp (1) REI MDs – CDS 53 retrosp. 164 prosp. Pregnancy NS – CDS was as effective as clinicians in using ovarian stimulation with FSH in resulting in pregnancy. 
Linares et al. (2011). 70  Cohort (Prosp) US Hosp (1) Medical wards MDs – CDS 251  Antimicrobial-AE a C.diff from treatment  NNT 10 – Asymptomatic bactiuria and culture-negative pyuria had decreased complications when a computerized alert reminded providers that those U/A and culture results did not require treatment. 
MacIvor et al. (2009). 71 2005–2007 (31) Observ (Retro) US Hosp (16) Medical wards NR N/A HIE 16 Mis-transfusions 38% increase – Centralized patient database detected 38% more ABO typing errors and prevented 6 mis-transfusions 
Madaras-Kelly et al. (2006). 72 2001–2004 (36) Time series (Prosp) US Hosp (1) Medical and Surgical ICU MDs – CDS NR MRSA Infection rates 0.0074% ARR .02 Rate of nosocomial MRSA infections decreased using a CDS intervention to decrease flouroquinolone use. Rate of nosocomial gram-negative organisms significantly increased by 22.66%. Rate of trimethoprim-sulfamethoxazole and piperacillin-tazobactam increased in usage. 
Maynard et al. (2010). 73 2005–2007 (36) Observ (both Retro and Prosp) US Hosp (1) All adult wards except psych and ob/gyn wards MDs – CPOE CDS 2924  Hospital-acquired VTE a health IT PPX-related bleeding  39% RRR – There was a reduction of the rate of HA VTE after introduction of CPOE and CDS. Neither health IT nor prophylaxis related bleeding was increased by this intervention. 
McCowan e al. (2001). 74 (6) RCT (Prosp) UK Amb (17) Primary Care MDs – CDS 477  Acute asthma exacerbations a Hospitalization ED visits  OR 0.43 – A significantly lower number of patients experienced asthma exacerbations if their physicians used CDS. Hospitalizations and ED visits had no effect. 
McMullin et al. (2006). 75 1999 (3) 2001–2002 (12) 2002–2003 (3) Cohort (Prosp) Canada Hosp (1) ICU MDs – CDS Phase 1: 68 Phase 2: 261 Phase 3: 101 VTE rates NS – DVT and pulmonary embolism rates were similar for all phases despite increased compliance with prophylaxis guidelines in phases 2 and 3. Phase 2 was characterized by behavioral approaches plus computerized alerts, while Phase 3 consisted of alerts alone. 
McMullin et al. (1999). 76 1994–1997 (48) Observ (Retro) US Hosp (1) All wards MDs – CDS 286 ADE NS – Using medication alerts did not significantly reduce the rates of adverse events as related to drug-interactions. Rates of dangerous drug combinations and length of time for which those combinations were applied was reduced. 
Menachemi et al. (2007). 77 NR Observ (Retro) US Hosp (98) All wards N/A – No. of IT Applications NR Eight patient safety indicators (PSIs) −1.82 RRR in rates of mortality .024 The greater the number of clinical IT applications adopted by a given hospital, the lower the adverse event rates in three of the eight PSIs: death in low-mortality DRGs, decubitus ulcers, and post-operative sepsis. The other five indicators did not reach statistical significance. 
Milani et al (2011). 78 2009–2010 (24) Cohort (Prosp) US Hosp (1) Medical wards MDs 35 CDS CPOE 47 written orders 33 CPOE  In-hospital bleeding a LOS 90-day mortality  ARR 52% .002 CPOE with decision support reduced hospital bleeding among patients with CKD admitted with ACS. LOS and 90-day mortality were not affected. 
Morriss et al. (2011). 79 2005–2006 (10) Observ (Prosp) US Hosp (1) Neonatal ICU RNs – Barcode 618 Preventable ADE associated with opioid administration 0.48 .045 Barcode medication administration system was found to significantly reduce the risk of preventable ADEs associated with opioid administration. 
Morriss et al. (2009). 80 NR (12.5) Cohort (Prosp) US Hosp (1) Neonatal ICU RNs and RTs – Barcode eMAR 958 Preventable ADEs ARR 3% .04 The barcode medication administration system was found to significantly decrease the risk of preventable ADEs. 
Novis et al. (2010). 81 2007–2008 (12) Observ (Prosp) US Hosp (1) Surgical wards MDs – CDS 800  Post op DVT at 30 days a at 60 days and at 90 days  NS – After implementation of CDS for VTE prophylaxis postoperatively, there was a trend toward decreased rates of post-operative VTEs, which did not reach significance, due to not being powered to do so. Rates of DVT prophylaxis ordering increased. 
Oliven et al. (2002). 82 (6) Cohort (Prosp) Israel Hosp (1) Two internal medicine wards MDs NR CPOE 1350 PEs prevented ARR 31% <.001 Although no the primary outcome, it was found that CPOE significantly reduced the rate of PEs. 
Overhage et al. (2002). 83 1995–1996 (12) RCT (Prosp) US Hosp (2) ED MDs 72 HIE 32 468  Hospital admission a Repeat ED visits  NS – There was no difference in admission rates or repeat ED visits for patients whose ED physicians had access to electronic medical records from another institution. Few of the physicians actually made use of the online system to check OSH records. 
Parente and McCullough. (2009). 84 1999–2002 (48) Observ (Retro) US Hosp (NR) Surgical wards MDs RNs – EHR CDS eMAR National sample Medicare claims data  Infection due to medical care a Post-op hemorrhage Post-op VTE  NR – Rates of hospital-associated infections significantly decreased with EHR use, but none of the other outcomes were reduced by either EHR or the other health IT tools studied. 
Paul et al. (2006). 85 2002–2004 (14) RCT (Prosp) Israel, Germany, and Italy Hosp (3) All wards MDs 199 cohort, NR RCT CDS 350 in cohort study/2326 in RCT  LOS a 30-day mortality  NS – At two of three sites, LOS was significantly reduced by using CDS for antibiotic selection. Overall, there was no significant reduction. 30-day mortality in the intention to treat analysis was not significantly different either at any of the individual sites nor overall. 
Peterson et al. (2005). 86 2001–2002 (6) Cohort (Prosp) US Hosp (1) Medicine and ICU wards MDs – CDS CPOE 3718  Hospital Fall rate a LOSDays of AMS  0.0036 ARR falls per 100 pt days .001 There was a significant reduction of patient in-hospital fall rates after implementation of CDS. No effect was found on hospital length of stay or days of altered mental status. 
Piontek et al. (2010). 87 2001 Observ (Retro) US Hosp (7) All adult wards Pharmacists – AED NR 230 000 admissions  Severity-adjusted Mortality rates a LOS Readmit Rates  decrease (amount NR) <.001 The primary outcome of severity-adjusted mortality rates was significantly lower in hospitals with AED as compared with concurrent control group and pre-intervention group. However, there was no significant difference in LOS or readmission rates. 
Poon et al. (2010). 88 2005 (9) Observ (Prosp) US Hosp (1) All adult wards including ICU RNs – Barcode eMAR 1726 Potential ADEs ARR 1.5% <.001 The rate of potential adverse drug events decreased significantly with use of barcode and eMAR. The rate of potential adverse events associated with timing errors did not change significantly. 
Potts et al. (2004). 89 2001 (2) 2002 (2) Observ (Prosp) US Hosp (1) Pediatric wards MDs – CPOE 514 Potential ADEs ARR 0.9 potential ADEs per 100 order <.001 Rates of potential ADEs were significantly reduced after CPOE implementation. 
Rollman et al. (2002). 90 1997–1998 (20) RCT (Prosp) US Amb (NR) Primary Care MDs 17 CDS EHR 226  Remission of depression at 3 a and 6 months  NS – There was no significant reduction in rates of recovery from depression when CDS was added to an EHR to actively remind physicians to manage patients' MDD. 
Ross et al. (2004). 91 2001–2002 (13) RCT (Prosp) US Amb (1) Cardiac RNs MDs – Patient Portal 107  Mortality a Hospitalizations ED visits  NS – Increased medication adherence approached but did not reach significance in the intervention group using a patient portal. The number of patients who visited the ED was not different, but the number of visits was higher in the intervention group. Mortality and hospitalizations were also not significantly different. 
Rind et al. (1994). 92 1990–1991 (18) Time Series (Prosp) US Hosp (1) All adult wards MDs – CDS 562 Renal impairment RR 0.45 – When computer-based alerts were activated during the intervention periods the rates of serious renal impairment was diminished. 
Rothschildet al. (2005). 93 2002 (11) Time series (Prosp) US Hosp (1) Cardiac Surgical ICU and step-down units RNs – Smart Pump 735  Serious ADEs a Non-intercepted potential ADEs  NS – There was no measurable impact of smart pumps on the serious medication error rate and non-intercepted potential adverse drug events. Authors postulated that lack of compliance played a factor in the lack of effectiveness observed. 
Schnipper et al. (2009). 94 2005–2006 (12) Observ (Prosp) US Hosp (1) 1 Medical ward Pas CDS CPOE 169  Percentage of patient days with Hypoglycemia a LOS  NS – Patient days of hypoglycemia did not change by adding CPOE to the admission order set for diabetic patients. Adjusted LOS did decrease by 25%. 
Shulman et al. (2005). 95 2001–2002 (15) Observ (Prosp) UK Hosp (1) ICU MDs – CPOE 387 ADEs NS .51 Rate of major or moderate errors affecting patients were unchanged in the CPOE group vs the handwritten prescription group. There was a reduction of major/moderate patient outcomes when non-intercepted and intercepted errors were combined (.01). 
Small et al. (2008). 96 2005 (5) Cohort (Prosp) UK Hosp (1) Oncology wards MDs CPOE NR ADEs ARR 8.6% <.0001 CPOE reduced adverse drug events. Error type distribution differed significantly among CPOE orders vs manual orders. CPOE was associated with fewer dose calculation error rates. CPOE decreased the rate of minor errors, with higher proportion of significant and life-threatening errors relative to total errors. 
Smith et al. (2010). 97 2006–2009 (31) RCT (Prosp) UK Amb (29) Primary Care MDs – CDS 911  Exacerbations a Hospitalizations  NS – There were no significant differences in exacerbations with use of CDS. Disaggregation of the composite outcome demonstrated that intervention of alerts was found to reduce the odds of patients requiring hospitalization. Rates of ED visits, after hours contacts with physicians was not significantly different. Rates of prednisolone use were increased by intervention. 
Tierney et al. (2005). 98 1994–1996 (36) RCT (Prosp) US Amb (4) Primary Care MDs and Pharmacists 274 MDs 20 pharmacists CDS 706  ED visits a Hospitalizations  NS – There was no significant difference in ED visits or hospitalizations rates between COPD patients of physicians who had CDS and those who did not. 
Upperman et al. (2005). 99 2002 (9) Cohort (both Retro and Prosp) US Hosp (1) All pediatric inpatients MDs – CPOE NR ADEs NS – Although in aggregate, rates ADEs were not significantly different after CPOE implementation, harmful ADEs were significantly reduced with CPOE usage with a NNT of 1 ADE per 64 patient-days. 
Van Doormaal et al. (2009). 100 2005–2008 (37) Cohort (Prosp) The Netherlands Hosp (2) Two hospital wards per hospital MDs NR CPOECDS NR  pADEs a AEs  NS – CPOE in combination with CDS was not statistically associated with a reduction in preventable ADEs. 
Walsh et al. (2008). 101 2001–2002 (16) Time Series (Prosp) US Hosp (1) All pediatric wards MDs – CPOE NR Serious Medication Errors NS – Overall, rates of non-intercepted serious medication errors were not significantly less. In the NICU and PICU, but not in the general pediatric wards, serious medication errors were significantly less after CPOE implementation. Time series analysis demonstrated that rates of medication errors varied significantly with time of year, irrespective of CPOE implementation, in which months earlier in the academic year yielded high error rates. 
Weingart et al. (2009). 102 2006 (6) Observ (Prosp) US Amb (NR) Adult Primary Care, Pediatric, Psychiatry, and other specialties MDs and physician-extenders 2321 CDS 60 352  pADEs a disabilityhospitalizations ED visits office visits  331 alerts needed to prevent 1 ADE – Alerts were responsible for preventing ADEs as well as hospitalizations, ED visits, and office visits. 
Yu et al. (2009). 103 2005–2006 (12) Case-control (Retro) US Hosp (122) All pediatric wards N/A – CPOE 1 151 932 ADEs OR 1.42 – Hospitals without CPOE had 42 percent higher rates of ADEs. 
Zanetti et al. (2003). 104 2000 (4) RCT (Prosp) US Hosp (1) OR MDs – CDS 449 cases Rate of surgical site infections ARR 2% .4 Surgical site-infection rates were reduced compared with pre-study period, but not significantly reduced compared to concurrent controls when CDS was used to alert physicians to give intraoperative antibiotics. 
Authors (year) Study Period (months) Study Design (Prosp or Retrosp) Quality Score (0–10)  Country Study Site ( n )  Setting Clinician Affected No. Clinicians Health IT Intervention No. Patients  Patient Outcome a Outcome  Effect Size b P -value  Key Findings 
Alvis et al. (1985). 36 NR RCT (Prosp) US Hosp (1) OR MDs – Smart Pump 30 Hemodynamic Stability (hypo- and hypertension) ARR 14% <.05 Patients who had computer-assisted, smartpump-mediated fentanyl infusion had greater hemodynamic stability. Fewer adjuvant drugs were used with smartpump technology than with manual bolus of fentanyl approach. 
Barenfanger, Short, and Groesch (2001). 37 1998–1999 (5) Cohort (Prosp) US Hosp (1) All wards MDs – CDS 378  Mortality a LOS  ARR 1.4% – Decreased mortality and LOS were observed when adjusted for severity of cases when using CDS surrounding antibiotic usage. Costs were also decreased. 
Buising et al. (2008). 38 2000–2007 (84) Time Series (Prosp) Australia Hosp (1) All wards MDs – CDS 189  GN bacteremia a In-hosp mortality LOS  NS – 30-day mortality, rates of GN bacteremia, and LOS remained unchanged when CDS was employed to encourage appropriate antibiotic usage. 
Chapuis et al. (2010). 39  Cohort (Prosp) France Hosp (1) Medical ICU RNs – AutoDisp 115 ADE NS – While automated dispensing system caused fewer medication errors as compared to baseline and concurrent control units, it did not reduce errors causing patient harm. Subjectively, nurses reported improved working conditions with automated dispensing system. 
Chertow et al. (2001). 40 1997–1998 (8) Cohort (Prosp) US Hosp (1) All adult wards MDs – CDS CPOE 17 828 admissions  Renal Impairment a LOS  NS – There was no significant decrease in their primary outcome of the number of patients who experienced a decline in function during hospitalization. However, there was a significant reduction in length of stay during intervention periods with physician alerts to adjust medications based on renal function. 
Cho et al. (2012). 41 2009–2010 (11) Observ (Prosp) Korea Hosp (1) All adult wards MDs – CDS 285 Contrast-induced AKI ARR 7% – Rates of contrast-induced AKI dropped after implementation of a computer alert program reminding providers to order prophylaxis for patients with an elevated GFR undergoing a CT. 
Cook et al. (2011). 42 2005–2009 (60) Observ (Retro) US Hosp (1) All adult wards MDs – CDS EHR CPOE 3612 charts  Nosocomial C. diff a Nosocomial MRSA  NS – Rates of nosocomial C. difficile infections did not change, but rates of MRSA infections decreased significantly after implementation of a EHR with CDS. 
Colpaert et al. (2006). 43 2004 (1.25) RCT (Prosp) Belgium Hosp (1) Surgical ICU IntensivistsResidents 5 3 8 Total CPOE CDS 90  Non-intercept potential ADEs a Rate of Non-intercepted potential ADEs Total ADEs Rate of ADEs  ARR 1.4 rates of Non-intercepted ADEs <.001 The potential ADEs and actual ADEs were reduced in both absolute number and rate in units which CPOE with CDS was used as opposed to paper-based units. CPOE with CDS was found to have the most impact on medication prescribing errors that had the least potential negative patient impact. 
Del Beccaro et al. (2006). 44 2002–2004 (26) Observ (Prosp) US Hosp (1) Pediatric ICU MDs – CPOE 2533 Mortality NS – No significant change in mortality post CPOE implementation in a PICU was found when morality was tracked for 13 months pre- and post- implementation. 
Dowding, Turley, and Garrido. (2012). 45 2003–2009 (84) Time series (Prosp) US Hosp (29) All wards RNs – EHReMARCPOECDS NR  Hospital-acquired pressure ulcers a Fall injuries  NS – EHR implementation with risk assessment tools was associated with decreased rates in HAPU, but not in fall rates, but finding was not significant when controlled for over time. 
Dykes et al. (2010). 46 2009 (6) RCT (Prosp) US Hosp (4) Medicine wards RNs – CDS 10 264 Fall Rates ARR 1.03 falls per 1000 patient days – Use of CDS fall prevention tool was associated with decreased fall rate in intervention units. Patients over 65 years old had greatest decrease in fall rates. There was no significant change in falls resulting in patient injury. 
Etchells et al. (2011). 47 2008 (6) Cohort (Prosp) Canada Hosp (2) Medicine wards MDs – CDS AED 271 Rates AEs NS – Real-time alerting system with simultaneous decision support, but had no significant impact on rates of adverse events. 
Evans et al. (1998). 48 1992–1995 (36) Observ (Prosp) US Hosp (1) ICU MDs – CDS 1681  ADEs a LOS  70% reduction .018 Significant reduction in the rate of ADE from antibiotics was reduced by an anti-infective CDS tool. LOS was also decreased. 
Evans et al. (1995). 49 1994–1995 (19) Observ (Retro) US Hosp (1) Adult ICU MDs – CDS 962  ADEs a NS – Using a CDS system for antibiotic selection did not decrease rates of ADEs. 
Evans et al. (1993). 50 1989–1992 (48) Observ (Retro) US Hosp (1) All adult wards MDs – AED CDS 92 649 ADEs ARR 13.6% <.001 Computerized surveillance of ADEs reduced the number of severe ADEs. Additionally, their surveillance system was used to create computer alerts to pharmacists when physicians prescribed medications to patients with previously known drug allergies, which also significantly reduced the rate of ADEs. 
Fitzmaurice et al. (1996). 51 1993–1994 (12) RCT (Prosp) US Amb (2) Primary Care MDs – CDS 49  Mortality a Thrombotic events Hemorrhagic events  NS – There was no difference in adverse event rates for patients who were randomized to CDS dosing of their warfarin. INR was better controlled in intervention group. 
Frances et al. (2001). 52 1997 (4) RCT (Prosp) US Amb (2) Primary Care MDs 63 CDS 730  Mortality a MI  NS – Rates of MI and mortality were not significantly different for patients whose physicians were randomized to have CDS reminding them to order appropriate post-MI medical management. 
Fiumara et al. (2010). 53 2006–2008 (23) Cohort (Prosp) US Hosp (1) Medical and Surgical wards MDs 425 CDS 880 90-day incidence of VTE NS – CDS to encourage DVT prophylaxis did not decrease rates of VTE at 90-days. 
Fleming et al. (2009). 54 2006–2008 (30) Cohort (Prosp) US Hosp (8) Medical wards MDs – CDS Alg 4454  Inpatient Mortality a 30-day mortality  2.9% ARR <.01 Unadjusted results showed reduction in both in-hospital mortality and 30-day mortality when order set was used. Results adjusted for covariates were not found to be significant with use of CDS. Adjusted results were of borderline significance. 
Gandhi et al. (2005). 55 1999–2000 Cohort (Prosp) US Amb (4) Internal Medicine MDs 34 CPOE 661 ADEs NS – Rates of ADE were the same in the CPOE vs. handwritten medication orders in the ambulatory care setting. Non-significant trend towards increase in ADE was found. 
Glassman et al. (2007). 56 2001–2002 (8) Cohort (Prosp) US Amb (NR) Internal Medicine MDs – AED CPOE 913 ADEs NS – Rates of ADEs were the same in the retrospective error detection system for a CPOE system. The AED focused primarily on drug-drug interactions and drug-disease interactions. 
Graumlich et al. (2009). 57 2004–2007 (39) RCT (Prosp) US Hosp (1) Medicine wards MDs 69 eMedRecCPOE 631  Readmission rate a ED visit rate ADEs post-discharge  NS – Discharge software did not affect readmission rates, ED visit rates or adverse events post-discharge. 
Gurwitz et al. (2008). 58 (12) RCT (Prosp) US and Canada LTC (2) All wards MDs PA NPs 37 CPOE CDS 1118  ADE rate a Preventable ADE rate  NS – CPOE with CDS did not reduce adverse drug event rate or preventable adverse drug event rate. 
Han et al. (2005). 59 2001–2003 (18) Observ (Retro) US Hosp (NR) Pediatric ICU MDs – CPOE 1942 Mortality Neg OR 3.71 – There was an unexpected increase in mortality coincident with CPOE implementation from 13 months before implementation and 5 months after CPOE implementation. 
Holdsworth et al. (2007). 60 2000–2001, 2004 (9+7) Cohort (Prosp) US Hosp (1) Pediatric ICU NR – CPOE CDS Baseline: 1210 post-CPOE: 1197  Rates of ADE a Total ADEs Rates of potential and preventable ADEs  OR 0.76 – There was significant reduction in the total ADEs, preventable ADEs, and potential ADEs after implementation of CPOE system. Sub-group analyses found that there were significant reductions in adverse events associated with certain antibiotic drug classes. LOS remained nearly identical between the ADE and potential ADE groups both before and after CPOE implementation. 
Jani, Barber, and Wong. (2010). 61 2005–2006 (13) Cohort (Prosp) UK Hosp (1) Pediatric wards MDs – CPOE 1590 Med dosing errors varying severity ARR 1% <.001 Electronic prescribing can reduce dosing errors without CDS. 
Jha et al. (2008). 62 2004–2005 (24) Cohort (Prosp) US Hosp (1,672) Medicine wards N/A – CPOE NR  Adjusted 30-day Mortality Rates for: (1) AMI a (2) CHF (3) PNA  ARR 1.5% <.002 Hospitals that had implemented CPOE and participated in reporting were found to have lower rates of adjusted 30-day mortality in both AMI and PNA patients. There was no difference in mortality in CHF patients. 
Jimenez-Munoz et al. (2011). 63 2006–2007 (4) Cohort (Prosp) Spain Hosp (1) Medicine wards MDs RNs – AutoDisp CPOE NR AEs ARR 2.73% <.003 CPOE and automated dispensing systems reduced medication errors affecting patient in two phases drug administering phases: transcription and administration when compared with the paper-based group. Error prevalence rates requiring monitoring or not, were not affected in the prescription phase. 
Keene et al. (2007). 64 1995–1997 (39) Observ (Prosp) US Hosp (1) Pediatric and Neonatal ICUs MDs – CPOE 1291 Mortality NS – Rates of mortality did not change with CPOE implementation. 
King et al. (2003). 65 1993–1996 1997–1999 (72) Cohort (Retro) Canada Hosp (1) Pediatric: Medical and Surgical wards MDs Residents – CPOE 36 103 AEs NS – There was no effect of rate ratios of ADEs in the pre- and post-CPOE implementation periods. There was a significantly decreased rate of potential ADEs in the control wards (with handwritten orders). By contrast, medication error rates were also significantly reduced in the CPOE wards. 
Kucher et al. (2005). 66 2000–2004 (41) RCT (Prosp) US Hosp (1) Medical and Surgical wards MDs 120 CDS 2506  VTE at 90 days a Mortality at 30 and 90 days  3.3% ARR <.001 For patients for whom thromboembolic prophylaxis had not been ordered, a computer generated alert to physicians reduced VTE rates at 90 days with no difference in mortality between groups. 
Kuperman et al. (1999). 67 1994–1995 (4) RCT (Prosp) US Hosp (1) Adult Medical and Surgical wards MDs – CDS NR  Mortality a Arrest Transfer to ICU MI Delirium Stroke Renal Insufficiency AKI Dialysis Return to OR All outcomes  NS – There was no difference in adverse event rates for patients who were randomized to having their physicians receive alerts regarding concerning laboratory values in their medical records. The intervention group had shorter median time intervals before an appropriate treatment was ordered. 
Lecumberri et al. (2008). 68 2005–2007 (18) Observ (Prosp) Spain Hosp (1) Medical and Surgical Wards MDs – CDS 19 338 VTE OR 0.53 – Rates of VTE were not significantly reduced by alerts to physicians overall in the post-intervention period, but in sub-group analysis of surgical patients, a significant reduction in VTE events was identified, which was stable over time. 
Lesourd et al. (2002). 69 2001 (NR) RCT (both Retro and Prosp) France Hosp (1) REI MDs – CDS 53 retrosp. 164 prosp. Pregnancy NS – CDS was as effective as clinicians in using ovarian stimulation with FSH in resulting in pregnancy. 
Linares et al. (2011). 70  Cohort (Prosp) US Hosp (1) Medical wards MDs – CDS 251  Antimicrobial-AE a C.diff from treatment  NNT 10 – Asymptomatic bactiuria and culture-negative pyuria had decreased complications when a computerized alert reminded providers that those U/A and culture results did not require treatment. 
MacIvor et al. (2009). 71 2005–2007 (31) Observ (Retro) US Hosp (16) Medical wards NR N/A HIE 16 Mis-transfusions 38% increase – Centralized patient database detected 38% more ABO typing errors and prevented 6 mis-transfusions 
Madaras-Kelly et al. (2006). 72 2001–2004 (36) Time series (Prosp) US Hosp (1) Medical and Surgical ICU MDs – CDS NR MRSA Infection rates 0.0074% ARR .02 Rate of nosocomial MRSA infections decreased using a CDS intervention to decrease flouroquinolone use. Rate of nosocomial gram-negative organisms significantly increased by 22.66%. Rate of trimethoprim-sulfamethoxazole and piperacillin-tazobactam increased in usage. 
Maynard et al. (2010). 73 2005–2007 (36) Observ (both Retro and Prosp) US Hosp (1) All adult wards except psych and ob/gyn wards MDs – CPOE CDS 2924  Hospital-acquired VTE a health IT PPX-related bleeding  39% RRR – There was a reduction of the rate of HA VTE after introduction of CPOE and CDS. Neither health IT nor prophylaxis related bleeding was increased by this intervention. 
McCowan e al. (2001). 74 (6) RCT (Prosp) UK Amb (17) Primary Care MDs – CDS 477  Acute asthma exacerbations a Hospitalization ED visits  OR 0.43 – A significantly lower number of patients experienced asthma exacerbations if their physicians used CDS. Hospitalizations and ED visits had no effect. 
McMullin et al. (2006). 75 1999 (3) 2001–2002 (12) 2002–2003 (3) Cohort (Prosp) Canada Hosp (1) ICU MDs – CDS Phase 1: 68 Phase 2: 261 Phase 3: 101 VTE rates NS – DVT and pulmonary embolism rates were similar for all phases despite increased compliance with prophylaxis guidelines in phases 2 and 3. Phase 2 was characterized by behavioral approaches plus computerized alerts, while Phase 3 consisted of alerts alone. 
McMullin et al. (1999). 76 1994–1997 (48) Observ (Retro) US Hosp (1) All wards MDs – CDS 286 ADE NS – Using medication alerts did not significantly reduce the rates of adverse events as related to drug-interactions. Rates of dangerous drug combinations and length of time for which those combinations were applied was reduced. 
Menachemi et al. (2007). 77 NR Observ (Retro) US Hosp (98) All wards N/A – No. of IT Applications NR Eight patient safety indicators (PSIs) −1.82 RRR in rates of mortality .024 The greater the number of clinical IT applications adopted by a given hospital, the lower the adverse event rates in three of the eight PSIs: death in low-mortality DRGs, decubitus ulcers, and post-operative sepsis. The other five indicators did not reach statistical significance. 
Milani et al (2011). 78 2009–2010 (24) Cohort (Prosp) US Hosp (1) Medical wards MDs 35 CDS CPOE 47 written orders 33 CPOE  In-hospital bleeding a LOS 90-day mortality  ARR 52% .002 CPOE with decision support reduced hospital bleeding among patients with CKD admitted with ACS. LOS and 90-day mortality were not affected. 
Morriss et al. (2011). 79 2005–2006 (10) Observ (Prosp) US Hosp (1) Neonatal ICU RNs – Barcode 618 Preventable ADE associated with opioid administration 0.48 .045 Barcode medication administration system was found to significantly reduce the risk of preventable ADEs associated with opioid administration. 
Morriss et al. (2009). 80 NR (12.5) Cohort (Prosp) US Hosp (1) Neonatal ICU RNs and RTs – Barcode eMAR 958 Preventable ADEs ARR 3% .04 The barcode medication administration system was found to significantly decrease the risk of preventable ADEs. 
Novis et al. (2010). 81 2007–2008 (12) Observ (Prosp) US Hosp (1) Surgical wards MDs – CDS 800  Post op DVT at 30 days a at 60 days and at 90 days  NS – After implementation of CDS for VTE prophylaxis postoperatively, there was a trend toward decreased rates of post-operative VTEs, which did not reach significance, due to not being powered to do so. Rates of DVT prophylaxis ordering increased. 
Oliven et al. (2002). 82 (6) Cohort (Prosp) Israel Hosp (1) Two internal medicine wards MDs NR CPOE 1350 PEs prevented ARR 31% <.001 Although no the primary outcome, it was found that CPOE significantly reduced the rate of PEs. 
Overhage et al. (2002). 83 1995–1996 (12) RCT (Prosp) US Hosp (2) ED MDs 72 HIE 32 468  Hospital admission a Repeat ED visits  NS – There was no difference in admission rates or repeat ED visits for patients whose ED physicians had access to electronic medical records from another institution. Few of the physicians actually made use of the online system to check OSH records. 
Parente and McCullough. (2009). 84 1999–2002 (48) Observ (Retro) US Hosp (NR) Surgical wards MDs RNs – EHR CDS eMAR National sample Medicare claims data  Infection due to medical care a Post-op hemorrhage Post-op VTE  NR – Rates of hospital-associated infections significantly decreased with EHR use, but none of the other outcomes were reduced by either EHR or the other health IT tools studied. 
Paul et al. (2006). 85 2002–2004 (14) RCT (Prosp) Israel, Germany, and Italy Hosp (3) All wards MDs 199 cohort, NR RCT CDS 350 in cohort study/2326 in RCT  LOS a 30-day mortality  NS – At two of three sites, LOS was significantly reduced by using CDS for antibiotic selection. Overall, there was no significant reduction. 30-day mortality in the intention to treat analysis was not significantly different either at any of the individual sites nor overall. 
Peterson et al. (2005). 86 2001–2002 (6) Cohort (Prosp) US Hosp (1) Medicine and ICU wards MDs – CDS CPOE 3718  Hospital Fall rate a LOSDays of AMS  0.0036 ARR falls per 100 pt days .001 There was a significant reduction of patient in-hospital fall rates after implementation of CDS. No effect was found on hospital length of stay or days of altered mental status. 
Piontek et al. (2010). 87 2001 Observ (Retro) US Hosp (7) All adult wards Pharmacists – AED NR 230 000 admissions  Severity-adjusted Mortality rates a LOS Readmit Rates  decrease (amount NR) <.001 The primary outcome of severity-adjusted mortality rates was significantly lower in hospitals with AED as compared with concurrent control group and pre-intervention group. However, there was no significant difference in LOS or readmission rates. 
Poon et al. (2010). 88 2005 (9) Observ (Prosp) US Hosp (1) All adult wards including ICU RNs – Barcode eMAR 1726 Potential ADEs ARR 1.5% <.001 The rate of potential adverse drug events decreased significantly with use of barcode and eMAR. The rate of potential adverse events associated with timing errors did not change significantly. 
Potts et al. (2004). 89 2001 (2) 2002 (2) Observ (Prosp) US Hosp (1) Pediatric wards MDs – CPOE 514 Potential ADEs ARR 0.9 potential ADEs per 100 order <.001 Rates of potential ADEs were significantly reduced after CPOE implementation. 
Rollman et al. (2002). 90 1997–1998 (20) RCT (Prosp) US Amb (NR) Primary Care MDs 17 CDS EHR 226  Remission of depression at 3 a and 6 months  NS – There was no significant reduction in rates of recovery from depression when CDS was added to an EHR to actively remind physicians to manage patients' MDD. 
Ross et al. (2004). 91 2001–2002 (13) RCT (Prosp) US Amb (1) Cardiac RNs MDs – Patient Portal 107  Mortality a Hospitalizations ED visits  NS – Increased medication adherence approached but did not reach significance in the intervention group using a patient portal. The number of patients who visited the ED was not different, but the number of visits was higher in the intervention group. Mortality and hospitalizations were also not significantly different. 
Rind et al. (1994). 92 1990–1991 (18) Time Series (Prosp) US Hosp (1) All adult wards MDs – CDS 562 Renal impairment RR 0.45 – When computer-based alerts were activated during the intervention periods the rates of serious renal impairment was diminished. 
Rothschildet al. (2005). 93 2002 (11) Time series (Prosp) US Hosp (1) Cardiac Surgical ICU and step-down units RNs – Smart Pump 735  Serious ADEs a Non-intercepted potential ADEs  NS – There was no measurable impact of smart pumps on the serious medication error rate and non-intercepted potential adverse drug events. Authors postulated that lack of compliance played a factor in the lack of effectiveness observed. 
Schnipper et al. (2009). 94 2005–2006 (12) Observ (Prosp) US Hosp (1) 1 Medical ward Pas CDS CPOE 169  Percentage of patient days with Hypoglycemia a LOS  NS – Patient days of hypoglycemia did not change by adding CPOE to the admission order set for diabetic patients. Adjusted LOS did decrease by 25%. 
Shulman et al. (2005). 95 2001–2002 (15) Observ (Prosp) UK Hosp (1) ICU MDs – CPOE 387 ADEs NS .51 Rate of major or moderate errors affecting patients were unchanged in the CPOE group vs the handwritten prescription group. There was a reduction of major/moderate patient outcomes when non-intercepted and intercepted errors were combined (.01). 
Small et al. (2008). 96 2005 (5) Cohort (Prosp) UK Hosp (1) Oncology wards MDs CPOE NR ADEs ARR 8.6% <.0001 CPOE reduced adverse drug events. Error type distribution differed significantly among CPOE orders vs manual orders. CPOE was associated with fewer dose calculation error rates. CPOE decreased the rate of minor errors, with higher proportion of significant and life-threatening errors relative to total errors. 
Smith et al. (2010). 97 2006–2009 (31) RCT (Prosp) UK Amb (29) Primary Care MDs – CDS 911  Exacerbations a Hospitalizations  NS – There were no significant differences in exacerbations with use of CDS. Disaggregation of the composite outcome demonstrated that intervention of alerts was found to reduce the odds of patients requiring hospitalization. Rates of ED visits, after hours contacts with physicians was not significantly different. Rates of prednisolone use were increased by intervention. 
Tierney et al. (2005). 98 1994–1996 (36) RCT (Prosp) US Amb (4) Primary Care MDs and Pharmacists 274 MDs 20 pharmacists CDS 706  ED visits a Hospitalizations  NS – There was no significant difference in ED visits or hospitalizations rates between COPD patients of physicians who had CDS and those who did not. 
Upperman et al. (2005). 99 2002 (9) Cohort (both Retro and Prosp) US Hosp (1) All pediatric inpatients MDs – CPOE NR ADEs NS – Although in aggregate, rates ADEs were not significantly different after CPOE implementation, harmful ADEs were significantly reduced with CPOE usage with a NNT of 1 ADE per 64 patient-days. 
Van Doormaal et al. (2009). 100 2005–2008 (37) Cohort (Prosp) The Netherlands Hosp (2) Two hospital wards per hospital MDs NR CPOECDS NR  pADEs a AEs  NS – CPOE in combination with CDS was not statistically associated with a reduction in preventable ADEs. 
Walsh et al. (2008). 101 2001–2002 (16) Time Series (Prosp) US Hosp (1) All pediatric wards MDs – CPOE NR Serious Medication Errors NS – Overall, rates of non-intercepted serious medication errors were not significantly less. In the NICU and PICU, but not in the general pediatric wards, serious medication errors were significantly less after CPOE implementation. Time series analysis demonstrated that rates of medication errors varied significantly with time of year, irrespective of CPOE implementation, in which months earlier in the academic year yielded high error rates. 
Weingart et al. (2009). 102 2006 (6) Observ (Prosp) US Amb (NR) Adult Primary Care, Pediatric, Psychiatry, and other specialties MDs and physician-extenders 2321 CDS 60 352  pADEs a disabilityhospitalizations ED visits office visits  331 alerts needed to prevent 1 ADE – Alerts were responsible for preventing ADEs as well as hospitalizations, ED visits, and office visits. 
Yu et al. (2009). 103 2005–2006 (12) Case-control (Retro) US Hosp (122) All pediatric wards N/A – CPOE 1 151 932 ADEs OR 1.42 – Hospitals without CPOE had 42 percent higher rates of ADEs. 
Zanetti et al. (2003). 104 2000 (4) RCT (Prosp) US Hosp (1) OR MDs – CDS 449 cases Rate of surgical site infections ARR 2% .4 Surgical site-infection rates were reduced compared with pre-study period, but not significantly reduced compared to concurrent controls when CDS was used to alert physicians to give intraoperative antibiotics. 

a Primary outcomes noted with asterisks. b All effect sizes are reported for the primary outcome listed.

Abbreviations: Prosp = prospective; Retrosp or Retro = retrospective; Health IT = Health information technology; Conf Int = confidence interval; Observ = observational study; RCT = randomized control trial; Hosp = hospital setting; Amb = ambulatory care setting; NR = not reported; NS = not significant primary outcome; LTC = long term care facility; ICU = intensive care unit; REI = reproductive endocrinology and infertility; ED = emergency department; OR = operating room or odds ratio; MDs = physicians; RNs = nurses; PA = physicians’ assistants; NPs = nurse practitioners; N/A = not applicable; CDS = clinical decision support; CPOE = computerized provider order entry; EHR = electronic health record; HIE = Health information exchange; AutoDisp = automated dispensation of medication; eMAR = electronic medication administration record; AED = automated error detection system; eMedRec = electronic medication reconciliation; Alg = electronic clinical pathway; GN = gram negative; LOS = length of stay; C. diff =  Clostridium difficile ; MRSA = Methicillin-resistant Staphylococcus aureus ; ADE = adverse drug event; AE = adverse event; pADE = preventable adverse drug event; AKI = acute kidney injury; AMI or MI = acute myocardial infarction; CHF = congestive heart failure; PNA = pneumonia; VTE = venous thromboembolism; PPX = prophylaxis; DVT = deep venous thrombosis; PE = pulmonary embolism; post-op = post-operatively; AMS = altered mental status; M = not significant or mixed results study; P = positive study; Neg = negative study (health IT was found to be harmful); COPD = chronic obstruction pulmonary disease; ARR = absolute risk reduction; NNT = number needed to treat.

We performed searches in bibliographic databases, Ovid Medline, Ovid EMBASE, the Cumulative Index to Nursing Allied Health (CINAHL) via Ebscohost, and Cochrane Library from January 2001 to June 2012. Conference proceedings were reviewed as well as bibliographies of selected articles. Citations of all identified prior systematic reviews were also reviewed. The search strategy included combinations of keywords and controlled vocabulary. A validated filter to represent patient safety was applied. 105 Appendix A illustrates the detailed search strategy for the four databases .

All citations, index terms, and abstracts (if available) were reviewed and rated as “potentially relevant” or “not relevant.” In accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for reporting of systematic reviews, one reviewer reviewed the entire set first of titles, followed by the abstracts. Articles that were potentially relevant were included in the set reviewed by 2 independent reviewers (see Figure 1 ). 106 Articles were reviewed independently, and studies were included in the review if 1) the study participants were health professionals in clinical practice or postgraduate training, 2) the intervention was health IT studied in a clinical setting, and 3) the outcomes (even if secondary and not primary) that were assessed included at least one direct patient safety outcome (including any aspect of patient well-being, with process measures considered insufficient). Only English-language studies were included. All disagreements were resolved by consensus.

Figure 1:

Study identification and selection.

Figure 1:

Study identification and selection.

Study Evaluation

Two authors independently assessed all selected studies for methodological quality. A previously described 10-point Methodological Quality Assessment was adapted to the purposes of this study. 9,11,17 This methodological rating scale assesses for 5 potential sources of bias, each scored either 0, 1, or 2, including (A) the method of allocation to study groups (random vs selected concurrent controls vs non-concurrent controls), (B) the unit of allocation (ward or clinic vs physician vs patient), (C) baseline differences between groups which could potentially be linked to the study outcome (no baseline differences and/or appropriate statistical adjustments made for differences vs baseline differences apparent without statistical adjustment vs unable to assess), (D) the type of safety outcome measure (objective outcome or subjective outcome with blinded assessment vs objective outcome with no blinding vs subjective outcome without blinding of assessors), and (E) completeness of follow-up (>90% vs 80%-90% vs <80% and/or unable to assess). 11 As such, a score of 10 represents studies whose design had the lowest amount of bias ( Table 2 ). Disagreements were resolved by discussion to reach consensus. Reviewer agreement and inter-rater reliability was analyzed by the kappa statistical method. Since one reviewer reviewed all of the articles, and multiple reviewers were paired with the principal reviewer, a quadratic-weighted kappa was chosen. 107

Table 2:

Study Quality Rating Scale

Potential Source of Bias  Score
 
Allocation Bias Randomized Quasi-randomized Concurrent controls 
Unit of Allocation Bias Cluster-analysis (i.e.,: practice or ward) Physician-based analysis Patient-level analysis 
Baseline Group Characteristics No baseline differences or appropriate statistical adjustments for differences Baseline differences present with no statistical adjustments Baseline differences not reported 
Objectivity of Outcome Objective outcomes with blinded assessment Objective outcomes without blinding Subjective outcomes with no blinding and poorly defined 
Completeness of Follow-up >90% 80–90% <80% or not described 
Potential Source of Bias  Score
 
Allocation Bias Randomized Quasi-randomized Concurrent controls 
Unit of Allocation Bias Cluster-analysis (i.e.,: practice or ward) Physician-based analysis Patient-level analysis 
Baseline Group Characteristics No baseline differences or appropriate statistical adjustments for differences Baseline differences present with no statistical adjustments Baseline differences not reported 
Objectivity of Outcome Objective outcomes with blinded assessment Objective outcomes without blinding Subjective outcomes with no blinding and poorly defined 
Completeness of Follow-up >90% 80–90% <80% or not described 

Adopting the methodology employed by a prior systematic review (Chaudhry et al.) 4 , quantitative reports were considered “hypothesis-testing” if the investigators compared data between groups or across time periods, using statistical tests to assess differences. We further categorized hypothesis-testing studies into 5 study types. RCTs were defined as studies that had a control and experimental arm for which the intervention (health IT) was randomly assigned. Cohort trials were defined as non-randomized studies for which a concurrent control arm was included. Observational studies were most often before-and-after studies in which the “before” group served as the only control. Time series analyses were studies for which time-series statistical analyses were conducted. Lastly, case-control studies were studies for which cases and controls were picked retrospectively, based on exposure to health IT. 4

Data Extraction and Analysis

For each article included, both reviewers extracted information regarding patients, clinicians involved, setting, intervention, and outcomes for each of the studies. The safety outcomes evaluated were categorized into the following groups: 1) ADEs or adverse events (AEs); 2) mortality; 3) thrombosis or bleed; 4) length of stay (LOS); 5) infection rates; 6) readmission, admission, or emergency department (ED) visits; 7) fall rates or pressure ulcer; 8) hemodynamic instability or intensive care unit (ICU) transfer; 9) myocardial infarction (MI) or cardiac events; 10) chronic disease exacerbations; and 11) altered mental status (AMS) or stroke incidence.

Adapting methodology used in prior reviews, positive studies were those in which the primary outcome studied showed statistically significant improvement. Negative studies were those for which there were statistically significant worse patient safety outcomes. Mixed or non-significant studies were those for which the primary outcome had a non-significant result but secondary outcomes had a positive result, or studies in which all outcomes were non-significant. 15 Studies for which patient safety outcomes were not the primary outcome studied were included and the non-patient safety outcome endpoints were not analyzed. Consensus was reached during review discussions. A narrative synthesis method was used to integrate the findings into descriptive summaries. Sub-analyses of positive studies, mixed studies, and randomized controlled trials were conducted.

RESULTS

The search strategy identified 6138 articles. After removal of duplicate articles and articles available only in a foreign language, there were 4736 articles that underwent title review. Based on title alone, 817 (17%) were considered not appropriate for the study. Another 344 articles were added based on a review of the references of the systematic reviews found during the title review process. A total of 4263 articles then underwent abstract review, with 135 articles included for full two-person review. Sixty-eight articles met all of the study inclusion criteria ( Figure 1 ).

Reviewer Agreement

Of the 135 papers reviewed by 2 reviewers, agreement about eligibility for inclusion in the systematic review was excellent 90.4% ( k  = 80.9%; 95% CI, 71.0-90.8%). Of the 69 studies included in the final review, the level of chance-corrected agreement for scientific merit between reviewers was excellent, with a quadratic-weighted k statistic of 88.9% (95% CI, 84.6-93.3%).

Descriptive Analysis of All Studies

Types of health IT and outcomes studied

There was at least one article for every type of health IT pre-identified. More than one health IT was analyzed in 22 studies (31%), and in those cases, all of the health IT tools studied were included in the analysis. The most common health IT interventions were CDS ( n  = 40) and CPOE ( n  = 27) ( Table 3 ). Four health IT tools (electronic medication reconciliation, electronic clinical pathways, patient portal, and smart pumps) were included in only one study, and another 2 tools (HIE and automated medication dispensing) were only found in 2 studies each.

Table 3:

Summary of Findings

Characteristic Total (%) Positive Studies Non-significant or Mixed Results Studies Negative Studies 
Total (%) 69 (100) 25 (36) 43 (62) 1 (1) 
Study Design 
 Randomized Control Trial 18 (26) 5 (7) 13 (19) – 
 Cohort 21 (30) 8 (12) 13 (19) – 
 Observational 22 (31) 10 (14) 12 (17) 1 (1) 
 Time Series 7 (10) 2 (3) 5 (7) – 
 Case-Control 1 (1) 1 (1) – – 
Setting 
 Inpatient 59 (86) 25 (36) 33 (48) 1 (1) 
 Outpatient 10 (14) 1 (1) 9 (13) – 
 Long-Term Care 1 (1) – 1 (1) – 
 Multi-Center 19 (28) 4 (6) 15 (22) – 
Clinicians Affected 
 Physicians 55 (80) 19 (28) 36 (52) 1 (1) 
 Nurses 10 (14) 4 (6) 6 (9) – 
 Other 5 (7) 2 (3) 3 (4) – 
 Pharmacists 2 (3) 1 (1) 1 (1) – 
Country 
 United States 51 (75) 19 (28) 31 (46) 1 (1) 
 Non-United States 19 (28) 7 (10) 12 (17) – 
Methodological Quality Assessment Score 
 0–3 20 (29) 10 (14) 10 (14) – 
 4–6 34 (49) 11 (16) 22 (32) 1 (1) 
 7–10 15 (22) 4 (6) 11 (16) – 
Type of Health IT Intervention Studied 
 Clinical decision support (CDS) 40 (58) 15 (22) 25 (36) – 
 Computerized provider order entry (CPOE) 27 (39) 10 (14) 16 (23) 1 (1) 
 Automated error detection 4 (6) 2 (3) 2 (3) – 
 Electronic medication administration record (eMAR) 4 (6) 1 (1) 3 (4) – 
 Electronic health record (EHR) 4 (6) – 4 (6) – 
 Med Administration Barcodes 3 (4) 2 (3) 1 (1) – 
 Health information exchange (HIE) 2 (3) 1 (1) 1 (1) – 
 Automated dispensing 2 (3) 1 (1) 1 (1) – 
 Electronic medication reconciliation 1 (1) – 1 (1) – 
 Electronic Clinical Pathways 1 (1) – 1 (1) – 
 Patient Portal 1 (1) – 1 (1) – 
 Smart pumps 1 (1) 1 (1) 1 (1) – 
 No. of IT Applications 1 (1) – 1 (1) – 
Patient Outcomes Studied 
 Adverse Drug Events or Adverse Events 37 (53) 17 (25) 20 (29) – 
 Mortality 18 (26) 5 (7) 12 (18) 1 (1) 
 Readmission, admission, or Emergency dept. visits 16 (24) 4 (6) 12 (18) – 
 Thrombosis or Bleed 10 (14) 3 (4) 7 (10) – 
 Length of Stay 8 (12) 4 (6) 4 (6) – 
 Infection Rates 8 (10) 5 (7) 3 (4) – 
 Fall Rates 3 (4) 2 (3) 1 (1) – 
 Hemodynamic Instability or ICU transfer 3 (4) 1 (1) 2 (3) – 
 Myocardial Infarction or Cardiac Events 3 (4) – 3 (4) – 
 Chronic Disease Exacerbations 3 (4) 1 (1) 2 (3) – 
 Altered Mental Status or Stroke incidence 2 (3) – 2 (3) – 
 Pressure Ulcers 1 (1) – 1 (1)  
Characteristic Total (%) Positive Studies Non-significant or Mixed Results Studies Negative Studies 
Total (%) 69 (100) 25 (36) 43 (62) 1 (1) 
Study Design 
 Randomized Control Trial 18 (26) 5 (7) 13 (19) – 
 Cohort 21 (30) 8 (12) 13 (19) – 
 Observational 22 (31) 10 (14) 12 (17) 1 (1) 
 Time Series 7 (10) 2 (3) 5 (7) – 
 Case-Control 1 (1) 1 (1) – – 
Setting 
 Inpatient 59 (86) 25 (36) 33 (48) 1 (1) 
 Outpatient 10 (14) 1 (1) 9 (13) – 
 Long-Term Care 1 (1) – 1 (1) – 
 Multi-Center 19 (28) 4 (6) 15 (22) – 
Clinicians Affected 
 Physicians 55 (80) 19 (28) 36 (52) 1 (1) 
 Nurses 10 (14) 4 (6) 6 (9) – 
 Other 5 (7) 2 (3) 3 (4) – 
 Pharmacists 2 (3) 1 (1) 1 (1) – 
Country 
 United States 51 (75) 19 (28) 31 (46) 1 (1) 
 Non-United States 19 (28) 7 (10) 12 (17) – 
Methodological Quality Assessment Score 
 0–3 20 (29) 10 (14) 10 (14) – 
 4–6 34 (49) 11 (16) 22 (32) 1 (1) 
 7–10 15 (22) 4 (6) 11 (16) – 
Type of Health IT Intervention Studied 
 Clinical decision support (CDS) 40 (58) 15 (22) 25 (36) – 
 Computerized provider order entry (CPOE) 27 (39) 10 (14) 16 (23) 1 (1) 
 Automated error detection 4 (6) 2 (3) 2 (3) – 
 Electronic medication administration record (eMAR) 4 (6) 1 (1) 3 (4) – 
 Electronic health record (EHR) 4 (6) – 4 (6) – 
 Med Administration Barcodes 3 (4) 2 (3) 1 (1) – 
 Health information exchange (HIE) 2 (3) 1 (1) 1 (1) – 
 Automated dispensing 2 (3) 1 (1) 1 (1) – 
 Electronic medication reconciliation 1 (1) – 1 (1) – 
 Electronic Clinical Pathways 1 (1) – 1 (1) – 
 Patient Portal 1 (1) – 1 (1) – 
 Smart pumps 1 (1) 1 (1) 1 (1) – 
 No. of IT Applications 1 (1) – 1 (1) – 
Patient Outcomes Studied 
 Adverse Drug Events or Adverse Events 37 (53) 17 (25) 20 (29) – 
 Mortality 18 (26) 5 (7) 12 (18) 1 (1) 
 Readmission, admission, or Emergency dept. visits 16 (24) 4 (6) 12 (18) – 
 Thrombosis or Bleed 10 (14) 3 (4) 7 (10) – 
 Length of Stay 8 (12) 4 (6) 4 (6) – 
 Infection Rates 8 (10) 5 (7) 3 (4) – 
 Fall Rates 3 (4) 2 (3) 1 (1) – 
 Hemodynamic Instability or ICU transfer 3 (4) 1 (1) 2 (3) – 
 Myocardial Infarction or Cardiac Events 3 (4) – 3 (4) – 
 Chronic Disease Exacerbations 3 (4) 1 (1) 2 (3) – 
 Altered Mental Status or Stroke incidence 2 (3) – 2 (3) – 
 Pressure Ulcers 1 (1) – 1 (1)  

Note: Studies can be counted in more than one category where applicable. Abbreviations: dept. = department; ICU = Intensive care unit. Other category under clinicians refers to either not reported or not applicable study population.

The patient safety outcomes studied varied widely. The most common outcomes studied included: ADEs and adverse events (53% of studies), mortality (26%), thrombosis or bleed (14%), LOS (12%), and infection rates (10%). Secondary outcomes were included in the analysis to capture the broadest number of patient outcomes ( Table 3 ).

Study setting and participants

Most of the studies ( n  = 59, 86%) were performed in inpatient settings. A multicenter study design was employed in 19 (28%) of the studies with the majority (15 of these 19, 79%) of the multicenter trials resulting in non-significant clinical outcomes. Multicenter design was used in 75% (6 of 8) of outpatient studies as compared to only 22% (12 of 55) of inpatient studies. The vast majority (80%) of studies assessed physicians, rather than other healthcare practitioners and most studies were conducted in the United States (75%). There were studies for which authors did not specify the clinicians affected by their health IT, nor was it clear that clinicians were a subject under study from the text. In these cases, reviewers classified the clinicians as “NR,” for not reported ( Table 1 ).

Study quality

The study designs were roughly evenly distributed between RCTs, cohort, and observational design studies, with only a few time-series and case-control designed studies. Study quality was approximately evenly distributed across each grouping of ratings (0–3; 4–6; 7–10) ( Table 3 ). Unlike prior studies, we did not find that there was a significant increase in the quality of studies over time. 11,17 In terms of quality assessment, the weakest aspects of study design tended to be with regard to randomization and allocation. Specifically, the majority of studies failed to have randomization or even a concurrent control group as part of the study design, and most allocation was done at the patient, rather than unit level. Eighty-one percent of studies received a 1 for blinding of outcomes (which meant objective outcomes were assessed without blinding), and 61% of studies received a 2 for follow up (indicating >90% follow up achieved and reported). Reporting of baseline characteristics was variable and evenly distributed between scores of 0, 1, and 2. Notably, this pattern for quality assessment held true for all studies as well as in sub-analysis of positive vs mixed and negative studies Only 10 (24%) of the non-significant studies enrolled over 1000 patients whereas, 11 (44%) of the studies which found a positive effect of health IT on patient outcomes had enrolled more than 1000 patients ( Table 4 ). Larger studies ( n  > 1000 patients) were also more likely to be conducted more recently than smaller studies.

Table 4:

Analysis of Studies Categorized as Mixed Results Studies

Characteristic Total Mixed Results Studies (%) Non-significant Studies (%) Mixed Studies: some positive results, some non-significant results (%) 
Total (%) 43 (100) 24 (56) 19 (44) 
Study Design 
 Randomized Control Trial 13 (30) 9 (21) 4 (9) 
 Cohort 16 (37) 8 (19) 8 (19) 
 Observational 14 (33) 7 (16) 7 (16) 
Setting 
 Inpatient 23 (53) 16 (37) 17 (40) 
 Outpatient 9 (21) 7 (16) 2 (5) 
 Long-Term Care 1 (2) 1 (2) – 
 Multi-Center 15 (35) 9 (21) 6 (14) 
Methodological Quality Assessment Score 
 0–3 10 (23) 6 (14) 4 (9) 
 4–6 22 (51) 12 (28) 10 (23) 
 7–10 11 (26) 6 (14) 5 (12) 
Type of Health IT Intervention Studied 
 Clinical decision support (CDS) 26 (60) 15 (36) 11 (26) 
 Computerized provider order entry (CPOE) 17 (40) 7 (16) 11 (26) 
 Automated error detection 2 (5) 2 (5) – 
 Electronic medication administration record (eMAR) 3 (7) 1 (2) 2 (5) 
 Electronic health record (EHR) 3 (7) 2 (5) 1 (2) 
 Medication Administration Barcode 1 (2) – 1 (2) 
 Health information exchange (HIE) 1 (2) 1 (2) – 
 Automated dispensing 1 (2) 1 (2) – 
 Electronic medication reconciliation 1 (2) 1 (2) – 
 Patient Portal 1 (2) 1 (2) – 
 Smart pumps 1 (2) 1 (2) – 
 No. of IT Applications 1 (2) – 1 (2) 
Patient Outcomes Studied 
 Adverse Drug Events or Adverse Events 18 (42) 9 (21) 9 (21) 
 Readmission, admission, or Emergency dept. visits 12 (28) 9 (21) 3 (5) 
 Mortality 10 (23) 7 (17) 3 (5) 
 Thrombosis or Bleed 9 (21) 6 (14) 4 (9) 
 Length of Stay 4 (9) – 4 (9) 
 Infection Rates 4 (9) 1 (2) 3 (5) 
 Myocardial Infarction or Cardiac Events 3 (7) 2 (5) 1 (2) 
 Hemodynamic Instability or ICU transfer 2 (5) 2 (5) – 
 Chronic Disease Exacerbations 2 (5) – 2 (5) 
 Fall Rates 1 (2) 1 (2) – 
 Pressure Ulcers 1 (2) 1 (2)  
Characteristic Total Mixed Results Studies (%) Non-significant Studies (%) Mixed Studies: some positive results, some non-significant results (%) 
Total (%) 43 (100) 24 (56) 19 (44) 
Study Design 
 Randomized Control Trial 13 (30) 9 (21) 4 (9) 
 Cohort 16 (37) 8 (19) 8 (19) 
 Observational 14 (33) 7 (16) 7 (16) 
Setting 
 Inpatient 23 (53) 16 (37) 17 (40) 
 Outpatient 9 (21) 7 (16) 2 (5) 
 Long-Term Care 1 (2) 1 (2) – 
 Multi-Center 15 (35) 9 (21) 6 (14) 
Methodological Quality Assessment Score 
 0–3 10 (23) 6 (14) 4 (9) 
 4–6 22 (51) 12 (28) 10 (23) 
 7–10 11 (26) 6 (14) 5 (12) 
Type of Health IT Intervention Studied 
 Clinical decision support (CDS) 26 (60) 15 (36) 11 (26) 
 Computerized provider order entry (CPOE) 17 (40) 7 (16) 11 (26) 
 Automated error detection 2 (5) 2 (5) – 
 Electronic medication administration record (eMAR) 3 (7) 1 (2) 2 (5) 
 Electronic health record (EHR) 3 (7) 2 (5) 1 (2) 
 Medication Administration Barcode 1 (2) – 1 (2) 
 Health information exchange (HIE) 1 (2) 1 (2) – 
 Automated dispensing 1 (2) 1 (2) – 
 Electronic medication reconciliation 1 (2) 1 (2) – 
 Patient Portal 1 (2) 1 (2) – 
 Smart pumps 1 (2) 1 (2) – 
 No. of IT Applications 1 (2) – 1 (2) 
Patient Outcomes Studied 
 Adverse Drug Events or Adverse Events 18 (42) 9 (21) 9 (21) 
 Readmission, admission, or Emergency dept. visits 12 (28) 9 (21) 3 (5) 
 Mortality 10 (23) 7 (17) 3 (5) 
 Thrombosis or Bleed 9 (21) 6 (14) 4 (9) 
 Length of Stay 4 (9) – 4 (9) 
 Infection Rates 4 (9) 1 (2) 3 (5) 
 Myocardial Infarction or Cardiac Events 3 (7) 2 (5) 1 (2) 
 Hemodynamic Instability or ICU transfer 2 (5) 2 (5) – 
 Chronic Disease Exacerbations 2 (5) – 2 (5) 
 Fall Rates 1 (2) 1 (2) – 
 Pressure Ulcers 1 (2) 1 (2)  

Abbreviations: dept. = department; ICU = Intensive care unit.

Effects of health IT on patient safety outcomes

Of the 69 studies, the majority ( n  = 43, 63%) had either non-significant findings with respect to patient safety outcomes, or mixed outcomes. Only 25 (36%) studies showed a statistically significant positive effect of health IT on the primary patient safety outcome assessed. There was also 1 (1%) study that found that health IT resulted in an increased mortality rate. There was a significant increase in the number of studies published on health IT and patient safety outcomes over time ( Figure 2 ).

Figure 2:

Number of studies published as a function of publication year. Even though the number of studies published on health information technology (IT) has increased significantly, the number of randomized controlled trials (RTC) published annually has only seen a modest increase over time.

Figure 2:

Number of studies published as a function of publication year. Even though the number of studies published on health information technology (IT) has increased significantly, the number of randomized controlled trials (RTC) published annually has only seen a modest increase over time.

Analysis of Positive Studies

The 25 studies that found that health IT had a positive effect on the primary patient safety outcome were mostly observational trials (40%) or cohort trials (30%). The majority of the positive studies were single center trials ( n  = 20), conducted in the United States ( n  = 19).

The vast majority of studies that found a positive effect of health IT occurred in the inpatient setting ( n  = 24, 96%). There was only one trial demonstrating a positive effect of health IT on patient safety outcomes in the outpatient setting, and none in the long-term care setting. There was no significant difference in the sample sizes or quality score of the positive studies as compared the mixed result or null studies ( Table 3 ).

Positive benefit on patient safety outcomes was demonstrated in studies evaluating CDS, CPOE, HIE, automated error detection, eMAR, medication administration barcodes, automated dispensing, and smart pumps. The health outcomes involved were adverse events ( n  = 16 studies), mortality ( n  = 4), LOS ( n  = 4), readmission rates or ED visits ( n  = 2), prevention or reduction of thrombosis or bleeding ( n  = 2), infection rates ( n  = 2), and rates of pressure ulcers or falls ( n  = 2), AMS or stroke incidence ( n  = 1), and hemodynamic instability or ICU admission ( n  = 1). The patient safety outcomes for which there were more positive studies than mixed studies were LOS, renal impairment, and fall or pressure ulcer rates ( Table 3 ).

In conducting further sub-analysis of the studies characterized as mixed results studies, it was found that more than half of those studies had non-significant findings with respect to all patient safety outcomes. The remaining studies categorized as mixed results had some secondary patient safety outcomes that were positive ( Table 4 ).

In order to determine which types of outcomes were positively affected by which types of health IT, the effective combinations of the two were analyzed. Overall, CDS, CPOE, or CPOE combined with CDS accounted for 73% of the interventions that were successful. The only health outcomes for which those health IT interventions did not constitute the majority was for infection rates, pressure ulcers, or hemodynamic instability or transfer to the ICU ( Table 5 ).

Table 5:

Analysis of the Health IT Found to be Effective in Improving Specific Patient Safety Outcomes

Total studies = 25 + 18 = 43  Patient Outcome
 
 
Type of Health IT Intervention ADEs or AEs Mortality Length of Stay Thrombosis or Bleed Infection Rates Readm, adm, or ED visits Hemodynamic Instability or ICU transfer Fall Rates Chronic Disease Exacerbations AMS or CVA Pressure Ulcers Total 
CDS    21 
CDS/CPOE        10 
CPOE         10 
AED         
No. of IT Applications         
Medication Administr-ation Barcode/eMAR          
Smart Pumps           
EHR           
Barcode           
HIE           
AED/CDS           
CDS/EHR/CPOE           
Total 21  
Total studies = 25 + 18 = 43  Patient Outcome
 
 
Type of Health IT Intervention ADEs or AEs Mortality Length of Stay Thrombosis or Bleed Infection Rates Readm, adm, or ED visits Hemodynamic Instability or ICU transfer Fall Rates Chronic Disease Exacerbations AMS or CVA Pressure Ulcers Total 
CDS    21 
CDS/CPOE        10 
CPOE         10 
AED         
No. of IT Applications         
Medication Administr-ation Barcode/eMAR          
Smart Pumps           
EHR           
Barcode           
HIE           
AED/CDS           
CDS/EHR/CPOE           
Total 21  

Abbreviations: ADE = adverse drug event; AE = adverse event; ED = emergency department; Readm = readmission; adm = hospital admission; ICU = intensive care unit; AMS = altered mental status; CVA = stroke; CDS = clinical decision support; CPOE = computerized provider order entry; EHR = electronic health record; HIE = Health information exchange; AutoDisp = automated dispensation of medication; eMAR = electronic medication administration record; AED = automated error detection system.

Subgroup Analysis: RCTs Only

Of all the study types, RCTs had the smallest percentage of studies demonstrating positive effect of health IT on safety outcomes ( n  = 5, 28%), as compared with all other studies ( n  = 20, 40%). Again, as for the entire group of studies, inpatient studies, physician studies, and US studies were all more common among RCTs ( Table 6 ). There was a much smaller increase in the number of RCT studies published over time, as compared to all studies ( Figure 2 ).

Table 6:

RCT Trials

Characteristic Total (%) Positive Studies Non-significant or Mixed Results Studies 
Total (%) 18 (100) 5 (28) 13 (72) 
Setting    
 Inpatient 10 (56) 5 (28) 5 (28) 
 Outpatient 7 (39) – 7 (39) 
 Long-Term Care 1 (6) – 1(6) 
 Multi-Center 8 (44) 1 (6) 7 (39) 
Clinicians Affected    
 Physicians 17 (94) 4 (22) 13 (72) 
 Nurses 2 (11) 1 (6) 1 (6) 
 Other 1 (6) – 1 (6) 
 Pharmacists 1 (6) – 1 (6) 
Country    
 USA 12 (67) 3 (17) 9 (50) 
 Non-USA 6 (33) 2 (11) 4 (22) 
Study Quality    
 0–3 5 (28) 1 (6) 4 (22) 
 4–6 9 (50) 3 (17) 6 (33) 
 7–10 4 (22) 1 (6) 3 (17) 
Type of Health IT Intervention    
 Clinical decision support 14 (78) 4 (22) 10 (56) 
 Computerized provider order entry 3 (17) 1 (6) 2 (11) 
 Electronic health record 1 (6) – 1 (6) 
 Smart Pumps 1 (6) 1 (6) – 
 eMedical Reconciliation 1 (6) – 1 (6) 
 Health information exchange (HIE) 1 (6) – 1 (6) 
 Patient Portal 1 (6) – 1 (6) 
Patient Outcomes Studied    
 Mortality 6 (33) – 6 (33) 
 Readmission, admission, or Emergency dept. visits 6 (33) 1 (6) 5 (28) 
 Adverse drug events or adverse events 3 (17) 1 (6) 2 (11) 
 Chronic Disease Exacerbations 3 (17) 1 (6) 2 (11) 
 Hemodynamic Instability or intensive care unit transfer 3 (17) 1 (6) 2 (11) 
 Thrombosis or Bleed 2 (11) 1 (6) 1 (6) 
 Length of stay 2 (11) 1 (6) 1 (6) 
 Myocardial infarction or Cardiac Events 2 (11) – 2 (11) 
 Infection Rates 1 (6) – 1 (6) 
 Fall rates 1 (6) 1 (6) – 
 Altered mental status or stroke incidence 1 (6) – 1 (6) 
Characteristic Total (%) Positive Studies Non-significant or Mixed Results Studies 
Total (%) 18 (100) 5 (28) 13 (72) 
Setting    
 Inpatient 10 (56) 5 (28) 5 (28) 
 Outpatient 7 (39) – 7 (39) 
 Long-Term Care 1 (6) – 1(6) 
 Multi-Center 8 (44) 1 (6) 7 (39) 
Clinicians Affected    
 Physicians 17 (94) 4 (22) 13 (72) 
 Nurses 2 (11) 1 (6) 1 (6) 
 Other 1 (6) – 1 (6) 
 Pharmacists 1 (6) – 1 (6) 
Country    
 USA 12 (67) 3 (17) 9 (50) 
 Non-USA 6 (33) 2 (11) 4 (22) 
Study Quality    
 0–3 5 (28) 1 (6) 4 (22) 
 4–6 9 (50) 3 (17) 6 (33) 
 7–10 4 (22) 1 (6) 3 (17) 
Type of Health IT Intervention    
 Clinical decision support 14 (78) 4 (22) 10 (56) 
 Computerized provider order entry 3 (17) 1 (6) 2 (11) 
 Electronic health record 1 (6) – 1 (6) 
 Smart Pumps 1 (6) 1 (6) – 
 eMedical Reconciliation 1 (6) – 1 (6) 
 Health information exchange (HIE) 1 (6) – 1 (6) 
 Patient Portal 1 (6) – 1 (6) 
Patient Outcomes Studied    
 Mortality 6 (33) – 6 (33) 
 Readmission, admission, or Emergency dept. visits 6 (33) 1 (6) 5 (28) 
 Adverse drug events or adverse events 3 (17) 1 (6) 2 (11) 
 Chronic Disease Exacerbations 3 (17) 1 (6) 2 (11) 
 Hemodynamic Instability or intensive care unit transfer 3 (17) 1 (6) 2 (11) 
 Thrombosis or Bleed 2 (11) 1 (6) 1 (6) 
 Length of stay 2 (11) 1 (6) 1 (6) 
 Myocardial infarction or Cardiac Events 2 (11) – 2 (11) 
 Infection Rates 1 (6) – 1 (6) 
 Fall rates 1 (6) 1 (6) – 
 Altered mental status or stroke incidence 1 (6) – 1 (6) 

The quality of the RCTs was significantly higher than the non-RCT studies ( P  < .001). The quality of the RCT studies did not improve over time (mean RTCs before 2003, 6.9 and after 2003, 7.2), unlike previously reported. 11,17

Most RCTs studied patient mortality and readmission, admission, and ED visits. For these outcomes, only one study found a benefit of health IT ( Table 6 ). 102

DISCUSSION

Overall Significance

Our finding that most studies had mixed, rather than positive effects on patient safety outcomes, is consistent with almost all prior systematic reviews conducted on health IT and patient safety outcomes. We also found a paucity of outpatient studies, studies evaluating large numbers of patients, and randomized control trials. Given the national priority placed on adoption and use of health IT, our work highlights the urgent need to better evaluate the use of multiple types of health IT on a variety of patient safety outcomes and in a variety of healthcare settings.

Summary of Findings

Demonstrating the benefit of health IT is challenging for several reasons. First, adverse patient outcomes that can be expected to be modified by the implementation of health IT are generally rare events, necessitating large study samples. 108,109 We only found 21 studies (31%) that had > 1000 patient study subjects. In addition, randomized control trials evaluating health IT are difficult to conduct, limiting the quality of evidence on this topic. Randomization is generally not feasible within an individual unit or practice, and thus has to be conducted across settings. In addition, health IT is costly to purchase, resource-intensive to implement and typically purchased for an entire practice or institution. Not surprisingly, we found that the rates of randomized control trials assessing the effects of health IT on patient safety outcomes are increasing more slowly than the rate of research on this topic overall ( Figure 2 ).

Consistent with those constraints, in regards to the types of health IT studied, we found that CDS was the most commonly studied health IT intervention. This is likely due to its inherent nature—it is a software-based intervention that can be turned on and turned off, making it well suited for randomized control, before-and-after, or time series designs. Furthermore, because it is software-based it can be trialed at multiple institutions at once; as such, 63% of the multicenter trials focused on CDS. In contrast to CDS, however, many of the individual tools studied had only one or two quantitative publications.

We found only 10 studies conducted in the outpatient setting, despite the fact that the majority of care is given in the outpatient setting. 4,110 Similarly, we found only 1 study conducted in the long-term care setting. While it has been shown that ambulatory care settings have until recently lagged behind larger institutions in engaging in health IT adoption, 111,112 given the importance of primary care to population health and prevention, the ambulatory care setting stands to gain a lot from rigorous study of the use of health IT to improve patient safety outcomes.

Future Directions and Policy Implications

Given our findings, this review underscores important future directions for this field of research. First, additional large studies are needed to evaluate the effect of health IT on patient safety outcomes, particularly in the outpatient and long-term care settings. Second, a more uniform system for characterizing health IT tools will be needed to facilitate comparison between studies of health IT interventions. CDS, the most commonly studied health IT, for example, covers a very broad range of actual interventions. Third, as the field continues to develop, more cross-institutional studies and collaborations will be required in order to capture the impact of the newest of the emerging health IT tools, such as patient portals and HIE systems.

From a public policy perspective, discussions are occurring in both the academic community and among regulatory agencies as to how to best regulate health IT. The Federal, Food, Drug, and Cosmetic Act recently declared health IT a medical device under regulatory jurisdiction of the US Food and Drug Administration (FDA). 113 To date, the FDA has not yet exerted its regulatory authority over the vast majority of health IT tools. The Office of the National Coordinator for Health Information Technology has also published Safety Assurance Factors for EHR Resilience (SAFER) guides designed to help organizations assess and optimize health IT safety. 114 However, given the mixed findings of many research studies on the effects of health IT on patient safety outcomes, and one study demonstrating a hazardous effect, ongoing studies will be critical to ensure patients remain safe and to better determine which types and features of health IT actually improve care for patients.

Limitations

This review has several key limitations. The first is a direct correlate of the quantity and scope of the literature. Despite performing a comprehensive search, only a limited set of articles with quantitative data were identified. For many important types of health IT, only a few studies reporting the impact on actual patient outcomes were found, even among technologies that are being promoted by government policy. As with all systematic reviews, this review also faced the limitations imposed by publication bias, for which studies with positive results are more likely to be published than those with non-significant findings. Proportionally, however, we did find more studies with non-significant findings than not, which would suggest that our findings may be conservative in their estimate of the number of studies for which no significant effect of health IT was found. We also confined our search to English language publications, which may have precluded us from finding additional relevant studies. Given these limitations, it is possible that certain types of technology were underrepresented in this review, such as emerging technologies (like mobile technologies), patient portals, or HIE.

For this review, we chose to use a quality scale that has been previously used and published in measuring the quality of the study of health IT. 9,11,17 While there are other widely utilized scales that might have been chosen, such as the Cochrane rating system, the scale we utilized has additional bias analysis categories not contained in other scales which we felt made it most rigorous for the quality analysis we were employing. Lastly, there is considerable heterogeneity as to what defines certain types of health IT. For example, CDS has become an umbrella term for many different types of decision support that can be implemented in different ways. We relied on authors’ classifications for health IT tools in determining the type of health IT evaluated, rather than addressing this level of variability. This assumption may have led to an overrepresentation of CDS in the literature.

The authors also recognize that the impact of health IT is greatly influenced by technical, organizational, political, and social factors. Controlling for these in the context of a systematic review is extremely difficult given that authors of the original studies are often not able to measure or quantify these factors, and instead rely on well-matched controls to mitigate these effects. The rating system of study quality is the authors’ attempt to guide readers as to which studies may most effectively control for larger, broader factors.

CONCLUSION

This review has important implications relevant to multiple stakeholders in healthcare, including providers, consumers, policymakers, and vendors. As the nation invests more heavily in health IT, understanding the effects on patient safety outcomes is critical. While there are certain health IT tools that are well studied and are demonstrating safety benefits for patients, there are many areas that are vastly understudied. This review underscores the need for additional, high quality, large-scale studies in multiple settings to better understand how health IT is actually impacting patients. Without such research, we will not be able to identify which health IT tools are indeed effective and in what settings we can expect the greatest benefit.

Author Statements

Funding : This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Competing Interests: The authors have no competing interests to declare.

Author Contribution

S.B.: Study design, data collection, data analysis and interpretation, writing, editing and figure creation, and editing.

R.K.: Study design, data interpretation, writing, editing, and figure editing.

Z.G.: Data collection, data analysis and interpretation, manuscript editing.

C.J.: Data collection, manuscript editing.

I.K.: Data collection, manuscript editing.

R.A.: Study design, literature search, manuscript writing and editing.

D.D.: Study design, literature search, manuscript writing and editing.

E.A.: Study design, data collection, data analysis and interpretation, writing, editing, and figure editing.

SUPPLEMENTARY MATERIAL

Supplementary material is available online at http://jamia.oxfordjournals.org/ .

REFERENCES

1
Kohn
L
Corrigan
J
Donaldson
M
.
To Err is Human: Building a Safer Health System
  .
Washington DC
:
Institute of Medicine, Committee on Quality of Health Care in America 2000
.
2
Health Information Technology: Revisions to Initial Set of Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology. Final Rule, 45 CFR Part 170. USA: Department of Health and Human Services, Statute of the United States Government 2010.
3
Jha
AK
DesRoches
CM
Campbell
EG
et al
.
Use of electronic health records in U.S. hospitals
.
N Engl J Med.
 
2009
;
360
(
16
):
1628
1638
.
4
Chaudhry
B
Wang
J
Wu
S
et al
.
Systematic review: impact of health information technology on quality, efficiency, and costs of medical care
.
Ann Intern Med.
 
2006
;
144
(
10
):
742
752
.
5
Conroy
S
Sweis
D
Planner
C
et al
.
Interventions to reduce dosing errors in children: A systematic review of the literature
.
Drug Saf.
 
2007
;
30
(
12
):
1111
1125
.
6
Eslami
S
de Keizer
NF
Abu-Hanna
A
.
The impact of computerized physician medication order entry in hospitalized patients–a systematic review
.
Int J Med Inform.
 
2008
;
77
(
6
):
365
376
.
7
Georgiou
A
Williamson
M
Westbrook
JI
Ray
S
.
The impact of computerised physician order entry systems on pathology services: A systematic review
.
Int J Med Inform.
 
2007
;
76
(
7
):
514
529
.
8
Hider
P.
Electronic Prescribing: A Critical Appraisal of the Literature
  .
NZHTA Report. Christchurch, New Zealand Health Technology Assessment
;
2002
;
5
(
2
).
9
Jerant
AF
Hill
DB
.
Does the use of electronic medical records improve surrogate patient outcomes in outpatient settings? J Fam Pract
.
2000
  ;
49
(
4
):
349
358
.
10
Van Rosse
F
Maat
B
Rademaker
CMA
van Vught
AJ
Egberts
ACG
Bollen
CW
.
The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review
.
Pediatrics.
 
2009
;
123
(
4
):
1184
1190
.
11
Hunt
DL
Haynes
RB
Hanna
SE
Smith
K
.
Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review
.
JAMA.
 
1998
;
280
(
15
):
1339
1346
.
12
Mollon
B
Chong
JJ
Holbrook
AM
Sung
M
Thabane
L
Foster
G
.
Features predicting the success of computerized decision support for prescribing: A systematic review of randomized controlled trials
.
BMC Med Inform Decis Mak.
 
2009
;
9
(
11
):
1
9
.
13
Montgomery
AA
Fahey
T
.
A systematic review of the use of computers in the management of hypertension
.
J Epidemiol Community Health.
 
1998
;
52
(
8
):
520
525
.
14
Shojania
KG
Jennings
A
Mayhew
A
Ramsay
CR
Eccles
MP
Grimshaw
J
.
The effects of on-screen, point of care computer reminders on processes and outcomes of care
.
Cochrane Database Syst Rev.
 
2009
;(
CD001096
):
1
68
.
15
Bryan
C
Boren
SA
.
The use and effectiveness of electronic clinical decision support tools in the ambulatory/primary care setting: a systematic review of the literature
.
Inform Prim Care.
 
2008
;
16
(
2
):
79
91
.
16
Chang
J
Langberg
M
Silksa
P
Dietrich
A
.
Improving outcomes through the use of inpatient order sets: A systematic review
.
J Gen Intern Med.
 
2010
;
25
:
S308
S309
.
17
Garg
A
Adhikari
N
McDonald
H
et al
.
Effects of computerized clinical decision support systems on practitioner performance and patient outcomes
.
JAMA.
 
2005
;
293
(
10
):
1223
1238
.
18
Johnston
ME
Langton
KB
Haynes
RB
Mathieu
A
.
Effects of computer-based clinical decision support systems on clinician performance and patient outcome. A critical appraisal of research
.
Ann Intern Med.
 
1994
;
120
(
2
):
135
142
.
19
Schedlbauer
A
Prasad
V
Mulvaney
C
et al
.
What evidence supports the use of computerized alerts and prompts to improve clinicians’ prescribing behavior? J Am Med Inform Assoc.
.
2009
  ;
16
(
4
):
531
538
.
20
Chatellier
G
Colombet
I
Degoulet
P
.
An overview of the effect of computer-assisted management of anticoagulant therapy on the quality of anticoagulation
.
Int J Med Inform.
 
1998
;
49
(
3
):
311
320
.
21
Durieux
P
Trinquart
L
Colombet
I
et al
.
Computerized advice on drug dosage to improve prescribing practice
.
Cochrane Database Syst Rev.
 
2008
;
3
(
3
):
1
62
.
22
Shebl
NA
Franklin
BD
Barber
N
.
Clinical decision support systems and antibiotic use
.
Pharm World Sci PWS.
 
2007
;
29
(
4
):
342
349
.
23
Yourman
L
Concato
J
Agostini
JV
.
Use of computer decision support interventions to improve medication prescribing in older adults: a systematic review
.
Am J Geriatr Pharmacother.
 
2008
;
6
(
2
):
119
129
.
24
Walton
R
Dovey
S
Harvey
E
Freemantle
N
.
Computer support for determining drug dose: systematic review and meta-analysis
.
BMJ.
 
1999
;
338
:
984
990
.
25
Sintchenko
V
Magrabi
F
Tipper
S
.
Are we measuring the right end-points? Variables that affect the impact of computerised decision support on patient outcomes: A systematic review
.
Med Inform Internet Med.
 
2007
;
32
(
3
):
225
240
.
26
Wong
K
Yu
SKH
Holbrook
A
.
A systematic review of medication safety outcomes related to drug interaction software
.
Can J Clin Pharmacol.
 
2010
;
17
(
2
):
e243
e255
.
27
Jamal
A
McKenzie
K
Clark
M
.
The impact of health information technology on the quality of medical and health care: a systematic review
.
Heal Inf Manag J.
 
2009
;
38
(
3
):
26
37
.
28
Ludwick
DA
Doucette
J
.
Adopting electronic medical records in primary care: Lessons learned from health information systems implementation experience in seven countries
.
Int J Med Inform.
 
2009
;
78
(
1
):
22
31
.
29
Kaushal
R
Shojania
KG
Bates
DW
.
Effects of computerized physician order entry and clinical decision support systems on medication safety: a systematic review
.
Arch Intern Med.
 
2003
;
163
(
12
):
1409
1416
.
30
Van Rosse
F
Maat
B
Rademaker
CMA
van Vught
AJ
Egberts
ACG
Bollen
CW
.
The effect of computerized physician order entry on medication prescription errors and clinical outcome in pediatric and intensive care: a systematic review
.
Pediatrics.
 
2009
;
123
(
4
):
1184
1190
.
31
Ammenwerth
E
Schnell-Inderst
P
Machan
C
Siebert
U
.
The effect of electronic prescribing on medication errors and adverse drug events: a systematic review
.
J Am Med Inform Assoc.
 
2008
;
15
(
5
):
585
600
.
32
Oren
E
Shaffer
ER
Guglielmo
BJ
.
Impact of emerging technologies on medication errors and adverse drug events
.
Am J Heal Pharm.
 
2003
;
60
(
14
):
1447
1458
.
33
Wolfstadt
JI
Gurwitz
JH
Field
TS
et al
.
The effect of computerized physician order entry with clinical decision support on the rates of adverse drug events: a systematic review
.
J Gen Intern Med.
 
2008
;
23
(
4
):
451
458
.
34
Jones
SS
Rudin
RS
Perry
T
Shekelle
PG
.
Health information technology: an updated systematic review with a focus on meaningful use
.
Ann Intern Med.
 
2014
;
160
:
48
54
.
35
Delpierre
C
Cuzin
L
Fillaux
J
Alvarez
M
Massip
P
Lang
T
.
A systematic review of computer-based patient record systems and quality of care: more randomized clinical trials or a broader approach? Int J Qual Heal Care
.
2004
  ;
16
(
5
):
407
416
.
36
Alvis
JM
Reves
J
Govier
A V
et al
.
Computer-assisted continuous infusions of fentanyl during cardiac anesthesia: comparison with manual method
.
Anesthesiology.
 
1985
;
63
(
1
):
41
49
.
37
Barenfanger
J
Short
MA
Groesch
AA
.
Improved antimicrobial interventions have benefits
.
J Clin Microbiol.
 
2001
;
39
(
8
):
2823
2828
.
38
Buising
K
Thursky
K
Robertson
M
et al
.
Electronic antibiotic stewardhsip-reduced consumption of broad spectrum antibiotics using a computerized antimicrobial approval system in a hospital system
.
J Antimicrob Chemother.
 
2008
;
62
:
608
616
.
39
Chapuis
C
Roustit
M
Bal
G
et al
.
Automated drug dispensing system reduces medication errors in an intensive care setting
.
Crit Care Med.
 
2010
;
38
(
12
):
2275
2281
.
40
Chertow
G
Lee
J
Burdick
E
Seger
DL
Song
J
Komaroff
AL
.
Guided medication dosing for inpatients with renal insufficiency
.
Am Heart J.
 
2001
;
286
(
22
):
2839
2844
.
41
Cho
A
Lee
J
Yoon
J
et al
.
Effect of an electronic alert on risk of contrast-induced acute kidney injury in hospitalized patients undergoing computed tomography
.
Am J Kidney Dis.
 
2012
;
60
(
1
):
74
81
.
42
Cook
P
Rizzo
S
Gooch
M
Jordan
M
Fang
X
Hudson
S
.
Sustained reduction in antimicrobial use and decrease in methicillin-resistant Staphylococcus aureus and Clostridium difficile infections following implementation of an electronic medical record at a tertiary-care teaching hospital
.
J Antimicrob Chemother.
 
2011
;
66
(
1
):
205
209
.
43
Colpaert
K
Claus
B
Somers
A
Vandewoude
K
Robays
H
Decruyenaere
J
.
Impact of computerized physician order entry on medication prescription errors in the intensive care unit: a controlled cross-sectional trial
.
Crit Care.
 
2006
;
10
(
1
):
R21
.
44
Del Beccaro
MA
Jeffries
HE
Eisenberg
MA
Harry
ED
.
Computerized provider order entry implementation: no association with increased mortality rates in an intensive care unit
.
Pediatrics.
 
2006
;
118
(
1
):
290
295
.
45
Dowding
D
Turley
M
Garrido
T
.
The impact of an electronic health record on nurse sensitive patient outcomes: an interrupted time series analysis
.
J Am Informatics Assoc.
 
2012
;
19
(
4
):
615
620
.
46
Dykes
P
Carroll
D
Hurley
A
et al
.
Fall prevention in acute care hospitals: a randomized trial
.
JAMA.
 
2010
;
304
(
17
):
1912
1918
.
47
Etchells
E
Adhikari
NKJ
Wu
R
et al
.
Real-time automated paging and decision support for critical laboratory abnormalities
.
BMJ Qual Saf.
 
2011
;
20
(
11
):
924
930
.
48
Evans
RS
Pestotnik
SL
Classen
DC
et al
.
A computer-assisted management program for antibiotics and other antiinfective agents
.
N Engl J Med.
 
1998
;
338
(
4
):
232
238
.
49
Evans
RS
Classen
DC
Pestotnik
SL
Clemmer
TP
Weaver
LK
Burke
JP
.
A decision support tool for antibiotic therapy
.
Proc Annu Symp Comput Appl Med Care.
 
1995
:
651
655
.
50
Evans
RS
Pestotnik
SL
Classen
DC
Bass
SB
Burke
JP
.
Prevention of adverse drug events through computerized surveillance
.
Proc Annu Symp Comput Appl Med Care.
 
1993
:
437
441
.
51
Fitzmaurice
D
Hobbs
F
Murray
E
Bradley
C
Holder
R
.
Evaluation of computerized decision support for oral anticoagulation management based in primary care
.
Br J Gen Pr.
 
1996
;
46
(
410
):
533
535
.
52
Frances
CD
Alperin
P
Adler
JS
Grady
D
.
Does a fixed physician reminder system improve the care of patients with coronary artery disease? A randomized controlled trial
.
West J Med.
 
2001
;
175
(
3
):
165
166
.
53
Fiumara
K
Piovella
C
Hurwitz
S
et al
.
Multi-screen electronic alerts to augment venous thromboembolism prophylaxis
.
Thromb Haemost.
 
2010
;
103
(
2
):
312
317
.
54
Fleming
NS
Ogola
G
Ballard
DJ
.
Implementing a standardized order set for community-acquired pneumonia: impact on mortality and cost
.
Jt Comm J Qual Patient Saf.
 
2009
;
35
(
8
):
414
421
.
55
Gandhi
TK
Weingart
SN
Seger
AC
et al
.
Outpatient prescribing errors and the impact of computerized prescribing
.
J Gen Intern Med.
 
2005
;
20
(
9
):
837
841
.
56
Glassman
PA
Belperio
P
Lanto
A
et al
.
The utility of adding retrospective medication profiling to computerized provider order entry in an ambulatory care population
.
J Am Med Inform Assoc.
 
2007
;(
4
):
424
431
.
57
Graumlich
JF
Novotny
NL
Stephen Nace
G
et al
.
Patient readmissions, emergency visits, and adverse events after software-assisted discharge from hospital: cluster randomized trial
.
J Hosp Med.
 
2009
;
4
(
7
):
E11
E19
.
58
Gurwitz
JH
Field
TS
Rochon
P
et al
.
Effect of computerized provider order entry with clinical decision support on adverse drug events in the long-term care setting
.
J Am Geriatr Soc.
 
2008
;
56
(
12
):
2225
2233
.
59
Han
YY
Carcillo
JA
Venkataraman
ST
et al
.
Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system
.
Pediatrics.
 
2005
;
116
(
6
):
1506
1512
.
60
Holdsworth
MT
Fichtl
RE
Raisch
DW
et al
.
Impact of computerized prescriber order entry on the incidence of adverse drug events in pediatric inpatients
.
Pediatrics.
 
2007
;
120
(
5
):
1058
1066
.
61
Jani
YH
Barber
N
Wong
ICK
.
Paediatric dosing errors before and after electronic prescribing
.
Qual Saf Health Care.
 
2010
;
19
(
4
):
337
340
.
62
Jha
AK
Orav
EJ
Ridgway
AB
Zheng
J
Epstein
AM
.
Does the Leapfrog program help identify high-quality hospitals? Jt Comm J Qual Patient Saf
.
2008
  ;
34
(
6
):
318
325
.
63
Jimenez Munoz
AB
Muino Miguez
A
Rodriguez Perez
MP
et al
.
Comparison of medication error rates and clinical effects in three medication prescription-dispensation systems
.
Int J Health Care Qual Assur.
 
2011
;
24
(
3
):
238
248
.
64
Keene
A
Ashton
L
Shure
D
Napoleone
D
Katyal
C
Bellin
E
.
Mortality before and after initiation of a computerized physician order entry system in a critically ill pediatric population
.
Pediatr Crit Care Med.
 
2007
;
8
(
3
):
268
271
.
65
King
WJ
Paice
N
Rangrej
J
Forestell
GJ
Swartz
R
.
The effect of computerized physician order entry on medication errors and adverse drug events in pediatric inpatients
.
Pediatrics.
 
2003
;
112
(
3 Pt 1
):
506
509
.
66
Kucher
N
Koo
S
Quiroz
R
et al
.
Electronic alerts to prevent venous thromboembolism among hospitalized patients
.
N Engl J Med.
 
2005
;
352
(
10
):
969
977
.
67
Kuperman
GJ
Teich
JM
Tanasijevic
MJ
et al
.
Improving response to critical laboratory results with automation: results of a randomized controlled trial
.
J Am Med Inform Assoc.
 
1999
;
6
(
6
):
512
522
.
68
Lecumberri
R
Marqués
M
Díaz-Navarlaz
MT
et al
.
Maintained effectiveness of an electronic alert system to prevent venous thromboembolism among hospitalized patients
.
Thromb Haemost.
 
2008
:
699
704
.
69
Lesourd
F
Avril
C
Boujennah
A
Parinaud
J
.
A computerized decision support system for ovarian stimulation by gonadotropins
.
Fertil Steril.
 
2002
;
77
(
3
):
456
460
.
70
Linares
LA
Thornton
DJ
Strymish
J
Baker
E
Gupta
K
.
Electronic memorandum decreases unnecessary antimicrobial use for asymptomatic bacteriuria and culture-negative pyuria
.
Infect Control Hosp Epidemiol.
 
2011
;
32
(
7
):
644
648
.
71
MacIvor
D
Triulzi
DJ
Yazer
MH
.
Enhanced detection of blood bank sample collection errors with a centralized patient database
.
Transfusion.
 
2009
;
49
(
1
):
40
43
.
72
Madaras-Kelly
KJ
Remington
RE
Lewis
PG
Stevens
DL
.
Evaluation of an intervention designed to decrease the rate of nosocomial methicillin-resistant Staphylococcus aureus infection by encouraging decreased fluoroquinolone use
.
Infect Control Hosp Epidemiol.
 
2006
;
27
(
2
):
155
169
.
73
Maynard
GA
Morris
TA
Jenkins
IH
et al
.
Optimizing prevention of hospital-acquired venous thromboembolism (VTE): prospective validation of a VTE risk assessment model
.
J Hosp Med.
 
2010
;
5
(
1
):
10
18
.
74
McCowan
C
Neville
R
Ricketts
I
Warner
F
Hoskins
G
Thomas
G
.
Lessons from a randomized controlled trial designed to evaluate computer decision support software to improve the management of asthma
.
Med Informatics Internet Med.
 
2001
;
26
(
26
):
191
201
.
75
McMullin
J
Cook
D
Griffith
L
et al
.
Minimizing errors of omission: Behavioural rEenforcement of Heparin to Avert Venous Emboli: The BEHAVE Study
.
Crit Care Med.
 
2006
;
34
(
3
):
694
699
.
76
McMullin
ST
Reichley
RM
Watson
LA
Steib
SA
Frisse
ME
Bailey
TC
.
Impact of a Web-based clinical information system on cisapride drug interactions and patient safety
.
Arch Intern Med.
 
1999
;
159
(
17
):
2077
2082
.
77
Menachemi
N
Saunders
C
Chukmaitov
A
Matthews
MC
Brooks
RG
.
Hospital adoption of information technologies and improved patient safety: a study of 98 hospitals in Florida
.
J Healthc Manag.
 
2007
;
52
(
6
):
398
409; discussion 410
.
78
Milani
R V
Oleck
SA
Lavie
CJ
.
Medication errors in patients with severe chronic kidney disease and acute coronary syndrome: the impact of computer-assisted decision support
.
Mayo Clin Proc.
 
2011
;
86
(
12
):
1161
1164
.
79
Morriss
FJ
Abramowitz
P
Nelson
S
Milavetz
G
Michael
S
Gordon
S
.
Risk of adverse drug events in neonates treated with opiods and the effect of a bar-coded-assisted medication administration system
.
Am J Heal Pharm.
 
2011
;
68
:
57
62
.
80
Morriss
FH
Abramowitz
PW
Nelson
SP
et al
.
Effectiveness of a barcode medication administration system in reducing preventable adverse drug events in a neonatal intensive care unit: a prospective cohort study
.
J Pediatr.
 
2009
;
154
(
3
):
363
368, 368
.
e1
.
81
Novis
SJ
Havelka
GE
Ostrowski
D
et al
.
Prevention of thromboembolic events in surgical patients through the creation and implementation of a computerized risk assessment program
.
J Vasc Surg.
 
2010
;
51
(
3
):
648
654
.
82
Oliven
A
Zalman
D
Shilankov
Y
Yeshurun
D
Odeh
M
.
Prevention of prescription errors by computerized, on-line, individual patient related surveillance of drug order entry
.
Stud Heal Technol Informatics.
 
2002
;
90
:
632
634
.
83
Overhage
JM
Dexter
PR
Perkins
SM
et al
.
A randomized, controlled trial of clinical information shared from another institution
.
Ann Emerg Med.
 
2002
;
39
(
1
):
14
23
.
84
Parente
ST
McCullough
JS
.
Health information technology and patient safety: evidence from panel data
.
Health Aff.
 
2009
;
28
(
2
):
357
360
.
85
Paul
M
Andreassen
S
Tacconelli
E
et al
.
Improving empirical antibiotic treatment using TREAT, a computerized decision support system: Cluster randomized trial
.
J Antimicrob Chemother.
 
2006
;
58
(
6
):
1238
1245
.
86
Peterson
JF
Kuperman
GJ
Shek
C
Patel
M
Avorn
J
Bates
DW
.
Guided prescription of psychotropic medications for geriatric inpatients
.
Arch Intern Med.
 
2005
;
165
(
7
):
802
807
.
87
Piontek
F
Kohli
R
Conlon
P
Ellis
JJ
Jablonski
J
Kini
N
.
Effects of an adverse-drug-event alert system on cost and quality outcomes in community hospitals
.
Am J Health Syst Pharm.
 
2010
;
67
(
8
):
613
620
.
88
Poon
EG
Keohane
CA
Yoon
CS
et al
.
Effect of bar-code technology on the safety of medication administration
.
N Engl J Med.
 
2010
;
362
(
18
):
1698
1707
.
89
Potts
AL
Barr
FE
Gregory
DF
Wright
L
Patel
NR
.
Computerized physician order entry and medication errors in a pediatric critical care unit
.
Pediatrics.
 
2004
;
113
(
1 Pt 1
):
59
63
.
90
Rollman
B
Hanusa
B
Lowe
HJ
Gilbert
T
Kapoor
WN
Schulberg
HC
.
A randomized trial using computerized decision support to improve treatment of major depression in primary care
.
J Gen Intern Med.
 
2002
;
17
:
493
503
.
91
Earnest
MA
Ross
SE
Wittevrongel
L
Moore
LA
Lin
CT
.
Use of a patient-accessible electronic medical record in a practice for congestive heart failure: patient and physician experiences
.
J Am Med Inform Assoc.
 
2004
;
11
(
5
):
410
417
.
92
Rind
D
Safran
C
Phillips
R
et al
.
Effect of computer-based alerts on the treatment and outcomes of hospitalized patients
.
Arch Intern Med.
 
1994
;
154
:
1511
1517
.
93
Rothschild
JM
Keohane
CA
Cook
EF
et al
.
A controlled trial of smart infusion pumps to improve medication safety in critically ill patients
.
Crit Care Med.
 
2005
;
33
(
3
):
533
540
.
94
Schnipper
JL
Ndumele
CD
Liang
CL
Pendergrass
ML
.
Effects of a subcutaneous insulin protocol, clinical education, and computerized order set on the quality of inpatient management of hyperglycemia: results of a clinical trial
.
J Hosp Med.
 
2009
;
4
(
1
):
16
27
.
95
Shulman
R
Singer
M
Goldstone
J
Bellingan
G
.
Medication errors: a prospective cohort study of hand-written and computerised physician order entry in the intensive care unit
.
Crit Care.
 
2005
;
9
(
5
):
R516
R521
.
96
Small
MDC
Barrett
A
Price
GM
.
The impact of computerized prescribing on error rate in a department of Oncology/Hematology
.
J Oncol Pharm Pract.
 
2008
;
14
(
4
):
181
187
.
97
Smith
JR
Noble
MJ
Musgrave
SD
et al
.
The at-risk registers in severe asthma (ARRISA) study: A cluster-randomised controlled trial in primary care
.
Thorax.
 
2010
;
65
:
A62
.
98
Tierney
WM
Overhage
JM
Murray
MD
et al
.
Can computer-generated evidence-based care suggestions enhance evidence-based management of asthma and chronic obstructive pulmonary disease? A randomized, controlled trial
.
Health Serv Res.
 
2005
;
40
(
2
):
477
497
.
99
Upperman
JS
Staley
P
Friend
K
et al
.
The impact of hospitalwide computerized physician order entry on medical errors in a pediatric hospital
.
J Pediatr Surg.
 
2005
;
40
(
1
):
57
59
.
100
Van Doormaal
JE
van den Bemt
PMLA
Mol
PGM
et al
.
Medication errors: the impact of prescribing and transcribing errors on preventable harm in hospitalised patients
.
Qual Saf Health Care.
 
2009
;
18
(
1
):
22
27
.
101
Walsh
KE
Landrigan
CP
Adams
WG
et al
.
Effect of computer order entry on prevention of serious medication errors in hospitalized children
.
Pediatrics.
 
2008
;
121
(
3
):
e421
e427
.
102
Weingart
SN
Simchowitz
B
Padolsky
H
et al
.
An empirical model to estimate the potential impact of medication safety alerts on patient safety, health care utilization, and cost in ambulatory care
.
Arch Intern Med.
 
2009
;
169
(
16
):
1465
1473
.
103
Yu
F
Salas
M
Kim
Y-II
Menachemi
N
.
The relationship between computerized physician order entry and pediatric adverse drug events: a nested matched case-control study
.
Pharmacoepidemiol Drug Saf.
 
2009
;
18
(
8
):
751
755
.
104
Zanetti
G
Flanagan
HL
Jr
Cohn
LH
Giardina
R
Platt
R
.
Improvement of intraoperative antibiotic prophylaxis in prolonged cardiac surgery by automated alerts in the operating room
.
Infect Control Hosp Epidemiol.
 
2003
;
24
(
1
):
13
16
.
105
Tanon
A
Champagne
F
Contandriopoulos
A
Pomey
M
Vadeboncoeur
A
Nguyen
H
.
Patient safety and systematic reviews: finding papers indexed in MEDLINE, EMBASE and CINAHL
.
Qual Saf Health Care.
 
2010
;
19
(
5
):
452
461
.
106
Liberati
A
Altman
D
Tetzlaff
J
et al
.
The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration
.
BMJ.
 
2009
;
339
:
b2700
b2700
.
107
Viera
AJ
Garrett
JM
.
Understanding interobserver agreement: the kappa statistic
.
Fam Med.
 
2005
;
37
(
5
):
360
363
.
108
Evans
R
Pestotnik
S
Classen
D
Horn
S
Bass
S
Burke
J
.
Preventing adverse drug events in hospitalized patients
.
Ann Pharmacother.
 
1994
;
28
(
4
):
523
527
.
109
Shojania
KG
Duncan
BW
McDonald
KM
Wachter
RM
Markowitz
AJ
.
Making health care safer: a critical analysis of patient safety practices
.
Evid Rep Technol Assess.
 
2001
;(
43
):
i
x, 1-668
.
110
Eslami
S
Abu-Hanna
A
de Keizer
NF
.
Evaluation of outpatient computerized physician medication order entry systems: a systematic review
.
J Am Med Inform Assoc.
 
2007
;
14
(
4
):
400
406
.
111
Classen
DC
Bates
DW
.
Finding the meaning in meaningful use
.
N Engl J Med.
 
2011
;
365
(
9
):
855
858
.
112
Abramson
EL
Malhotra
S
Fischer
K
et al
.
Transitioning between electronic health records: Effects on ambulatory prescribing safety
.
J Gen Intern Med.
 
2011
;
26
(
8
):
868
874
.
113
Karsh
B-T
Weinger
MB
Abbott
PA
Wears
RL
.
Health information technology: fallacies and sober realities
.
J Am Med Inform Assoc.
 
2010
;
17
(
6
):
617
623
.
114
Technology TO of the NC for HI
.
HealthIT.gov SAFER Guides
  .
Available at:
http://www.healthit.gov/safer/safer-guides .
Accessed October 10, 2014
.

Comments

0 Comments