## Abstract

Background: Medication errors are common, and while most such errors have little potential for harm they cause substantial extra work in hospitals. A small proportion do have the potential to cause injury, and some cause preventable adverse drug events.

Objective: To evaluate the impact of computerized physician order entry (POE) with decision support in reducing the number of medication errors.

Design: Prospective time series analysis, with four periods.

Setting and participants: All patients admitted to three medical units were studied for seven to ten-week periods in four different years. The baseline period was before implementation of POE, and the remaining three were after. Sophistication of POE increased with each successive period.

Intervention: Physician order entry with decision support features such as drug allergy and drug-drug interaction warnings.

Main outcome measure: Medication errors, excluding missed dose errors.

Results: During the study, the non-missed-dose medication error rate fell 81 percent, from 142 per 1,000 patient-days in the baseline period to 26.6 per 1,000 patient-days in the final period (P < 0.0001). Non-intercepted serious medication errors (those with the potential to cause injury) fell 86 percent from baseline to period 3, the final period (P = 0.0003). Large differences were seen for all main types of medication errors: dose errors, frequency errors, route errors, substitution errors, and allergies. For example, in the baseline period there were ten allergy errors, but only two in the following three periods combined (P < 0.0001).

Conclusions: Computerized POE substantially decreased the rate of non-missed-dose medication errors. A major reduction in errors was achieved with the initial version of the system, and further reductions were found with addition of decision support features.

Medication errors occur commonly in hospitals.1–5 While most such errors are minor, a small proportion result in an injury or adverse drug event (ADE). In an earlier report, we found that about 1 in 100 medication errors actually results in an ADE, although about 7 per 100 had the potential to do so.5 Although most medication errors have little potential for harm, they are undesirable and do cause substantial extra work.5

Adverse drug events are important both for patients and hospitals. While most ADEs are minor—rashes and diarrhea, for example—some are serious, and a few even result in death.6–8 These events are also costly: Two recent studies independently estimated their costs to be more than $2,000 per event.8,9 Nationally, the cost of ADEs occurring in hospitals has been estimated at$2 billion.8,9

One intervention that has substantial potential for improving the medication ordering process is computerized physician order entry (POE), in which physicians write orders online.10 Physician order entry can improve ordering by ensuring complete, unambiguous, and legible orders. Also, the computer can assist the physician at the time of ordering by suggesting appropriate doses and frequencies, displaying relevant laboratory data, and screening orders for allergies and drug-drug and drug-laboratory interactions.

We recently reported the results of a study of the impact of POE on serious medication errors—errors that either had the potential for harm or actually resulted in an ADE.10 However, the impact of POE on all medication errors—both minor and serious—is important and was not addressed in that study, and the impact of POE was measured at only one stage in its development. Therefore, we performed a trial to evaluate the impact of POE on the medication error rate and also collected error rates at several stages—for a base-line period before POE and then for three separate time periods after implementation. Specific goals were to compare the medication error rates over time; to evaluate which types of medication errors were most likely to be affected; and to evaluate the errors not prevented with the most mature version, to guide future improvements.

## Methods

### Study Site and Participants

This study took place at Brigham and Women's Hospital, an academic tertiary-care hospital with approximately 700 beds. The study was performed on three medical units (two general care medical units and one medical intensive care unit) over four and a half years. Data were collected in four separate time periods, all seven to ten weeks long. All evaluations took place in October, November, and December, except for the period 3 analysis, which was delayed until March 1997 because of staffing shortages in the pharmacy. The study periods were as follows: baseline, 51 days, Oct-Nov 1992; period 1, 68 days, Oct-Dec 1993; period 2, 49 days, Nov-Dec 1995; and period 3, 52 days, Mar–Apr 1997. Participants were all patients admitted to a study floor during a study period.

An internally developed information system (BICS) manages the hospital's administrative, financial, and clinical data.11 The system includes clinical results reporting and a computer-based event detection application, which uses rules to detect a wide variety of events.12 Computerized physician order entry is another application within BICS which came online in May 1993.13,14 All orders are written using this application. Approximately 16,000 orders are written daily, 40 percent for medications. The BICS POE application checks each order for completeness and ensures that certain parameters (e.g., names of medications) come from standard lists. Suggested doses and frequencies are offered for medication orders. Entered orders are screened for problems, such as drug allergies and drug-drug interactions, and the system presents these problems to the physician immediately, when appropriate. At the time of the study, orders were printed out on the floor and carried manually to the pharmacy; a direct electronic link has subsequently been completed.

### Interventions

In the baseline period, orders were hand written in order books in the traditional fashion. Subsequent study periods took place after the introduction of POE and serial improvements to it (Table 1).

Table 1

System Characteristics By Period

 Period System Characteristics Baseline Orders written on paper No automated decision support Period 1 POE in place: Complete orders requiredMedication name, dose, and frequency selection from listsRelevant laboratories displayedDecreased transcriptionRudimentary drug-allergy checkingRedundant medication checkingRudimentary drug-drug interaction checkingNotification for several drug-laboratory problemsMany orders entered using preapproved order sets Period 2 Improved drug allergy checking Period 3 Improved potassium ordering Improved drug-drug interaction checking
 Period System Characteristics Baseline Orders written on paper No automated decision support Period 1 POE in place: Complete orders requiredMedication name, dose, and frequency selection from listsRelevant laboratories displayedDecreased transcriptionRudimentary drug-allergy checkingRedundant medication checkingRudimentary drug-drug interaction checkingNotification for several drug-laboratory problemsMany orders entered using preapproved order sets Period 2 Improved drug allergy checking Period 3 Improved potassium ordering Improved drug-drug interaction checking

Period 1 began five months after the implementation of POE. During this period, most basic system features were in place, but relatively little decision support was provided. Specifically, all orders required specification of the dose, the route, and the frequency of administration. Doses were ordered primarily from hospital-approved standard lists, so that egregious dosing errors were much less likely; medication names and frequencies were also selected from lists. Relevant laboratory results were displayed on the ordering screen. Transcription was greatly decreased but not eliminated; transcription to a paper medication administration record was still required. A simple system was in place to detect drug-allergy interactions for the drug families to which patients are most commonly allergic (e.g., penicillins and sulfa drugs). Duplicate orders (multiple orders for the same drug) generated a warning as well. Clinicians were notified about a small number of life-threatening drug-drug interactions and drug-laboratory checks by paging them after the fact. Many orders were entered through predefined order sets.

In period 2, the most important change was introduction of a comprehensive allergy warning system that included cross-sensitivity checking. By period 3, rules for the use of potassium chloride were significantly revised. All orders for amounts of intravenous potassium higher than 20 milliequivalents were amended to include explicit notation that it be administered in divided doses, the number of available sliding scales was reduced, and individualized scales were restricted. In addition, a more sophisticated drug-drug interaction checking system was implemented, which checked for approximately 100 pairs of serious interactions and also notified the physician at the time of ordering, instead of after the fact.15 Orders were also electronically communicated to the pharmacy (in periods 1 and 2, they were printed out on the floor and hand-carried to the pharmacy).

### Main Outcome Measures

The main outcome measure of the study was the non-missed-dose medication error. We defined medication errors as errors in the process of ordering, dispensing, or administering a medication, regardless of whether an injury occured or whether the potential for injury was present. Missed-dose errors were errors in which doses of medications were not available to nursing personnel at the time they were needed for administration; in most instances, the doses were eventually given. Because missed-dose errors tend to have less potential for harm than other errors and we did not expect them to be significantly decreased by the implementation of POE, we excluded them from the primary outcome. Applications that might be expected to decrease the frequency of missed dose errors are direct electronic transmission of orders to and from the pharmacy and integration of a computerized medication administration record. As noted earlier, direct electronic order transmission to the pharmacy was not in place in period 3, and we are still developing a computerized medication administration record.

As secondary outcomes, we collected data on adverse drug events (both nonpreventable and preventable)7 and potential adverse drug events, defined as errors with the potential for harm that did not result in an injury. Potential adverse drug events included errors that were intercepted before the medication reached the patient and non-intercepted errors that did reach the patient but did not cause injury. An example of the latter would be inadvertent administration of penicillin to a patient with a known penicillin allergy, who did not react. Serious medication errors were defined as those associated with a preventable ADE or a potential ADE; non-intercepted serious medication errors include only the preventable ADEs and non-intercepted potential ADEs.

### Case Finding

Medication errors, potential ADEs, and ADEs were detected in three ways: 1) pharmacists reported any prescribing errors, potential ADEs, or ADEs that they identified during the dispensing process, and reports were also solicited from nurses through daily visits by the study investigator; 2) a trained reviewer evaluated all medication sheets received by the pharmacy; and 3) the study case investigator reviewed all charts daily on weekdays for evidence of medication errors or ADEs.7 The bulk of medication errors were identified by evaluating the medication sheets, so that the detection process differentially detected ordering errors. The chart review included a careful reading of the progress notes in each chart and a more detailed investigation if the investigator identified indications of an ADE (e.g., major bleeding, new confusion, unanticipated intensive care unit transfer, or cardiac arrest). People reporting incidents were assured of anonymity; it was emphasized that we viewed errors and incidents as results of system flaws rather than as human failings.

### Classification

All incidents were evaluated as to whether they represented medication errors. The medication errors were then classified by type: dose error (overdose, underdose, missed dose, wrong dose form, dose omitted), route error (incorrect route, wrong route, route omitted), frequency error (incorrect frequency, frequency omitted), substitution (wrong drug given, wrong patient received drug), drug-drug interaction, inappropriate drug, illegible order, known allergy to drug, drug not available (nonformulary and not readily attainable), avoidable delay in treatment, and preparation error.

### Analysis

Rates of non-missed-dose medication errors were calculated using both the number of patient-days in the period and the number of admissions. Comparisons between periods were made using the chi-squared test for trend using StatXact. The test for trend assumes an ordering of the categories and was used because of the serial improvements in the system over the four periods.

## Results

During the study, the number of admissions in the three units varied among the study periods from 379 to 492, patient-days varied from 1,704 to 2,619, and medication orders varied from 10,070 to 15,025 (Table 2). The number of medication orders per admission remained relatively constant over the four periods, although the number of medication orders per patient-day was higher in the last two periods compared with the first two, reflecting shorter inpatient length of stay in later periods.

Table 2

Comparison Across Periods

 Baseline Period 1 Period 2 Period 3 Duration in days: 51 68 49 51 Patient-days: 1,704 2,619 1,784 1,878 Admissions: 379 492 471 475 Medication orders: 10,070 15,025 13,139 14,352 Medication orders/patient-days: 5.91 5.74 7.36 7.64 Medication orders/admission: 26.6 30.5 27.9 30.2
 Baseline Period 1 Period 2 Period 3 Duration in days: 51 68 49 51 Patient-days: 1,704 2,619 1,784 1,878 Admissions: 379 492 471 475 Medication orders: 10,070 15,025 13,139 14,352 Medication orders/patient-days: 5.91 5.74 7.36 7.64 Medication orders/admission: 26.6 30.5 27.9 30.2

The main outcome, the non-missed-dose error rate per 1,000 patient-days, fell from 142 during baseline to 51.2 in period 1, then rose to 74.0 in period 2, and then fell again to 26.6 in period 3 (P < 0.0001, Table 3). The largest differences were between the baseline and period 1 (a 64 percent drop), when POE was implemented, and between periods 2 and 3 (also a 64 percent drop), when the screens were changed to minimize the likelihood of potassium chloride errors (Figure 1).

Table 3

Medication Error and Event Rates, By Period

*

Of the 82 intercepted potential ADEs in period 2, 77 were potassium chloride errors; and in period 3, of the 106 intercepted potential ADEs, 101 were potassium chloride errors. In period 4, these errors were essentially eliminated by revising the potassium chloride ordering screens.

Figure 1

Event rates by period. Top, the non-missed-dose medication error rate per 1,000 patient-days, by period. This rate dropped 64 percent between baseline and period 1, then climbed 45 percent between periods 1 and 2, and finally fell 64 percent between periods 2 and 3; the overall decline was 81 percent (P < 0.0001). Middle, the missed dose error rate per 1,000 patient-days across periods, which climbed significantly during the study (P < 0.0001). Bottom, the non-intercepted serious medication error rate per 1,000 patient-days, which fell during the study (P = 0.0003). Non-intercepted serious medication errors were defined as either non-intercepted potential adverse drug events (ADEs) or preventable ADEs.

Figure 1

Event rates by period. Top, the non-missed-dose medication error rate per 1,000 patient-days, by period. This rate dropped 64 percent between baseline and period 1, then climbed 45 percent between periods 1 and 2, and finally fell 64 percent between periods 2 and 3; the overall decline was 81 percent (P < 0.0001). Middle, the missed dose error rate per 1,000 patient-days across periods, which climbed significantly during the study (P < 0.0001). Bottom, the non-intercepted serious medication error rate per 1,000 patient-days, which fell during the study (P = 0.0003). Non-intercepted serious medication errors were defined as either non-intercepted potential adverse drug events (ADEs) or preventable ADEs.

In contrast, the missed-dose error rate climbed substantially over the study period (Figure 1), almost doubling from 169 per admission in the baseline period, to 329 in period 3 (P < 0.0001).

The non-intercepted serious medication error rate (the combination of preventable ADEs and non-intercepted potential ADEs) fell over the four periods, from 7.6 per 1,000 patient-days in the baseline period to 7.3 per 1,000 patient-days in period 1 and to 1.7 and 1.1 in periods 2 and 3, respectively (Figure 1, P = 0.0003). The rate of preventable ADEs was highest in period 1, at 5.7 events per 1,000 patient-days, although it fell in the following two periods.

An obvious concern was whether the increase in preventable ADEs in period 1 was due to POE. We therefore examined them more closely by evaluating the case descriptions, in particular to assess whether they were caused by POE. None appeared to have been caused by POE. Of the 15 preventable ADEs, 3 might have been prevented by changes made in period 2 or 3, while 12 would not; most of the 12 were due to use of multiple sedating drugs, not addressed by POE.

The rate of intercepted potential ADEs climbed substantially from baseline to periods 1 and 2; it rose from 15.8 per 1,000 patient-days at baseline to 31.3 in period 1 and 59.4 in period 2 (P = 0.15) before falling to 0.5 in period 3 (Table 3). These increases in errors were largely related to POE's initial structure for potassium chloride orders, which made it easy to order large doses of intravenous potassium without explicitly specifying that it be given in divided doses (i.e., not more than 20 milliequivalents at a time). Standard nursing practice is to give it in divided doses, and all of these errors were intercepted by nurses so that no patient was injured. These potassium ordering errors accounted for 77 of the 82 intercepted potential ADEs in period 1 and for 101 of the 106 in period 2. Once the potassium ordering screen was changed to include this stipulation, the rate of intercepted potential ADEs fell to 0.5 per 1,000 patient-days in period 3. Only one intercepted event was identified in period 3.

Examination of the severity of ADEs and potential ADEs across the periods (Table 4) showed that in the baseline period all five errors associated with preventable ADEs were serious or life-threatening, and six of the eight errors associated with non-intercepted potential ADEs also fell within those categories. In periods 2 and 3 combined, there were only two such serious or life-threatening errors that were not intercepted.

Table 4

 Baseline Period 1 Period 2 Period 3 ADEs, not preventable: 20 24 17 16 Life-threatening 0 (0%) 2 (8%) 2 (12%) 0 (0%) Serious 3 (15%) 4 (17%) 6 (35%) 6 (38%) Significant 17 (85%) 18 (75%) 9 (53%) 10 (62%) ADEs, preventable: 5 15 2 2 Life-threatening 1 (20%) 4 (27%) 1 (50%) 0 (0%) Serious 4 (80%) 5 (33%) 0 (0%) 0 (0%) Significant 0 (0%) 6 (40%) 1 (50%) 2 (100%) Potential ADEs, not intercepted: 8 4 1 0 Life-threatening 1 (13%) 0 (0%) 0 (0%) 0 (0%) Serious 5 (63%) 1 (25%) 1 (100%) 0 (0%) Significant 2 (25%) 3 (75%) 0 (0%) 0 (0%) Non-intercepted serious medication errors†: 13 19 3 2 Life-threatening 2 (15%) 4 (21%) 1 (33%) 0 (0%) Serious 9 (69%) 6 (32%) 1 (33%) 0 (0%) Significant 2 (15%) 9 (47%) 1 (33%) 2 (100%) Potential ADEs, intercepted: 27 82 106 1 Life-threatening 3 (11%) 78 (95%) 99 (93%) 0 (0%) Serious 12 (44%) 2 (2%) 4 (4%) 0 (0%) Significant 12 (44%) 2 (2%) 3 (3%) 1 (100%)
 Baseline Period 1 Period 2 Period 3 ADEs, not preventable: 20 24 17 16 Life-threatening 0 (0%) 2 (8%) 2 (12%) 0 (0%) Serious 3 (15%) 4 (17%) 6 (35%) 6 (38%) Significant 17 (85%) 18 (75%) 9 (53%) 10 (62%) ADEs, preventable: 5 15 2 2 Life-threatening 1 (20%) 4 (27%) 1 (50%) 0 (0%) Serious 4 (80%) 5 (33%) 0 (0%) 0 (0%) Significant 0 (0%) 6 (40%) 1 (50%) 2 (100%) Potential ADEs, not intercepted: 8 4 1 0 Life-threatening 1 (13%) 0 (0%) 0 (0%) 0 (0%) Serious 5 (63%) 1 (25%) 1 (100%) 0 (0%) Significant 2 (25%) 3 (75%) 0 (0%) 0 (0%) Non-intercepted serious medication errors†: 13 19 3 2 Life-threatening 2 (15%) 4 (21%) 1 (33%) 0 (0%) Serious 9 (69%) 6 (32%) 1 (33%) 0 (0%) Significant 2 (15%) 9 (47%) 1 (33%) 2 (100%) Potential ADEs, intercepted: 27 82 106 1 Life-threatening 3 (11%) 78 (95%) 99 (93%) 0 (0%) Serious 12 (44%) 2 (2%) 4 (4%) 0 (0%) Significant 12 (44%) 2 (2%) 3 (3%) 1 (100%)
*

Percentage totals may not add to 100, because of rounding.

Non-intercepted serious medication errors are the sum of preventable ADEs and non-intercepted potential ADEs.

As expected, the non-missed-error rate was higher in the intensive care unit than in the general care units; it also fell more in the intensive care units than in the general care units (Table 5). In contrast, the missed-dose error rate did not display a consistent pattern in the intensive care unit, but in the general care units it rose substantially over the period of the study.

Table 5

Medication Error Rate, By Unit Type Across Periods

 Baseline Period 1 Period 2 Period 3 P Value Patient-days: 1,704 2,619 1,784 1,878 - ICU (n) 403 736 296 597 - Non-ICU (n) 1,301 1,883 1,488 1,281 - Non-missed-dose errors: ICU (n) 100 71 47 21 - ICU rate/1,000 pt-days 248 96.5 159 35.2 0.0001 Non-ICU (n) 142 63 85 29 - Non-ICU rate/1,000 pt-days 109 33.5 57.1 22.6 0.0001 Missed-dose errors: ICU (n) 127 185 119 172 - ICU rate/1,000 pt-days 315 251 402 288 0.09 Non-ICU (n) 161 315 281 445 - Non-ICU rate/1,000 pt-days* 124 167 189 347 0.0001
 Baseline Period 1 Period 2 Period 3 P Value Patient-days: 1,704 2,619 1,784 1,878 - ICU (n) 403 736 296 597 - Non-ICU (n) 1,301 1,883 1,488 1,281 - Non-missed-dose errors: ICU (n) 100 71 47 21 - ICU rate/1,000 pt-days 248 96.5 159 35.2 0.0001 Non-ICU (n) 142 63 85 29 - Non-ICU rate/1,000 pt-days 109 33.5 57.1 22.6 0.0001 Missed-dose errors: ICU (n) 127 185 119 172 - ICU rate/1,000 pt-days 315 251 402 288 0.09 Non-ICU (n) 161 315 281 445 - Non-ICU rate/1,000 pt-days* 124 167 189 347 0.0001

Note: Pt-days indicates patient-days; ICU, intensive care unit.

*

This rate increased over the study.

### Evaluation By Error Type

The results by type of error show that the rates for most subtypes fell over the course of the study (Table 6). Dose errors were the most frequent subtype, and these actually remained high in periods 1 and 2. However, most dose errors in periods 1 and 2 (86 and 89 percent, respectively) were potassium chloride ordering errors, and this rate fell substantially in period 3 after changes were made in potassium chloride ordering and usage practices. The next most common types of errors, frequency errors and route errors, both fell significantly between baseline and period 3 (P < 0.0001). Some of the rarer errors such as known allergy errors actually have the greatest potential for harm. There were ten known allergy errors in the first period, and only two in the following three periods combined (P < 0.0001). In period 1, approximately 50 orders per day hospital-wide were canceled because of a computer warning about drug allergies; in period 2, this number rose to 80 per day.

Table 6

Non-missed-dose Medication Errors, By Type and Period

 Baseline (n = 10,070) Period 1 (n = 15,025) Period 2 (n = 13,139) Period 3 (n = 14,352) P* Value Dose errors 81 (47.5) 90† (34.3) 114† (63.9) 40† (21.3) 0.03 Frequency errors 43 (25.2) 4 (1.5) 2 (1.1) 4 (2.1) 0.0001 Route errors 25 (14.7) 5 (1.9) 6 (3.3) 4 (2.1) 0.0001 Substitution errors 12 (7.0) 3 (1.1) 3 (1.7) 0 (0) 0.0001 Documented allergy 10 (5.9) 1 (0.4) 1 (0.6) 0 (0) 0.0001 Inappropriate drug 7 (4.1) 3 (1.1) 1 (0.6) 0 (0) 0.002 Avoidable delay 7 (4.1) 0 (0) 0 (0) 0 (0) 0.003 Drug-drug interaction 2 (1.2) 0 (0) 1 (0.6) 0 (0) 0.19 Inadequate follow-up 1 (0.6) 0 (0) 0 (0) 0 (0) 0.17 Other 54 (31.7) 28 (10.7) 4 (2.2) 2 (1.1) 0.0001 Total 242 (142.0) 134 (51.2) 132 (74.0) 50 (26.6) 0.0001
 Baseline (n = 10,070) Period 1 (n = 15,025) Period 2 (n = 13,139) Period 3 (n = 14,352) P* Value Dose errors 81 (47.5) 90† (34.3) 114† (63.9) 40† (21.3) 0.03 Frequency errors 43 (25.2) 4 (1.5) 2 (1.1) 4 (2.1) 0.0001 Route errors 25 (14.7) 5 (1.9) 6 (3.3) 4 (2.1) 0.0001 Substitution errors 12 (7.0) 3 (1.1) 3 (1.7) 0 (0) 0.0001 Documented allergy 10 (5.9) 1 (0.4) 1 (0.6) 0 (0) 0.0001 Inappropriate drug 7 (4.1) 3 (1.1) 1 (0.6) 0 (0) 0.002 Avoidable delay 7 (4.1) 0 (0) 0 (0) 0 (0) 0.003 Drug-drug interaction 2 (1.2) 0 (0) 1 (0.6) 0 (0) 0.19 Inadequate follow-up 1 (0.6) 0 (0) 0 (0) 0 (0) 0.17 Other 54 (31.7) 28 (10.7) 4 (2.2) 2 (1.1) 0.0001 Total 242 (142.0) 134 (51.2) 132 (74.0) 50 (26.6) 0.0001

Note: The number of occurrences of each error is shown, followed by the rate per 1,000 patient-days in parentheses.

*

The P value was determined by the chi-squared test for trend across the four periods.

Of these errors, 77 of 90 (86 percent) in period 1 and 101 of 114 (89 percent) in period 2 were potassium chloride errors, whereas none of the 40 dose errors in period 3 was a potassium chloride error.

### Analysis of Errors Missed in Period 3

We evaluated the errors still occurring in period 3 to see whether additional changes to POE could prevent similar errors in the future. Of the 50 errors in period 3, 48 could potentially be prevented with additional changes to POE. These changes fell into nine categories, but changes in two might prevent the majority. Eighteen errors were due to improper use of a “multiple routes” option, in which the physician could indicate more than one route for the medication, depending on the patient's condition (e.g., morphine sulfate IV/PO). These were considered errors if for one of the routes the dose would be inappropriate or if one route would not be feasible for the medication—for example, a drug that should not be given intravenously. The second large group (n = 13) of errors might be prevented with route restrictions for some oral medications, such as sustained-release preparations and gelatin capsules, which should not be ordered to be given by enterostomy.

## Discussion

More than 80 percent of non-missed-dose medication errors were eliminated by computerized POE. Three quarters of this reduction was achieved with a relatively simple system, which structured the entry of orders and included rudimentary order checking. Reductions were seen in a broad array of error types and in both general care and intensive care units. The number of non-intercepted serious medication errors—those with the potential to cause injury—also fell significantly.

Many other evaluations have demonstrated benefits of POE systems10,16 and computerized decision support17–20 for improving the quality and efficiency of care. For example, Tierney et al.16 demonstrated that implementation of a POE system on a medical service resulted in a decrease in average length of stay of 0.89 days and a 12.7 percent reduction in charges. Recently, Evans et al.19 found that computer-assisted decision support for ordering antibiotics in one intensive care unit resulted in substantially lower costs and improved quality of care, in part by improving the appropriateness of drug dosing and decreasing the number of allergic reactions.

In another evaluation of the same POE system evaluated in this study, using serious medication errors as the primary outcome and data from 1995, we found that this system prevented 55 percent of serious medication errors. The current study evaluated the impact of POE on all medication errors and included 1997 data, after the system had undergone additional refinements.10 The additional refinements probably account for some of the difference in effect size between the current study (an 88 percent reduction for serious medication errors between the baseline and period 3) and the 55 percent effect noted above. Order entry resulted in improvements both because of additional structuring of orders and because it allowed checking of orders for problems such as allergies and drug interactions.

The increase in the number of intercepted potential ADEs that occurred post-POE during periods 1 and 2 illustrates the potential that any change, especially a systems change with profound effects such as POE, has for causing new errors, even though this particular error was always intercepted and the overall effect was clearly positive.

The implications for designers of computerized ordering systems are clear. Physician order entry guides physicians toward particular ordering paths, which physicians are likely to follow. Thus, it is vitally important that the suggested paths and parameters are correct and are revised continually. Implementation of POE made it easier for users to order potassium sliding scales and large intravenous doses but did not initially include safeguards about the rate of intravenous potassium. While “all nurses should know” that potassium should not be given too rapidly intravenously, this was not explicitly stated in the entered order. Although this problem was detected in period 1, it was not fixed until period 3. During this interval, the program changes were in a long queue of changes to be made in the system and never quite reached the top. After implementation of POE, a host of valuable quality changes were quickly sought by practitioners, such as guidelines for drugs and implementation of critical pathways. This example underscores the need for significant programming resources after the introduction of POE.

Even the last version of POE we tested in this study was far from mature; we continue to refine it and add improvements. For example, this version of the application still did not include guided dose algorithms or dose adjustments for renal insufficiency, two changes that a prior study suggested might have the largest impact on adverse event prevention.21 An outstanding issue that we are still addressing is the problem of multiple sedating drugs; a simple algorithm will not deal with this problem, since many drugs are involved, and often the issue is that several of these drugs are given in close temporal proximity.

Other automation and system strategies will be necessary to further reduce the number of medication errors, in particular the number of missed-dose errors, which rose substantially during this study. Missed-dose errors have relatively low potential for harm5 but are costly because they cause additional work for providers. One estimate was that each missed-dose error causes about 15 minutes of extra work, primarily for nursing and pharmacy personnel.5 There are many possible causes of the increase in the missed-dose rate. Patient acuity and the number of transfers rose over the study period, while pharmacy staffing decreased. Also, the pharmacy satellites were closed as part of a programmatic change to deploy pharmacists on floors to take advantage of their clinical skills (although this probably resulted in better patient care overall).22 Furthermore, until the final period of the study, drug orders were not communicated electronically to the pharmacy but were printed out and hand carried. Another issue is that nurses had increased drug delivery expectations, because POE made orders immediately visible to them. Eventually, moving to point-of-care delivery devices linked with bar-coding and computerization of the medication administration record could reduce the number of missed doses.22,23

One limitation of this study is that we evaluated only three units at one tertiary care institution, so the impact could be different in other settings. However, Raschke et al.20 found important benefits of computerized medication alerts in a community hospital. Another limitation of this study is that it was a time series study, not a randomized trial. Because of the complexity of introducing POE, a randomized trial was not feasible. However, it is unlikely that unmeasured temporal effects accounted for the very large effect seen, which appeared immediately after the introduction of the intervention and persisted for the duration of the study. Also, the fourth period of the study was conducted later in the year than the first three, so that the house officers had more experience, although in prior evaluations we saw little evidence of a temporal correlation with error rate. A further limitation is that our detection methodology was better for detecting errors in ordering than errors in medication administration. One method used to assess the frequency of medication errors is the direct observation approach, which finds about one medication error per patient per day, and most of these are administration errors.1,2 Ordering errors appear more often than administration errors to result in patient injuries, although both are important.7 Another limitation is that we did not formally assess the reliability of our medication error detection approach in this study, although we did assess the reliability of our assessment of events, and this reliability was good.

We conclude that computerized POE resulted in a very large decrease in the frequency of non-missed-dose medication errors, the errors that are most likely to harm patients. Systems such as these have the potential to both fix and cause problems, and require evaluation. The reductions occurred because order entry both structured orders and facilitated the checking of them. Further reductions should be possible with additional decision support and refinement of the system. Such systems should be used more widely.

The authors thank Priscilla Dasse, RN, of the Risk Management Foundation, and William Churchill, PharmD, of the Brigham and Women's Hospital, for their help and support. The POE development team, particularly Jennifer Schmiz, Nina Plaks, Cynthia Spurr, RN, Debra Thomas, RN, and Eileen O'Connell, RN, were invaluable in the analysis and production of the system enhancements.

## References

1
Allan
EL
Barker
KN
.
Fundamentals of medication error research
.
Am J Hosp Pharm

1990
;
47
:
555
71
.
2
Barker
KN
Allan
EL
.
Research on drug-use-system errors
.
Am J Health Syst Pharm

1995
;
52
:
400
3
.
3
Lesar
TS
Briceland
LL
Delcoure
K
Parmalee
JC
Masta-Gornic
V
Pohl
H
.
Medication prescribing errors in a teaching hospital
.
JAMA

1990
;
263
:
2329
34
.
4
Folli
HL
Poole
RL
Benitz
WE
Russo
JC
.
Medication error prevention by clinical pharmacists in two children's hospitals
.
Pediatrics

1987
;
79
:
718
22
.
5
Bates
DW
Boyle
DL
Vander Vliet
MB
Schneider
J
Leape
LL
.
Relationship between medication errors and adverse drug events
.
J Gen Intern Med

1995
;
10
:
199
205
.
6
Classen
DC
Pestotnik
SL
Evans
RS
Burke
JP
.
Description of a computerized adverse drug event monitor using a hospital information system
.
Hosp Pharm

1992
;
27
:774,
776
9
,
783
.
7
Bates
DW
Cullen
D
Laird
N
et al
.
Incidence of adverse drug events and potential adverse drug events: implications for prevention
.
JAMA

1995
;
274
:
29
34
.
8
Classen
DC
Pestotnik
SL
Evans
RS
Lloyd
JF
Burke
JP
.
Adverse drug events in hospitalized patients: excess length of stay, extra costs, and attributable mortality
.
JAMA

1997
;
277
;
301
6
.
9
Bates
DW
Spell
N
Cullen
DJ
et al
.
The costs of adverse drug events in hospitalized patients
.
JAMA

1997
;
227
:
307
11
.
10
Bates
DW
Leape
LL
Cullen
DJ
et al
.
Effect of computerized physician order entry and a team intervention on prevention of serious medication errors
.
JAMA

1998
;
280
:
1311
6
.
11
Teich
JM
Glaser
JP
Beckley
RF
et al
.
Toward cost-effective, quality care; the Brigham Integrated Computing System
.
Proc 2nd Nicholas E. Davies CPR Recognition Symposium

1996
:
3
.
12
Kuperman
GJ
Teich
JM
Bates
DW
McLatchey
J
Hoff
TG
.
Representing hospital events as complex conditionals
.
Proc AMIA Annu Fall Symp

1995
;
19
:
137
41
.
13
Teich
JM
Hurley
JF
Beckley
RF
Aranow
M
.
Design of an easy-to-use physician order entry system with support for nursing and ancillary departments
.
Proc Annu Symp Comput Appl Med Care

1992
:
99
103
.
14
Teich
JM
Spurr
CD
Flammini
SJ
et al
.
Response to a trial of physician-based inpatient order entry
.
Proc Annu Symp Comput Appl Med Care

1993
:
316
320
.
15
Paterno
M
Teich
JM
Seger
DL
Bates
DW
.
A practical method for presenting drug interactions to clinicians
.
Proc AMIA Annu Fall Symp

1996
:
872
.
16
Tierney
WM
Miller
ME
Overhage
JM
McDonald
CJ
.
Physician inpatient order writing on microcomputer workstations: effects on resource utilization
.
JAMA

1993
;
269
:
379
83
.
17
Hunt
DL
Haynes
RB
Hanna
SE
Smith
K
.
Effects of computer-based clinical decision support systems on physician performance and patient outcomes
.
JAMA

1998
;
280
:
1339
46
.
18
Evans
RS
Pestotnik
SL
Classen
DC
Horn
SD
Bass
SB
Burke
JP
.
Preventing adverse drug events in hospitalized patients
.
Ann Pharmacother

1994
;
28
:
523
7
.
19
Evans
RS
Pestotnik
SL
Classen
DC
et al
.
A computer-assisted management program for antibiotics and other anti-infective agents
.
N Engl J Med

1998
;
338
:
232
8
.
20
Raschke
RA
Golihare
B
Wunderlich
TA
et al
.
A computer alert system to prevent injury from adverse drug events: development and evaluation in a community hospital
.
JAMA

1998
;
280
:
1317
20
.
21
Bates
DW
O'Neil
AC
Boyle
D
et al
.
Potential identifiability and preventability of adverse events using information systems
.
J Am Med Inform Assoc

1994
;
1
:
404
11
.
22
Bates
DW
Burrows
AM
Grossman
DG
Schneider
PJ
Strom
BL
.
Top-priority actions for preventing adverse drug events in hospitals: recommendations of an expert panel
.
Am J Health Syst Pharm

1996
;
53
:
747
51
.
23
Bates
DW
.
Medication errors: how common are they and what can be done to prevent them?
Drug Safety

1996
;
5
:
303
10
.
This work was supported in part by the Risk Management Foundation, Boston, Massachusetts.