The Utility of Adding Retrospective Medication Profiling to Computerized Provider Order Entry in an Ambulatory Care Population

Abstract

Background: We assessed whether medication safety improved when a medication profiling program was added to a computerized provider order entry system.

Design: Between June 2001 and January 2002 we profiled outpatients with potential prescribing errors using computerized retrospective drug utilization software. We focused primarily on drug interactions. Patients were randomly assigned either to Provider Feedback or to Usual Care. Subsequent adverse drug event (ADE) incidence and other outcomes, including ADE preventability and severity, occurring up to 1 year following the last profiling date were evaluated retrospectively by a pharmacist blinded to patient assignment.

Measurements: Data were abstracted using a study-designed instrument. An ADE was defined by an Adverse Drug Reaction Probability scale score of 1 or more. Statistical analyses included negative binomial regression for comparing ADE incidence.

Results: Of 913 patients in the analytic sample, 371 patients (41%) had one or more ADEs. Incidence, by individual, was not significantly different between Usual Care and Provider Feedback groups (37% vs. 45%; p = 0.06; Coefficient, 0.19; 95% CI: −0.008, 0.390). ADE severity was also similar. For example, 51% of ADEs in the Usual Care and 58% in the Provider Feedback groups involved symptoms that were not serious (95% CI for the difference, −15%, 2%). Finally, ADE preventability did not differ. For example, 16% in the Usual Care group and 17% in the Provider Feedback group had an associated warning (95% CI for the difference, −7 to 5%; p = 0.79).

Conclusion: Medications safety did not improve with the addition of a medication profiling program to an electronic prescribing system.

Introduction

Adverse drug events (ADEs) are unintended medication-related events that harm patients and waste health care resources.1,2 ADEs are also common. In one study, by Classen et al., ADEs complicated approximately 2.4% of hospital admissions, increased costs and raised mortality risk.3 And, in a meta-analysis of 39 studies from U.S. hospitals, Lazarou et al. estimated that the incidence of serious and fatal adverse drug reactions in hospitalized patients, including patients hospitalized because of an adverse reaction, was about 7%.4 More recently, ADE incidence among Medicare enrollees in the ambulatory setting was estimated at about 50 per 1,000 person-years, with preventable ADEs accounting for just under 14 per 1,000 person-years5; there were also consequential increases in health care costs.6

While not all ADEs can be predicted or eliminated, many are preventable.5,7–9 Electronic order entry shows promise for improving prescribing practices and reducing medication errors, especially when coupled with related decision support.10,11 Such systems are not perfect, however. For example, in a seminal study by Bates and colleagues, preventable adverse drug events decreased by only 17% after a computer order entry system was introduced.11 Importantly, information systems have gaps where errors may occur either due to the technology itself or to how the technology is used.10,12–14 Such suboptimal effects may mitigate as information technology evolves—perhaps with improved decision support that generates proportionally more actionable alerts15—but meanwhile other approaches to improve medication safety need to be considered.

One possibility is to add an additional technology to an electronic prescribing system in order to capture overlooked prescribing errors. In that regard, drug utilization review is a method variously utilized by Medicaid and private concerns in an attempt to alter prescribing and improve medication safety and/or to reduce pharmaceutical-related costs.16–18 The technology includes prospective interventions (i.e., before dispensing) or retrospective reviews (i.e., after dispensing). For example, computerized retrospective drug utilization review typically uses claims-based information to develop post-dispensing medication profiles based on pre-determined criteria. A profile might involve a warning that a patient has duplicate therapy (i.e., two drugs from the same drug class) or has two drugs that interact. The medication profiles are run at intervals and are generally reviewed by pharmacists and/or overseeing physicians for relevance. Prescribing clinicians, who are usually apprised of pertinent information by letter, have the responsibility of determining any appropriate clinical action.16,17

We speculated that adding a retrospective medication profiling system with warnings about possible prescribing errors might help fill in gaps that could bypass an electronic order entry system. More specifically, we hypothesized that the two technologies together would reduce the incidence of drug-related adverse events, along with improving other pertinent safety outcomes. We present details of our study, below.

Methods

Study Site

The study was conducted within a Southern California VA health care system, affiliated with a nearby University of California medical school. It has numerous training programs, with outpatient clinics located at the main hospital and at various sized ambulatory care centers. At the time of the study, it served approximately 80,000 patients and dispensed, by mail or on-site, over 1 million prescriptions per year.19 In fiscal year 2001, there were nearly 8,500 admissions to the health care system's tertiary care hospital and almost 910,000 outpatient visits, of all types (data from Austin VHA Support Service Center [VSSC]).

The Electronic Medical Record and Computerized Provider Order Entry

The health care system uses the Veteran's Healthcare System's electronic medical record, known as the Computerized Patient Record System (CPRS). Clinicians enter prescriptions via computerized provider order entry that includes embedded order checks (drug alerts) for, as examples, drug-drug interactions, drug (class) duplications, and drug-allergy warnings. These appear in a pop-up box when providers renew or enter new prescriptions.19,20 Alerts are labeled as significant or critical, with the latter requiring a notation to continue prescribing. There were approximately 2,000 drug-drug interaction rules embedded at the time of our study.19 Pharmacists typically review orders and relevant alerts before processing the prescription for either mail or window dispensing.

Retrospective Drug Utilization Review and Medication Profiling

For our study, we licensed a proprietary computerized retrospective drug utilization review program (RationalMed™) that extracted and utilized archived data from the Veterans Health Information Systems and Technology Architecture (VISTA) system. The archived data were in electronic VISTA files that included prescription as well as patient and clinical information. From these data, the computerized drug utilization review system developed medication profiles, prioritizing patients who might be at heightened risk for selected types of prescribing errors, and including all patients receiving medications within a specified time frame. The program allowed for setting parameters such as the number of profiles to be run at any one time. Each profile contained one or more possible prescribing errors (also known as “conflicts”), among other data (see below). Profiles were based on prescription and clinical data approximately 30 to 45 days old.

Our study focused on possible prescribing errors involving drug-drug and drug-disease interactions and, to a nominal extent, drug duplications. The categories were similar to those noted by Hennessey et al. in studying six Medicaid programs.21 As general examples, a conflict might inform clinicians that a non-steroidal anti-inflammatory agent may worsen renal insufficiency or that verapamil may interact with simvastatin, potentially leading to a toxic effect. Specific therapeutic recommendations were not provided. Diseases or conditions were based on extracted data (e.g., ICD codes); although in some cases the system extrapolated certain medications as indicative of specified conditions/diseases. For example, a patient on an oral hypoglycemic drug would be considered to have diabetes mellitus.

The computerized medication profiling program began in June 2000 and continued into January 2001. We ran the profiling system every 2 weeks during this period, on a specified day (the profiling date), attempting to include approximately 60 consecutive, eligible outpatients in each 2 week period in order to achieve the requisite analytic sample size in the planned time frame while attempting to include the higher priority conflicts (i.e., those with the higher likelihood of having clinical relevance). The numbers of profiles included in any one run varied somewhat, depending on the number of available profiles and number of eligible patients. Profiles underwent brief review by the study coordinator (PB, a clinical pharmacist), and were excluded if the patient was no longer alive, was an inpatient, or was under the direct care of the principal investigator (PAG), or if the profile was an exact duplicate of an earlier profile. Possible prescribing errors (“conflicts”) were excluded as follows: (1) a medication was not listed as “active” (an active prescription referring to a prescription entered in CPRS, with or without refills, that has not expired or been discontinued); and/or (2) based on a limited number of predetermined rules for exclusion (e.g., a HMG CoA Reductase Inhibitor (“statin”)—peptic disorder interaction or an insulin-aspirin interaction). The review did not assess clinical relevance but involved checking for life or limb-threatening prescribing errors (e.g., a contravention to a black box warning) so that, if one were to be found, urgent provider notification could be initiated (regardless of patient assignment). An independent clinical pharmacist completed a second review for any perceived life and limb threatening prescribing errors, as well.

Patients whose profiles were included were assigned by the study coordinator (PB), using a preprinted list developed by random number generator (by the statistician), to Provider Feedback or to Usual Care groups. Patient assignment was to one group only, even if the patient had multiple profiles generated over the course of the profiling phase. For patients in the Provider Feedback group we attempted to identify relevant clinicians so that we could inform them of the conflicts. The letter was sent from a central location by intra-office mail. The letter contained background on the educational and informative nature of the program and a summary of potential conflict(s). Included as well were a hardcopy medication profile and a voluntary feedback form. The medication profile itself included selected demographics, as well as co-morbid conditions, potential therapeutic issues (i.e., conflicts) and literature citations related to the therapeutic problem(s). An electronic mail message inquiring whether the information was received was sent approximately one week after the letter. We offered to send a copy, if not. We did not formally assess whether the information was useful to providers but we monitored whether a provider at least received some information as determined by provider response to hardcopy or electronic mail or by receipt of electronic mail. Of note, profiles and letters were not integrated into the VA's electronic medical record (CPRS) and would not have been available to clinicians (or pharmacists) while viewing the electronic medical record.

The study received approval from the VA facility's Institutional Review Board.

Determining Adverse Drug Events and Related Outcomes

Our primary hypothesis was that medication profiling and provider feedback would decrease ADE incidence. Secondarily, we speculated that medication profiling would decrease the number of preventable ADEs as well as decrease the severity of ADEs since they might be discovered at an early stage, before more severe harm occurred.

To evaluate these and other end points associated with conflicts and suspected ADEs, another clinical pharmacist, blinded to the profile/patient assignment, retrospectively reviewed and abstracted patient and clinical data using a study-designed hardcopy instrument. The review of the electronic medical record (CPRS) included an assessment of clinician notes (outpatient and inpatient), listed adverse drug reactions and allergies, and any other data deemed appropriate by the reviewer in judging that an ADE had occurred (e.g., laboratory tests, medication history) for all randomized patients, with data abstracted for up to 1 year after the date of the last generated profile. The reviewer was instructed to specifically look for any suspected ADEs that might be directly associated with the listed conflict(s). The overall goal was to identify reactions that were plausibly and causally related to pharmaceutical use—particularly related to the drugs identified in the conflicts on the medication profiles—leading to a change in therapy or clinical approach or to harm.1,22,23

Among its items, the study-designed instrument included the 10-item Adverse Drug Reaction Probability (ADRP) Scale,24 used to assess the likelihood of an ADE. Scores on the ADRP scale range from −4 to +13: below 1 indicates a doubtful ADE; 1 to 4, possible; 5 to 8, probable; and 9 or higher, definite. The reviewer entered the individual components of the scale but was not asked to complete the scoring. ADE severity was categorized by the reviewer as: 1) Laboratory or test abnormality; 2) Symptoms: not serious or serious, with the latter defined as requiring intervention to prevent permanent impairment or damage; 3) Disability, cognitive or physical; or 4) Death. ADE preventability was determined by the presence of an associated conflict (i.e., a possible prescribing error as documented on the profile), a relevant order check for a drug-drug interaction (based on a printed list of drug-drug interaction order checks available at the time) or the finding of a specific warning in the medical record (e.g., a pharmacist note). The instrument also allowed the reviewing pharmacist to judge whether a clinical action (e.g., change in therapy) might have prevented the event.

Sample Size Determination

We originally estimated that 20% of our outpatient sample would have an annual average rate of 1.1 ADEs or, conversely, that 80% of outpatients would not have an ADE.25 Our goal was to include at least 900 individuals in the analytic sample to allow 90% power to detect a 40% reduction in the ADE rate, assuming a type 1 error rate (alpha) of 5%.

Data Analysis

Based on prior review of literature, an ADRP score of 1 or greater (i.e., a possible, probable, or definite ADE per the scale) was the predetermined basis for an ADE.25,26 In assessing ADE incidence, the primary outcome, we anticipated that the distribution would be Poisson in nature. However, because of the over dispersion in the data, to compare incidence of ADEs between Usual Care and Provider Feedback groups we used negative binomial regression to account for varying risks of events across the patient population.27 In analyzing the results, we used the number of individuals as the denominator in determining rates because about 89% of the subjects in the study had a single profile and of the remaining 11%, there were similar numbers in both study groups who had multiple (two or three) profiles. Therefore, under these circumstances, event rates should be reasonably approximated by using sample sizes in the denominator of these calculations. For comparing severity of outcomes (e.g., laboratory or test abnormality, symptoms), the worst outcome was used when multiple outcomes were listed. For that analysis, we used the Cochran-Armitage Trend Test. For most other analyses we used chi-square tests (e.g., when assessing preventability) and the analysis of variance (ANOVA). When comparing complications from ADEs, we used an exact procedure based on the multinomial distribution to compute the p-value. Two-sample t-tests for quantitative data were used when the appropriate transformation to normality was found. Otherwise, we used the Mann-Whitney test (e.g., ADE incidence among individuals with ADEs).

No adjustment for multiple evaluations of secondary end points was incorporated into the analysis. Although we considered a cluster-adjustment analysis,28 when appropriate, this was not necessary due to a lack of significant findings as such an analysis only reduces statistical significance.

Results

Study Sample

We initially collected data on 932 patients. Because we later estimated that approximately 1 month was necessary for the letter and profiles to be received and acted upon by designated providers, we excluded 18 patients post-hoc, in the pre-analysis stage, because in retrospect they appeared to have left VA within 1 month of, or before, the profiling date (10 died, 4 were institutionalized in a non-VA setting, 4 had left the VA for an unknown reason). We excluded one patient due to an initially-missed exclusion criterion. In three patients initially included, we excluded four subsequent profiles for similar reasons.

Our overall analytic sample included 913 of 932 patients (98%) with 458 in Provider Feedback and 455 in Usual Care groups.

Table 1 presents information and selected characteristics of patients in the analytic sample, with demographics derived from patient data at baseline or within 1 year from the first profile assignment. More specifically, the 913 patients had 1,024 eligible profiles (514 Usual Care, 510 Provider Feedback) with an overall mean of 1.1 profiles (range 1 to 3, SD 0.35) and no difference in means between groups (p = 0.49). The profiles contained 1,452 conflicts (Usual Care 738; Provider Feedback 714) for an overall mean of 1.6 conflicts per person. Of the conflicts, 57% were drug-drug interactions, 42% drug-disease interactions and 1% drug duplications. There was no significant difference in the mean number of conflicts (p = 0.31) or types of conflicts (p= 0.39) between groups. Over the course of the assessment period, 817 (90%) persons had clinical information available for at least 1 year from the first profiling date (data not in Table).

Table 1

Selected Patient and Profiling Information*

DEMOGRAPHIC CHARACTERISTICS	Usual Care (n = 455)	Provider Feedback (n = 458)
Age: Mean (SD)	67.3 (10.6)	67.2 (11.0)
Gender, n (%)
Male	447 (98%)	446 (98%)
Female	8 (2%)	12 (3%)
VA Primary Care Assignment n (%)
Individual Primary Care Physician	230 (53%)	237 (54%)
Specific Team with Providers	179 (41%)	178 (40%)
Unassigned	28 (6%)	26 (6%)
VA Medical Subspecialty Care, n (%)
Yes	380 (84%)	394 (86%)
VA Practice Site Assignment for Primary Care, n (%)
Tertiary Care Center	154 (36%)	165 (38%)
Large Ambulatory Care Center	145 (34%)	141 (33%)
Other Ambulatory Care Centers/Community Clinics	130 (30%)	125 (29%)
Prior ADE Mean (SD)	1.1 (1.4)	1.2 (1.5)
Documentation of Prior ADE (1 or more), n (%)†	244 (54%)	275 (60%)
Number of Medications Mean (SD)	11.1 (6.7)	11.1 (7.0)
Selected Health Conditions (highest incidence), %
Hypertension	79%	78%
Diabetes	49%	49%
Ischemic Heart Disease	36%	38%
Major Depressive Disorder	31%	32%
Arthritis	30%	30%
GERD†	31%	25%
Profiles
Total (n = 1,024)	514	510
Mean (SD)	1.1 (0.35)	1.1 (0.36)
Conflicts
Total (N = 1,452)	738	714
Mean (SD)	1.6 (1.0)	1.6 (0.90)
Conflict Types (n = 1,452)
Drug-Drug Interactions	433 (59%)	398 (56%)
Drug-Disease Interactions	300 (41%)	308 (43%)
Drug/Drug Class Duplications	5 (1%)	8 (1%)

*

Percentages are rounded and represent valid percentages. Demographic information based on patient data at baseline and/or within first year from first profile assignment. GERD refers to Gastroesophageal Reflux Disease. Prior ADE refers to a patient having one or more ADEs documented in CPRS prior to the first profiling date.

†

Prior ADE: p = 0.05; GERD: p =0.03; All other p > 0.05

ADE Determination and Incidence

Three hundred and seventy six individuals (376/913, or 41%) had 571 suspected adverse events (range 1 to 6) of which 565 events (in 371 individuals, or 41%) had an ADRP score of 1 or more and hence met criteria for inclusion. The mean ADRP score was 5.2 for all eligible ADE, with no difference between the two groups (Usual Care 5.3, Provider Feedback 5.1; p = 0.32).

Table 2 illustrates that of the total number of 565 eligible events, 255 ADEs occurred among 166 persons (of 455) in the Usual Care group and 310 ADEs occurred among 205 persons (of 458) in the Provider Feedback group. The percentage of individuals who had an ADE was not significantly different between groups, though a trend (p = 0.06) was noted toward fewer affected persons in the Usual Care group (37%) when compared to the Provider Feedback group (45%). However, the mean number of events among the 371 persons who had an ADE was not statistically significant (Usual Care 1.54; Provider Feedback 1.51; 95% CI: −0.16 to −0.22; p = 0.54). (data not in Table.)

Table 2

Adverse Drug Events: Incidence, Attribution, Severity, and Preventability*

Outcome Measures	Usual Care	Provider Feedback	95% CI for Difference	P-Value
Totals
Individuals in each group (total = 913)	455	458	—	—
Number of ADEs (total = 565)	255	310
Number of individuals with ADEs (total = 371)	166	205	—
ADE incidence† (total sample 913 individuals)
ADE incidence per group n (%)	166/455 (37%)	205/458 (45%)	−15, −2	—
ADE incidence per patient mean (SD)	0.57 (0.91)	0.68 (0.97)	−0.23, 0.01	—
Coefficient for Group (95% CI)	0.19 (−0.008, 0.390)		—	0.06
ADE Severity‡ (563 events)
Laboratory or test abnormality, n (%)	76 (30%)	76 (25%)	−2%, 13%	0.53
Symptoms (Not Serious), n (%)	130 (51%)	178 (58%)	−15%, 2%
Symptoms (Serious), n (%)	46 (18%)	54 (18%)	−6%, 7%
Disability/Death, n (%)	2 (1%)	1 (0%)	−1%, 3%
Total ADE	254	309
Other ADE Severity Measures (565 events)
Hospitalizations attributed to an ADE, n (%)	18 (7%)	20 (7%)	−4%, 5%	0.77
Emergency Room visits attributed to an ADE, n (%)	28 (11%)	33 (11%)	−5%, 6%	0.90
ADE Preventability (564 events)
Warning (profile), n (%)	37 (15%)	46 (15%)	−6%, 6%	0.90
Warning (profile, order check, or pharmacist), n (%)	40 (16%)	51 (17%)	−7%, 5%	0.79
Warning (any of above or clinical action), n (%)	69 (27%)	75 (24%)	−4%, 10%	0.45
ADE Attribution§ (565 events)
Single Drug, n (%)	194 (76%)	246 (79%)	−10%, 4%	0.71
Drug-Drug interaction, n (%)	47 (18%)	47 (15%)	−3%, 10%
Dosing or other prescribing error, n (%)	9 (4%)	9 (3%)	−3%, 4%
Patient error, n (%)	5 (2%)	7 (2%)	−3%, 2%
Total ADE	255	310

*

Percentages are rounded and represent valid percentages. Denominators are based on the following: Of 913 individuals in the analytic sample (Usual Care 455, Provider Feedback 458), 371 persons (Usual Care 166; Provider Feedback 205) had 565 eligible adverse drug events (Usual Care 255, Provider Feedback 310). An ADE was defined as having an Adverse Drug Reaction Probability Scale score of 1 or more (see text for details). Some data may be missing from selected analyses, as noted (e.g., two missing data points for assessing overall ADE severity, one for ADE preventability); CI refers to Confidence Interval.

†

For ADE incidence among eligible population: Statistical comparison was by negative binomial regression (see text for details) and the coefficient and its 95% Confidence Interval (CI) were derived accordingly; other presented 95% CI (e.g., percentages and means) were based on unadjusted data.

‡

In determining severity, the worst outcome was used when there were multiple outcomes listed. Hospitalizations and Emergency Room visits include initial or subsequent events attributed to an ADE.

§

ADE attribution refers to the causative agent(s) or prescribing error associated with the particular ADE.

ADE Severity and Preventability

Most events involved laboratory or test abnormalities (27%) or symptoms that were categorized as not serious (55%). Serious symptoms accounted for 18% of outcomes; disability/death accounted for less than 1% (data not in Table). Table 2 presents the non-significant difference in severity determinations between the two groups as well as the non-significant differences in number of hospitalizations and emergency room visits attributed to adverse drug events.

Approximately 26% of ADEs were judged potentially preventable, with only about 16% preventable in terms of having an order check or warning letter (data not in Table). There were no significant differences in the proportion of possibly preventable events, variously defined, between groups (Table 2).

Other ADE Measures

Seventy-eight percent (78%) of ADEs were attributed to a single drug, 17% to a drug-drug interaction, 3% to a dosing or other prescription error and 2% to patient error (data not in Table). Table 2 presents the non-significant differences across the two groups. Of the suspected ADEs found by the reviewer, 95% of these were documented by clinicians in their notes in the electronic medical record (data not in Table).

Table 3 alphabetically lists the types of drugs most commonly associated, either alone or in combination, with adverse events. Table 4 lists the complications of the ADE, per reviewer assessment, in 13 categories. There was no significant difference between groups (p = 0.13).

Table 3

Types of Drugs Commonly Associated with Adverse Drug Events (in Alphabetical Order)

Alpha-blockers/Related (CV150)

Analgesics (opioid and non-opioid) (CN101, CN103)

Angiotensin converting enzyme inhibitors (CV800)

Anticoagulants (e.g., warfarin, heparin, heparinoids) (BL110)

Anticonvulsants (CN400)

Antidepressants (tricyclics, non-tricyclics) (CN609, CN601)

Lipid Lowering Agents (CV350, VT103)

Beta-blockers/Related (CV100)

Calcium-channel blockers (e.g., verapamil, diltiazem, dihydropyridine agents) (CV200)

Diuretics (e.g., loop, thiazides, potassium-sparing) (CV701, CV702, CV704)

Glucocorticoids (HS051)

Nonsalicylate nonsteroidal anti-inflammatory agents (MS102)

Oral hypoglycemic agents (HS502)

Drugs were categorized based on the VA's Drug Classification codes (in parentheses, as of April 2005). For current categorization, please see http://www.pbm.va.gov/NationalFormulary.aspx.

Table 4

Classification of ADE Complications

Classification of ADE Complications (565 ADE)	Usual Care (255 ADE)	Provider Feedback (310 ADE)
Cardiovascular	59 (23%)	54 (17%)
Renal	40 (16%)	50 (16%)
Gastrointestinal	40 (16%)	44 (14%)
Central Nervous System	22 (9%)	39 (13%)
Hematological	34 (13%)	26 (8%)
Metabolic	19 (8%)	18 (6%)
Respiratory	10 (4%)	19 (6%)
Genitourinary	5 (2%)	13 (4%)
Cutaneous	7 (3%)	7 (2%)
Allergic	5 (2%)	9 (3%)
Musculoskeletal	2 (1%)	9 (3%)
Psychological	3 (1%)	7 (2%)
Other	9 (4%)	15 (5%)

Percentages are rounded and represent valid percentages. Information based on the type of adverse drug event, by system affected (e.g., gastrointestinal) or by mechanism (e.g., allergic) per reviewer. Comparison across the two groups by exact calculation of p-value based on the multinomial distribution, p = 0.13.

Discussion

We added a retrospective medication profiling to a computerized provider order entry system with order checks, focusing primarily on potential drug-drug and drug-disease interactions in an ambulatory care population. We found that the two systems combined did not change the overall incidence, severity, or preventability of subsequent adverse drug events. While a statistical trend favoring Usual Care regarding ADE incidence was noted, the difference in mean events was nominal and adjustment for clustering would have further reduced statistical significance. Hence, the statistical trend is likely due to the large sample along with the analytic methodology employed. We briefly discuss the possible reasons for our non-significant findings, below.

First, a stand-alone, broad-based retrospective drug utilization program may not be particularly useful in reducing adverse events. Although these programs have now been utilized for over 2 decades to our knowledge there has not been a properly designed and implemented large-scale randomized control trial published in the peer-reviewed literature to assess their impact on relevant clinical events. While some data indicates that more targeted medication management programs—meaning those focused on a particular problem or set of problems—may change prescribing behavior, 29–31 whether such programs improve outcomes in an outpatient population is questionable.16,17,21 Indeed, the most comprehensive assessment on stand-alone retrospective drug utilization programs on clinical outcomes in outpatients found a null effect.21 Such programs do have intuitive merit, but there are issues that make the impact of such programs debatable, such as lack of timeliness of prescribing information and relevance of warnings.17

That said, Javitt et al. recently published a randomized study testing a sentinel system with over 1,000 rules that involved directed recommendations (e.g., “Stop metformin in patients with renal insufficiency” or “Add ACE inhibitor in congestive heart failure”). This study found that such a system—which also included recommendations that did not directly involve drug therapy—improved adherence to clinical guidelines and decreased hospital admissions and per member costs.32 Thus, it may be that a comprehensive medication profiling system with specific recommendations, rather than with warnings alone, would have a more positive additional clinical benefit. In this regard, further study may be warranted.

Second, the medication profiling program may not have provided additional benefit because it duplicated, in some respects, the function of the computerized system's order checks. More specifically, both advised on drug-drug interactions, although our intervention included additional drug-disease interactions. VA order checks also include drug duplication and drug-allergy alerts. Perhaps different results would have accrued had we focused on a different set of potential prescribing errors (e.g., dosing or durational problems).

Third, focusing on adverse outcomes due to drug interactions may not have been optimal in terms of finding a change in outcomes. A recent study by Lasser et al. suggests that drug interactions are probably an infrequent cause of adverse events, finding that less than 1% of outpatients were subsequently harmed despite receiving a prescription with a black box warning for a drug-drug, drug-disease, or drug-laboratory interaction.33 Thus, even if providers were to have changed therapy in accordance with warnings—an effect which we did not systematically measure—a difference in incidence or severity of adverse outcomes may have been missed because the subsequent events, were such to occur, were too infrequent.

Fourth, our non-significant results may have been due to missed opportunities to impart clinically useful information. One important consideration was in trying to ascertain and contact the supervising and/or treating provider(s). In anticipating the general problem, we contacted more than one clinician, when applicable. Over 1,000 letters were mailed (referring to the analytic sample, Provider Feedback Group) with a mean of about two letters per profile. Moreover, to help assure contact with providers, we utilized more than one contact method. Still, we recognize that in some instances the identified clinician(s) was not the treating clinician. Another possible issue is that we did not pre-review conflicts for relevance. However, Hennessey et al.21 found no effect on outcomes using data from Medicaid programs that typically review alerts prior to contacting providers16 so we are not convinced that pre-review would make a difference. Finally, substantive delays between prescribing and profile receipt occurred. Not all claims-based systems are subject to similar delays18 but the less timely the information, the less chance it will be relevant, as drug therapy may already have been changed17,34 and/or clinical problems may have resolved. While clinicians frequently over-ride real-time electronic alerts,20,35 it is unclear how often clinicians find retrospective warnings useful. In our study we did not comprehensively assess subsequent behavioral changes after received warnings so we do not know how often clinicians found the information useful. In retrospect, this would have been helpful in determining the effect of the profiling.

Limitations

We utilized the program within a single public health care system, albeit a large and diverse one, and this may affect generalizability to other settings. Providers were not chosen as the unit of randomization and this may have reduced our ability to detect differences between groups. We likely excluded some conflicts when drugs were still being taken (e.g., the prescription had expired) and included some when drugs were no longer being used (i.e., not yet cancelled). We used data abstracted by a single reviewer and there are likely to be misclassified, miscoded, and misattributed events. However, the randomized protocol as well as a blinded assessment method reduces the likelihood of systematic bias. We undoubtedly missed some events that were observed outside the VA or that were due to drugs prescribed by non-VA providers. On the other hand, 95% of the ADEs found by the reviewer were already documented in the electronic medical record, suggesting that we captured many ADEs that were clinically meaningful. Our analytic sample development might have introduced bias but a post-hoc evaluation of 15 patients without predetermined exclusion criteria (of 19 excluded) indicated that one ADE was missed. Conversely, in 16 patients who had no substantive pharmacy utilization during the assessment period one ADE was included. These minor differences would not qualitatively affect our results.

Conclusion

We found that adding a retrospective medication profiling program to computerized provider order entry with order checks did not alter ADE incidence, severity, or preventability. To improve medication safety, health care systems that utilize electronic prescribing may be better served by increasing the usability of that technology along with the functionality of embedded order checks.

The authors gratefully acknowledge funding support for this study from VA Health Services Research and Development, SAF-98-144-1 (“Patient Profiling and Provider Feedback to Reduce Adverse Drug Events”).

We thank Mrs. Patricia Preyer for clerical support and Ms. Alissa Simon for work on abstraction forms. We are grateful to Jonathan Nebeker, MD and Steven Asch, MD for their thoughtful comments on the manuscript. We also thank Jocelyn Hayes-Hunter, PharmD for her efforts on the grant proposal and for comments on the manuscript. Finally, we appreciate the help of our anonymous reviewers.

Substantive data contained in this manuscript has not been presented or published previously. Some preliminary data (primarily economic) was presented at the VA HSR and D national meeting in 2004.

Other than providing technical support and services and a discounted licensing fee for the computerized retrospective drug utilization review program (RationalMed™), the licensor and/or its oversight company(ies) were not engaged in the study itself or in the writing or approval of the manuscript.