Overrides of medication-related clinical decision support alerts in outpatients

Abstract

Background Electronic prescribing is increasingly used, in part because of government incentives for its use. Many of its benefits come from clinical decision support (CDS), but often too many alerts are displayed, resulting in alert fatigue.

Objective To characterize the override rates for medication-related CDS alerts in the outpatient setting, the reasons cited for overrides at the time of prescribing, and the appropriateness of overrides.

Methods We measured CDS alert override rates and the coded reasons for overrides cited by providers at the time of prescribing. Our primary outcome was the rate of CDS alert overrides; our secondary outcomes were the rate of overrides by alert type, reasons cited for overrides at the time of prescribing, and override appropriateness for a subset of 600 alert overrides. Through detailed chart reviews of alert override cases, and selective literature review, we developed appropriateness criteria for each alert type, which were modified iteratively as necessary until consensus was reached on all criteria.

Results We reviewed 157 483 CDS alerts (7.9% alert rate) on 2 004 069 medication orders during the study period. 82 889 (52.6%) of alerts were overridden. The most common alerts were duplicate drug (33.1%), patient allergy (16.8%), and drug–drug interactions (15.8%). The most likely alerts to be overridden were formulary substitutions (85.0%), age-based recommendations (79.0%), renal recommendations (78.0%), and patient allergies (77.4%). An average of 53% of overrides were classified as appropriate, and rates of appropriateness varied by alert type (p<0.0001) from 12% for renal recommendations to 92% for patient allergies.

Discussion About half of CDS alerts were overridden by providers and about half of the overrides were classified as appropriate, but the likelihood of overriding an alert varied widely by alert type. Refinement of these alerts has the potential to improve the relevance of alerts and reduce alert fatigue.

Meaningful Use, Safety, Alert Fatigue, Efficiency, Electronic Health Records, Electronic Prescribing

Introduction

Clinical decision support (CDS) represents an important tool for promoting patient safety and quality of care,1 and medication-related CDS is especially useful.2 Physicians can choose to accept or override nearly all CDS alerts, and understanding how physicians respond to these alerts is critical to obtaining benefit from them.

Simply having an electronic health record appears to have little impact on quality by itself.3,4 However, a variety of randomized controlled trials clearly demonstrate that CDS can improve care.5,6 To improve both quality and safety, key system functionality must be both implemented and used, though prior work suggests that uptake of such functionality has been highly variable among adopters of electronic health records.7

Therefore, since the start of 2011, providers have had the opportunity to be rewarded by the government for achieving core objectives of ‘meaningful use’ of certified systems in Stage 1 of this program, and Stage 2 will raise the bar further.8 Relatively few data are available regarding the impact of meaningful use of medication-related CDS, though this does appear to be one of the first areas in which conversion to an electronic medical record may improve care, especially safety.9

Overall, use of electronic prescribing has the potential to improve safety, efficiency, and quality, though a variety of work suggests that this does not occur in all instances.10,11 For example, data from our center show that there is substantial room for improvement in the meaningful use of electronic medication lists, a basic component of electronic prescribing. We studied more than 4000 clinicians and found that only 66% appeared to be meaningfully using medication lists.12 Furthermore, although we have extensively modified the CDS system at our center to promote user acceptance and reduce alert fatigue,13 we have continued to observe a high level of CDS alert overrides.14 With the implementation of electronic health records and CDS, we now have an additional resource: logs capturing individual providers' responses to CDS relating to medication safety and efficiency. While these have been available in many electronic health records, others do not track them, and we believe they have been underutilized.

A better understanding of what alerts are delivered, whether providers override them, and what responses providers offer when they respond to them may provide insights into how the alerts themselves should be delivered, and also into policies around meaningful use. Therefore, we performed a study with the following three aims: (1) to characterize the types and numbers of alerts delivered in the ambulatory setting; (2) to characterize the frequency with which they are overridden; and (3) to describe providers' reasons for overriding them and the appropriateness of those reasons.

Materials and methods

Study setting and design

We obtained CDS alert override rates and the system-coded reasons for overrides selected by providers at the time of prescribing from outpatient clinics and ambulatory hospital-based practices at a large academic health care center over a period of 3 years from January 1, 2009 to December 31, 2011. All provider types were studied, including attending physicians, house-staff, and non-physicians with prescribing authority. We included the following eight types of alerts: patient allergies, drug–drug interactions, duplicate drugs, drug–class interactions, class–class interactions, age-based recommendations, renal recommendations, and formulary substitutions. The study was approved by the Partners Human Research Committee.

Electronic prescribing system

Our electronic prescribing system is an application within the Longitudinal Medical Record (LMR), an electronic medical record that was developed in-house and implemented in 2000. The LMR is used for multiple functions including writing all clinical orders, note writing, access to laboratory, radiology, and pathology reports, and electronic prescribing, among other functions. Each prescription is placed sequentially in the LMR and constitutes a separate order for drug contraindication checking by the CDS system. The current medication list is continually updated in real time as new medications are entered or discontinued. The computerized prescribing application ensures the completeness of all prescriptions by providing clinicians with a coded dictionary of medication names, available strengths including default doses, and required fields for drug dose, number of pills or units dispensed, and refills. Prescriptions cannot be signed and completed until all relevant information has been entered by the prescriber. Additionally, the application ensures legibility by printing or electronically sending typed prescriptions to pharmacies.

Appropriateness criteria

Through detailed chart reviews of alert override cases and selective literature review, a group of four trained clinicians (one physician (KCN), two pharmacists (SPS, DLS), and one nurse (IC)) developed criteria for appropriateness for each of the alert domains. The final criteria, shown in online supplementary appendix A, were modified iteratively as necessary until consensus was reached on all criteria. Group members independently reviewed random samples of 30 overrides from each alert domain for appropriateness in order to ensure that inter-rater reliability was greater than 80%.

A random sample of 100 alert overrides was then selected from each of the following alert domains: patient allergies, drug–drug interactions, age-based recommendations, renal recommendations, and formulary substitutions. We also selected a random sample of 100 alert overrides combined for duplicate drug, drug–class interaction, and class–class interaction domains. This provided a total sample size of 600 overrides for analysis of alert appropriateness. Each group member analyzed 1–2 of these samples to assess whether the overrides were appropriate.

Outcomes

Our primary outcome was the rate of CDS alert overrides. Our secondary outcomes were the rate of overrides by alert type, the reasons cited for overrides at the time of prescribing, and the appropriateness of the overrides.

Analysis

We compared CDS alert override rates and the appropriateness of overrides among the different alert types in our sample. Comparisons are presented as counts with percentages, and p values were calculated using the χ² test.

Results

A total of 1718 providers received 157 483 CDS alerts (7.9% alert rate) on 2 004 069 medication orders during the three-year study period. Provider types included staff physicians (N=855, 49.8%), house staff (N=616, 35.9%), nurse practitioners (N=79, 4.6%), and physicians' assistants (N=35, 2.0%). A total of 133 providers (7.7%) did not have a provider type listed. Providers overrode 82 889 (52.6%) of the alerts generated. Table 1 shows alert and override rates as well as the most common reason cited for overrides at the time of prescribing in each of the eight alert types studied. The most common alert types were duplicate drug (33.1%), patient allergy (16.8%), and drug–drug interaction (15.8%). Table 2 shows the annual alert override rates over the three-year study period. The alert rates and the alert override rates decreased in year 2 (p<0.001) and increased in year 3 (p<0.001). Although these differences were highly statistically significant because of the large sample size, the differences were small.

Table 1

Breakdown of alert overrides

	Total alerts		Alert overrides
Alert type	N	(%)	N	(%)	Most common reason for override
Patient allergy	26 408	16.8	20 430	77.4	Patient took previously without allergic reaction
Drug–drug interaction	24 849	15.8	14 966	60.2	Will monitor as recommended
Duplicate drug	52 113	33.1	14 917	28.6	Patient requires different strengths of same drug
Drug–class interaction	19 593	12.4	4782	24.4	Transitioning from one drug to the other
Class–class interaction	4184	2.7	2918	69.7	Patient on long term therapy with combination
Age-based suggestion	10 501	6.7	8297	79.0	Patient has tolerated this drug in the past
Renal suggestion	3890	2.5	3035	78.0	Patient has tolerated this drug in the past
Formulary substitution	15 945	10.1	13 554	85.0	Intolerance/failure of suggested substitution
Total	157 483	100.0	82 899	52.6

Table 2

Breakdown of orders, alerts, and overrides by year

		Alerts		Overrides
Year	Prescriptions (N)	(N)	(%)	(N)	(%)
2009	495 645	49 112	9.9	26 138	53.2
2010	712 057	46 590	6.5	23 789	51.1
2011	796 367	61 781	7.8	32 972	53.4
Total	2 004 069	157 483	7.9	82 899	52.6

We found significant variation in override rates by alert type (p<0.001), with rates varying widely from 24.4% to 85.0%. The most likely alerts to be overridden were formulary substitutions (85.0%), age-based recommendations (79.0%), renal recommendations (78.0%), and patient allergies (77.4%). In our subset of alert overrides analyzed for appropriateness, we found that the appropriateness of alert overrides also varied significantly by alert type (p<0.001), from 12% for renal recommendation alerts, to 92% for medication allergy alerts. Interestingly, overrides of drug–class and class–class interactions were more appropriate (88% and 69% appropriate, respectively) than overrides of individual drug–drug interactions (12% appropriate). Table 3 shows the rates of appropriateness for alert overrides by alert type, and table 4 shows examples of appropriate and inappropriate overrides by alert type. Our inter-rater reliability for assessment of alert appropriateness was very good (κ=0.89).

Table 3

Appropriateness of overrides by alert type

Alert type	Appropriate (%)	Not appropriate (%)	Exclusions* (%)
Patient allergy†	92	8
Drug–drug interaction†	12	88
Duplicate drug‡	82	18
Drug–class interaction‡	88	12
Class–class interaction‡	69	31
Age-based suggestion†	39	60	1
Renal suggestion†	12	85	3
Formulary substitution†	57	43
Average	53	47

*Exclusions due to inability to access patient electronic record or medication list.

†Appropriateness analysis based on a random sample of 100 alert overrides from each of these alert domains.

‡Appropriate analysis based on a combined random sample of 100 alert overrides from these alert domains.

Table 4

Example of appropriate and inappropriate overrides by alert type

Alert type	Example of inappropriate override	Example of appropriate override
Patient allergy	Antibiotic prescribed to a patient with a listed allergy to that antibiotic	Narcotic prescribed to a patient with a listed allergy to a different narcotic
Drug–drug interaction	Warfarin prescribed in conjunction with a fluoroquinolone and no monitoring of INR	Warfarin prescribed in conjunction with a fluoroquinolone and close monitoring of INR
Duplicate drug	Duplicate prescription for an antihypertensive after a dose increase, when the lower dose was not discontinued	Steroid taper prescribed with different doses of the same steroid
Drug–class interaction	ACE inhibitor prescribed to a patient already on a different ACE inhibitor	ACE inhibitor prescribed to a patient already on a different ACE inhibitor only while transitioning from one drug to the other
Class–class interaction	Hypnotic and benzodiazepine prescribed simultaneously	Hypnotic and benzodiazepine prescribed simultaneously only while transitioning from one drug to the other
Age-based suggestion	Long-acting benzodiazepine prescribed as first-line therapy in a benzodiazepine-naïve elderly patient	Short-acting benzodiazepine prescribed at minimum dose as second-line therapy in elderly patient who has tolerated benzodiazepines in the past
Renal suggestion	Metformin prescribed to a patient with creatinine clearance <50	Metformin prescribed to a patient with creatinine clearance >50
Formulary substitution	Prescriber selected proton pump inhibitor that patient has taken in the past instead of choosing formulary substitution	Prescriber selected selective serotonin reuptake inhibitor that patient has been taking long-term instead of choosing formulary substitution

Providers cited a variety of reasons for overriding the CDS alerts, with the most common overall being that the patient had previously tolerated the drug. The reasons cited for alert overrides varied by alert type. For example, the most common reason cited for overriding drug–drug interaction alerts was that the provider ‘will monitor as recommended’ (42%). The most common reason cited for overriding formulary substitutions was ‘failure or intolerance to suggested substitution’ (44%). Table 5 shows override reasons by alert type.

Table 5

Override reasons by alert type

Alert type	Override reason*
Patient allergy	Patient took previously without allergic reaction
	Low risk cross sensitivity
	Will monitor
	No reasonable alternatives
	Other
Drug–drug interaction, drug–class interaction, and class–class interaction	Will monitor as recommended
	Will adjust dose as recommended
	Patient has already tolerated this combination
	No reasonable alternatives
	Other
Duplicate drug	Patient requires different strengths of same drug
	Transitioning from one drug to the other
	Patient on long-term therapy with combination
	Advice from a consultant
	New evidence supports duplicate therapy of this type
	Other
Age-based suggestion	Patient has tolerated this drug in the past
	Advice from a consultant
	New evidence supports therapy of this type
	Other
Renal suggestion	Patient has tolerated this drug in the past
	New evidence supports therapy of this type
	Advice from a consultant
	Other
Formulary substitution	Intolerance/failure of suggested substitution
	Patient has a diagnosis requiring 80 mg atorvastatin
	Patient preference
	Patient currently taking prescribed medication
	Insurance does not allow above suggestion
	Written originally by another physician
	Physician preference
	Other

*Reasons listed in order of decreasing frequency. Counts/percentages not included as many alerts had multiple reasons selected.

Discussion

In this study, we found that about half of CDS alerts were overridden by providers, and the rate of overrides varied substantially by alert type. Both the alert and override rates decreased in year 2 and increased modestly in year 3. Half of the overrides were appropriate, and the proportion that was appropriate varied even more by alert type. Few overrides of renal dosage recommendations, drug–drug interactions, or age-related dosage recommendations were appropriate, while the vast majority of drug allergy, drug–class, duplicate drug, and drug formulary alerts were, suggesting that this last group might be particularly good targets for refinement to reduce alert fatigue by reducing the number of alerts. Notably, the appropriateness of overrides of drug–class and class–class alerts was much higher than that of individual drug–drug alerts. This could indicate that while providers generally agree with alerts based on larger categories of drugs and their potential interactions, they may feel that exceptions do exist for individual drugs. However, the high rates of inappropriate overrides for these individual drug–drug interactions indicate the need for further education and intervention.

Our results are consistent with existing literature, which reports override rates ranging from 50% to more than 90%, with drug–drug interactions being one of the most common alert types to be overridden, although few previous studies have compared such a broad array of alert types, and we believe these results may help organizations consider which ones to focus on.15–18 While the overall override rate in this study is at the lower end of what is reported in the literature, it is higher than the rate we reported from our ambulatory care practices in a prior study in 2006.14 At that time, we found that 33% of interruptive CDS alerts in outpatient systems were overridden, with override rates ranging widely by the type of warning, from 23% for duplicate class warnings to 90% for drug–pregnancy interactions.14 Our current higher overall rate of overrides suggests that the number of alerts that our clinicians experience has probably grown, which could lead to alert fatigue.19–21 There are two possible reasons for this increased number of alerts. First, we may have a lower threshold for alerting. The natural tendency is for committees that manage alerts to add alerts incrementally, and unless an aggressive approach is taken to eliminate unnecessary alerts, more are generally included over time. One recent study examined the impact of tiering alerts for drug–drug interaction to reduce the total number of alerts.22 In that study, override rates with drug–drug interaction alerts were significantly lower with tiered alerts than at the non-tiered site. For the most severe alerts, none were overridden at the tiered site, versus a 66% override rate at the non-tiered site, showing that there were substantial override rates even for serious warnings when tiering was not used. Second, an increase in the number of known drug interactions requiring alerting could also increase the total number of alerts. In fact, our center has engaged in a continuous improvement process to update and increase the number of alerts to reflect newly recognized interactions and improve patient safety,23 and in addition more drugs are being used all the time.

Our study provides some important additions to the existing literature. First, we assessed the override responses to a very broad range of medication alert types, such as age-based and renal recommendation, which most prior alert studies have not assessed. Second, we evaluated the appropriateness of alert overrides, and found that appropriateness varies dramatically by alert type. Few studies have evaluated the appropriateness of alert overrides,24 and little prior data exist about the appropriateness of overrides by alert type. Appropriateness can be hard to evaluate, but our inter-rater reliability for assessment was good (κ=0.89). Of course, providers may have had information that the reviewers did not, but this was likely the case in only a small minority of cases as reviewers had access to the complete electronic medical record. Further research into the appropriateness of alert overrides may help identify a number of ways in which alerts can be improved. For example, in many instances alerts might be made more specific, with one example being suppressing alerts if a patient has been taking a combination of interacting drugs for some time, or bringing in additional factors such as the presence of laboratory tests or symptoms. Additional research may also enable characterization of the variation in appropriateness of alert overrides at the physician level in order to formulate targeted interventions aimed at prescribers who have high inappropriate override rates.

Our study has several limitations. First, our results represent data from ambulatory care practices affiliated with a large, academic healthcare center. We did not study inpatient practices. Second, the alerts that were identified originated from a single internally developed CDS system. Our sample did not include other internally developed and/or commercially available CDS systems. Third, as mentioned in the methods section, our analysis on the appropriateness of alert overrides was based on a randomly-selected subset of overrides. In addition, we did not specifically assess whether patients were harmed as a result of alert overrides. We also did not specifically examine clustering of CDS overrides or their appropriateness by provider. Due to their cost and complexity, analyses of patient harm are typically focused on a narrower range of events or smaller sample size.25–27

In summary, more than half of the CDS alerts in our sample were overridden by providers. While some of the alerts were for formulary substitutions, others warn of potential significant patient harm such as overly high drug dosages given the patient's renal function or age. Despite their potential for patient harm, these alerts have override rates above 60%. Physicians' reasons for overriding alerts varied widely depending on the type of alert. Future research should investigate the appropriateness of overrides given the specific clinical context, in order to optimize alert types and frequencies to increase their relevance for patient care, as well as the clustering of overrides by provider in order to design effective interventions aimed at reducing inappropriate alert overrides, which could range from extinguishing unnecessary warnings to targeting physicians with inappropriately high override rates.

Contributors

KCN: conception and design, acquisition of data, analysis and interpretation of data, drafting of the manuscript, critical revision of the manuscript for important intellectual content, statistical analysis. KCN had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. SPS: acquisition of data, analysis and interpretation of data. DLS: acquisition of data, analysis and interpretation of data, statistical analysis. IC: acquisition of data, analysis and interpretation of data. JMF: analysis and interpretation of data, administrative, technical or material support. LMR: acquisition of data, administrative, technical, or material support. LAV: administrative, technical, or material support, supervision. DWB: conception and design, analysis and interpretation of data, critical revision of the manuscript for important intellectual content, obtaining funding, supervision.

Funding

This work was supported by the Agency for Healthcare Research and Quality (AHRQ, Rockville, MD) grant number 1U19HS021094-01. The content is solely the responsibility of the authors and does not necessarily represent the official views of AHRQ.

Competing interests

None.

Ethics approval

Partners Human Research Committee.

Provenance and peer review

Not commissioned; externally peer reviewed.