https://orcid.org/0000-0002-3491-6215
Abstract
Opioid-related overdose (OD) deaths continue to increase. Take-home naloxone (THN), after treatment for an OD in an emergency department (ED), is a recommended but under-utilized practice. To promote THN prescription, we developed a noninterruptive decision support intervention that combined a detailed OD documentation template with a reminder to use the template that is automatically inserted into a provider’s note by decision rules. We studied the impact of the combined intervention on THN prescribing in a longitudinal observational study.
ED encounters involving an OD were reviewed before and after implementation of the reminder embedded in the physicians' note to use an advanced OD documentation template for changes in: (1) use of the template and (2) prescription of THN. Chi square tests and interrupted time series analyses were used to assess the impact. Usability and satisfaction were measured using the System Usability Scale (SUS) and the Net Promoter Score.
In 736 OD cases defined by International Classification of Disease version 10 diagnosis codes (247 prereminder and 489 postreminder), the documentation template was used in 0.0% and 21.3%, respectively (P < .0001). The sensitivity and specificity of the reminder for OD cases were 95.9% and 99.8%, respectively. Use of the documentation template led to twice the rate of prescribing of THN (25.7% vs 50.0%, P < .001). Of 19 providers responding to the survey, 74% of SUS responses were in the good-to-excellent range and 53% of providers were Net Promoters.
A noninterruptive decision support intervention was associated with higher THN prescribing in a pre-post study across a multiinstitution health system.
INTRODUCTION
Introduction of new therapies in routine care is difficult, even in the most extreme settings of need. A case in point is the role of take-home naloxone (THN) for the mitigation of harm from opioid overdose (OD).1 Even within emergency departments (EDs) with existing naloxone distribution programs, the overall distribution of TNH remains low.2 This article examines the impact of an alternative approach that uses a “noninterruptive” alert delivered during provider documentation to encourage providers to order and dispense THN during an ED visit for an OD.
More than 78 000 Americans died from opioid-related OSs between October 2020 and October 2021.3 Opioid-related deaths have continued to surge throughout the COVID pandemic, with 10 states reporting a 98% increase in opioid OS deaths compared to the previous year.4,5 Many complex factors may be contributing to this increase in OS during the pandemic, for example, social isolation and reduced access to care and support services.6 EDs bear the brunt of the burden of patients presenting with opioid OSs and are in a unique position within the healthcare system for interventions to improve management and address the opioid OD crisis. The ED visit provides an opportunity to engage these patients in discussion about their substance use, perform brief interventions, provide referrals to treatment, supply the patient with medications for opioid use disorder if appropriate, and, as is the focus of this study, provide patients with education and supplies for overdose harm reduction.
One possible approach to enhance the uptake of THN is an electronic health record pop-up style reminder for the physician during the care of an OS patient. Previous work with such a reminder for THN in the ED has shown to improve the rates of prescription; however, the effects of the alert were relatively small on average and varied widely across institutions in a healthcare system studied.1,2,7,8 Clinical decision support tools have been shown to improve naloxone prescription in hospital-wide settings for patients receiving prescriptions for high-dose opioids; however, this is a very different patient population than those presenting to the ED with opioid OS.9–14 The limitations of pop-up style blocking alerts for changing provider behavior for preventative care are well known.15 Alert fatigue is a commonly reported issue and many institutions closely regulate pop-up style alerts to preserve their effectiveness16 and limit their contributions to physician burnout.17
Rather than relying on a pop-up alert for THN, we developed a novel 2-step noninterruptive approach: (1) insertion of a reminder to access a template for documentation of ED-based opioid OS care into providers’ notes based on the patient’s chief complaint and other documentation and (2) a corresponding documentation template for OS care, which when inserted into the note, provided support to help providers capture critical aspects of the history, physical exam, and to take actions to mitigate future risk, including prescription of THN. The use of this template created both detailed text documentation of the visit and coded data for an OS research registry. Herein, the combination of this 2-step approach (reminder and template for care) is referred to as the Opioid Overdose Decision Support Tool (OODST). The evaluation was conducted as a quality improvement study with the following objectives:
Evaluate the impact of the dynamic reminder on the use of the OODST documentation template,
Evaluate the sensitivity and specificity of the OODST reminder for patients presenting to the ED with opioid OS,
For patients identified to have an opioid OS, assess for differences in the rate of THN ordering when the OODST was used versus when it was not used, and
Measure ED providers’ ratings of usability and satisfaction with the OODST.
MATERIALS AND METHODS
OODST development
The OODST was initially developed as a tool to improve the documentation of opioid OS care in EDs for the National Institutes of Health-funded registry of post-OS care.18 It was developed in a collaborative, iterative process using the Institute for Healthcare Improvement’s Model for Improvement framework.19 Stakeholder groups included emergency medicine attendings, residents, and EHR development specialists using an agile, scrum-based development methodology. After initial requirements were elicited, prototypes were developed, reviewed by providers, and iteratively improved. Throughout the process we measured our rate of THN distribution among the group of patients presenting with a suspected opioid OS with the goal of making modifications to improve THN rates. A picture of the OODST reminder is presented in Figure 1 and that of the OODST documentation template is presented in Figure 2. The insertion of the OODST reminder into provider notes is triggered under the following conditions:
Screenshot of the OODST reminder inserted into the progress note. Used with permission from Epic Systems.
A screenshot of the OODST documentation template and the generated clinical note. The OODST included 3 tabs: Opioid Pre-Hospital, Opioid ED, and Opioid Treatment Plan, which enable the generation of a comprehensive clinical note. Used with permission from Epic Systems.
Chief complaint for the visit was entered as “Drug Overdose” by the triage nurse on intake using an existing menu for ED admission types OR
The text adjunctive documentation for the chief complaint field contained one of the following terms: “naloxone” or “Narcan®.”
As shown in Figure 1, the reminder part of the OODST inserted text red that had special characters within it that prevented providers from being able to close the note and complete documentation without, either: (1) inserting the second part of the intervention, the documentation template into the note or (2) deleting the reminder (with its special characters) from the note. This 2-stage approach was implemented based on discussions with providers, as it was deemed less disruptive to their workflow than the automatic insertion of the documentation template in all cases of suspected OS.
Figure 2 shows a screenshot of the second part of the OODST intervention—the documentation template. The OODST form is invoked using a proprietary method for Epic (Epic Systems, Verona, Wisconsin) EHR systems called a “.Phrase.” Typing “.OODST” results in submitting an electronic object with 2 functions: users click on values in the template to document different aspects of care of a patient presenting with an OS, and the object automatically inserts text into the provider’s note with the features selected and deletes the special characters that prevent closing the note. The documentation template included 3 tabbed sections: Opioid Pre-Hospital, Opioid ED, and Opioid Treatment Plan, helping providers generate a comprehensive clinical note for ED care. One of the buttons that could be selected automatically generated an order for a THN kit and added documentation to the note that the kit had been ordered and distributed to the patient.
Prior to clinical implementation, ED attendings and resident physicians received a brief tutorial on the use of the OODST in clinical care. After implementation, the OODST’s usability, user satisfaction and impacts on care were evaluated using a quality improvement-based process model. The design was reviewed and approved as a quality improvement study based on MUSC’s IRB guidelines for quality improvement.
Patient population
The OODST was implemented at the MUSC Charleston Emergency Department, as well as at 4 recently acquired regional MUSC Hospitals. All sites are on the same EMR (Epic) system. These hospitals included Lancaster Medical Center, Chester Medical Center, Florence Medical Center, and Marion Medical Center. Training for providers in the use of the documentation template was provided at the Charleston site in-person sessions. The regional hospitals had the system activated without training.
To assess the OODST and reminder performance, electronic medical record (EMR) data were retrieved for all ED patients between July 30, 2020, and February 28, 2022, who met 1 or more of the following conditions:
OODST reminder was triggered, or
OODST was used, or
International Classification of Disease version 10 (ICD-10) diagnostic code(s) consistent with opioid OS based on phenotypes developed by Vivolo-Kantor et al20 and Slavova et al21 (list of ICDs available in appendix).
The following variables were abstracted from electronic records: patient age, sex, race and ethnicity, patient admission date, whether the smart form reminder was triggered, whether the OODST was used, whether an ICD-10 code associated with opioid OS was recorded, whether the naloxone kit was ordered, and text fields describing the reason for visit and medical insurance type. To assist in measuring the sensitivity and specificity of rules for insertion of the reminder to use the OODST, the total number of ED visits not involving opioid OS during the same timeframe was determined.
Statistical analyses
We first calculated summary statistics for demographic and insurance variables stratified by smart form use using chi square, Fisher’s exact or t tests to test for differences between these groups, as appropriate.
Rate of OODST use pre- and postautomated reminder insertion into the note
The documentation template (part 2 of the OODST, as shown in Figure 2) was developed and deployed first, with activation on July 30, 2020. The initial implementation required physicians to remember or look up the appropriate .Phrase (“.OODST”) and to insert the template using this method. The reminder portion of the OODST (Figure 1) was activated on February 25, 2021. The sensitivity and specificity of the reminder and the rates of use of the OODST documentation template and of prescription of THN were subsequently followed using electronic data. The precision of the insertion of the reminder was evaluated by comparing insertions on a case-by-case basis to the ICD-10 opioid OS case definition. In addition, we manually reviewed false-positive and false-negative cases (ie, either potentially wrongful insertion of the reminder or failure to insert the reminder, respectively), to determine the true sensitivity and specificity of insertion and the positive predictive value. The manual review was conducted by 2 clinicians, independently with the adjudication of differences by a third. Clinicians reviewed the chief complaint and the final ED diagnosis to make a determination of whether the case was a likely opioid OS event. The sensitivity, specificity, and the false-positive rate of insertion of the reminder were then recomputed.
Fisher’s exact test was used to determine if insertion of the reminder (part 1 of the intervention) increased the rate of use of OODST for documentation among opioid OS cases. We also conducted an interrupted time series analysis to assess how monthly rates for OODST use changed in response to implementation of the reminder.22 For this calculation, we used the ICD-10 definition of an opioid OS to identify cases.
To assess the OODST’s effect on the clinician’s decision to order a naloxone kit, we formed a 2 × 2 table that included all opioid OS ED visits based on the ICD-10 opioid OS phenotype and examined whether the OODST was used (yes/no) versus whether a naloxone kit was ordered (yes/no). We used a chi square test to test for a difference between the groups. To examine temporal effects of use of the OODST on naloxone kit ordering, we ran a generalized linear model with a log link (Poisson model) to estimate the monthly number of kits ordered, with smart form use status, month and their interaction as predictor terms. We used log (monthly OOD counts) as the offset. We tested whether the interaction term was significant to determine whether the trends over time were different between smart form use groups.
Usability and satisfaction
Users of the OODST were sent surveys to assess its usability and user satisfaction. Users were emergency medicine attending physicians, resident physicians, and advanced practice providers (APPs) working at MUSC’s Emergency Departments during the study period. Usability was measured by the System Usability Scale (SUS).23 Satisfaction was also measured by the Net Promoter® Score (NPS), which captures user experience and predicts growth in uptake. NPSs are grouped as follows: promoters (score 9–10), passives (7–8), and detractors (0–6).24
RESULTS
During the study period, there were 736 visits involving an opioid OS per the ICD-10 phenotype, with 103 (14%) having OODST use over the full study timeframe (before and after reminder implementation). Table 1 provides a summary of demographic and insurance variables for this group stratified by whether the OODST was used (yes/no). The mean age was 39.6 years, 36.3% were female, and 24.6% were non-Hispanic Black. More than two-thirds (66.3%) of cases were reported in the regional hospital network (MUSC Lancaster, Chester, Marion, and Florence), where the rate of OODST use was significantly less than at the main University hospital (10.0% vs 21.8%%, P < .0001).
OODST use among ED patients with ≥1 opioid overdose diagnostic code
| Factor | Level | Total | OODST used | P-Value | |
|---|---|---|---|---|---|
| Yes | No | ||||
| Group size | N (%) | 736 | 103 (14%) | 633 (86%) | – |
| Age | Mean (SD) | 39.6 (16.6) | 40.6 (14.2) | 39.5 (16.8) | .37 |
| Sex | Male | 469 (63.7%) | 70 (14.9%) | 399 (85.1%) | .38 |
| Female | 267 (36.3%) | 33 (12.4%) | 234 (87.6%) | ||
| Race and ethnicity | Non-Hispanic White | 539 (73.2%) | 72 (134%) | 467 (86.6%) | .08 |
| Non-Hispanic Black | 181 (24.6%) | 26 (14.4%) | 155 (85.6%) | ||
| Hispanic | 9 (1.2%) | 4 (44.4%) | 5 (55.6%) | ||
| Other or missing | 7 (1%) | 1(14.3%) | 6 (85.7%) | ||
| Insurance provider | Medicaid | 156 (21.2%) | 15 (9.6%) | 141 (90.4%) | .32 |
| Medicare | 66 (9%) | 7 (10.6%) | 59 (89.4%) | ||
| Blue Cross Blue Shield | 64 (8.7%) | 9 (14.1%) | 55 (85.9%) | ||
| Other provider | 70 (9.5%) | 11 (15.7%) | 59 (84.3%) | ||
| Unknown or missing | 380 (51.6%) | 61 (16.1%) | 319 (84.0%) | ||
| Network | University | 248 (33.7%) | 54 (21.8%) | 194 (78.2%) | <.0001 |
| Regional | 488 (66.3%) | 49 (10.0%) | 439 (90.0%) | ||
| Factor | Level | Total | OODST used | P-Value | |
|---|---|---|---|---|---|
| Yes | No | ||||
| Group size | N (%) | 736 | 103 (14%) | 633 (86%) | – |
| Age | Mean (SD) | 39.6 (16.6) | 40.6 (14.2) | 39.5 (16.8) | .37 |
| Sex | Male | 469 (63.7%) | 70 (14.9%) | 399 (85.1%) | .38 |
| Female | 267 (36.3%) | 33 (12.4%) | 234 (87.6%) | ||
| Race and ethnicity | Non-Hispanic White | 539 (73.2%) | 72 (134%) | 467 (86.6%) | .08 |
| Non-Hispanic Black | 181 (24.6%) | 26 (14.4%) | 155 (85.6%) | ||
| Hispanic | 9 (1.2%) | 4 (44.4%) | 5 (55.6%) | ||
| Other or missing | 7 (1%) | 1(14.3%) | 6 (85.7%) | ||
| Insurance provider | Medicaid | 156 (21.2%) | 15 (9.6%) | 141 (90.4%) | .32 |
| Medicare | 66 (9%) | 7 (10.6%) | 59 (89.4%) | ||
| Blue Cross Blue Shield | 64 (8.7%) | 9 (14.1%) | 55 (85.9%) | ||
| Other provider | 70 (9.5%) | 11 (15.7%) | 59 (84.3%) | ||
| Unknown or missing | 380 (51.6%) | 61 (16.1%) | 319 (84.0%) | ||
| Network | University | 248 (33.7%) | 54 (21.8%) | 194 (78.2%) | <.0001 |
| Regional | 488 (66.3%) | 49 (10.0%) | 439 (90.0%) | ||
Abbreviations: ED: emergency department; OODST: Opioid Overdose Decision Support Tool; SD: standard deviation.
Note: University Hospital refers to the Medical University of South Carolina (MUSC) in Charleston. “Regional” refers to MUSC Lancaster, Chester, Florence, and Marion ED facilities. Chi square, Fisher exact, or t tests, as appropriate, with statistical significance set at P < .05.
OODST use among ED patients with ≥1 opioid overdose diagnostic code
| Factor | Level | Total | OODST used | P-Value | |
|---|---|---|---|---|---|
| Yes | No | ||||
| Group size | N (%) | 736 | 103 (14%) | 633 (86%) | – |
| Age | Mean (SD) | 39.6 (16.6) | 40.6 (14.2) | 39.5 (16.8) | .37 |
| Sex | Male | 469 (63.7%) | 70 (14.9%) | 399 (85.1%) | .38 |
| Female | 267 (36.3%) | 33 (12.4%) | 234 (87.6%) | ||
| Race and ethnicity | Non-Hispanic White | 539 (73.2%) | 72 (134%) | 467 (86.6%) | .08 |
| Non-Hispanic Black | 181 (24.6%) | 26 (14.4%) | 155 (85.6%) | ||
| Hispanic | 9 (1.2%) | 4 (44.4%) | 5 (55.6%) | ||
| Other or missing | 7 (1%) | 1(14.3%) | 6 (85.7%) | ||
| Insurance provider | Medicaid | 156 (21.2%) | 15 (9.6%) | 141 (90.4%) | .32 |
| Medicare | 66 (9%) | 7 (10.6%) | 59 (89.4%) | ||
| Blue Cross Blue Shield | 64 (8.7%) | 9 (14.1%) | 55 (85.9%) | ||
| Other provider | 70 (9.5%) | 11 (15.7%) | 59 (84.3%) | ||
| Unknown or missing | 380 (51.6%) | 61 (16.1%) | 319 (84.0%) | ||
| Network | University | 248 (33.7%) | 54 (21.8%) | 194 (78.2%) | <.0001 |
| Regional | 488 (66.3%) | 49 (10.0%) | 439 (90.0%) | ||
| Factor | Level | Total | OODST used | P-Value | |
|---|---|---|---|---|---|
| Yes | No | ||||
| Group size | N (%) | 736 | 103 (14%) | 633 (86%) | – |
| Age | Mean (SD) | 39.6 (16.6) | 40.6 (14.2) | 39.5 (16.8) | .37 |
| Sex | Male | 469 (63.7%) | 70 (14.9%) | 399 (85.1%) | .38 |
| Female | 267 (36.3%) | 33 (12.4%) | 234 (87.6%) | ||
| Race and ethnicity | Non-Hispanic White | 539 (73.2%) | 72 (134%) | 467 (86.6%) | .08 |
| Non-Hispanic Black | 181 (24.6%) | 26 (14.4%) | 155 (85.6%) | ||
| Hispanic | 9 (1.2%) | 4 (44.4%) | 5 (55.6%) | ||
| Other or missing | 7 (1%) | 1(14.3%) | 6 (85.7%) | ||
| Insurance provider | Medicaid | 156 (21.2%) | 15 (9.6%) | 141 (90.4%) | .32 |
| Medicare | 66 (9%) | 7 (10.6%) | 59 (89.4%) | ||
| Blue Cross Blue Shield | 64 (8.7%) | 9 (14.1%) | 55 (85.9%) | ||
| Other provider | 70 (9.5%) | 11 (15.7%) | 59 (84.3%) | ||
| Unknown or missing | 380 (51.6%) | 61 (16.1%) | 319 (84.0%) | ||
| Network | University | 248 (33.7%) | 54 (21.8%) | 194 (78.2%) | <.0001 |
| Regional | 488 (66.3%) | 49 (10.0%) | 439 (90.0%) | ||
Abbreviations: ED: emergency department; OODST: Opioid Overdose Decision Support Tool; SD: standard deviation.
Note: University Hospital refers to the Medical University of South Carolina (MUSC) in Charleston. “Regional” refers to MUSC Lancaster, Chester, Florence, and Marion ED facilities. Chi square, Fisher exact, or t tests, as appropriate, with statistical significance set at P < .05.
Sensitivity and specificity of the OODST reminder insertion
Among 489 opioid OS cases, as defined by ICD-10 codes only, the OODST reminder was triggered 384 times (sensitivity = 78.5%); among 181 725 nonopioid OS cases, the template was not triggered 180 615 times (specificity = 99.6%). The positive predictive value was 36.8%. The results of the manual review of cases where the OODST was inserted showed that the insertion rules for the OODST were more sensitive and precise than the ICD-10-based case definition rules for opioid overdose cases. Reviewers identified 221 additional opioid OS cases among those labeled as false positives by ICD-10 coding alone and found that only 26 of the 105 false-negative cases were actual opioid OS events. The revised estimate sensitivity and specificity of the insertion rule were 95.9% and 99.8%, respectively, with a positive predictive value of 58%. The most common reason for erroneous insertion of the OODST reminder was an OS with another substance with empiric treatment with naloxone in the field.
Rate of OODST use pre- and postreminder implementation
Table 2 provides a comparison of smart form use before and after reminder implementation among those with an opioid OS. The OODST was never used prior to implementation of the reminder (despite education and training on the OODST) but was used 104 times (21.3%) following implementation of the reminder (P < .0001). Using an interrupted time series analysis where the interruption was at the end of February 2021 (when the reminder was implemented), there was a significant change in the rate of OODST use (increase of 25% overall, P = .0005), but the postimplementation trend (slope) was not statistically significant (P > .05), indicating that the reminder led to a significant increase in template use when implemented, but that the rate of use did not continue to change after implementation.
Rate of OODST use pre- and postreminder implementation among patients with opioid OD (defined by ICD-10 codes)
| Smart form used | Total | ||
|---|---|---|---|
| Yes | No | ||
| Reminder implemented | |||
| Pre | 0 (0%) | 247 (100%) | 247 |
| Post | 103 (21.1%) | 386 (78.9%) | 489 |
| Total | 103 | 633 | 736 |
| P-Value | <.0001 (Fisher’s exact test) | ||
| Smart form used | Total | ||
|---|---|---|---|
| Yes | No | ||
| Reminder implemented | |||
| Pre | 0 (0%) | 247 (100%) | 247 |
| Post | 103 (21.1%) | 386 (78.9%) | 489 |
| Total | 103 | 633 | 736 |
| P-Value | <.0001 (Fisher’s exact test) | ||
Abbreviations: OD: overdose; OODST: Opioid Overdose Decision Support Tool.
Rate of OODST use pre- and postreminder implementation among patients with opioid OD (defined by ICD-10 codes)
| Smart form used | Total | ||
|---|---|---|---|
| Yes | No | ||
| Reminder implemented | |||
| Pre | 0 (0%) | 247 (100%) | 247 |
| Post | 103 (21.1%) | 386 (78.9%) | 489 |
| Total | 103 | 633 | 736 |
| P-Value | <.0001 (Fisher’s exact test) | ||
| Smart form used | Total | ||
|---|---|---|---|
| Yes | No | ||
| Reminder implemented | |||
| Pre | 0 (0%) | 247 (100%) | 247 |
| Post | 103 (21.1%) | 386 (78.9%) | 489 |
| Total | 103 | 633 | 736 |
| P-Value | <.0001 (Fisher’s exact test) | ||
Abbreviations: OD: overdose; OODST: Opioid Overdose Decision Support Tool.
Rate of naloxone kit ordering when OODST used versus not used
Among those with an opioid OS, naloxone kits were ordered 25.7% of the time when the OODST was not used versus 50% of the time when it was used (P < .0001). Figure 3 shows the trends in naloxone kit ordering over time among OOD cases according to whether the smart form was used. While the slopes of the lines fitted to these data were not significantly different (P = .39), the rate of naloxone kit ordering was significantly greater overall when the smart form was used compared to when it was not used (P < .0001).
Trends in naloxone kit ordering over time among OOD cases according to whether the smart form was used. The dashed vertical line indicates where the reminder was implemented. The rate of naloxone kit ordering was significantly greater overall when the smart form was used compared to when it was not used (P < .0001).
Usability and satisfaction
The surveys were sent to a total of 70 emergency medicine providers (attending physicians, resident physicians and APPs who had used the OODST during the study period). Complete responses were obtained from 19 providers (27.1%). Fourteen of the 19 completed surveys (73.7%) were from MUSC Charleston, and 5 (26.3%) were from nonuniversity hospitals (MUSC Lancaster, Marion, Chester, or Florence). Of the 19 respondents to the SUS, 73.7% scored in the excellent and good range, 0% okay, and 26.3% in the poor range. With regard to the NPS, 52.6% scored as promoters, 15.8% as passives, and 31.6% as detractors. Examples of positive, neutral, and negative qualitative feedback from the evaluation were “I wouldn’t use it if it wasn’t for the reminder” (positive); “It asks all the right questions and helps the provider not miss questions” (positive); “It sometimes does not apply” (neutral); and “it makes my charting way more cumbersome” (negative).
DISCUSSION
Patients who present with 1 opioid OS are at high risk for a future fatal OS, with some studies reporting a 10% mortality rate within 1 year.25 Furthermore, the risk of a fatal OS after an ED visit for a nonfatal OS is highest in the first 2 days after ED discharge.26 Naloxone is a life-saving medication in the context of opioid OS, and distributing naloxone to high-risk groups for potential future OSs may reduce mortality.27,28 Because of the potential benefit to patients, providing a prescription for naloxone or distributing a naloxone kit directly to patients has been identified as a key quality measure for EDs caring for patients with OD.29–32 Therefore, when designing the OODST, we aimed to streamline naloxone kit ordering with the intent of improving the rate of distribution from our ED where we can provide free naloxone kits through grant funding from our state’s Department of Health and Human Services. Free kits are available for distribution from our ED since December 2017.33
Although free naloxone kits were available to all patients, the rate of kit distribution in the case of true opioid OSs was relatively low at baseline (29.1%), highlighting the need for a focused intervention. As demonstrated by our analysis, the OODST, when used, resulted in significant improvements in naloxone kit ordering, doubling the rate of THN kit distribution as compared to cases when OODST was not used. Moreover, the observed improvement in naloxone distribution due to the OODST was higher than those due to some of the other previously reported EMR interventions to improve ED-based naloxone distribution.1,8 While it seems logical that providers should be well versed in the documentation of OS events in the ED and “remember” to follow guidelines for harm mitigation by prescription of naloxone for take-home use, adherence to the guidelines is inconsistent. The foundation for most programs is the use of a pop-up reminder alert within the EHR, typically interrupting the provider at the time of ordering of medications or other interventions. While prior work has shown this approach to be effective in some studies, there are still significant gaps in providing THN to patients with presenting to EDs with opioid OS.32
Our work with the OODST focused on a noninterruptive alert triggered during note writing. While this is not appropriate for all types of care, we believe that many types of decision support reminders are delivered in similar contexts where the alert is present to not to avoid an error but to encourage the adoption of good population health practices. We used a trigger to insert a reminder into the note. This trigger had a relatively high sensitivity (78.5%) and specificity (99.8%). There was no interruption of the time-sensitive ordering workflow during OS care, but instead, the prompt occurred during the documentation process. Our experience showed that some kind of reminder was necessary to ensure that emergency clinicians use the documentation template. The impact of the OODST reminder and template was relatively similar in the presence and absence of accompanying clinician training on the tools, and usability and satisfaction data were also similar.
The mechanism for the OODST’s effect needs further study. The OODST ties the automated generation of documentation for the visit and order of a discharge naloxone kit together. Given that the ordering of immediate care for a patient and the documentation of that care typically occur at separate times in the ED, it may be that reminders for longer-term care of the patient are better accepted during documentation activities. Decision support at documentation time may result in providers missing the opportunity to be reminded in some patients if documentation is completed after discharge. Even with this limitation, documentation time might be a preferred time for decision support for providers for some applications.
Noninterruptive alerts are a growing area of interest in decision support systems. This implementation is somewhat novel, focusing on the decision support tool embedded within clinician notes. Other published implementations of noninterruptive decision support have used banners,34 alerts in additional windows,35 and nudges within ordering systems36 to modify behaviors.
Why might noninterruptive alerts be more effective for certain types of decision support? During the process of care for patients in demanding environments, such as the ED, physicians may be operating in a rapid pattern-matching mode, based on the so-called “system-1” reasoning (“thinking fast”).37 An interruptive alert during system-1 reasoning is truly a distraction from the specific care activity that the provider is trying to complete. At the time of documentation, particularly when a clinician is using a template to generate a note, he or she may be likely to be using more deliberate “system-2 reasoning processes” (ie, thinking “slow”) and as a result may be much more open to reminders to make care more complete and of higher quality, particularly if little effort is required.
Overall, documentation template utilization was moderate (21.1%) but, when utilized, was highly effective in the promotion of THN prescription (50% compliance). Default insertion of the OODST into provider notes upon triggering, rather than the 2-step process used in this study, might enhance use. However, even with the highly precise insertion rule used in this study, this would result in insertion of the “wrong” template for documentation in about 4 in 10 cases (42%) and, accordingly, the present 2-step process might be the optimal one. Furthermore, research into modifications to the reminder triggers to more accurately identify cases of opioid OS may be helpful. Additional analysis of barriers and facilitators to template adoption may be useful to improve the OODST acceptability, thereby increasing utilization. Prior work has demonstrated several barriers to the use of clinical decision support (CDS) including time constraints and difficulty of competing clinical demands, both of which would be particularly relevant in the ED setting.38 An important facilitator to the use of CDS is the potential to improve the quality of care, which is particularly relevant to this initiative.38–40
Limitations
This study has several limitations. The project was implemented as a quality improvement study in a single hospital system. Regional hospitals did not receive training in use of the OODST prior to or during implementation. This may have resulted in less uptake and acceptance of the tool at those sites. ICD-10 codes were used as the gold standard in comparisons to “true” opioid OSs for the evaluation of impact of the reminder on OODST insertion and effects of the OODST on THN prescribing. Our manual review suggested that this definition both undercounted cases and erroneously included cases, which is consistent with prior studies examining the relevance of ICD-10 codes for the identification of opioid OSs.41 These differences should not impact assessments of the impact of the use of the OODST on prescription of THN, which was our primary measure of effectiveness.
This study did not directly compare the OODST with pop-up style alerts and there may be other options for decision support for the dispensing of THN, such as standing (default) orders and dedicated order sets, that have demonstrated effectiveness.42,43 A randomized trial (or a set of related trials) of different implementation decision support strategies should be performed to identify the optimal (set of) strategies to promote THN adoption. As Austrian et al44 have described, multiple rapid cycle testing trials may be the ideal approach to identify the optional combination of features.
CONCLUSION
The OODST doubled the rate of prescription of THN when used, with more than 50% adherence of providers to recommended practice. Further research into the use of noninterruptive decision support using prompted insertion of documentation templates to guideline compliance is warranted along with a comparison of this approach to other approaches for decision support.
FUNDING
Funding for this project was provided by NIH/NCATS # 5U01TR002628, “Developing a Data Infrastructure to Monitor and Combat the Opioid Epidemic,” PI Leslie A. Lenert, MD.
AUTHOR CONTRIBUTIONS
Lindsey Jennings led the design and evaluation project and contributed to writing of the article. Ralph Ward contributed to the design of the project, analysis, and writing of the article. Jenna McCauley, Ekaterina Pekar, Elizabeth Szwast, Luke Sox, Joseph Hying, and Jihad S. Obeid contributed to the design of the project, the implementation of parts of the project, and the authorship of the article. Leslie Lenert conceived of the project, obtained funding, led the overall project, and contributed to the writing of the article.
CONFLICT OF INTEREST STATEMENT
None declared.
DATA AVAILABILITY
Data supporting this study are available on request to the corresponding author. Software products described in this study are available to other Epic customers on request to the corresponding author.
