The use of artificial intelligence to optimize medication alerts generated by clinical decision support systems: a scoping review

Abstract Objective Current Clinical Decision Support Systems (CDSSs) generate medication alerts that are of limited clinical value, causing alert fatigue. Artificial Intelligence (AI)-based methods may help in optimizing medication alerts. Therefore, we conducted a scoping review on the current state of the use of AI to optimize medication alerts in a hospital setting. Specifically, we aimed to identify the applied AI methods used together with their performance measures and main outcome measures. Materials and Methods We searched Medline, Embase, and Cochrane Library database on May 25, 2023 for studies of any quantitative design, in which the use of AI-based methods was investigated to optimize medication alerts generated by CDSSs in a hospital setting. The screening process was supported by ASReview software. Results Out of 5625 citations screened for eligibility, 10 studies were included. Three studies (30%) reported on both statistical performance and clinical outcomes. The most often reported performance measure was positive predictive value ranging from 9% to 100%. Regarding main outcome measures, alerts optimized using AI-based methods resulted in a decreased alert burden, increased identification of inappropriate or atypical prescriptions, and enabled prediction of user responses. In only 2 studies the AI-based alerts were implemented in hospital practice, and none of the studies conducted external validation. Discussion and Conclusion AI-based methods can be used to optimize medication alerts in a hospital setting. However, reporting on models’ development and validation should be improved, and external validation and implementation in hospital practice should be encouraged.


Background and significance
Preventing adverse drug events (ADEs) is an essential part of medication safety efforts worldwide. 13][4] A recent retrospective cohort study in 11 hospitals in the United States showed that ADEs, despite various medication safety efforts, are still the most frequent (39.0%) type of adverse events patients encounter during their hospital stay, and are often deemed preventable (26.8%). 5Among the most broadly deployed tools to prevent ADEs in hospitalized patients, are clinical decision support systems (CDSSs). 6SSs generate medication alerts when the content of the electronic hospital record (EHR) of a patient matches with pre-defined rules in a knowledge database upon which CDSSs operate.The alerts are primarily shown during the medication prescribing phase.The goal of these alerts is to support healthcare providers in checking dosages, drug-drug interactions, contra-indications, duplicate therapy, drug allergies, and intolerances, and by doing so reducing the ADE risk of hospitalized patients.However, current CDSSs generate a high alert volume containing medication alerts with no or limited clinical relevance, resulting in alert fatigue and override rates as high as 96%.This alert fatigue is concerning as it may result in missing clinically significant alerts, which compromises patient safety.[7][8][9][10][11] This is mainly caused by the fact that the current medication alerts fail to account for the complexity of care and patient specific variables.10,12,13 In the past years, several attempts have been made to improve appropriateness and clinical value of medication alerts.A recent scoping review of Ledger et al identified 6 types of interventions for medication alerts in hospitals: alert inactivation, alert severity reclassification, information provision, use of contextual information, threshold adjustment, and encounter suppression.14 A study by Bakker et al showed the effect of alert inactivation and severity reclassification; only drug-drug interaction alerts assessed as clinically relevant in the intensive care setting were turned on, which resulted in 12% decrease in the number of high-risk combinations.15 Selecting the alerts to be inactivated by employing a multidisciplinary committee has shown to be an effective approach.16 A way to use contextual information to optimize alerts is by designing computerized decision tree rules, to context-dependently suppress irrelevant drug-drug interaction alerts, which has been shown to result in less alerts and a higher positive predictive value (PPV).17 However, given the high number of medication alerts and high override rates, room for improvement remains.Medication alerts should be more specific, for example by targeting only high risk contraindications, or by tailoring the alerts to specific medical specialties.18,19 Artificial Intelligence (AI) methods may contribute to further optimization of medication alerts generated by CDSSs, eg, by predicting physician responses, generating AI-based medication alerts, or by developing a triage system.In contrast to rigid and simple decision tree like logic upon which current CDSSs operate, AI-based methods can take into account large amounts of EHR data, recognize complex patterns, and provide individualized predictions.20 In medicine, the most often used AI-based methods are machine learning (ML), deep learning (DL), and natural language processing (NLP).21 ML and DL can be applied to structured data, whereas NLP can be used on unstructured data.In ML data analytical algorithms are developed to extract features from data, which can be used to cluster patients characteristics or predict the probability of disease outcomes.DL is an extension of ML, which can be described as networks with a large number of layers, consequently leading to the ability to explore more complex nonlinear patterns in the data.In NLP, information is extracted from unstructured data, such as clinical notes from physicians, to supplement and enrich medical data.22 Use of these methods to create new or optimize existing medication alerts may help to reduce alert fatigue, for example by prioritizing the alerts based on appropriateness and usefulness, but also in preventing ADEs which are currently missed by the existing CDSSs.12 Several literature reviews have already been performed on AI and CDSS in specific domains such as in oncology, dentistry, or infectious disease.[23][24][25] However, none of these reviews focused on medication alerts generated by CDSS in hospitalized patients.
Therefore, the aim of this scoping review was to provide a comprehensive overview of the current state of applying AIbased methods to optimize medication alerts generated by CDSS in a hospital setting.By synthesizing the available evidence, this review aims to inform on the potential AI offers for improvement of CDSS and to identify opportunities for future research.

Approach
The proposed scoping review was conducted in accordance with the Joanna Briggs Institute (JBI) methodology for scoping reviews 26 and reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR). 27The protocol was registered in the Open Science Framework. 28formation sources and search strategy A preliminary search of Medline and Embase databases was conducted on May 10, 2023 before conducting this scoping review and no systematic or scoping reviews were identified on the specific topic of this review.Furthermore, at that time, no registered research protocols were found in the Open Science Framework or PROSPERO on this specific topic.
The literature search was conducted in Medline, Embase, and Cochrane Library (reviews and trials) databases on May 25, 2023 using the definite search strategy.No filters were applied with regard to publication year and country.The search was limited to the English language.The search strategy consisted of 3 segments: 1 related to AI-based methods, 1 related to pharmacotherapy, and 1 related to CDSS.The exact search strategies for each database can be found in Supplement Appendix S1.Additionally, the references of relevant studies were screened to identify possible other relevant studies.Forward citation was performed in Web of Science and in Scopus to identify additional relevant studies, using the studies initially labeled relevant during the title/abstract screening phase.If the studies identified via these additional searches were not already identified via the primary search strategy, the search strategy was reviewed and adjusted if needed to be able to include all relevant studies.

Eligibility criteria
This scoping review aimed to include all studies that explore the use of AI-based methods to optimize medication alerts generated by CDSS in a hospital setting.We included a wide range of AI-based methods including supervised and unsupervised ML, NLP, and DL methods.Quantitative and mixedmethod studies of any observational or interventional design (including but not limited to cohort studies, randomized controlled trials, and controlled trials) were eligible for inclusion.Scoping and systematic reviews or meta-analyses that met the inclusion criteria were not included but their references lists were screened for relevant studies.Qualitative studies, case reports, abstracts of congresses, expert opinions, editorials, and narrative reviews were excluded.
This review focusses on optimization of medication alerts generated by CDSSs.Optimization of medication alerts by applying AI can be achieved in different ways, eg, by predicting physicians responses, generating AI-based medication alerts, or developing a triage system.Regarding the type of medication alerts included, we considered alerts generated at prescribing and monitoring stages as we were interested in medication alerts for prescribers and pharmacists.Furthermore, these 2 stages hold the highest risk for ADEs. 29,30hese alerts could warn for various risks, such as over-and underdosing, drug-drug interactions, contra-indications (including pregnancy and lactation), duplicate therapy, and drug intolerances and -allergies.
Regarding the use of AI-based methods, studies that did not report any performance measures or clinical outcomes were excluded.No restrictions were made to the type of performance measures or clinical outcomes used.

Data extraction and synthesis
The titles and abstracts of the studies generated by the search strategy were collected and uploaded into Endnote# citation manager to remove duplicates.The first reviewer (J.G.) screened the titles and abstracts using ASReview version 1.0. 31ASReview is a tool designed to accelerate the screening of large numbers of literature references using active learning, a type of ML, with the main principle to achieve higher accuracy with fewer training data if the algorithm can choose the data from which it learns. 32,33The algorithm does not choose the included studies but merely presents the studies to the researchers in order of probability of relevance based on prior knowledge.It does not influence the search, but only uses the results of the search.
ASReview requires researchers to specify relevant and irrelevant papers related to a specific research question as prior knowledge, to train its algorithm.In this study, a total of 10 relevant and 10 irrelevant studies (as assessed by J.G.) were used to train the algorithm.Based on this prior knowledge the algorithm predicted a ranking of the relevance of all papers uploaded in the tool.Thereafter the screening of the title and abstract of the studies could start.van de Schoot et al reported that the number of relevant abstracts found after reading 10% of the abstracts ranges from 70% to 100%.Furthermore, 8% to 33% of all abstracts have to be screened to find 95% of the relevant studies. 30Following these findings, it was decided that after screening at least 10% of the abstracts and titles and 50 studies were consecutively identified as irrelevant based on the ranking of ASReview, the title and abstract screening process will be terminated (ie, stopping rule).
The results using ASReview ranking algorithm were verified by a second reviewer (R.M.) by manually screening a sample of the studies included through ASReview's screening process, comprising 5% of the total number of studies and including the prior knowledge, to confirm that they would be included.Furthermore, the first and second reviewer (J.G. and J.M.) verified the results using ASReview by manually screening a random sample (comprising 5% of the total number of studies) of the studies excluded through ASReview's screening process to confirm that they would be excluded.The process of the title and abstract screening is visualized in Figure 1.
At least 80% of the findings from the first and second reviewer had to be identical to validate the abstract screening and start the full text screening.If this was not the case, the first and second author had to screen another 10%.Subsequently, discrepancies in the findings were discussed to reach consensus and the inclusion criteria were reviewed and adjusted for clarification if needed.Thereafter the full text screening and data extraction could start.The full-text review was done manually.Two reviewers (J.G., R.M.) screened 10% of the studies included in the full text screening to establish information to extract from the studies.The remaining studies for full-text screening were divided equally between these 2 reviewers.In case of doubt, the 2 reviewers tried to reach consensus.If this was not possible, third reviewer (P.v.d.B.) decided.All decisions were documented.
The information extracted from the included studies consisted of basic characteristics of the studies (title, author, affiliations, year of publication, journal, and country of origin), the study aims, study setting (eg, oncology, cardiology, intensive care, pediatrics, emergency), study methods, medication alerts characteristics (eg, drug-drug interactions, dosages, allergies alerts), AI-based methods used and their statistical performance, CDSS characteristics (eg, developer, targeted at physicians, pharmacist, or other healthcare professional), clinical outcomes studied and the findings, as well as most important conclusions.The extracted data were recorded in an Excel version 2307 (Microsoft, Redmond, WA, United States).

Selection of included studies
The flowchart of the selection process is shown in Figure 2. In total, 7553 citations were identified.After removal of 1928 duplicates, 5625 citations were left to be screened based on the title and abstract.After screening 10% of all citations in ASReview by the first reviewer, the stopping rule was reached since 126 subsequent irrelevant citations were found.The title and abstract screening eventually resulted in inclusion of 64 studies for full-text screening.Of these, 10 studies met the inclusion criteria and were included for dataextraction and analysis.No additional relevant studies were identified by checking references of relevant studies and forward citation in Web of Science and Scopus.

Main characteristics of the included studies
In Table 1, a summary of the main characteristics of the 10 included studies is provided.The studies were published between 2013 and 2022.5][36][37][38][39][40][41][42] Four studies (40%) stated the medical specialty studied. 34,38,40,42Segal et al included data from the internal medicine department 42 and Lee et al from the pediatric department. 40Beaudoin et al focused on (inappropriate) antimicrobial prescriptions, in specific piperacillin-tazobactam prescriptions. 34Hogue et al included data from 7 different departments: obstetricsgynecology and nursery, general pediatrics, surgery, oncology, specialized pediatrics, neonatal intensive care unit, and pediatric intensive care unit. 3841,43 However, Kawazoe et al did mention including data from the 8 medicines most frequently causing alerts: ursodeoxycholic acid, carvedilol, sennoside, loxoprofen, brotizolam, nifedipine, famotidine, and pravastatin. 35

Types of alerts optimized
Several types of medication alerts were generated by the CDSSs studied; over-and underdosing, drug-drug interactions, contra-indications (including pregnancy and lactation), duplicate therapy, and drug intolerances and -allergies.Except for the study by Kawazoe et al, where specifically dosing alerts were targeted, all others focused on optimizing the process related to the medication alerts.The study by  Kawazoe et al specifically focused on optimizing dosing alerts by trying to identify appropriate dosing thresholds for the alerts. 35he approach for optimizing medication alerts varied between the studies.In 4 studies, the goal was to identify and prevent prescription errors. 36,38,40,42Liu et al, Poly et al, and Corny et al pursued similar objectives regarding refining the relevance of alerts; focusing on predicting physician's responses 37,41 or prioritizing prescription checks. 43Balestra et al developed a model based on past recommendations, focusing on designing a model for identifying medication orders requiring an intervention. 39Similarly, Beaudoin et al developed a model using past recommendations, evaluating a CDSSs consisting of a knowledge database linked to a model that extracts classification rules for alerts of inappropriate prescriptions. 34

CDSS used
In 2 studies a commercial system called Medaware ® (Raanana, Israel) was used. 37,42Corny et al tested the accuracy of Lumio Medication ® , developed by Lumio Medical (Paris, France). 4339,43 Liu et al aimed at filtering out alerts from the user's view, without specifically mentioning who they were targeting as end-user.42 Four other studies developed a model aimed to be incorporated into the CDSS, by predicting user responses, by serving as a triage system or by filtering medication alerts generated by the regular CDSS.[37][38][39]41 The 2 remaining studies developed a hybrid system linking an ML model to the regular CDSS.40,43 AI-based methods used, their statistical performance, and outcomes Nine studies reported using ML models (Table 3).Of these studies, 5 reported using supervised learning ML methods, 34,36,37,42,43 Hogue et al reported using an unsupervised ML method, 38 and 3 studies did not state whether they applied supervised or unsupervised ML. 35,39,41 One study reported developing DL models based on supervised learning.40 None of the studies mentioned using NLP.The choice for these AI models and methods was not explicitly stated in the studies, but this generally depends on the specific requirements of the problem, the nature of the data, and the available computational resources.However, none of the included studies mentioned who chose the method of optimization nor was the field of expertise of the study team described.
Only 2 of the included studies reported implementing the developed AI-based model in daily practice (Table 3). 40,42egal et al integrated Medaware into an existing EHR system in 1 university hospital in Israel.Initially, the system operated in a "silent mode" for several months. 41During this period analyses and monitoring were performed.Once the performance level was acceptable, the system switched to live mode in a single internal medicine department and the physicians started receiving alerts in the EHR system and could respond to them.The model by Lee et al was deployed in 1 university hospital in Korea, targeting pediatric outpatients and a selected number of medications. 401][42][43] In these studies the alert burden was decreased by 14%-90%.39][40][41]43 The most often used performance measure was PPV broadly ranging from 9% to 100%.Lower PPV's were reported in the studies of Hogue et al, Liu et al, and Balestra et al 38,39,41 ranging from 9% to 49%, and higher PPV's were reported in the studies of Lee et al, Poly et al, Corny et al, and Beaudoin et al ranging from 73% to 100%. 34,37,40,43Furthermore, the highest sensitivity and specificity of included studies were reported in the studies from Hogue et al (looking at the pharmacological profiles) and Poly et al ranging from 57% to 100%. 37,38Only 4 studies reported both statistical performance and outcomes. 34,40,41,43

Main findings
This scoping review provides a comprehensive overview of currently available evidence on the use of AI-based methods to optimize medication alerts generated by CDSS in the hospital setting.A decrease in alert burden was the most often studied outcome, varying from 14% to 90% in the included studies.Seven studies reported PPV as a performance measures, which broadly ranged from 9% to 100%.The highest PPV's were reported in the studies of Lee et al, Poly et al, Corny et al, and Beaudoin et al ranging from 73% to 100%.These results indicate that AI-based methods have the potential to optimize medication alerts, but at the same time the results also show that there is substantial room for further improvement in application of these methods for this goal and reporting about such applications.Also, given the lack of external validation, the generalizability of the models and potential for implementation in hospital practice is limited.
To the best of our knowledge, this is the first review on the use of AI-based methods to optimize medication alerts generated by CDSS in the hospital setting.Several studies have been performed on AI in decision support systems in healthcare, but studies on medication alerts specifically are lacking.Furthermore, this review demonstrates that AI represents a novel approach to optimizing medication alerts.A recent scoping review of Ledger et al identified 6 types of interventions for optimization of medication alerts in hospitals; alert inactivation, alert severity reclassification, information provision, use of contextual information, threshold adjustment, and encounter suppression. 14AI is particularly well-suited for conducting these interventions, as is also shown by the studies included in this review.
The developed AI-based models have shown to decrease the alert burden and help identify more inappropriate or atypical prescriptions, compared to the regular CDSSs, subsequently leading to a decreased alert fatigue. 15In general, when more alerts are shown to physicians, they are less likely to intervene on these alerts. 44However, for AI-based medication alerts to be effective and safe, the models generating these alerts must have an optimal balance between sensitivity and specificity. 45The higher the specificity of the models, the less false alerts will be shown, leading to less alert fatigue.The higher the sensitivity, the better the models' ability to produce alerts which warn about events with high probability of patient harm.None of the studies showed a sensitivity and specificity higher than 90%.This shows that optimizing medication alerts in the vast array of all prescriptions and medication alerts is a challenging task, also for AI-based methods.
Despite the promising results showing a decreased alert burden and high PPV values, in most studies the scope is limited as the study was conducted in a single hospital 36,37,[39][40][41][42][43] and/or with a focus on specific medication or departments. 40,42Furthermore, in only 2 studies, the optimized alerts or alert process were implemented in practice.
Although not explicitly stated in the included studies, lack of alignment between the needs from hospital practice with AIbased efforts, and insufficient gains in terms of clinical outcomes may explain low uptake in hospital practice. 46,47Also, before an AI-based model can be adopted in hospital practice, extensive validation has to be performed, internally and externally.However, in most studies, the AI algorithm was developed using a limited dataset of only 1 hospital and none of the studies mention conducting external validation.Successful implementation may be achieved when more focus is given on the process of implementation in practice, rather than focusing solely on statistical performance measures. 43,45][50] None of the included studies mentioned information about model development and validation.This limits transparency and replicability of the research and makes it difficult to  52 which may explain why the relevant information about models development and validation was often missing.Adhering to the TRI-POD statement would ensure that enough information is provided for the reader to fully understand how the model was developed and validated.This transparency would support further research in the use of AI to improve CDSSs.

Strengths, limitations, and future perspectives
Strengths of this scoping review include the comprehensive search in different electronic databases and the screening of the references of relevant studies together with forward citation.Furthermore, the title and abstract screening was validated by a second screener, to maintain consistency in the inclusion of studies.Additionally, the methodology and execution of this scoping review was structurally assessed since it was conducted in accordance with the JBI methodology for scoping review and the PRISMA-ScR. 26,27imitations include the restriction to only include peerreviewed studies.Non peer-reviewed studies may provide a more recent overview of the current state of AI-based methods for optimization of medication alerts generated by CDSS.On the other hand, novel methods for CDSSs in clinical settings must be assessed via rigorously peer-reviewed studies.Another limitation is the lack of critical appraisal of the included studies, but this is in accordance with the guidelines on scoping reviews we used. 26,27Also, given the high heterogeneity of studies included in terms of setting, methods and outcomes studies, a critical appraisal is of limited value.Furthermore, the utilization of ASReview may have resulted in missing relevant studies.Nonetheless, the inherent techniques of ASReview and the applied methodologies as described have mitigated this likelihood to a minimum.Additionally, these potentially missed studies were expected to be identified by forward citation.Finally, important contributions from Non-English speaking countries may have been missed.
Future studies on AI-based methods for optimization of medication alerts generated by CDSS in hospital settings should include larger datasets, to be able to extensively validate the models internally and externally.Prior to and during the development of CDSS using AI-based methods, it is essential to ensure sufficient support and collaboration with healthcare professionals, to facilitate trust, clinical value, and the implementation of the models in hospital practice.Moreover, such studies should use reporting guidelines, such as the upcoming ML focused TRIPOD (TRIPOD-ML) to enable critical appraisal of the results presented. 53Once the models are developed and validated, adequate attention must be given to their implementation into hospital practice.

Conclusions
This scoping review provides an overview of the use of AIbased methods for optimizing medication alerts generated by CDSS in hospital setting.The then studies included show that AI has the capacity to adequately reduce alert burden and identify inappropriate prescriptions, but the datasets used were relatively small and the models lack formal validation.Most have not been implemented in hospital practice.Future studies should focus on validation and implementation of AIbased methods, and use reporting guidelines such as the TRI-POD Statement to report their work.

Figure 1 .
Figure 1.Process of title and abstract screening using ASReview software.

Figure 2 .
Figure 2. Preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow diagram.

Table 1 .
Main characteristics of the included studies.Journal of the American Medical Informatics Association, 2024, Vol.31, No. 6 a NICU, neonatal intensive care unit.b PICU, pediatric intensive care unit.

Table 2 .
Type of alerts or process optimized and CDSS used.

Table 3 .
AI-based methods used, implementation, and validation.
b UL, unsupervised learning.c NN, neural network.d CART, classification and regression trees.e RF, random forest.f GBT, gradient boosted trees.g SVM, support vector machine.h NB, naïve Bayes.

Table 4 .
51sults, outcomes, and statistical performance measures reported.inaclassificationmodelto decide the predicted class of an observation, with the model assigning the observation to one class if its predicted probability surpasses the threshold and to the other Journal of the American Medical Informatics Association, 2024, Vol.31, No. 6assess the risk of bias and potential usefulness of the prediction model.This reflects a wider trend in reporting of clinical prediction model studies which has already been recognized for several years.51Additionally,none of the included studies reported applying reporting guidelines such as Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) Statement, a Post prescribing surveillance ¼ monitoring changing in the patient record to identify potential ADEs.b RF performed better than the BAG and CART algorithm, all correlation coefficients are shown in figures in the original manuscript.c Sensitivity value was set to ≥99%.