Effectiveness of clinical dashboards as audit and feedback or clinical decision support tools on medication use and test ordering: a systematic review of randomized controlled trials

Abstract Background Clinical dashboards used as audit and feedback (A&F) or clinical decision support systems (CDSS) are increasingly adopted in healthcare. However, their effectiveness in changing the behavior of clinicians or patients is still unclear. This systematic review aims to investigate the effectiveness of clinical dashboards used as CDSS or A&F tools (as a standalone intervention or part of a multifaceted intervention) in primary care or hospital settings on medication prescription/adherence and test ordering. Methods Seven major databases were searched for relevant studies, from inception to August 2021. Two authors independently extracted data, assessed the risk of bias using the Cochrane RoB II scale, and evaluated the certainty of evidence using GRADE. Data on trial characteristics and intervention effect sizes were extracted. A narrative synthesis was performed to summarize the findings of the included trials. Results Eleven randomized trials were included. Eight trials evaluated clinical dashboards as standalone interventions and provided conflicting evidence on changes in antibiotic prescribing and no effects on statin prescribing compared to usual care. Dashboards increased medication adherence in patients with inflammatory arthritis but not in kidney transplant recipients. Three trials investigated dashboards as part of multicomponent interventions revealing decreased use of opioids for low back pain, increased proportion of patients receiving cardiovascular risk screening, and reduced antibiotic prescribing for upper respiratory tract infections. Conclusion There is limited evidence that dashboards integrated into electronic medical record systems and used as feedback or decision support tools may be associated with improvements in medication use and test ordering.


INTRODUCTION
With the widespread uptake of electronic medical records, massive health-related datasets have been generated and they continue to grow at unprecedented rates. 1,2 Despite the potential impact of using these datasets to improve patient care, clinicians are often overwhelmed with the complexity of processing electronic medical record data. 3,4 To better utilize these routinely collected data, clinical dashboards have been developed and integrated into electronic medical record systems to help clinicians make informed decisions and ensure the quality and safety of the care delivered. 1,2 Clinical dashboards are interactive data visualization tools that provide a visual summary of decision-related clinical information displayed in graphs, charts, or interactive tables. 5 They are commonly used in healthcare as clinical decision support systems (CDSS) or audit and feedback (A&F) tools to help clinicians make informed decisions and to provide feedback on variations in care. Clinical dashboards integrated into electronic medical record systems can display critical indicators to clinicians allowing them to recognize suboptimal care, which has been used to motivate better performance. 6,7 Suboptimal care related to medication prescription and test ordering has been extensively reported in the literature. Examples of this include overuse of medications (such as antibiotics and opioid analgesics) 8 and unnecessary referrals for diagnostic imaging or laboratory tests, 9 despite numerous guidelines endorsing rational use of these interventions. Optimizing medication prescription and test ordering are central to high-quality healthcare and clinical dashboards are therefore promising tools for enabling clinicians to reflect on their practice and identify areas to change.
Traditional methods of CDSS and A&F without the use of clinical dashboards have been shown to improve healthcare delivery. For instance, a recent systematic review revealed that CDSS integrated into electronic medical record systems increased the proportion of patients receiving desired care by 5.8% compared with usual care. 10 A Cochrane review showed that A&F interventions resulted in a 4.3% absolute increase in healthcare professionals' compliance with the desired practice. 11 None of these reviews, however, considered clinical dashboards as CDSS and A&F mechanisms, despite the increasing use in the last decade. A previous narrative review of 11 studies on the effects of clinical dashboards included only 1 randomized controlled trial, which showed no effect on antibiotic prescribing for acute respiratory infection in primary care. 5 This review is now 9 years old and did not conduct systematic searches nor assess the risk of bias. A more recent systematic review 12 focused on critical care units and included a wide range of data visualization techniques, that is, not only clinical dashboards.
Despite the increasing popularity in healthcare, there is limited knowledge on the effectiveness of clinical dashboards in changing clinician or patient's behavior. In this systematic review, we aimed to assess the effectiveness of clinical dashboards used as CDSS or A&F tools (as a standalone intervention or part of a multifaceted intervention) in primary care or hospital settings on medication prescription, adherence, and test ordering.

METHODS
This systematic review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) recommendations. 13 The protocol has been published in the Open Science Framework. 14

Searches
Electronic literature searches were conducted in the following databases: MEDLINE via Ovid, EMBASE via Ovid, and CINAHL via EBSCO, CENTRAL via Cochrane Library, INSPEC, ACM Digital Library, and IEEEXplore, from inception to August 2021. We combined the following terms and their variations to construct the search strategies: dashboard, decision support, electronic health record, and quality indicators. The reference lists of included studies and relevant systematic reviews were screened for additional relevant citations. We did not restrict our searches to any language or date of publication. The search strategies used for the selected databases are outlined in Supplementary File S1.
Titles and abstracts of records retrieved from our electronic searches were screened independently by 2 reviewers. Full texts of potentially eligible articles were screened independently by 2 reviewers according to the eligibility criteria, and disagreements were resolved by consensus or in consultation with a third reviewer.

Eligibility criteria
Types of studies Eligible studies had to be randomized controlled trials published in peer-reviewed journals. In our protocol, we stated we would consider observational studies (eg, prospective or retrospective cohorts) but we later decided to include only randomized controlled trials, which allowed us to focus on the highest level of evidence to investigate the effects of dashboard interventions. Conference abstracts and study protocols were excluded.

Types of participants and settings
Studies including clinicians or patients as participants, investigating any health condition in primary care or hospital settings (eg, emergency departments, hospital wards, and outpatient clinics) were considered. Studies including healthy populations or healthcare students were excluded.

Types of interventions and comparators
We considered clinical dashboard interventions used as CDSS or A&F. Clinical dashboards included those involving graphical user interfaces containing measures of clinical performance or clinical indicators to enable decision-making. We also considered clinical dashboards that provided a visual summary of decision-related information displayed in graphs, charts, or interactive tables. We also included studies with multifaceted/multicomponent interventions including a clinical dashboard as a core component. Studies comparing the effectiveness of clinical dashboard interventions with any type of control were considered, including usual care, no intervention, and a similar intervention without the dashboard component.

Types of outcome measures
The 2 outcomes of interest for this review were: (1) medication use, including the rate of medication prescribed/administered and medication intake adherence; (2) test ordering, such as the rate of imaging referrals and the count of routine laboratory test orders. We focused on these outcomes (rather than clinical or patient-reported outcomes) as we aimed to evaluate whether clinical dashboards with CDSS or A&F features achieve their main objective-changing clinician or patient's behavior.

Data extraction
A standardized spreadsheet was developed, and 2 reviewers independently extracted the data from the included studies. The extracted data from the included studies were: study design, sample size, sample characteristics (source, health condition, age, sex), healthcare setting, country, type of dashboard, dashboard features, intervention characteristics, outcome measures, and time points. The effect sizes (eg, mean difference, risk ratio, and odds ratio) and their 95% confidence intervals (CIs) were also extracted.

Risk of bias assessment
The Cochrane Risk of Bias (RoB) 2.0 tool 15 was used to assess the risk of bias of included randomized controlled studies. The risk of bias of each domain was judged as high risk of bias, low risk of bias, and some concerns, and overall risk of bias for each included study was also provided. Studies were considered as having an overall low risk of bias when all domains were judged as low risk, whereas studies were considered as having an overall high risk of bias when at least 1 bias domain was judged as high risk. 15 For studies with some concerns in at least 1 domain, we recorded it as having some concerns.

Certainty of evidence
The Grading of Recommendations of Assessment, Development and Evaluation (GRADE) approach 16 was used to assess the quality of the body of evidence for the primary outcomes of included studies. The GRADE ratings were summarized as either high, moderate, low, or very low across 5 domains: study limitations, inconsistency of results, indirectness of evidence, imprecision of estimates and publication bias. 16 Since we did not conduct a meta-analysis, assessment against inconsistency and publication bias was not applicable in this review.

Data synthesis
Since high heterogeneity exists in health conditions and outcome measures, we were unable to perform a meta-analysis. Therefore, a narrative synthesis of the findings was conducted. We descriptively reported the effect sizes for both primary and secondary outcomes related to medication use and test ordering, the intervention time frame, and the number of participants in each trial. Relevant data were grouped and assessed based on the types of interventions (ie, CDSS, A&F, standalone, or multifaceted interventions) and types of comparators (eg, usual care/no intervention). Results are presented in the summary of findings tables along with the GRADE assessment.

Outcomes
Six studies evaluated the changes in medication prescription as the primary outcome, 17,18,22,23,26,27 with one study recording it as the secondary outcome. 21 Two studies 19,24 assessed medication adherence as the primary outcome, while 2 trials focused on test ordering. 20,21 One study 25 had coprimary outcomes on both medication prescription and test ordering.

Risk of bias
Overall, 4 trials had the risk of bias judged as "some concerns," 17,19,23 5 trials had a "low" risk of bias, 18,21,25-27 and 2 trials were considered at "high" risk of bias. 20,22,24 One issue leading to a judgment of an overall "high" risk of bias was a lack of reporting on missing outcome data, and no analysis methods for correcting this bias. In cluster randomized trials, the main issue was the lack of concealment of the cluster allocation, which is likely to lead to selection bias. The risk of bias for individual trials is summarized in Figure 2.

Quality of the evidence (GRADE)
The quality of evidence is assessed against risk of bias, indirectness, and imprecision. Half of the trials 17,20,23,25,27 had a moderate quality of evidence, the common reason for downgrading was due to some concerns within the risk of bias. Three trials 18,21,26 had a high quality of evidence, 1 trial 19 had a low quality of evidence, while 2 trials 22,24 had a very low quality of evidence. The main issue leading to the judgment of low quality was a high risk of bias and uncertainty about imprecision of the estimates due to a lack of effect size reporting. Table 2 provides the summary of findings for all included studies, including effect sizes (if reported) for both primary and secondary outcomes, the intervention time frame, and the number of participants in each trial. The effects of interventions were classified for dashboards as standalone interventions and dashboards as a core component of a multifaceted intervention, respectively.

Clinical dashboards as standalone interventions
Four trials investigated changes in antibiotic prescribing, 17,22,26,27 with 2 trials having a moderate quality of evidence, 17,27 one 26 having a high quality of evidence and another 22 having a very low quality of evidence. One trial including 2900 primary care physicians 26 found a feedback dashboard did not lower nationwide antibiotic prescribing in primary care over 2 years (between-group difference À1.73%, 95% CI À5.07% to 1.72%). Another trial with 573 clinicians also found no effects on antibiotic prescribing for acute respiratory infection (odds ratio 0.97, 95% CI 0.70-1.40). 16 One trial including 2566 dentists 27 reported a significant reduction in antibiotic prescribing across all NHS general dental practices (between-group difference À5.7%, 95% CI À10.2% to À1.1%). Another crossover trial 22 with 163 physicians revealed that a feedback dashboard reduced antibiotic prescribing in primary care by an average of 4% per 10-day period (coef. À0.04, 95% CI À0.07 to À0.01).
One trial 18 providing a high quality of evidence investigated the effects of a feedback dashboard on statin prescribing for patients with atherosclerotic cardiovascular disease. The trial included 96 primary care physicians and showed no significant difference between the dashboard only intervention arm and the control arm (adjusted difference in percentage points 4.1%, 95% CI À0.8% to 13.1%). However, a significant increase in statin prescribing was seen in the dashboard with peer comparison group (adjusted difference in percentage points 5.8%, 95% CI 0.9% to 13.5%), when compared to the control group.
Patient adherence to prescribed medication was reported in 2 trials with low quality of evidence. 19,24 One trial with 111 patients 19 reported a higher proportion of patients diagnosed with inflammatory arthritis adhering to their prescribed disease-modifying antirheumatic drug therapy in the dashboard group compared to the control group (87% vs 43%, P < .01). In another trial 24 with 114 South Korean kidney transplant recipients, there was no significant   (continued) between-group difference in adherence to immunosuppressive medications (no effect size provided). One trial with moderate evidence quality involving 114 medical interns 20 found no difference in the count of routine laboratory test orders placed by a physician per patient-day in the dashboard intervention group-the ordering dropped by 0.14 less laboratory tests per patient-day among physicians in the intervention group (95% CI À0.56 to 0.27), when compared to the control group.
Clinical dashboards as a core component of a multifaceted intervention One trial providing a high quality of evidence investigated a feedback dashboard as a component of a multifaceted intervention involving staff training and provision of education materials to support guideline-endorsed care of low back pain in emergency departments. 21 This trial involved 269 clinicians and 4625 patients and found no effects on lumbar imaging referrals (odds ratio 0.77, 95% CI 0.47-1.26), but revealed a significant reduction in opioid administration (odds ratio 0.57, 95% CI 0.38-0.85).
Peiris and colleagues 25 assessed a multicomponent cardiovascular disease intervention across 60 general practices with 38 725 patients, the quality of evidence was considered moderate. The inter-vention consisted of computerized decision support, A&F tools, and staff training, where the dashboard was the major component. There was a higher proportion of patients receiving appropriate screening of cardiovascular disease risk factors in the dashboard group versus the control group (risk ratio 1.25, 95% CI 1.04-1.50). This trial 25 reported no difference in the proportion of patients at high cardiovascular disease risk receiving recommended medication prescription (risk ratio 1.11, 95% CI 0.97-1.27).
One trial with 45 primary care clinicians 23 found that education plus a feedback dashboard significantly decreased antibiotic prescription rates for upper respiratory tract infections (interaction term ratio 0.60, 95% CI 0.47-0.77) and bronchitis (interaction term ratio 0.42, 95% CI 0.32-0.55), but not for sinusitis (interaction term ratio 1.05, 95% CI 0.91-1.21) and pharyngitis (interaction term ratio 0.91, 95% CI 0.76-1.09), compared to a control group that received education only (moderate quality of evidence).

Summary of main results
This systematic review assessed the effects of clinical dashboards used as A&F or CDSS on medication prescription, adherence, and  Not reported Moderate test ordering. In standalone interventions, there was conflicting evidence on the effects of dashboards on prescription of antibiotics and statins: 3 trials found no effects on antibiotic or statin prescribing, while 2 trials detected a significant reduction in antibiotic prescribing. Dashboards improved medication adherence in patients with inflammatory arthritis but did not increase the adherence to immunosuppressive medicines in kidney transplant recipients. In multicomponent interventions, dashboards reduced opioid use for low back pain and antibiotic prescribing for upper respiratory tract infections. For test ordering, dashboards increased the proportion of patients receiving appropriate cardiovascular risk screening but had no effect on the rate of imaging referrals for low back pain.

Comparison with existing studies
A previous narrative systematic review investigating the effects of dashboards was conducted by Dowding and colleagues in 2015. 5 Eleven studies were identified with empirical evaluations of dashboards; however, the majority of the studies were nonrandomized pre-post designs with only 1 randomized controlled trial included. Whilst our review focused on investigating dashboards on medication use and test orders through a comprehensive review of randomized controlled studies, we included 11 relevant trials. Another systematic review on the effectiveness of information display interventions on patient care outcomes was published in 2019. 12 Of 22 eligible studies included, only 5 were randomized controlled studies. The findings suggested that there was limited evidence that dashboards significantly improve patient outcomes. However, this systematic review was conducted solely in critical care settings and included various information display interventions, such as physiologic and laboratory monitoring, expert systems, and multipatient dashboards. Notably, there was a recent review on patient safety dashboards published in late 2021. 28 It analyzed 33 time-series studies and case studies and concluded limited evidence for dashboards directly or indirectly impacting patient safety. Clinical dashboards are not only used as standard standalone interventions but are also frequently used as part of multifaceted interventions for improving healthcare performance and processes. Our review found the effects vary greatly in terms of medication prescription, medication adherence, and test ordering. This high level of heterogeneity aligns with findings from other systematics reviews evaluating the impacts of electronic A&F 29 and CDSS 30,31 without the dashboard component. The heterogeneity of our findings might result from dashboards being adopted using different formats, using different technologies, in different health settings, for different endusers. In the included studies of our review, intervention effects were assessed either solely on the dashboard itself or by incorporating dashboards into multicomponent interventions. Therefore, it is difficult to arrive at a definitive conclusion why some dashboard interventions contribute to improvements in healthcare performance while others do not.

Explanations and implications for future research
One issue leading to a lack of significant intervention effects might be related to data quality. This is evidenced by one trial 26 that failed to reduce the antibiotic prescription rate, where the authors discussed that incompleteness of the routine health data may have hindered comprehensive feedback on prescribing rates to clinicians. The issues of accuracy, completeness, interoperability, and reliability that are associated with routine health data have been widely acknowledged and yet no perfect solutions have been brought forward. Therefore, provided that the digital dashboard, by its nature, is a data visualization and analytics tool, it would inevitably suffer from the deficiencies of healthcare data. This may result in the inability of digital health dashboards to accurately display a full picture of patient information or healthcare use, which might impede effective feedback and decision-making.
Another possible reason for the nonsignificant effects might be a lack of dashboard use by clinicians or patients. 32 For effective implementation of dashboards into routine care there is a need for a thorough inspection of the organizational environment preimplementation. When implementing health informatics tools such as clinical dashboards, healthcare organizations should take multilevel factors into account, such as people, process, technology, and their interactions, 33 to identify factors that might impede the health interventions from achieving their full potential. 34 That is, to implement a health technology, not only should evaluation efforts focus on the intervention itself, but they should also consider the underlying infrastructure that supports the devices, and the human factors such as clinicians' readiness and digital literacy, as well as the healthcare organizational environment and resources for properly implementing the technology.
Behavior change theories are encouraged to be used when designing interventions aiming to change clinicians' practice, including clinical dashboards. 11,35 Recently, Dowding and colleagues 36 proposed a theory to guide the design of clinical dashboards, including 3 domains: the cues of the intervention message; the nature of the task or behavior to be performed; and situational/personality variables. Cues of the intervention message focus on providing specific tasks and performance goals as opposed to more generalized feedback. 36 One trial in our review 27 designed an A&F dashboard for dentists using a similar behavior change technique involving "instructions on how to perform the behavior" and "provision of information about health consequences of performing the behavior." The authors found that this dashboard led to a significant reduction in antibiotic prescribing. 27 The nature of the task to be performed concerns cognitive resources-the more cognitively demanding a task is, the less effective the intervention would be. 36 Another trial in our review 17 developed a quality dashboard for acute respiratory infections, which included data unrelated to study outcomes (eg, distribution of patient visits, billing information) displayed with 10 other reports unrelated to the trial's activities. The high cognitive demand tasks associated with this dashboard could explain the lack of effects of the intervention on antibiotic prescribing for acute respiratory infections. 17 Situational/personality factors relate to baseline performance. 36 In one of our included trials, 24 the authors claimed that the nonsignificant improvement in medication adherence was in part due to already high baseline adherence. Another trial 26 also explained that the feedback dashboard was not associated with reduced prescribing rates possibly because Switzerland has the lowest antibiotic prescription rates in Europe. The same factor was observed in the SHaPED trial, 21 which had low preintervention lumbar imaging rates. Thus, given the already high-level baseline performance, it would be more challenging for dashboard intervention to make a difference. None of the other trials included in this review explicitly stated that they have used theory to design dashboard components.
With the capability of integrating and sharing real-time health data, digital health dashboards are deemed as an important building block for developing a learning health system. 37,38 A learning health system learns from routine health data and feeds the evidence back into practice to create cycles of continuous improvement. 37 Within this process, the dashboard has therefore moved from a one-way linear data output model (health data input-dashboard-analyzed data presentation) to a cyclical data output model (data input from electronic health record-dashboard and other health technologies-integrated data presentation-update data into electronic health record). By harnessing the power of capturing real-time health data, and integrating data from various sources, digital health dashboards can be a critical enabler in accelerating the uptake of evidence into practice, thereby improving healthcare performance, patient safety, and quality of care. This significance has been increasingly recognized especially during the COVID-19 pandemic. 39 With the urgent need to timely acquire patient's demographics, COVID-19 severity, risk factors, and test results, population health dashboards/national dashboards have been developed to monitor pandemics and assist in making clinical decisions and public health policies. 40 The evidence generated from disease diagnosis and management is then updated in the routine healthcare databases and shown in the dashboards to better inform practice. This learning health system has its unique role in promising us to rapidly adapt to public health emergencies, and the dashboard is no doubt a key enabler.

Strengths and limitations
The strengths of this study include the adoption of a robust methodology for systematically searching, screening, extracting, and summarizing the existing evidence. Meanwhile, we only included randomized controlled trials which helped to reduce the heterogeneity of included studies. Nevertheless, there was still a high level of heterogeneity regarding the study populations, health conditions, and outcomes measures in included trials, which is the main limitation of this review. Secondly, there is a deviation from the study protocol, 14 where we initially planned to consider a wide range of study designs since the previous review on this topic included only 1 randomized trial. However, in our searches we were able to find 11 eligible randomized trials investigating the effects of dashboards, which allowed us to focus on the highest level of evidence. We also narrowed down the outcome measures into medication use and test orders since these are relevant outcomes related to quality and safety of healthcare and more likely to be influenced by dashboard interventions. We believe that we did not miss any relevant trials considering our sensitive search strategy and we manually screened potentially eligible studies cited in the included studies. Finally, as there were 2 dashboard types assessed in the review, when it comes to multifaceted interventions, assigning specific effects to the dashboard component inevitably became equivocal.

CONCLUSION
There is limited evidence indicating the positive impact of introducing clinical dashboards into routine practice on medication use and test ordering. Dashboards seem to have become an integral component of healthcare organizations with a prior assumption that they are useful, but the evidence from our review contradicts this assumption to some extent. When designing and implementing dashboards in healthcare, important aspects, such as design theories, data quality, healthcare processes, human factors and available resources, warrant further attention as they might influence the effects of dashboards on healthcare performance and quality of care.

AUTHOR CONTRIBUTIONS
CX, GCM, and CGM conceived the study. QC, LH, and CH supported the data extraction. All authors contributed to the planning of the study and the manuscript.

SUPPLEMENTARY MATERIAL
Supplementary material is available at Journal of the American Medical Informatics Association online.

PATIENT AND PUBLIC INVOLVEMENT
There was no involvement from patients or the public in the design, conduct, or outcome of this work.