Improving Clinician Decisions and Communication in Critical Care Using Novel Information Technology

Abstract

Introduction

The electronic medical record (EMR) is presumed to support clinician decisions by documenting and retrieving patient information. Research shows that the EMR variably affects patient care and clinical decision making. The way information is presented likely has a significant impact on this variability. Well-designed representations of salient information can make a task easier by integrating information in useful patterns that clinicians use to make improved clinical judgments and decisions. Using Cognitive Systems Engineering methods, our research team developed a novel health information technology (NHIT) that interfaces with the EMR to display salient clinical information and enabled communication with a dedicated text-messaging feature. The software allows clinicians to customize displays according to their role and information needs. Here we present results of usability and validation assessments of the NHIT.

Materials and Methods

Our subjects were physicians, nurses, respiratory therapists, and physician trainees. Two arms of this study were conducted, a usability assessment and then a validation assessment. The usability assessment was a computer-based simulation using deceased patient data. After a brief five-minute orientation, the usability assessment measured individual clinician performance of typical tasks in two clinical scenarios using the NHIT. The clinical scenarios included patient admission to the unit and patient readiness for surgery. We evaluated clinician perspective about the NHIT after completing tasks using 7-point Likert scale surveys. In the usability assessment, the primary outcome was participant perceptions about the system’s ease of use compared to the legacy system.

A subsequent cross-over, validation assessment compared performance of two clinical teams during simulated care scenarios: one using only the legacy IT system and one using the NHIT in addition to the legacy IT system. We oriented both teams to the NHIT during a 1-hour session on the night before the first scenario. Scenarios were conducted using high-fidelity simulation in a real burn intensive care unit room. We used observations, task completion times, semi-structured interviews, and surveys to compare user decisions and perceptions about their performance. The primary outcome for the validation assessment was time to reach accurate (correct) decision points.

Results

During the usability assessment, clinicians were able to complete all tasks requested. Clinicians reported the NHIT was easier to use and the novel information display allowed for easier data interpretation compared to subject recollection of the legacy EMR.

In the validation assessment, a more junior team of clinicians using the NHIT arrived at accurate diagnoses and decision points at similar times as a more experienced team. Both teams noted improved communication between team members when using the NHIT and overall rated the NHIT as easier to use than the legacy EMR, especially with respect to finding information.

Conclusions

The primary findings of these assessments are that clinicians found the NHIT easy to use despite minimal training and experience and that it did not degrade clinician efficiency or decision-making accuracy. These findings are in contrast to common user experiences when introduced to new EMRs in clinical practice.

Critical care, Burn Units, Information Technology, Electronic Health Record, Patient Care Team, Decision Making, Simulation

INTRODUCTION

Care for Burn Intensive Care Unit (BICU) patients is complex and requires a team of highly trained, experienced clinicians that synchronize efforts to achieve optimal outcomes. Care is complicated by risks posed from information overload,¹ alarm fatigue,² task saturation,³ interruptions,⁴ and communication challenges caused by differences in clinician perceptions, care priorities, and professional identities.⁵ Individual and team decisions are best made with current, accurate information. Effective presentation of the most important, or salient, information^6,7 promises to improve clinical decision making.⁸

The electronic medical record (EMR) has evolved as the presumed means to support clinician decisions by documenting and retrieving needed patient information. Some have shown the EMR improves patient care,⁹ or supports compliance,¹⁰ while others^11–14 question its effectiveness on patient outcomes, or indicate mixed outcomes using an EMR and its impact on clinical decision making.^15–18

Research on decision aids suggests that the way a problem is presented can improve, or degrade, clinicians’ cognitive work.¹⁹ Cook’s patient safety research “strongly supports the use of decision support tools to improve human performance”.²⁰ Artifacts such as checklists are representations clinicians use to perform cognitive work that shape decision making and collaboration.²¹ Well-designed representations can make a task easier by integrating multiple kinds of technical and clinical information in a compact, efficient manner.^22,23 Making data patterns easier for clinicians to identify through well-designed information displays could improve clinical judgments and decisions.^24,25

Using Cognitive Systems Engineering²⁶ methods, our research team developed a novel health information technology (NHIT) software for a 16-bed BICU to display salient clinical information, interface with the electronic medical record (that had been in use in the BICU for years) and enable communication amongst the clinical team. Our research^27–29 identified eighteen barriers to cognitive work, which yielded thirty-nine software requirements, and seven key modules (Supplemental Tables S1–S3). The NHIT prototype incorporated over 320 elements of information in a Patient View (Fig. 1) and integrated text messaging (Supplemental Figure S1, Supplemental Table S4) to support team communication. The NHIT incorporates multiple data sources into a single interface by extracting information from the EMR and other databases. Clinicians can configure displays according to role and information needs, clinical tasks, and patient condition.

FIGURE 1

Novel IT System Prototype Patient View, © 2019 Applied Research Associates. Used by permission

We evaluated clinician perception of the NHIT as well as the system’s effect on individual and team performance in a BICU by conducting two studies: a usability assessment and a validation assessment.

METHODS

We performed usability and validation assessments to determine whether the NHIT improved clinician performance and whether clinicians accepted it. Both assessments were performed according to protocols approved by the local institutional review board and the Department of Defense Human Research Protection Office.

Usability Assessment

The objective of the usability assessment was to determine if the NHIT was a valid tool acceptable for helping clinicians complete their work. We recruited eleven physicians, twenty nurses, and ten respiratory therapists (RTs) of various experience levels for this assessment. Each participant received a 5-minute orientation and used the NHIT to complete two clinical scenarios – determine if a patient was ready for the operating room and complete a patient admission – and then rated the usability of the NHIT versus their experience with the legacy EMR (Fig. 2, panel A). RTs only completed the pre-operative scenario.

FIGURE 2

Validation Assessment Study Design.

The usability assessment employed Rubin’s³⁰ approach to evaluate the NHIT using simulated task performance, and verbal protocol analysis (also known as “thinking aloud”) to reveal information they considered, how they found it, and how they interpreted it. This approach included the collection of quantitative data by having the participant perform a task noting the time and steps to complete the task. It also included collection of qualitative data such as the participants’ subjective ease of use and confidence in decisions. We collected data on how clinicians used patient information to make clinical decisions, and how they rated the novel interface in terms of effort and efficiency of supporting cognitive work. During the usability assessment, the NHIT was populated with deceased patient data.

Following the scenarios, participants used a 7-point scale rating the NHIT on its ease of use, its impact on their ability and speed finding information, and how it affected their confidence in decision making. Additional assessments compared the NHIT’s ability to support clinician cognitive work as compared to the legacy system (Supplemental Figure S2).

Validation Assessment

The purpose of the validation assessment was to study how the NHIT affected team performance as measured by decision making accuracy (was the right decision made) and efficiency (how long did it take to get to the decision) and to gather additional information about user’s subjective experience with the NHIT. The validation assessment observed two clinician teams each consisting of an attending burn surgeon, a bedside nurse, and a resident physician while they completed two high-fidelity simulation-based patient care scenarios. Scenarios called for teams to treat a “patient” with acute respiratory distress syndrome (ARDS) and a “patient” with intra-abdominal sepsis because therapeutic interventions in both circumstances are challenging to implement, potentially risky, and require team decision making. We developed, piloted, and refined scenarios with input from experienced burn care providers. Simulation materials are available at: http://usaisr.amedd.army.mil/pubsupp.html.

In each scenario, teams cared for simulated patients (SimMan 3G, Laerdal®, Stavanger, Norway). Scenarios lasted approximately 6-hours and took place in an actual patient room in order to increase the fidelity of the clinician experience (Fig. 2, panel B). When participants requested consultation or support, researchers performed necessary roles including RT, rehabilitation specialist, nutritionist, pharmacist, family member, and subspecialty consultants. Simulation scripts accounted for scenario evolution depending on clinician decisions, including tasks, lab values, imaging, notes, patient care needs, and changes in patient condition. Scenarios helped assess how well either system supported the teams’ key decisions. Each team was oriented to the IT system and the simulation environment for approximately 1 hour the night before their first validation assessment. Scenarios began at change of shift (6:45 AM) when researchers, simulating the “night-shift,” gave standardized handoffs to the resident and nurse subjects. Teams then had time to prepare for multidisciplinary rounds, conduct rounds, and perform post-rounds care.

Table I

Usability Assessment: Participant Ratings of Novel IT System and Legacy System

	Overall (n = 41)	Physician (n = 11)	Nurse (n = 20)	RT (n = 10)
Scenario 1-Preparing for SurgeryMean (SD)
I am confident in my decision/recommendation.	5.4 (1.2)	5.7 (1.1)	5.2 (1.3)	n/a
The system was easy to use to make this decision.	4.9 (1.4)	5.6 (1.1)	4.7 (1.7)	n/a
The system enabled me to quickly find the information I needed.	5.18 (1.6)	5.7 (1.4)	4.9 (1.8)	n/a
It was straightforward to find the information I needed.	4.95 (1.6)	5.3 (1.4)	4.75 (1.8)	n/a
Scenario 2-New Admission Mean (SD)
I am confident in my decision/recommendation.	5.9 (0.75)	6.1 (0.73)	5.85 (0.67)	5.7 (0.76)
The system was easy to use to make this decision.	5.5 (0.95)	6.1 (0.57)a	5.6 (1.2)	4.8 (1.3)a
The system enabled me to quickly find the information I needed.	5.1(1.2)	5.7 (1.2)a	5.5 (1.2)	4.8 (1.3)a
It was straightforward to find the information I needed.	5.2(1.1)	6.2 (0.92)a,b	5.5 (1.1)b	4.9 (1.4)a
Information Search		Disagree (1–3)	Neutral 4	Agree (5–7)
I can find the information I need in [the novel IT system] morequicklythan I can using [the legacy system]		24.4%	9.7%	65.9%
I can find the information I need moreeasilythan I can using [the legacy system]		14.6%	19.5%	65.9%
Usability
The [the novel IT system] iseasierto use than [the legacy system]		14.5%	19.0%	66.0%
I would feel moreconfident making future clinical decisions and recommendations using [the novel IT system] than using [the legacy system]		19.5%	22.0%	58.5%
[The novel IT system] supportsthe way I do my work better than [the legacy system]		17.1%	19.5%	63.4%

^aIndicates significant difference between identified groups (p < 0.05) by MANOVA.

^bIndicates significant difference between identified groups (p < 0.05) by MANOVA.

Team 1 used the NHIT plus the legacy EMR system for the ARDS scenario, then the legacy EMR system only for the Sepsis scenario. Team 2 used the legacy EMR system for the ARDS scenario, then the NHIT plus the legacy EMR system for the Sepsis scenario. We used simulated patient data and the legacy EMR system to enter/record all patient data which the NHIT could then re-represent through real-time integration.

We used both Rubin’s evaluation methods and McGrath’s³¹ model of group process to discover how either IT system affected individual, group, and environmental factors to satisfy outcome measures. Three observers shadowed team members and collected data on these outcome measures: activities performed, the time it took the team to arrive at key decisions, information seeking behaviors, communication patterns, decision-making processes, and use of the NHIT messaging feature. Care provided at the same level of quality in a shorter timeframe was considered more efficient. Scenarios took approximately six hours to complete.

We also collected data after the scenarios. Using a brief survey (Supplemental Figure S3), participants rated team decision making, communication, and performance on a 7-point scale, and compared both IT systems. They provided feedback about their experience in a semi-structured group interview.

Statistical Analysis and Outcome Measures

We analyzed the usability assessment’s primary outcome measures (participants’ ratings of NHIT ease of use, efficiency of use, and decision-making support) using one-way multivariate-analysis-of variance (MANOVA), and secondary measures (time to complete tasks, ratings of decision-making confidence) using a combination of t-tests, one-way analysis-of-variance, and multivariate-analysis-of variance (MANOVA). We used descriptive statistics (e.g., frequency, standard deviation) to analyze the validation assessment’s primary outcome measure (time to reach key decisions) and to tabulate comments from post-scenario interview sessions.

RESULTS

Usability assessment

Ninety percent of the nurses, 53% of the physicians, and 50% of the RTs had more than 7 years of service in the BICU. All participants were able to complete all tasks successfully. For the admission scenario, MANOVAs on the effect of role on use of the system were significant as follows: physicians rated the NHIT as easier to use and easier to find information compared to RTs, p < 0.05, p < 0.05; physicians also rated it easier than nurses did for finding needed information, p < 0.05. All participants rated themselves as confident in their decisions and rated effort needed to use the NHIT to be low compared to the legacy system (Table I).

Due to complexities using the legacy system in a simulated fashion, only one experienced and technology savvy BICU nurse completed the usability tasks using the legacy system. Data from this experience provides context for the mean times of 20 BICU nurses using the NHIT (Supplemental Table S5).

Table II

Validation Assessment: Elapsed Time from start of the scenario to Key Decisions (hours: minutes)

	Legacy	Novel	Difference
Scenario 1: Abdominal Sepsis	Team 1	Team 2	L-N
Decision to initiate antibiotic therapy (EMR)	2:59	4:15	+1:16
Diagnose Sepsis (OBS)	3:44	4:00	+0:16
Communicate with the family/patient’s decision maker (OBS)	3:12	2:16	−0:56
Decision to perform exploratory laparotomy or transition to palliative care (scenario end) (OBS)	6:38	6:43	+0:05
Scenario 2: Severe ARDS	Team 2	Team 1
Decision to initiate antibiotic therapy (EMR)	2:15	0:59	−1:16
Decision to paralyze the patient (EMR)	2:03	2:05	+0:03
Ordered the rotaprone bed (OBS)	3:00	NP	NA
Initiate inhaled nitric oxide therapy (EMR & OBS)	3:15	3:16	+0:01
Communicate with the family/patient’s decision maker (OBS)	1:04	2:42	+1:38
Decision to cannulate or forgo cannulation for ECMO (scenario end) (OBS)	5:07	4:33	−0:33

OBS, Observation; EMR, Electronic Medical Record; L, Legacy; N, Novel; NP, Not Performed; NA, Not Applicable.

Validation Assessment

Team 1 had more overall experience than Team 2, but team experience was similar in the BICU work setting. Years of experience for each team were: the Team 1 attending physician had 10+ years in practice and 10+ years working in the BICU, the resident had 4–6 years in practice and less than one year in the BICU, and the nurse had 10+ years in practice and 1–3 years working in the BICU; the Team 2 attending physician had 10+ years in practice and 7–9 years working in the BICU, the resident had less than one year in practice and less than one year working in the BICU, and the nurse had 7–9 years in practice and 1–3 years working in the BICU. Team 1 had one female member, while Team 2 was all male. Demographic information is in Supplemental Table S6. Key decisions and elapsed times are shown in Table II and Supplemental Table S7. The more experienced team consistently gave antibiotics faster, but otherwise decision-making efficiency was similar. In the sepsis scenario, Team 1 (legacy IT) decided to perform an exploratory laparotomy 6 hours and 38 minutes into the scenario, while the less experienced Team 2 (NHIT) arrived at the same decision in 6 hours and 43 minutes. In the ARDS scenario, Team 1 (NHIT) decided on ECMO treatment at 4 hours 33 minutes, while less experienced Team 2’s (legacy IT) decision came at 5 hours 7 minutes.

Field note analysis demonstrated that the NHIT supported efficient and distributed decision making by allowing for flexible and asynchronous communication. The NHIT messaging feature allowed participants to check patient or test status, consult requests, and medication orders. Team communication strategies differed in how they used the NHIT messaging feature. Team 1 had 11 communication threads, 82% of which received replies. This experienced team communicated one-to-one (e.g., resident to attending) in 9 of the 11 threads. In contrast, Team 2 used the feature to broadcast information one-to-many (e.g., nurse to resident, charge nurse, pharmacist) for 70% of their 10 threads (Supplemental Table S8).

During interviews, both teams discussed how clinically challenging the scenarios were and that because “patients” were “so sick,” team members were at the bedside more and relied on the IT systems less. Participants consistently felt like the scenarios were “real” and that the NHIT was easy to use and helped prepare for rounds more efficiently, stating that “[it was] easier to see trends of information [using the NHIT].” Participants found that the NHIT’s messaging service made communication easier.

Finally, post-validation assessment survey results (Supplemental Table S9), showed that all but one participant rated performance, communication, and decision making as greater with the NHIT, and the NHIT as superior to the legacy system for ease of use and information finding. In every question regarding use of the systems, the average score for each team rated the NHIT as better than the legacy system with the exception of finding information and decision confidence from Team 2, which were rated as being equal. No statistical analysis was made due to the small sample size, but differences ranged from zero under Team 2, to 3.3 with an average advantage to NHIT of 1.45 over all questions and both teams.

DISCUSSION

Primary findings of these assessments are that clinicians found the NHIT easy to use despite minimal training and experience using the NHIT, and that it did not degrade clinician efficiency or decision-making accuracy. These findings contrast common experiences with new EMR implementation.^32–35 Clinicians favored the NHIT and integrated communications software over traditional EMR displays and traditional means of communication. The NHIT appears to maintain or improve clinician confidence in decision making and communication while reducing the effort required finding salient information. The consistency of our data in favor of the NHIT suggests that information displays created using cognitive systems engineering methods to develop design requirements produce valid and acceptable IT solutions to real clinical challenges.

Usability assessment results demonstrate that the NHIT meets individual clinician needs for decision support, measured as their confidence in the final decision during the study. It also improved self-reported efficiency without compromising accuracy. Validation assessment results suggest that using the NHIT may enable a team with less experienced trainees to perform similarly to a team with more experienced trainees, as their time to reach sepsis diagnosis was similar. The NHIT may also improve experienced team performance, as it took 10 minutes less time to reach a decision about use of prone positioning to manage ARDS. All participants rated the NHIT as more effective than the legacy system to support decision making and communication. Improving team communication could also reduce communication errors, one of the top causes of sentinel events.³⁵

Effective IT systems support team efforts to develop shared understanding, enable members to detect problems earlier, evaluate options and provide better care.⁹ The NHIT supported both teams despite different experience levels. Consistent with naturalistic decision-making research on experts,³⁶ the more experienced resident in the validation assessment considered fewer options, and then evaluated each option until it could be ruled out. The resident with less than 1 year of experience used the NHIT to evaluate more potential diagnoses, including the correct diagnosis.

More accurate and efficient decisions may optimize patient outcomes by reducing potentially inappropriate care and avoiding unnecessary morbidity.⁸ Each team used the NHIT to share basic information about the patient’s current state, improving situation awareness.³⁷ The NHIT supported option evaluation, problem detection, sense making, and decision making,³⁸ enabling teams to develop a shared understanding of a patient’s status and evaluate plans for future care.

Reducing training time to learn new IT systems could save healthcare organizations significant costs. During the usability assessment, participants using the NHIT easily completed six information search and decision-making tasks and rated the experience favorably compared to the legacy EMR. During validation assessment, the NHIT improved clinician ability to find and share information effectively and efficiently.

Like the work of Patel³⁹ and Pickering,⁴⁰ we conducted field studies of clinicians to develop and refine novel health IT that supports cognitive work in the ICU. Pickering et.al.⁴¹ developed an interface to manage information overload and demonstrated that some interfaces effectively reduce clinician error and improve task load management.¹ Despite independent development, the NHIT shares many features of that system. Both use automated alerts and clinical decision support such as smart checklists to monitor EMR data, identify patterns, and trigger information presentation to clinicians.^25,42–44 These similarities suggest essential features for health IT systems supporting ICU clinician cognitive work: information organized by organ system for patients, a unit level view of all patients, automated alerts, and smart checklists. Our research also identified requirements for features that: display current care team members and facilitate their communication via a single, integrated system, that help clinicians synchronize daily schedules across patients, and that uniquely identify each patient and key data (Supplemental Table S3). Finally, the NHIT can be configured according to different clinical tasks, roles, experience levels, and patient types. Its modular design enables customized displays to match user preferences or evidence-based configurations and to be reconfigured as technology and clinical practice evolve.

Some limitations should be noted. The project is a single center study with a small number of subjects. In the usability assessment, we were only able to complete one legacy system assessment because it was impossible to remove outcomes in the historical data, making it difficult to blind subjects to previously made decisions. In the validation assessment, there were only two small subject teams and two clinical scenarios. Given a small sample, our analyses focused on process descriptions and consistent themes across different data sources. Our sample size and methodology limits generalizability of our findings, however the consistency of themes regarding ease of use and enhanced communication may be transferrable. During validation assessments, the simulated patients’ condition deteriorated rapidly, leading team members to spend more time at bedside and away from the IT systems. This limited time spent using the IT system after approximately half way through each scenario. Finally, teams only had to care for one patient during scenarios, rather than multiple ICU patients that typically compete for clinician time and thought. Even so, the conduct of these assessments using clinically relevant and complex scenarios and in a natural work setting (in the case of the validation assessment) contributed to the robustness of our findings. Those who conduct studies like our validation assessment should consider a more gradual change in patient condition and simulating more than one patient.

Future Work

We plan to test the NHIT further with a larger sample in a different facility to generalize our findings. We also plan to add three features we initially identified as being essential to clinical work: data entry, scheduling, and checklists. The NHIT will also be paired with other clinical management systems (e.g., medical devices, databases), to serve as a data integrator. Finally, validating the system’s machine learning algorithms will make it possible to identify patterns in data such as trends, comparable patients, and care plans.

Use of a Cognitive Systems Engineering approach has resulted in a novel health IT system that effectively supports clinician cognitive work and that clinicians favored over a traditional EMR. The NHIT is designed to be modular and to evolve with science, new technologies, and changes in clinical practice. When compared with a legacy system that participants had used for years, the NHIT was rated as easier to use, despite minimal training, and yielded equivalent to slightly improved results in diagnostic efficiency. This work contributes to a growing body of research that demonstrates how rigorous cognitive research can yield IT systems that improve clinician performance, and, as a consequence, could improve patient outcomes.

CONCLUSIONS

This is one of few manuscripts describing the effect of IT solutions on a medical joint cognitive system.^45,46 Following minimal training, clinicians effectively used the novel IT while performing realistic patient care for highly complex, simulated patients. For similarly complex care scenarios, a less experienced team performed similarly to a more experienced team. In both studies, clinicians expressed preference for the NHIT, particularly its messaging support and data organization/trend visualizations. Two findings – similar or improved decision making plus clinician preferences – suggest that novel IT systems such as the NHIT, could improve team based clinical decision making and ultimately patient safety and outcomes. Finally, this study demonstrates how cognitive systems engineering approaches to developing health IT may produce intuitive solutions that match clinical priorities and workflows and consequently require minimal training for adoption and use.

FUNDING SOURCES

This work is supported by the Joint Program Committee 1 Health Information Technology and Informatics portfolio under Contract Number W81XWH-12-C-0126

CONFLICT OF INTEREST

The authors have no conflicts of interest to disclose.

ACKNOWLEDGMENTS

In the conduct of research where humans are the subjects, the investigator(s) adhered to the policies regarding the protection of human subjects as prescribed by Code of Federal Regulations (CFR) Title 45, Volume 1, Part 46; Title 32, Chapter 1, Part 219; and Title 21, Chapter 1, Part 50 (Protection of Human Subjects).

The authors are grateful to their colleagues at AISR (LTC Kevin Chung, MD, LTC Elizabeth Mann-Salinas, PhD, Nicole Caldwell, RN), at ARA (Christopher Argenta, Charlie Fisher, Anna Grome, Jeff Brown, Shilo Anders, PhD, Rob Strouse, Elizabeth Papautsky, PhD, Cindy Dominguez, PhD) and Beth Crandall (Crandall Consulting), as well as MAJ Heather Delaney, MD, Chief of Medical Simulation, Mr. Robert Coffman, Program Coordinator, and Mr. Thomas Kai, Simulation Technician, COL Booker King, Director and COL John Melvin, Chief Nurse, U.S. Army Burn Center, as well as the nurse and support staff of the U.S. Army Burn Center Burn Intensive Care Unit and the clinicians who gave their time and support to this project. Finally, we appreciate the editorial support of Ms. Stacie Barczak during the final preparation of the manuscript.

The views, opinions and/or findings contained in this manuscript are those of the authors and do not necessarily reflect the views of the Department of Defense and should not be construed as an official DoD/Army position, policy or decision unless so designated by other documentation. No official endorsement should be made. Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the U.S. Government.