Abstract

Context Failure to notify patients of test results is common even when electronic health records (EHRs) are used to report results to practitioners. We sought to understand the broad range of social and technical factors that affect test result management in an integrated EHR-based health system.

Methods Between June and November 2010, we conducted a cross-sectional, web-based survey of all primary care practitioners (PCPs) within the Department of Veterans Affairs nationwide. Survey development was guided by a socio-technical model describing multiple inter-related dimensions of EHR use.

Findings Of 5001 PCPs invited, 2590 (51.8%) responded. 55.5% believed that the EHRs did not have convenient features for notifying patients of test results. Over a third (37.9%) reported having staff support needed for notifying patients of test results. Many relied on the patient's next visit to notify them for normal (46.1%) and abnormal results (20.1%). Only 45.7% reported receiving adequate training on using the EHR notification system and 35.1% reported having an assigned contact for technical assistance with the EHR; most received help from colleagues (60.4%). A majority (85.6%) stayed after hours or came in on weekends to address notifications; less than a third reported receiving protected time (30.1%). PCPs strongly endorsed several new features to improve test result management, including better tracking and visualization of result notifications.

Conclusions Despite an advanced EHR, both social and technical challenges exist in ensuring notification of test results to practitioners and patients. Current EHR technology requires significant improvement in order to avoid similar challenges elsewhere.

Background

Failure to follow up on abnormal test results (‘missed test results’) is a global patient safety concern.15 Electronic health records (EHRs) are increasingly used to notify practitioners of abnormal test results.68 EHR-based test result notification systems can reduce, if not eliminate, many of the communication problems inherent with paper-based records.912 However, ensuring that test results receive appropriate follow-up remains challenging even with electronic transmission.5,13,14 Previous studies of practitioner responses to abnormal test result notifications through EHRs found that 7–8% of abnormal test alerts had no evidence of timely follow-up, even when practitioners had electronically acknowledged result receipt.10,11,14

Although the proposed Stage 2 meaningful use regulations15 to be implemented in 2014 in the USA include structured laboratory result reporting functionality, little is known about how to optimize EHR-based test result management. Currently, several major EHR vendors employ a reporting functionality whereby providers receive notification of results in their inboxes (similar to email). This functionality is being increasingly adopted across EHRs and is likely to have a great impact on future test result management practices.

In complex healthcare settings that use EHRs, test result management encompasses many contextual factors such as clinical workflow, user behaviors, and organizational policies and procedures.16,17 Electronically transmitted test results may be missed or overlooked for a variety of reasons related to EHR design and use.18,19 For instance, while most EHRs deliver abnormal results to practitioners through an inbox, the inbox often also holds less important data (‘noise’) that could dilute important or urgent information (‘signals’).19 Other factors contributing to missed test results include policies and procedures related to the use of EHRs, such as unclear responsibility for follow-up of abnormal results when multiple providers are involved in patient care.10 Thus, contextual factors20 that affect test results management derive from both technical and social dimensions of the EHR-enabled health care system.

To better understand contextual factors that affect practitioners' test results management practices within the setting of a comprehensive EHR, we designed and administered a nationwide survey to primary care practitioners (PCPs) practicing in all facilities of the Department of Veterans Affairs (VA). Our objective was to use survey data on PCPs' experiences, practices, and preferences to create a body of knowledge upon which to base future improvements in EHR-based test result reporting systems.

Methods

Between June 10, 2010 and November 5, 2010, we conducted a cross-sectional, web-based survey of PCPs in VA settings nationwide. The local institutional review board approved the study. Within the VA system, most patients are assigned to staff PCPs, who serve as a coordinating hub for most care and thus depend heavily on the clinical information sent to their EHR inboxes. A smaller number of patients are assigned to trainees or subspecialists who see patients on a part-time basis, usually one half-day per week. Thus, these practitioners have a smaller panel (the total number of patients a PCP is responsible for) due to their other competing responsibilities. Our previous experience suggested that staff PCPs' experiences, practices, and preferences related to test result management might be different from those of trainees and subspecialists who serve as PCPs.16,18 Thus, we used a large administrative VA database, the Veterans Health Administration Support Service Center Clinical Care: Primary Care Management Module, to identify all PCPs with a minimum primary care patient panel size of 250 (N=5001). This strategy allowed us to exclude trainees and subspecialists because they are generally expected to have small panel sizes.

Survey development

Survey development was guided by literature review1,6,2134 and by a conceptual model describing multiple inter-related dimensions of EHR use (see ‘Survey content areas’ section below; also see table 1 which lists model dimensions).35 A psychometrician (CS) guided the survey development process, which included item writing and refinement, soliciting input from subject matter experts in EHR use, and iterative content review. After refining all survey items, we pilot tested the survey with 10 PCPs for readability, clarity, and ease of completion in a web-based format. With the exception of demographic items and 10 open-ended items, items were rated on a 5-point Likert-type scale with response options ranging from ‘strongly disagree’ to ‘strongly agree.’ Survey completion time was approximately 20–25 min.

Table 1

Eight interactive socio-technical health information technology (HIT) dimensions addressed for missed test results

Dimension Definition 
Hardware and software Equipment and software required to run the healthcare applications 
Clinical content Data, information, and knowledge entered, displayed, or transmitted in EHRs 
Human computer interface Aspects of the EHR system that users interact with (eg, see, touch, or hear) 
People Humans involved in the design, development, implementation, and use of HIT 
Workflow and communication Work processes needed to ensure that each patient receives the care they need at the time they need it 
Organizational policies and procedures Internal culture, structures, policies, and procedures that affect all aspects of HIT management and healthcare 
External rules, regulations, and pressures External forces that facilitate or place constraints on the design, development, implementation, use, and evaluation of HIT in the clinical setting 
System measurement and monitoring Measurement of system availability, use, effectiveness, and unintended consequences of system use 
Dimension Definition 
Hardware and software Equipment and software required to run the healthcare applications 
Clinical content Data, information, and knowledge entered, displayed, or transmitted in EHRs 
Human computer interface Aspects of the EHR system that users interact with (eg, see, touch, or hear) 
People Humans involved in the design, development, implementation, and use of HIT 
Workflow and communication Work processes needed to ensure that each patient receives the care they need at the time they need it 
Organizational policies and procedures Internal culture, structures, policies, and procedures that affect all aspects of HIT management and healthcare 
External rules, regulations, and pressures External forces that facilitate or place constraints on the design, development, implementation, use, and evaluation of HIT in the clinical setting 
System measurement and monitoring Measurement of system availability, use, effectiveness, and unintended consequences of system use 

EHR, electronic health record.

Survey content areas

Hardware and software, content, and user interface

The VA uses the Computerized Patient Record System (CPRS) as its EHR in all its facilities. CPRS uses the ‘View Alert’ notification system to communicate test results (as well as other important clinical information) to practitioners through an inbox. The View Alert system also displays notifications as alerts with various priorities. Practitioners see their patients' alerts each time they log in to the system or switch between patient records (figure 1). Alerts remain within the inbox until read by the practitioner, but they may be removed automatically if unopened after a certain time period (eg, 14 or 30 days). Although this functionality is used across the VA, individual facilities have discretion over which types of alerts practitioners must receive (eg, they can allow flexibility for practitioners to turn off certain notifications, such as normal test results). Our survey items assessed PCPs' perceptions of the View Alert notification system, including their views on the content and quantity of alerts received, perceived ease of use, and use of the EHR's features for alert processing and follow-up.

Figure 1

Alert notification window in the VA Computerized Patient Record System (CPRS).

Figure 1

Alert notification window in the VA Computerized Patient Record System (CPRS).

People

Our survey assessed several characteristics of the respondents including age, gender, race, job classification (ie, academic physician, non-academic physician, nurse practitioner, physician assistant), years employed at the VA, native language (English or other), perceived adequacy of training related to the EHR notification system, and prior use of an EHR other than CPRS.

Workflow and communication

Workflow items assessed perceptions of alert burden, processes related to notifying patients of their test results, and practices used to support alert management.

Organizational features

These items assessed institutional cultural norms and expectancies regarding management of alerts, technical support for alert notifications, organizational support to facilitate patient notification, and the amount of protected time (ie, specifically designated and compensated time) clinicians were given by their organizations to manage alerts. Protected time was measured in hours per week; all other response choices and coding were the same as for other item categories on the 5-point Likert-type scale.

External rules and regulations

We assessed whether practitioners were aware of and adhered to a national VA policy36 released in 2009 requiring notification of patients within 14 days for both normal and abnormal test results.

System measurement and monitoring

We assessed perceptions of practices related to measuring and monitoring alert follow-up at the system level.

New features and functions to improve EHR-based notification

We inquired whether PCPs would endorse several potential strategies to improve EHR-based notification. Strategies were selected based on thematic areas of improvement identified in previous work8,10,11,16,18,37 and addressed both technical and social dimensions.

Survey administration

We solicited support from the section chiefs of primary care at 142 VA facilities nationwide by asking them to email all PCPs at their respective sites to orient them to the project. We subsequently invited all participants by sending a personalized email from the principal investigator (HS) that described the study and provided a link to the web-based survey instrument. To increase response rates, invitation emails and subsequent reminders were followed by telephone attempts to reach non-respondents. In keeping with VA policies, we did not use monetary or other incentives for participation.

Data analysis

Data were downloaded from the internet survey administration service and were analyzed using SPSS statistical software. We generated descriptive statistics to summarize the characteristics of respondents and to aggregate responses to other survey items using descriptive statistics. Likert-type item responses were collapsed into three categories of ‘agree or strongly agree,’ ‘disagree or strongly disagree,’ and ‘neither agree nor disagree.’

Results

Of 5001 PCPs invited, 2590 (51.8%) responded. Figure 2 shows the geographic distribution of the respondents by Veterans Integrated Service Network (regional divisions of VA). Table 2 shows the characteristics of the respondents. Characteristics of non-respondents were not available for comparison with the respondent group. The vast majority of respondents had considerable experience with the VA EHR, having worked within the VA for 2 or more years. Less than half (45.7%) reported having received sufficient training on the View Alert system, and only a minority (13.7%) reported any refresher training (data not shown in the table). Nevertheless, the majority believed they had the knowledge (74.4%) and proficiency (81.8%) necessary to use the View Alert system (data not shown in the table). Nearly half (46.6%) also had prior experience using a non-VA EHR. Of these, only 19% of providers thought that the non-VA EHRs they used were better than VA's CPRS; 55% indicated that the non-VA EHR they used was overall inferior to CPRS, and 26% perceived it was about the same. No particular non-VA EHR was consistently identified as better or worse than CPRS.

Table 2

Characteristics of survey respondents (n=2590)

Characteristic n (%) 
Age 
   20–39 338 (13.1) 
   40–49 685 (26.4) 
   50–59 961 (37.1) 
   60 and over 402 (15.5) 
   Missing 204 (7.9) 
Gender 
   Male 1080 (41.7) 
   Female 1343 (51.9) 
   Missing 167 (6.4) 
Race 
   White 1630 (62.9) 
   Black 118 (4.6) 
   Asian 431 (16.6) 
   Other 188 (7.3) 
   Missing 223 (8.6) 
Job classification 
   Physician, academic 438 (16.9) 
   Physician, non-academic 1228 (47.4) 
   Nurse practitioner 561 (21.7) 
   Physician assistant 204 (7.9) 
   Missing 159 (6.1) 
Years at VA 
   <2 437 (16.9) 
   2–10 1219 (47.1) 
   11–20 589 (22.7) 
   >20 201 (7.8) 
   Missing 144 (5.6) 
Native language 
   English 1911 (73.8) 
   Other 498 (19.2) 
   Missing 181 (7.0) 
Characteristic n (%) 
Age 
   20–39 338 (13.1) 
   40–49 685 (26.4) 
   50–59 961 (37.1) 
   60 and over 402 (15.5) 
   Missing 204 (7.9) 
Gender 
   Male 1080 (41.7) 
   Female 1343 (51.9) 
   Missing 167 (6.4) 
Race 
   White 1630 (62.9) 
   Black 118 (4.6) 
   Asian 431 (16.6) 
   Other 188 (7.3) 
   Missing 223 (8.6) 
Job classification 
   Physician, academic 438 (16.9) 
   Physician, non-academic 1228 (47.4) 
   Nurse practitioner 561 (21.7) 
   Physician assistant 204 (7.9) 
   Missing 159 (6.1) 
Years at VA 
   <2 437 (16.9) 
   2–10 1219 (47.1) 
   11–20 589 (22.7) 
   >20 201 (7.8) 
   Missing 144 (5.6) 
Native language 
   English 1911 (73.8) 
   Other 498 (19.2) 
   Missing 181 (7.0) 

VA, Department of Veterans Affairs.

Figure 2

Distribution of respondents within 21 Veterans Affairs Networks across the USA (Veterans Integrated Service Networks or VISNs).

Figure 2

Distribution of respondents within 21 Veterans Affairs Networks across the USA (Veterans Integrated Service Networks or VISNs).

Table 3 shows the distribution of responses of selected items related to use and perceptions of the EHR technology (hardware/software, clinical content, and human–computer interface). Although most PCPs found the EHR alert notification system easy to use, more than half believed that the EHR software did not have convenient features for notifying patients of test results. Less than two-thirds of practitioners used a basic sorting feature for alerts, and just over a third used an enhanced functionality to process alerts. Both of these features are intended to promote efficiency when managing alerts. In addition, a significant proportion of PCPs had not altered their alert notification filters to help customize the amount and types of alerts they received. Thus, they received only notifications determined important (and set as default) by the facility (43.0%; data not shown).

Table 3

Technology-related items associated with EHR-based test result management

 Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing 
Item n (%) n (%) n (%) n (%) 
Hardware and software 
   CPRS has convenient features for notifying patients of test results 628 (24.2) 1437 (55.5) 421 (16.3) 104 (4.0) 
   Has a convenient way of generating letters in CPRS for patient notification 1091 (42.1) 1042 (40.2) 347 (13.4) 110 (4.2) 
   Has an automated voice messaging system that can notify patients of test results 38 (1.5) 2337 (90.2) 113 (4.4) 102 (3.9) 
   Uses My HealtheVet* to notify patients of test results 45 (1.7) 2308 (89.1) 134 (5.2) 103 (4.0) 
Clinical content 
   Acknowledges (ie, clicks on) all high priority alert notifications 2365 (91.3) 80 (3.1) 112 (4.3) 33 (1.3) 
   Follows up on all high priority alert notifications received 2372 (91.6) 77 (3.0) 97 (3.7) 44 (1.7) 
   Acknowledges all alert notifications received regardless of priority 2169 (83.7) 228 (8.8) 146 (5.6) 47 (1.8) 
Human–computer interface 
   Finds the alert notification system in CPRS easy to use† 1842 (71.1) 326 (12.6) 393 (15.2) 29 (1.1) 
   Follows up on all alert notifications received 1968 (76.0) 311 (12.0) 258 (10.0) 53 (2.0) 
   Alert notification system in CPRS makes it possible for providers to miss test results 1440 (55.6) 555 (21.4) 506 (19.5) 89 (3.4) 
   Sorts alert notifications when necessary according to urgency, patient name, location, or alert date/time 1625 (62.7) 562 (21.7) 381 (14.7) 22 (0.8) 
   Uses the Process All function of the system when necessary‡ 981 (37.9) 1219 (47.1) 367 (14.2) 23 (0.9) 
 Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing 
Item n (%) n (%) n (%) n (%) 
Hardware and software 
   CPRS has convenient features for notifying patients of test results 628 (24.2) 1437 (55.5) 421 (16.3) 104 (4.0) 
   Has a convenient way of generating letters in CPRS for patient notification 1091 (42.1) 1042 (40.2) 347 (13.4) 110 (4.2) 
   Has an automated voice messaging system that can notify patients of test results 38 (1.5) 2337 (90.2) 113 (4.4) 102 (3.9) 
   Uses My HealtheVet* to notify patients of test results 45 (1.7) 2308 (89.1) 134 (5.2) 103 (4.0) 
Clinical content 
   Acknowledges (ie, clicks on) all high priority alert notifications 2365 (91.3) 80 (3.1) 112 (4.3) 33 (1.3) 
   Follows up on all high priority alert notifications received 2372 (91.6) 77 (3.0) 97 (3.7) 44 (1.7) 
   Acknowledges all alert notifications received regardless of priority 2169 (83.7) 228 (8.8) 146 (5.6) 47 (1.8) 
Human–computer interface 
   Finds the alert notification system in CPRS easy to use† 1842 (71.1) 326 (12.6) 393 (15.2) 29 (1.1) 
   Follows up on all alert notifications received 1968 (76.0) 311 (12.0) 258 (10.0) 53 (2.0) 
   Alert notification system in CPRS makes it possible for providers to miss test results 1440 (55.6) 555 (21.4) 506 (19.5) 89 (3.4) 
   Sorts alert notifications when necessary according to urgency, patient name, location, or alert date/time 1625 (62.7) 562 (21.7) 381 (14.7) 22 (0.8) 
   Uses the Process All function of the system when necessary‡ 981 (37.9) 1219 (47.1) 367 (14.2) 23 (0.9) 

Italicized items are discussed in the text.

*My HealtheVet is a free, online personal health record for VA patients.

†CPRS (Computerized Patient Record System) is the Department of Veterans Affairs electronic health record software.

‡The Process All feature allows clinicians to efficiently process alerts one after another without returning to the View Alert window.

EHR, electronic health record; VA, Department of Veterans Affairs.

Table 4 shows the distribution of responses for survey items related to workflow and communication. A majority of practitioners stayed after hours or came in on weekends to address alerts; only about a third had remote access to the EHR to manage alerts after hours. More than half of PCPs electronically assigned a surrogate or covering practitioner to handle their alerts when out of the office, a process we defined as ‘electronic hand-off.’ Just over a third reported having the help needed for notifying patients of test results. Patient notification varied between normal and abnormal results. Almost half of PCPs reported that they did not immediately notify patients of normal test results and relied on the patient's next visit to notify them, whereas about one-fifth relied on the next visit to notify patients of abnormal results.

Table 4

Workflow and communication items associated with EHR-based test result management

 Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing 
Item n (%) n (%) n (%) n (%) 
Workflow and communication 
   Alert burden 
      The number of alerts exceeds what they can effectively manage 1803 (69.6) 321 (12.4) 432 (16.7) 34 (1.3) 
      Receives too many alerts to easily focus on most important ones 2078 (80.2) 215 (8.3) 244 (9.4) 53 (2.0) 
      In the past year, missed test results that led to delayed patient care 772 (29.8) 1212 (46.8) 537 (20.7) 69 (2.7) 
      Uses remote access after hours or on weekends to manage alerts 893 (34.5) 1327 (51.2) 320 (12.4) 50 (1.9) 
      Has remote access to CPRS 953 (36.8) 1599 (61.7) – 38 (1.5) 
      Stays after hours or comes in on weekends to manage alerts 2218 (85.6) 219 (8.5) 105 (4.1) 48 (1.9) 
      Wishes the system provided them with more alerts 151 (5.8) 2190 (84.6) 208 (8.0) 41 (1.6) 
      Receives too many alert notifications per day 2251 (86.9) 164 (6.3) 166 (6.4) 9 (0.3) 
      Uses additional paper-based methods to help follow test results 1451 (56.0) 1103 (42.6) – – 
      Gets too many FYI only alert notifications that require a signature even though no action on their part is required 2071 (80.0) 250 (9.7) 258 (10.0) 11 (0.4) 
      Often receives alert notifications where they are unsure as to why they were sent to them 1601 (61.8) 539 (20.8) 432 (16.7) 18 (0.7) 
   Patient notification 
      Consistently notifies patients of abnormal test results 2146 (82.9) 151 (5.8) 185 (7.1) 108 (4.2) 
      Has the help needed for notifying patients of test results 981 (37.9) 1180 (45.6) 332 (12.8) 97 (3.7) 
      Consistently notifies patients of normal test results 1167 (45.1) 902 (34.8) 415 (16.0) 106 (4.1) 
      Relies on a patient's next visit to notify them of their abnormal test results 521 (20.1) 1556 (60.1) 407 (15.7) 106 (4.1) 
      Relies on a patient's next visit to notify them of their normal test results 1193 (46.1) 858 (33.1) 431 (16.6) 108 (4.2) 
   Use of surrogates 
      Alert notifications related to surrogates create new safety concerns 1339 (51.7) 482 (18.6) 730 (28.2) 39 (1.5) 
      Has support staff to assist with management of test result alert notifications 873 (33.7) 1304 (50.3) 311 (12.0) 102 (3.9) 
      Assigns a surrogate to take care of alert notifications when out of the office 1525 (58.9) 788 (30.4) 239 (9.2) 38 (1.5) 
 Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing 
Item n (%) n (%) n (%) n (%) 
Workflow and communication 
   Alert burden 
      The number of alerts exceeds what they can effectively manage 1803 (69.6) 321 (12.4) 432 (16.7) 34 (1.3) 
      Receives too many alerts to easily focus on most important ones 2078 (80.2) 215 (8.3) 244 (9.4) 53 (2.0) 
      In the past year, missed test results that led to delayed patient care 772 (29.8) 1212 (46.8) 537 (20.7) 69 (2.7) 
      Uses remote access after hours or on weekends to manage alerts 893 (34.5) 1327 (51.2) 320 (12.4) 50 (1.9) 
      Has remote access to CPRS 953 (36.8) 1599 (61.7) – 38 (1.5) 
      Stays after hours or comes in on weekends to manage alerts 2218 (85.6) 219 (8.5) 105 (4.1) 48 (1.9) 
      Wishes the system provided them with more alerts 151 (5.8) 2190 (84.6) 208 (8.0) 41 (1.6) 
      Receives too many alert notifications per day 2251 (86.9) 164 (6.3) 166 (6.4) 9 (0.3) 
      Uses additional paper-based methods to help follow test results 1451 (56.0) 1103 (42.6) – – 
      Gets too many FYI only alert notifications that require a signature even though no action on their part is required 2071 (80.0) 250 (9.7) 258 (10.0) 11 (0.4) 
      Often receives alert notifications where they are unsure as to why they were sent to them 1601 (61.8) 539 (20.8) 432 (16.7) 18 (0.7) 
   Patient notification 
      Consistently notifies patients of abnormal test results 2146 (82.9) 151 (5.8) 185 (7.1) 108 (4.2) 
      Has the help needed for notifying patients of test results 981 (37.9) 1180 (45.6) 332 (12.8) 97 (3.7) 
      Consistently notifies patients of normal test results 1167 (45.1) 902 (34.8) 415 (16.0) 106 (4.1) 
      Relies on a patient's next visit to notify them of their abnormal test results 521 (20.1) 1556 (60.1) 407 (15.7) 106 (4.1) 
      Relies on a patient's next visit to notify them of their normal test results 1193 (46.1) 858 (33.1) 431 (16.6) 108 (4.2) 
   Use of surrogates 
      Alert notifications related to surrogates create new safety concerns 1339 (51.7) 482 (18.6) 730 (28.2) 39 (1.5) 
      Has support staff to assist with management of test result alert notifications 873 (33.7) 1304 (50.3) 311 (12.0) 102 (3.9) 
      Assigns a surrogate to take care of alert notifications when out of the office 1525 (58.9) 788 (30.4) 239 (9.2) 38 (1.5) 

Italicized items are discussed in the text.

CPRS, Computerized Patient Record System; EHR, electronic health record; FYI, for your information.

Table 5 summarizes responses to items that assessed people and organizational features affecting alert management. Only about a third of respondents reported having or accessing an assigned technical contact to help with alerts; most admitted receiving help with alert management from colleagues. About a third of respondents reported receiving protected time for alert management; of these, most respondents reported it to be at least 4 hours per week (69.8%).

Table 5

Organizational and people items associated with EHR-based test result management

 Agree or strongly agree
 
Disagree or strongly disagree Neither agree nor disagree
 
Missing
 
Item n (%) n (%) n (%) n (%) 
Organizational policies and procedures 
   Technical support 
      Clinic has an assigned technical contact that can be accessed for help with alert notifications 908 (35.1) 947 (36.6) 651 (25.1) 84 (3.2) 
      IT help person provides useful resources 621 (24.0) 882 (34.1) 988 (38.1) 99 (3.8) 
      With questions about or problems with alert notifications, gets the help needed from IT 887 (34.2) 809 (31.2) 807 (31.2) 87 (3.4) 
      With questions about or problems with alert notifications, gets the help needed from colleagues 1564 (60.4) 348 (13.4) 587 (22.7) 91 (3.5) 
   Protected time 
      Has protected clinical time slots to manage alert notifications779 (30.1) 1773 (68.5)  38 (1.5) 
People 
   Supportive norms 
      Colleagues believe the alert notifications in CPRS help them get their job done effectively 640 (24.7) 849 (32.8) 1041 (40.2) 60 (2.3) 
      Supervisor believes alert notifications in CPRS are an essential component of effective primary care 1403 (54.2) 143 (5.5) 981 (37.9) 63 (2.4) 
      Senior management has emphasized the importance of the use of alert notifications in high quality care 1359 (52.5) 189 (7.3) 981 (37.9) 61 (2.4) 
   Performance expectancy 
      Using alert notifications in CPRS enhances providers' ability to provide safe patient care 2098 (81.0) 167 (6.4) 275 (10.6) 50 (1.9) 
      Using alert notifications in CPRS enhances providers' effectiveness on the job 1803 (69.6) 311 (12.0) 423 (16.3) 53 (2.0) 
      Using alert notifications in CPRS increases providers' productivity 952 (36.8) 858 (33.1) 723 (27.9) 57 (2.2) 
      Using alert notifications in CPRS allows providers to meet performance standards 1312 (50.7) 542 (20.9) 679 (26.2) 57 (2.2) 
 Agree or strongly agree
 
Disagree or strongly disagree Neither agree nor disagree
 
Missing
 
Item n (%) n (%) n (%) n (%) 
Organizational policies and procedures 
   Technical support 
      Clinic has an assigned technical contact that can be accessed for help with alert notifications 908 (35.1) 947 (36.6) 651 (25.1) 84 (3.2) 
      IT help person provides useful resources 621 (24.0) 882 (34.1) 988 (38.1) 99 (3.8) 
      With questions about or problems with alert notifications, gets the help needed from IT 887 (34.2) 809 (31.2) 807 (31.2) 87 (3.4) 
      With questions about or problems with alert notifications, gets the help needed from colleagues 1564 (60.4) 348 (13.4) 587 (22.7) 91 (3.5) 
   Protected time 
      Has protected clinical time slots to manage alert notifications779 (30.1) 1773 (68.5)  38 (1.5) 
People 
   Supportive norms 
      Colleagues believe the alert notifications in CPRS help them get their job done effectively 640 (24.7) 849 (32.8) 1041 (40.2) 60 (2.3) 
      Supervisor believes alert notifications in CPRS are an essential component of effective primary care 1403 (54.2) 143 (5.5) 981 (37.9) 63 (2.4) 
      Senior management has emphasized the importance of the use of alert notifications in high quality care 1359 (52.5) 189 (7.3) 981 (37.9) 61 (2.4) 
   Performance expectancy 
      Using alert notifications in CPRS enhances providers' ability to provide safe patient care 2098 (81.0) 167 (6.4) 275 (10.6) 50 (1.9) 
      Using alert notifications in CPRS enhances providers' effectiveness on the job 1803 (69.6) 311 (12.0) 423 (16.3) 53 (2.0) 
      Using alert notifications in CPRS increases providers' productivity 952 (36.8) 858 (33.1) 723 (27.9) 57 (2.2) 
      Using alert notifications in CPRS allows providers to meet performance standards 1312 (50.7) 542 (20.9) 679 (26.2) 57 (2.2) 

Italicized items are discussed in the text.

*Of the 779 providers who reported receiving protected time, 224 (28.8%) reported <4 h/week, 544 (69.8%) reported ≥4 h/week, and 11 (1.4%) did not report the number of hours of protected time.

CPRS, Computerized Patient Record System; EHR, electronic health record.

About half of respondents (54.2%) were aware of the 2009 Veterans Health Administration (VHA) policy (ie, external rules and regulations) regarding notification of patients regarding test results. However, among those aware of the policy only 20% reported that they changed their result notification practices accordingly (data not shown in tables). Almost a third (31.9%) reported that their supervisors monitored how they managed notifications, but fewer (23.9%) received feedback on their follow-up practices (ie, system measurement and monitoring).

Table 6 summarizes PCPs' assessments of potential new features and functions to improve EHR-based notification. PCPs strongly endorsed four functionalities to reduce loss of alert information in the EHR (preventing automated deletion of alerts, being able to retrieve deleted alerts, having a back button, and being able to access alerts in the inbox for at least 30 days). To improve alert management options, most agreed with the need for a feature to remind them at a later date of the necessity to take a follow-up action. In addition, most PCPs endorsed the need for a separate messaging system within the EHR to allow providers to communicate, rather than the traditional method of including these human-generated messages along with the EHR-generated notification system that was used for result notification. About two-thirds of respondents endorsed one or more new visualization techniques, including a separate window for high-priority alerts or a method to filter or color-code alerts based on type. Most PCPs supported strategies to improve safety related to hand-offs, including being able to assign responsibility of test result follow-up and to display who is responsible for follow-up. About half were in favor of receiving feedback about their alert management performance.

Table 6

New features and functions to improve EHR-based notification

 Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing 
Item n (%) n (%) n (%) n (%) 
Hardware and software 
   Improving communication and alert management options 
      I would like to be able to set reminders for myself for future actions 2160 (83.4) 64 (2.5) 213 (8.2) 153 (5.9) 
      I would like to have a messaging system within CPRS that would allow providers to communicate with one another - this would be outside the View Alert system 1826 (70.5) 174 (6.7) 444 (17.1) 146 (5.6) 
Human–computer interface 
   Improving alert visualization 
      I would like to receive high priority test result notifications in one window, and all other alert notifications in another window 1611 (62.2) 474 (18.3) 370 (14.3) 135 (5.2) 
      I would like an option to display only certain alert notifications at a time (ie, filter to display only surrogate, inpatient, or high priority alerts) 1722 (66.5) 321 (12.4) 398 (15.4) 149 (5.8) 
      I would like to have my alert notifications color-coded according to type (eg, surrogate, inpatient, or high priority alerts) 1720 (66.4) 316 (12.2) 414 (16.0) 140 (5.4) 
   Better processing and tracking of alerts 
      I would like to be able to retrieve my deleted alert notifications 2036 (78.6) 193 (7.5) 228 (8.8) 133 (5.1) 
      High priority alert notifications should not disappear until I actively delete them after taking follow-up action 1913 (73.9) 266 (10.3) 273 (10.5) 138 (5.3) 
      There should be a mechanism in the alert notification system in CPRS to display the name of the person responsible for following up on the test result alert 1596 (61.6) 292 (11.3) 557 (21.5) 145 (5.6) 
      I would like to have a ‘back button’ in CPRS to retrieve the prior window 2173 (83.9) 67 (2.6) 212 (8.2) 138 (5.3) 
      All unacknowledged alert notifications should stay in the alert notification window for at least 30 days 1689 (65.2) 318 (12.3) 445 (17.2) 138 (5.3) 
Workflow and communication 
   Improving the surrogate process 
      As a surrogate, I should only receive high priority alert notifications for patients assigned to the provider who is out of the office 1659 (64.1) 432 (16.7) 360 (13.9) 139 (5.4) 
      Currently it is not possible for a provider to directly assign, without IT assistance, more than one surrogate in CPRS. This capability should be introduced 1555 (60.0) 264 (10.2) 618 (23.9) 153 (5.9) 
System measurement and monitoring 
   Improving feedback  
      I would like to receive feedback about my performance related to follow-up of high priority alert notifications 1264 (48.8) 479 (18.5) 708 (27.3) 139 (5.4) 
 Agree or strongly agree Disagree or strongly disagree Neither agree nor disagree Missing 
Item n (%) n (%) n (%) n (%) 
Hardware and software 
   Improving communication and alert management options 
      I would like to be able to set reminders for myself for future actions 2160 (83.4) 64 (2.5) 213 (8.2) 153 (5.9) 
      I would like to have a messaging system within CPRS that would allow providers to communicate with one another - this would be outside the View Alert system 1826 (70.5) 174 (6.7) 444 (17.1) 146 (5.6) 
Human–computer interface 
   Improving alert visualization 
      I would like to receive high priority test result notifications in one window, and all other alert notifications in another window 1611 (62.2) 474 (18.3) 370 (14.3) 135 (5.2) 
      I would like an option to display only certain alert notifications at a time (ie, filter to display only surrogate, inpatient, or high priority alerts) 1722 (66.5) 321 (12.4) 398 (15.4) 149 (5.8) 
      I would like to have my alert notifications color-coded according to type (eg, surrogate, inpatient, or high priority alerts) 1720 (66.4) 316 (12.2) 414 (16.0) 140 (5.4) 
   Better processing and tracking of alerts 
      I would like to be able to retrieve my deleted alert notifications 2036 (78.6) 193 (7.5) 228 (8.8) 133 (5.1) 
      High priority alert notifications should not disappear until I actively delete them after taking follow-up action 1913 (73.9) 266 (10.3) 273 (10.5) 138 (5.3) 
      There should be a mechanism in the alert notification system in CPRS to display the name of the person responsible for following up on the test result alert 1596 (61.6) 292 (11.3) 557 (21.5) 145 (5.6) 
      I would like to have a ‘back button’ in CPRS to retrieve the prior window 2173 (83.9) 67 (2.6) 212 (8.2) 138 (5.3) 
      All unacknowledged alert notifications should stay in the alert notification window for at least 30 days 1689 (65.2) 318 (12.3) 445 (17.2) 138 (5.3) 
Workflow and communication 
   Improving the surrogate process 
      As a surrogate, I should only receive high priority alert notifications for patients assigned to the provider who is out of the office 1659 (64.1) 432 (16.7) 360 (13.9) 139 (5.4) 
      Currently it is not possible for a provider to directly assign, without IT assistance, more than one surrogate in CPRS. This capability should be introduced 1555 (60.0) 264 (10.2) 618 (23.9) 153 (5.9) 
System measurement and monitoring 
   Improving feedback  
      I would like to receive feedback about my performance related to follow-up of high priority alert notifications 1264 (48.8) 479 (18.5) 708 (27.3) 139 (5.4) 

CPRS, Computerized Patient Record System; EHR, electronic health record.

Interpretation

Test result reporting has substantial patient safety implications and is now being considered in the meaningful use criteria for EHRs.38 However, our data suggest that despite the use of an advanced EHR system, both social and technical challenges exist in ensuring the reliability of test result notification to practitioners and patients within one of the world's largest healthcare systems. Other healthcare systems, most of which have a shorter history of EHR use, are likely to face similar challenges as they begin adopting EHRs. Several technical as well as social (personal, workflow, and organizational) factors need to be addressed in order for EHR-based test reporting functionality to be successful. PCPs in our sample endorsed several new features and functions to reduce loss of information in the EHR and to improve visualization of alerts that communicate test results.

We found that most providers do not routinely notify patients of normal test results, and a substantial proportion use the next scheduled visit to notify patients of both normal and abnormal results. A 1996 study conducted outside the VA reached similar conclusions, suggesting that this pattern has changed little in the last two decades.30 Recent initiatives within the VA are developing additional guidance to address test result management, which might change some of the patterns we found.39 As healthcare quality improvement efforts increasingly emphasize patient engagement, alternative methods are being explored for patients to receive test results. For instance, results are increasingly accessible to patients through online portals,40 although adoption has been slower than expected.41,42 Furthermore, the Department of Health and Human Services recently proposed a rule43 allowing patients to access test results directly from the laboratory upon patient request (ie, bypassing the provider). However, providers' interpretation of test results within the context of the patient's other clinical conditions remains essential.44 Because most PCPs receive hundreds of test results a week,31 one area for potential improvement is PCPs' resources to facilitate patient notification. Just over a third of the PCPs reported having administrative support for patient notification, presenting an opportunity to leverage the current shift towards team-based models of care (eg, medical home teams45). For example, with appropriate task delegation and clarity of roles, other members of the team could be responsible for tracking results and notifying patients.45 Future EHR development should focus on innovative methods to facilitate patient notification of test results.

Our findings underscore the additional time burden induced by EHR-based alert systems, with the majority of respondents staying after hours, coming in on weekends, or using remote access from home to deal with notifications. Unfortunately, there is little or no reimbursement for non-face-to-face time related to documentation, follow-up, and patient notification of test results. Although the interventions proposed in the survey focused mostly on technological features and workflow, several organizational and policy interventions are also needed to improve the outcomes of EHR-based test results reporting.

Based on our data, we recommend several strategies for improving test results reporting though EHRs. First, being able to easily retrieve and readily access critical test result data is an absolute requirement for safe and efficient care, and the ability to do so (eg, to retrieve previously viewed alerts) was endorsed by a large majority of respondents. For instance, providers might accidently delete a result notification or need to return to an alert they processed some time ago, and these workflow requirements should inform EHR design. Second, to effectively address alert-related information overload,19 providers need better methods to display, sort, and visualize test result information46 according to patient, date, type, priority, and origin of alerts. Usability testing is essential to evaluate EHR support features for planning and prioritizing among high volumes of alerts. Third, most providers agreed with the need for strategies to improve electronic hand-offs in the EHR, including assignment of responsibility. There is surprisingly little knowledge to guide effective policy setting or practice management in this area. While health information technology can prevent communication breakdowns,4750 vulnerabilities related to teamwork and care coordination51,52 have received less attention, and novel methods to reduce ambiguities related to these hand-offs are needed. Lastly, EHRs need to be able to support the ‘prospective’ memory53 of providers (memory of an intent to perform a future task) by facilitating alert tracking and self-reminders for future tasks. Some of these features should be deemed as universal requirements in EHRs.

Most of our results are generalizable outside the VA even though there are differences in EHR features, usage, organizational, workflow, and policy-related factors. This is due to the fact that the VA has been a leader in the successful integration and use of comprehensive EHRs, and many of our findings are valuable for other healthcare institutions that are moving to more integrated, technology-enabled care environments as required to meet the new patient management challenges prescribed by accountable care organizations.54 Although the alert notification system was specific to the VA's EHR, most major EHRs use similar methods to notify providers of important clinical findings, and others will follow in compliance with meaningful use requirements. Our study had a relatively low response rate. However, this is comparable to or higher than the response rates of many other physician surveys,5558 which traditionally are not very high.59,60 In fact, the large number of completed surveys represents one of the largest total responses regarding EHR characteristics ever reported.61,62 Studies that report higher response rates for physician surveys often utilize monetary incentives for participants (which may lead to a different bias in the results), a practice not permitted for data collection within the VA. Lastly, we did not have access to any demographic or practice-based characteristics of non-respondent physicians for comparison.

Overall, our survey data suggest that current capabilities for test result management are limited even within a well-established, mature EHR. A comprehensive socio-technical approach is needed to optimize EHR-based test result management. Key components of this socio-technical approach will include: (1) design, development, and testing of EHR features and functions to support physicians' test result management workflows and inclusion of these new features in all EHRs; (2) new policies and procedures at the local institutional and national level regarding appropriate methods and timeliness of patient notification; and (3) commitments from organizational leadership and payers to acknowledge and support the additional, non-face-to-face work required to provide care for patients in an EHR-enabled healthcare system.

Acknowledgements

We thank Annie Bradford, PhD for assistance with medical editing.

Funding

The research reported here was supported by the Department of Veterans Affairs National Center for Patient Safety and in part by the Houston VA HSR&D Center of Excellence (HFP90-020), the National Library of Medicine (R01 LM006942), and the Office of the National Coordinator for Health Information Technology (ONC #10510592).

Competing interests

None.

The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or other funding agencies. The corresponding author had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Ethics approval

Baylor College of Medicine Institutional Review Board approved this study.

Provenance and peer review

Not commissioned; externally peer reviewed.

References

1
Wahls
TL
Cram
PM
.
The frequency of missed test results and associated treatment delays in a highly computerized health system
.
BMC Fam Pract
 
2007
;
8
:
32
.
2
Schiff
GD
Hasan
O
Kim
S
et al
.
Diagnostic error in medicine: analysis of 583 physician-reported errors
.
Arch Intern Med
 
2009
;
169
:
1881
7
.
3
Lorincz
CY
Drazen
E
Sokol
PE
et al
.
Research in Ambulatory Patient Safety 2000–2010: a 10-year review
.
Am Med Assoc
 
2011
. . (accessed 3 Dec 2012).
4
Wynia
MK
Classen
DC
.
Improving ambulatory patient safety
.
JAMA
 
2011
;
306
:
22 2504
5
.
5
Callen
JL
Westbrook
JI
Georgiou
A
et al
.
Failure to follow-up test results for ambulatory patients: a systematic review
.
J Gen Intern Med
 
2011
.
6
Murff
HJ
Gandhi
TK
Karson
AK
et al
.
Primary care physician attitudes concerning follow-up of abnormal test results and ambulatory decision support systems
.
Int J Med Inform
 
2003
;
71
:
137
49
.
7
Poon
EG
Wang
SJ
Gandhi
TK
et al
.
Design and implementation of a comprehensive outpatient Results Manager
.
J Biomed Inform
 
2003
;
36
:
80
91
.
8
Singh
H
Vij
MS
.
Eight recommendations for policies for communicating abnormal test results
.
Jt Comm J Qual Patient Saf
 
2010
;
36
:
226
32
.
9
Tenner
CT
Shapiro
NM
Wikler
A
.
Improving health care provider notification in an academic setting: a cascading system of alerts
.
Arch Intern Med
 
2010
;
170
:
392
.
10
Singh
H
Thomas
EJ
Mani
S
et al
.
Timely follow-up of abnormal diagnostic imaging test results in an outpatient setting: are electronic medical records achieving their potential?
Arch Intern Med
 
2009
;
169
:
1578
86
.
11
Singh
H
Thomas
EJ
Sittig
DF
et al
.
Notification of abnormal lab test results in an electronic medical record: do any safety concerns remain?
Am J Med
 
2010
;
123
:
238
44
.
12
Singh
H
Arora
H
Vij
M
et al
.
Communication outcomes of critical imaging results in a computerized notification system
.
J Am Med Inform Assoc
 
2007
;
14
:
459
66
.
13
Casalino
LP
Dunham
D
Chin
MH
et al
.
Frequency of Failure to Inform Patients of Clinically Significant Outpatient Test Results
.
Arch Intern Med
 
2009
;
169
:
1123
9
.
14
Elder
NC
McEwen
TR
Flach
J
et al
.
The management of test results in primary care: does an electronic medical record make a difference?
Fam Med
 
2010
;
42
:
327
33
.
15
Health Information Technology: Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology, 2014 Edition; Revisions to the Permanent Certification Program for Health Information Technology
.
Office of the National Coordinator for Health Information Technology (ONC), Department of Health and Human Services
.
Federal Register
 
2012
(cited 2012 July 25);
77
:
13845
(accessed 3 Dec 2012)
16
Hysong
SJ
Sawhney
MK
Wilson
L
et al
.
Understanding the management of electronic test result notifications in the outpatient setting
.
BMC Med Inform Decis Mak
 
2011
;
11
:
22
.
17
Final Report: Summary review: Evaluation of Veterans Health Administration procedures for communicating abnormal test results
.
Report number 01-01965-24. 11-25-2002. Ref Type: Report
.
18
Hysong
SJ
Sawhney
MK
Wilson
L
et al
.
Provider management strategies of abnormal test result alerts: a cognitive task analysis
.
J Am Med Inform Assoc
 
2010
;
17
:
71
7
.
19
Singh
H
Spitzmueller
C
Petersen
NJ
et al
.
Information overload and missed test results in electronic health record-based settings
.
Arch Intern Med.
 
In press
.
20
Taylor
SL
Dy
S
Foy
R
et al
.
What context features might be important determinants of the effectiveness of patient safety practice interventions?
BMJ Qual Saf
 
2011
;
20
:
611
17
.
21
Yano
EM
Simon
B
Canelo
I
et al
.
1999 VHA survey of primary care practices
.
VA HSR&D Center of Excellence for the Study of Healthcare Provider Behavior. 00-MC12. 2000. VA HSR&D Center of Excellence for the Study of Healthcare Provider Behavior. Ref Type: Report
.
22
Forward momentum: Hospital use of information technology
. (
2005
(accessed 3 Dec 2012).
23
Lewis
JR
.
IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use
.
Int J Hum Comput Interact
 
1995
;
7
:
57
78
.
24
Clason
DL
Dormody
TJ
.
Analyzing data measured by individual Likert-type items
.
J Agricultural Educ
 
2000
;
35
:
31
6
.
25
Lyons
SS
Tripp-Reimer
T
Sorofman
BA
et al
.
VA QUERI informatics paper: information technology for clinical guideline implementation: perceptions of multidisciplinary stakeholders
.
J Am Med Inform Assoc
 
2005
;
12
:
64
71
.
26
Campbell
EG
Regan
S
Gruen
RL
et al
.
Professionalism in medicine: results of a national survey of physicians
.
Ann Intern Med
 
2007
;
147
:
795
802
.
27
Jha
AK
Ferris
TG
Donelan
K
et al
.
How common are electronic health records in the United States? A summary of the evidence
.
Health Aff (Millwood)
 
2006
;
25
:
w496
507
.
28
Cutler
DM
Feldman
NE
Horwitz
JR
.
U.S. adoption of computerized physician order entry systems
.
Health Aff (Millwood)
 
2005
;
24
:
1654
63
.
29
Ash
JS
Gorman
PN
Seshadri
V
et al
.
Computerized physician order entry in U.S. hospitals: results of a 2002 survey
.
J Am Med Inform Assoc
 
2004
;
11
:
95
9
.
30
Boohaker
E
Ward
RE
Uman
JE
et al
.
Patient notification and follow-up of abnormal test results. A physician survey
.
Arch Intern Med
 
1996
;
156
:
327
31
.
31
Poon
E
Gandhi
T
Sequist
T
et al
.
"I wish I had seen this test result earlier!": dissatisfaction with test result management systems in primary care
.
Arch Intern Med
 
2004
;
164
:
2223
8
.
32
Hickner
J
Graham
DG
Elder
NC
et al
.
Testing process errors and their harms and consequences reported from family medicine practices: a study of the American Academy of Family Physicians National Research Network
.
Qual Saf Health Care
 
2008
;
17
:
194
200
.
33
Elder
NC
Dovey
SM
.
Classification of medical errors and preventable adverse events in primary care: a synthesis of the literature
.
J Fam Pract
 
2002
;
51
:
927
32
.
34
Elder
NC
Pallerla
H
Regan
S
.
What do family physicians consider an error? A comparison of definitions and physician perception
.
BMC Fam Pract
 
2006
;
7
:
73
.
35
Sittig
DF
Singh
H
.
A new sociotechnical model for studying health information technology in complex adaptive healthcare systems
.
Qual Saf Health Care
 
2010
;
19
(
Suppl 3
):
i68
74
.
36
VHA Directive 2009–019: Ordering and Reporting Test Results. Veterans Health Administration
2009
(accessed 3 Dec 2012)
37
Singh
H
Wilson
L
Reis
B
et al
.
Ten strategies to improve management of abnormal test result alerts in the electronic health record
.
J Patient Saf
 
2010
;
6
:
121
23
.
38
Blumenthal
D
Tavenner
M
.
The "Meaningful Use" Regulation for Electronic Health Records
.
N Engl J Med
 
2010
;
363
:
501
4
.
39
Sittig
DF
Singh
H
.
Improving test result follow-up through electronic health records requires more than just an alert
.
J Gen Intern Med
 
2012
;
27
:
1235
37
.
40
Fraser
HS
Allen
C
Bailey
C
et al
.
Information systems for patient follow-up and chronic management of HIV and tuberculosis: a life-saving technology in resource-poor areas
.
J Med Internet Res
 
2007
;
9
:
e29
.
41
Archer
N
Fevrier-Thomas
U
Lokker
C
et al
.
Personal health records: a scoping review
.
J Am Med Inform Assoc
 
2011
;
18
:
515
22
.
42
Sittig
DF
.
Personal health records on the internet: a snapshot of the pioneers at the end of the 20th Century
.
Int J Med Inform
 
2002
;
65
:
1
6
.
43
CLIA Program and HIPAA Privacy Rule
.
Patients' Access to Test Reports (42 CFR 493, 45 CFR 164)
.
Fed Regist
 
2011
;
76
.
44
Davis
Giardina T
Singh
H
.
Should patients get direct access to their laboratory test results? An answer with many questions
.
JAMA
 
2011
;
306
:
2502
3
.
45
Singh
H
Graber
M
.
Reducing diagnostic error through medical home-based primary care reform
.
JAMA
 
2010
;
304
:
463
4
.
46
Woods
DD
Patterson
E
Roth
EM
.
Can we ever escape from data overload? A cognitive systems diagnosis
.
Cogn Technol Work
 
2002
;
4
:
22
36
.
47
Brantley
SD
Brantley
RD
.
Reporting significant unexpected findings: the emergence of information technology solutions
.
J Am Coll Radiol
 
2005
;
2
:
304
7
.
48
Kuperman
GJ
Teich
JM
Tanasijevic
MJ
et al
.
Improving response to critical laboratory results with automation: results of a randomized controlled trial
.
J Am Med Inform Assoc
 
1999
;
6
:
512
22
.
49
Petersen
LA
Orav
EJ
Teich
JM
et al
.
Using a computerized sign-out program to improve continuity of inpatient care and prevent adverse events
.
Jt Comm J Qual Improv
 
1998
;
24
:
77
87
.
50
Singh
H
Naik
A
Rao
R
et al
.
Reducing diagnostic errors through effective communication: harnessing the power of information technology
.
J Gen Int Med
 
2008
;
23
:
489
94
.
51
Woolf
SH
Kuzel
AJ
Dovey
SM
Jr
et al
.
A string of mistakes: the importance of cascade analysis in describing, counting, and preventing medical errors
.
Ann Fam Med
 
2004
;
2
:
317
26
.
52
Gandhi
TK
Sittig
DF
Franklin
M
et al
.
Communication breakdown in the outpatient referral process
.
J Gen Intern Med
 
2000
;
15
:
626
31
.
53
Dismukes
K
.
Remembrance of things future: prospective memory in laboratory, workplace, and everyday settings
.
Hum Factors Ergon Soc
 
2010
;
6
:
79
122
.
54
Fisher
ES
McClellan
MB
Safran
DG
.
Building the path to accountable care
.
N Engl J Med
 
2011
;
365
:
2445
7
.
55
Kellerman
SE
Herold
J
.
Physician response to surveys. A review of the literature
.
Am J Prev Med
 
2001
;
20
:
61
7
.
56
Singh
H
Thomas
EJ
Wilson
L
et al
.
Errors of diagnosis in pediatric practice: a multisite survey
.
Pediatrics
 
2010
;
126
:
70
9
.
57
Asch
DA
Jedrziewski
MK
Christakis
NA
.
Response rates to mail surveys published in medical journals
.
J Clin Epidemiol
 
1997
;
50
:
1129
36
.
58
Cummings
SM
Savitz
LA
Konrad
TR
.
Reported response rates to mailed physician questionnaires
.
Health Serv Res
 
2001
;
35
:
1347
55
.
59
Halpern
SD
Kohn
R
Dornbrand-Lo
A
et al
.
Lottery-based versus fixed incentives to increase clinicians' response to surveys
.
Health Serv Res
 
2011
;
46
:
1663
74
.
60
Nicholls
K
Chapman
K
Shaw
T
et al
.
Enhancing response rates in physician surveys: the limited utility of electronic options
.
Health Serv Res
 
2011
;
46
:
1675
82
.
61
Poon
EG
Wright
A
Simon
SR
et al
.
Relationship between use of electronic health record features and health care quality: results of a statewide survey
.
Med Care
 
2010
;
48
:
203
9
.
62
Simon
SR
Soran
CS
Kaushal
R
et al
.
Physicians' use of key functions in electronic health records from 2005 to 2007: a statewide survey
.
J Am Med Inform Assoc
 
2009
;
16
:
465
70
.
This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (http://creativecommons.org/licenses/by-nc-nd/3.0/) which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact journals.permissions@oup.com

Comments

0 Comments