Abstract

Objective: To describe advanced clinical information systems in the context in which they have been implemented and are being used.

Design: Case series of five U.S. hospitals, including inpatient, ambulatory and emergency units. Descriptive study with data collected from interviews, observations, and document analysis.

Measurements: The use of computerized results, notes, orders, and event monitors and the type of decision support; data capture mechanisms and data form; impact on clinician satisfaction and clinical processes and outcomes; and the organizational factors associated with successful implementation.

Results: All sites have implemented a wide range of clinical information systems with extensive decision support. The systems had been well accepted by clinicians and have improved clinical processes. Successful implementation required leadership and long-term commitment, a focus on improving clinical processes, and gaining clinician involvement and maintaining productivity.

Conclusion: Despite differences in approach there are many similarities between sites in the clinical information systems in use and the factors important to successful implementation. The experience of these sites may provide a valuable guide for others who are yet to start, or are just beginning, the implementation of clinical information systems.

In 1991 an Institute of Medicine report recommended the widespread adoption of the computer-based patient record (CPR).1 However, in 1997 it was noted that there was little evidence of progress and that “there is no common data model for the CPR … and no common set of scenarios that are supported.”2 What has happened in the past decade is an increased interest in the use of computers in health care amongst policy makers, payers, and legislators.3 A number of studies have shown that clinical information systems—computers used by clinicians in the provision of care—can improve quality and reduce costs.4–7 However, few published data describe the types of systems in use across different sites and the context in which they have been implemented and are being used. To examine these issues, we studied five U.S. hospitals that have advanced clinical information systems (including computerized notes and ordering with associated decision support). Our goals were to identify the systems in use, how and by whom they were used, their impact, and the organizational factors associated with implementation.

Methods

Site Selection

A review was undertaken of articles published between January1986 and January 2002 using the MEDLINE database and the following medical subject headings: computer systems, hospital information systems, clinical decision support systems, and computerized medical record systems. The results of this review were used to create a typology of clinical information systems and to identify sites. The major categories of clinical information systems identified were computerized results, notes, ordering, and event monitoring systems. Also identified was computerized decision support, defined as the provision of patient and clinically relevant information to help clinicians make decisions.4,8 Another category of clinical administration systems was developed to include systems such as computerized access to clinical reference databases and patient tracking systems. Another source of data used were the Proceedings of the Annual Computer-based Patient Record Institute Nicholas E. Davies' Award, an annual award established in 1995 to recognize major achievements and to communicate lessons learned from successful implementations.9 The criteria used to select sites were the use of computerized results, notes, and orders with associated decision support across inpatient, ambulatory, and emergency settings. A deliberate mix was chosen to include sites that had developed their own systems and those that had implemented systems developed elsewhere, and the cases were limited to five sites that were determined to be achievable within the scope of the research.10

All five sites selected had won the Computer-Based Patient Record Institute Davies' Award:

  1. LDS Hospital, Salt Lake City (LDSH) in 199511

  2. Wishard Memorial Hospital, Indianapolis (WMH) in 199712

  3. Brigham and Women's Hospital, Boston (BWH) in 199613

  4. Queen's Medical Center, Honolulu (QMC) in 199914

  5. Veteran's Affairs Puget Sound Healthcare System, Seattle and Tacoma (VAPS) in 200015

The sites differed in size, but all were teaching hospitals providing services to a predominantly adult population (Table 1). We included associated ambulatory clinics at LDSH and QMC to make them comparable to the other sites. A significant difference among sites was that at WMH, BWH and VAPS house staff physicians provided most direct patient care and ordering, whereas at LDSH and QMC attending physicians provided direct care (including orders) to more than half of all patients. The systems at LDSH had been largely internally developed since 1963. At WMH, the closely associated Regenstrief Institute had developed the systems since 1972. The systems at BWH had been internally developed since 1984. In 1995 QMC installed a number of commercial systems, and in 1997 VAPS was the third pilot site for the Veterans' Affairs Computerized Patient Record System, with both implementing most systems in under two years.

Table 1

Description of Cases

 LDSH* WMH BWH QMC† VAPS 
Location of Hospital Salt Lake City, Utah Indianapolis, Indiana Boston, Massachusetts Honolulu, Hawaii Seattle and Tacoma, Washington 
Acute beds 378 246 716 530 315 
Annual admissions 21,201 16,734 39,848 19,000 11,924 
Annual outpatient visits 490,000 711,518 463,544 200,000 482,701 
Annual ED visits 33,354 95,756 49,077 35,000 38,078 
Full Time Employees 3,200 5,000 6,450 3,000 2,643 
Attending Physicians 530 affiliated and 39 staff 800 affiliated 2,485 affiliated 15 staff 1,200 affiliated, 38 part time staff 148 full time and 
Resident Physicians 31 150 610 50 147 
Profit Status Private not for profit County-owned Private not for profit Private not for profit Veteran's Affairs 
Clinical Service Profile Tertiary - Adults and maternity Tertiary—adults, maternity, and pediatrics Quaternary—adults and maternity Tertiary—adults, maternity and pediatrics Tertiary—adults 
 LDSH* WMH BWH QMC† VAPS 
Location of Hospital Salt Lake City, Utah Indianapolis, Indiana Boston, Massachusetts Honolulu, Hawaii Seattle and Tacoma, Washington 
Acute beds 378 246 716 530 315 
Annual admissions 21,201 16,734 39,848 19,000 11,924 
Annual outpatient visits 490,000 711,518 463,544 200,000 482,701 
Annual ED visits 33,354 95,756 49,077 35,000 38,078 
Full Time Employees 3,200 5,000 6,450 3,000 2,643 
Attending Physicians 530 affiliated and 39 staff 800 affiliated 2,485 affiliated 15 staff 1,200 affiliated, 38 part time staff 148 full time and 
Resident Physicians 31 150 610 50 147 
Profit Status Private not for profit County-owned Private not for profit Private not for profit Veteran's Affairs 
Clinical Service Profile Tertiary - Adults and maternity Tertiary—adults, maternity, and pediatrics Quaternary—adults and maternity Tertiary—adults, maternity and pediatrics Tertiary—adults 
*

LDSH includes Bryner and Salt Lake Clinics (housing 79 affiliated physicians) located within 5 miles of hospital.

QMC includes two physician office buildings (housing 237 affiliated physicians) located on campus.

Data Collection

Data collection took place over a two-year period starting in June 2000. The literature review described earlier was used to develop a data collection survey designed to measure the use of different systems across settings and the levels of decision support (Figure 1). The survey measured which functions were computerized and which were not, how the data were entered into the computer, and whether the data were free text or coded (free text data cannot be easily retrieved and used for other purposes, whereas coded data is structured and classified and can be processed by a computer16). The survey was tested and modified at BWH between August and December 2000. The survey was then sent to the other sites before visits were conducted between January and May 2001. Because of resource constraints the site visits were conducted by one researcher (DD), who is a physician with experience in the implementation of clinical information systems. During site visits, data were obtained from semistructured interviews with clinicians, informatics personnel, and managers as well as nonparticipant observations in clinical areas. In total there was 38 interviews and 41 hours of direct observation at the five sites. Interviews were conducted in person using a standard set of questions as a guide and were audiotaped and transcribed. Observations were conducted of physicians and other clinicians as they worked in inpatient units, ambulatory clinics, and emergency departments. Most observations took place during the day and early evening from Monday to Friday. Brief handwritten notes were taken during observations, and a full record was completed within 24 hours. Internal documents such as data entry and output forms were collected during visits.

Figure 1

Excerpt from the Data Collection Survey

Figure 1

Excerpt from the Data Collection Survey

Data Analysis

After site visits the data were analyzed by two of the researchers (DD and DB). Data relating to the use of different systems and the level of decision support were analysed using the categories of the survey. Data from different sources were used to check the integrity of the results in a process of triangulation.17 For example, data pertaining to the type of systems in use were first collected using the survey and the literature and then were compared with interviews and observations. Numerous discrepancies discovered when completing the survey from different sources required follow-up. After formatting, the data were sent back to the key site contact(s) for respondent validation.18 The data relating to impact measures and implementation factors were coded and indexed using analytical categories that were developed and refined during the course of the study.19 All researchers had input into the design of the survey and the interview guide, which were used rigorously at all sites to ensure a consistent approach.

Results

Approach to Computerization

All sites implemented computerized results in the early stages but differed in the path taken to computerizing other functions. At LDSH the initial focus was on capturing data from equipment such as monitors and ventilators and computerizing nurses' and therapists' notes in inpatient units.11,20 Decision support was focussed on selected areas such as infectious diseases. WMH began with computerized notes and ordering in clinics and providing decision support for preventive care interventions and reducing medication errors.12 This was later introduced into inpatient units and then emergency departments. At BWH the early focus was on the computerized notes in ambulatory clinics, inpatient physician ordering to prevent medication errors, and the development of event monitors.13 Physician ordering was later implemented in ambulatory clinics and emergency departments. At QMC and VAPS the initial focus was on inpatient physician order entry, although VAPS also targeted computerization of notes.14,15

Clinical Information Systems in Use

Computerized Results

All sites had numerical results and reports of diagnostic tests available on the computer (Table 2). Only WMH and VAPS had radiology images in the clinical information system, although each site had a digital radiography system that allowed diagnostic quality images to be viewed on a limited number of dedicated monitors. Computerized results were captured directly from pathology and other departmental information systems. Pathology and hematology results were coded, as were microbiology reports. Other reports of diagnostic tests were generally free text. Each system had at least some chronological and graphical display of numerical results.

Table 2

Computerized Results (All Settings)

Department Results LDSH WMH BWH QMC VAPS 
Pathology Laboratory 
 Hematology 
 Anatomic pathology reports 
 Anatomic pathology images     
 Cytology reports 
 Cytology images     
 Microbiology reports 
Radiology* Reports 
 Radiograph images    
 CT images    
 MRI images    
 Ultrasound images    
Nuclear medicine Reports 
 Images     
Cardiology EKG reports  
 EKG tracings  + (ED)  
 Angiography and echocardiography reports  
 Angiography and echocardiography images     
Respiratory Reports  
 Images (eg waveforms)      
Neurology EEG reports   
 EMG reports   
Department Results LDSH WMH BWH QMC VAPS 
Pathology Laboratory 
 Hematology 
 Anatomic pathology reports 
 Anatomic pathology images     
 Cytology reports 
 Cytology images     
 Microbiology reports 
Radiology* Reports 
 Radiograph images    
 CT images    
 MRI images    
 Ultrasound images    
Nuclear medicine Reports 
 Images     
Cardiology EKG reports  
 EKG tracings  + (ED)  
 Angiography and echocardiography reports  
 Angiography and echocardiography images     
Respiratory Reports  
 Images (eg waveforms)      
Neurology EEG reports   
 EMG reports   

CT = computed tomography, MRI = magnetic resonance imaging, EKG = electrocardiogram, EEG = electroencephalograph, EMG = electromyography.

*

Only WMH and VAPS had radiology images available within the clinical information system and accessible in all clinical settings. All sites had radiology images available via dedicated viewing monitors in limited areas.

Computerized Notes

In inpatient units, WMH and VAPS had almost complete computerized notes and LDSH had complete computerization of nurses' and therapists' notes (Table 3). All sites had computerized ambulatory physician notes and emergency department summary notes or discharge summaries. Medication and allergy lists were computerized except in the QMC emergency department. At LDSH, QMC, and VAPS nurses used a computerized medication administration record. Problem lists were widely used except in LDSH inpatient units and LDSH, BWH, and QMC emergency departments. In inpatient units, procedure and discharge notes were usually captured from dictation systems, whereas physician progress notes were directly entered into the computer. In ambulatory clinics at BWH, most physician notes were obtained from dictation systems; at QMC and VAPS there was mostly direct entry; and at LDSH and WMH, physicians used both methods. At WMH physicians also entered data by hand onto printed templates from which coders extracted data. Emergency physicians at WMH and VAPS directly entered notes, whereas other sites used dictation systems. Physicians often entered problem lists and allergies using menu lists to help in the selection of terms.

Table 3

Computerized Notes*

 LDSH WMH BWH QMC† VAPS 
Inpatient Physicians: admission, allergies, medications, procedures, discharge summary Physicians: problem list, allergies, medica-tions, admission note, progress (60% house staff, 100% attending physicians), proce-dures (50%), dis-charge summary Physicians: problem list, allergies, medica-tions, some progress in the form of a hand-(over summary) procedure notes, discharge summary Physician: problem list, allergies, medica-tions, history, procedures Physicians: problem list (85%), allergies, medi-cations, admission,progress, procedures, discharge summary 
 Nurses: Initial assess-ment, progress, vita-signs, handover, medication admin-istration record,admin-discharge summary Nurses: initial assess-ment (70%), vital sign  Nurses: initial assess-ment, vital signs (general wards), operative notes, medi-cation administration record Nurses: initial assess-ment, progress notes, vital signs (50%), medication admin-istration record (80%), discharge summary 
 Therapists: all notes Therapists: most notes   Therapists: all notes 
Ambulatory Physicians: problem list, allergies, medi-cations, history and physical findings, procedures Physicians: problem list, allergies, medi-cations, history and physical findings (50%), vital signs (70%),procedures Physicians: problem list, allergies, medi-cations, history and physical findings, procedures Physicians: problem list, allergies, medi-cations, history and physical findings, procedures Physicians: problem list (85%), allergies, medi-cations, history and physical findings, procedures 
 Nurses: vital signs Nurses: vital signs Nurses: vital signs Nurses: initial assess-ment and vital signs Nurses: initial assess-ment (80%), vital signs (90%) 
     Therapists: all notes 
Emergency Physicians: allergies, medications, dis-charge summary Physicians: problem list, allergies, medica-tions (discharge only), discharge summary Physicians: allergies, medications, sum-mary note by attend-ing physician on all physician on all patients Physicians: summary note by attending physician on all patients Physicians: problem list 85%), allergies, medi-(cations (67%), history and physical findings, progress, discharge 
 Nurses: initial assess-ment, progress, vital signs, transfer, ,medi-cation administration record,discharge summary   Nurses: initial assess-ment, discharge summary Nurses: initial assess-ment, vital signs (10%) 
 LDSH WMH BWH QMC† VAPS 
Inpatient Physicians: admission, allergies, medications, procedures, discharge summary Physicians: problem list, allergies, medica-tions, admission note, progress (60% house staff, 100% attending physicians), proce-dures (50%), dis-charge summary Physicians: problem list, allergies, medica-tions, some progress in the form of a hand-(over summary) procedure notes, discharge summary Physician: problem list, allergies, medica-tions, history, procedures Physicians: problem list (85%), allergies, medi-cations, admission,progress, procedures, discharge summary 
 Nurses: Initial assess-ment, progress, vita-signs, handover, medication admin-istration record,admin-discharge summary Nurses: initial assess-ment (70%), vital sign  Nurses: initial assess-ment, vital signs (general wards), operative notes, medi-cation administration record Nurses: initial assess-ment, progress notes, vital signs (50%), medication admin-istration record (80%), discharge summary 
 Therapists: all notes Therapists: most notes   Therapists: all notes 
Ambulatory Physicians: problem list, allergies, medi-cations, history and physical findings, procedures Physicians: problem list, allergies, medi-cations, history and physical findings (50%), vital signs (70%),procedures Physicians: problem list, allergies, medi-cations, history and physical findings, procedures Physicians: problem list, allergies, medi-cations, history and physical findings, procedures Physicians: problem list (85%), allergies, medi-cations, history and physical findings, procedures 
 Nurses: vital signs Nurses: vital signs Nurses: vital signs Nurses: initial assess-ment and vital signs Nurses: initial assess-ment (80%), vital signs (90%) 
     Therapists: all notes 
Emergency Physicians: allergies, medications, dis-charge summary Physicians: problem list, allergies, medica-tions (discharge only), discharge summary Physicians: allergies, medications, sum-mary note by attend-ing physician on all physician on all patients Physicians: summary note by attending physician on all patients Physicians: problem list 85%), allergies, medi-(cations (67%), history and physical findings, progress, discharge 
 Nurses: initial assess-ment, progress, vital signs, transfer, ,medi-cation administration record,discharge summary   Nurses: initial assess-ment, discharge summary Nurses: initial assess-ment, vital signs (10%) 
*

Indicates 100% of the listed function unless otherwise stated.

At QMC internal hospital ambulatory clinics and 11 out of 237 affiliated physician clinics using computerized notes.

Medications were captured directly from order entry systems and medication administration records. Nurses and therapists directly entered their own notes. In intensive care units at LDSH, data were captured directly from monitors and other equipment. At WMH vital signs were captured directly from bedside measuring equipment. Most computerized notes were free text with the exception of LDSH and WMH, where nursing and therapist's notes were coded. Problem, medication and allergy lists, and vital signs were mostly coded. Templates designed around types of notes (e.g., admission, diabetes visit) were commonly used for directly entering notes. All sites provided printed reports, including clinician patient lists and patient summaries.

Computerized Ordering

There was widespread use of computerized ordering for most functions in inpatient units and for medications in ambulatory clinics and diagnostic tests in emergency departments (Table 4). However, only at WMH and VAPS did most ambulatory clinics use computerized ordering, and at BWH the most extensive computerized ordering was in the emergency department. There were high levels of direct physician entry of orders across all settings at WMH, BWH and VAPS (Figure 2). Orders for tests and medications were coded, whereas other orders were free text. Decision support was widely used in computerized ordering systems for medications and tests with the most extensive at WMH and BWH (Tables 5 and 6). All sites used order sets based on guidelines for specific clinical conditions (e.g., community-acquired pneumonia) and types of admissions (e.g., routine preoperative). At LDSH, physicians used the anti-infective agent ordering system to decide on therapy, even though they did not use it to order these agents and a number of intensive care units used decision support in the setting of ventilators.

Figure 2

Proportion of computerized orders directly entered by physicians*†‡

*The percentages refer to the proportion of direct physician entry of computerized orders, not the proportion of direct physician entry of all computerized and noncomputerized orders.

LDSH physicians entered total parental nutrition and one-third of blood orders on the inpatient units and one-third of blood orders in the emergency department (apart from one ICU, in which they also ordered anti-infective agents and ventilator settings). Therefore, 1% direct physician entry has been used in both these situations for illustrative purposes.

At QMC inpatient units the overall proportion of direct physician entry of computerized orders was 62%, although proportion of direct entry for residents was 80%. (Attending physicians entered majority of tests and proportion of direct entry by them was 48%.)

Figure 2

Proportion of computerized orders directly entered by physicians*†‡

*The percentages refer to the proportion of direct physician entry of computerized orders, not the proportion of direct physician entry of all computerized and noncomputerized orders.

LDSH physicians entered total parental nutrition and one-third of blood orders on the inpatient units and one-third of blood orders in the emergency department (apart from one ICU, in which they also ordered anti-infective agents and ventilator settings). Therefore, 1% direct physician entry has been used in both these situations for illustrative purposes.

At QMC inpatient units the overall proportion of direct physician entry of computerized orders was 62%, although proportion of direct entry for residents was 80%. (Attending physicians entered majority of tests and proportion of direct entry by them was 48%.)

Table 4

Computerized Ordering (indicates 100% of the listed function unless otherwise stated)

 LDSH* WMH BWH† QMC‡ VAPS§ 
Inpatient      
   Medications and other therapeutics + (Except chemotherapy and total parental nutrition) 
   Diagnostic tests 
   Patient care and therapy + (Nurse-generated only) 
    Consults and referrals 
Ambulatory      
   Medications and other therapeutics 
    Diagnostic tests   
    Patient care and therapy    
    Consults and referrals   
Emergency department      
    Medications and other therapeutics + (Blood only) only) + (Discharge + (Oral only) 
   Diagnostic tests + (Except cardiology and respiratory) + (Except respiratory and neurology) 
    Patient care and therapy    
    Consults and referrals  
 LDSH* WMH BWH† QMC‡ VAPS§ 
Inpatient      
   Medications and other therapeutics + (Except chemotherapy and total parental nutrition) 
   Diagnostic tests 
   Patient care and therapy + (Nurse-generated only) 
    Consults and referrals 
Ambulatory      
   Medications and other therapeutics 
    Diagnostic tests   
    Patient care and therapy    
    Consults and referrals   
Emergency department      
    Medications and other therapeutics + (Blood only) only) + (Discharge + (Oral only) 
   Diagnostic tests + (Except cardiology and respiratory) + (Except respiratory and neurology) 
    Patient care and therapy    
    Consults and referrals  
*

At LDSH computerized ordering was used for ventilator settings in a number of ICU's. 24 out of 79 affiliated physician clinics at LDSH used computerized ordering.

No computerized ordering in BWH neonatal intensive care unit. 57% of BWH ambulatory clinics used computerized ordering.

At QMC internal hospital clinics and 11 out of 237 affiliated physician clinics used computerized ordering

§

No computerized ordering in VAPS bone marrow unit. Only nonacute section of VAPS emergency department used computerized ordering for medications.

Table 5

Decision Support for Computerized Ordering of Medications (Inpatient)

Type of decision support LDSH WMH BWH QMC VAPS 
Drug name checking 
Default administration route (simple) 
Listing by formulary 
Patient drug allergies (simple - specific drug) 
Patient drug allergies (advanced - drug families) 
Protocol or diagnosis based therapy 
Duplicate order checking  
Drug - drug interactions  
Substitute therapy suggestion  
Subsequent or corollary orders (e.g., serum level for aminoglycosides)  
Default doses (simple)  
Relevant information display (e.g., last potassium when ordering digoxin)  
Drug cost display   
Guided dosing calculation   
Time based checks to ensure optimum timing and duration   
Default doses (advanced e.g., adjusted for age, renal function)    
Administration route change (adjusted for patient route for other drugs)    
Drug–laboratory interactions    
Type of decision support LDSH WMH BWH QMC VAPS 
Drug name checking 
Default administration route (simple) 
Listing by formulary 
Patient drug allergies (simple - specific drug) 
Patient drug allergies (advanced - drug families) 
Protocol or diagnosis based therapy 
Duplicate order checking  
Drug - drug interactions  
Substitute therapy suggestion  
Subsequent or corollary orders (e.g., serum level for aminoglycosides)  
Default doses (simple)  
Relevant information display (e.g., last potassium when ordering digoxin)  
Drug cost display   
Guided dosing calculation   
Time based checks to ensure optimum timing and duration   
Default doses (advanced e.g., adjusted for age, renal function)    
Administration route change (adjusted for patient route for other drugs)    
Drug–laboratory interactions    
Table 6

Decision Support for Computerized Ordering of Diagnostic Tests (Inpatient)

Type of decision support LDSH WMH BWH QMC VAPS 
Duplicate order checking 
Diagnosis or problem based order sets 
Suggestions for alternate tests  
Prerequisite or subsequent tests  
Redundant test check   
Display of cost    
Display of recent results    
Type of decision support LDSH WMH BWH QMC VAPS 
Duplicate order checking 
Diagnosis or problem based order sets 
Suggestions for alternate tests  
Prerequisite or subsequent tests  
Redundant test check   
Display of cost    
Display of recent results    

Computerized Event Monitoring and Notification

All sites except QMC used event monitoring and notification systems. These systems searched through computerized data, identified items of clinical significance, and notified the relevant clinician. They were used to alert for critical results (e.g., high potassium), screen for possible adverse drug events (e.g., naloxone use), and provide reminders based on therapeutic protocols (e.g. aspirin for patients with myocardial infarction) and health maintenance guidelines (e.g., Pap smears). At LDSH, event monitors were also used for sophisticated prophylactic antibiotic and microbiology alerts. These systems generated computer alerts or sent a page or e-mail to the physician. Health maintenance and therapeutic reminders were listed on a printed patient summary but were also available on the computer. A list of patients with signals suggesting the presence of an adverse drug event was printed daily for pharmacists. Microbiol-ogy alerts at LDSH were printed out for the infectious diseases' team.

Clinical Administration Systems

All sites provided a variety of information systems that were designed to help clinicians manage their workload. These included patient tracking systems in inpatient and emergency units, scheduling systems in ambulatory clinics, and e-mail and other message systems. Each site also provided computerized access to clinical reference sources such as drug pharmacopoeias, on-line clinical texts and journals, and clinical protocols and guidelines.

Impact of Clinical Information Systems

Clinician Satisfaction and Utilization

At LDSH, physicians rated the availability of laboratory results as the greatest benefit.21 At QMC and VAPS, physicians appreciated access to patient results and notes and least liked the extra time required for computerized ordering.14,15 Although physicians at WMH and BWH spent more time using computerized ordering, there were high satisfaction levels, and physicians believed that it improved the quality of their work.5,22–24 These findings were reflected in the interviews and observations. Physicians appreciated the improved access to patient information and the decision support in computerized ordering and event monitors. Other clinicians valued the clarity and completeness of physician notes and orders. In observing clinicians at work, it was apparent that the computer was the primary source of information. Clinicians were constantly using computers to find patients, look up results, and enter notes and orders. Printed patient summaries were widely used on inpatient units and were often the primary information source during rounds. In ambulatory clinics physicians relied heavily on the computer to find results and notes and the patient summaries printed by clerical staff at the start of clinics. In most cases, the paper chart was not called for or used. Physicians often wrote notes on the printed patient summaries and later entered the full details into the computer or dictation system. In emergency departments with computerized tracking systems, the computer had replaced the “greaseboard” for finding patients, monitoring their prog-ress, and handing over their care to other clinicians.

Changes to Clinical Processes and Patient Care

At LDSH, Evans et al. found that computerized ordering for anti-infective agents reduced excess drug doses, susceptibility mismatches, adverse drug events, length of stay, and overall costs.25 Tierney et al. found that medical teams at WMH using computerized ordering generated 13% lower charges and 13% lower costs.5 In another study at this hospital, suggestions for corollary orders (e.g., drug levels) after the ordering of drugs increased compliance from 22% to 46%.26 In another WMH study, simply displaying the cost of tests at the time of computerized ordering reduced by 14% the number of tests ordered.27 At BWH, Bates et al showed a 55% reduction in serious medication errors.6 Over 41/2 years, this system reduced the medication error rate by 81%.28 Computerized ordering at BWH improved drug and frequency choices,29 improved ordering for patients with renal insufficiency,30 and reduced by 24% the ordering of redundant laboratory tests.31

At LDSH, computerized ordering and event monitors for anti-infective agents reduced adverse drug events by 30%, increased patients receiving appropriately timed preoperative antibiotics from 40% to 99%, and decreased by 23% overall antibiotic use.7 The event monitoring system at BWH reduced by 38% the response time to critical laboratory results but had no effect on outcomes.32 At BWH, computerized monitoring detected ten times as many adverse drug events as did a voluntary reporting system.33 In inpatient units and ambulatory clinics at WMH, the provision of reminders significantly increased compliance with preventive care guidelines.34–36

At BWH, the use of electronic notes in the ambulatory clinics resulted in cost savings in chart pulls and dictation.37 Ambulatory physicians interviewed at LDSH reported a decrease in dictation costs because more physicians directly entered their own notes into the computer. A study conducted at WMH showed only marginal benefits from sharing patient results and notes with emergency departments at two other local hospitals.38 When interviewed, managers and physicians stated that the investment in these systems had paid for itself in improvements to care and cost savings. However, no studies have assessed the overall return on investment of good clinical information systems. At all sites, except VAPS, the information systems were designed to generate data for performance monitoring, quality improvement projects and clinical research, with savings in time and effort for these activities.11,12,14,39 At LDSH and WMH these systems were used to feed billing systems without the need for additional data entry.12

Organizational Factors Associated with Successful Implementation

Data obtained from interviews, observations and the literature identified five major factors associated with successful implementation of these systems (Table 7).11–15,40–42

Table 7

Factors Associated with Successful Information—Representative Quotations from Interviews

Having organizational leadership, commitment and vision “(The CEO) invested in it when there was very little reason to invest in it. He was way ahead of his time. When there was tremendous cost constraints and against all of his senior managers advice, including my own, he invested a lot of money in IS. Because it takes so much time to develop, especially when you are developing it for the first time, he was right on target.” Physician manager 
 “When there were bumps and bruises along the way and some people questioned whether they should be doing this they would get a friendly call from (the CEO) that this is the direction we are going in and everyone is going to march in this direction.” Deputy CIO 
 “I think strong physician leadership. This has grown out of the physician community, folks who are well regarded as clinicians and as part of the group and colleagues. Chief of Medicine 
 “You have to have leadership that has a vision of where you are going and you have to stay the course. Any time you are on the cutting edge there are going to be fresh starts and you have to be able to re-articulate the vision so people can see where it is going to lead.” CEO 
Improving clinical processes and patient care “My objective overall was to improve patient care. As with most facilities we struggled with having clinical information available to the physicians and practitioners at the point of contact where it was in the wards or in the clinics.” CEO 
 “We are able to standardize care around best practices in a much more effective manner when it's enforced by an order entry system. It aids the physicians in terms of not being forced to remember huge amounts of choices for drugs or other clinical scenarios.” Physician 
 “Safety is a big part of quality management and once a gap is noted that could cause errors we would like to close that gap quickly and comprehensively. With paper systems it is just impossible to do that”. Physician manager 
Involving clinicians in design and modification of system “We had over 530 people involved, and doctors hired to help us design screens and everything. The doctors were very much part of the effort.” IT Manager 
 “We developed screens and asked for their input before we settled on a certain pathway or a certain format. We involved them as much as we could without getting bogged down in all the minutiae.” Physician 
 “We said we want any suggestions, if this thing sucks and you think it does then tell us about it. Being that open minded and being willing to go back and change things and modify it was very helpful. It showed you were interested.” Physician 
 “They didn't just wait until someone got frustrated enough with a process that they sent them an email or showed up in their office. They set up regular meetings on a weekly basis so that those who were users of the system had automatic input into the development of the system. Getting that feedback has been criti-caland responding to it also.” Chief of Medicine 
Maintaining or improving clinical productivity “It's easier to get the physician to the workstation if you let them do all of their work there, all of their ordering there and that's the approach that we have taken.” Physician and informatics professional 
 “I think that training was one. They all had to be trained and we spent a lot of time training them, sometimes one on one. I can recall some of those who were not as quick as others we would send some trainers to spend one on one time with them.” Physician 
 “We would go out and hover in the areas, we made rounds frequently, often times we would be there before the physician, just as they were getting ready to call, we'd say ‘here we are.’” Nurse and informatics professional 
Building momentum and support amongst clinicians “Once he convinced us we became his disciples and then all of us went out and had a meet with all the medical staff in division meetings and department meetings. We demonstrated and talked about it and evangelized the clinical staff that this was something good, something sexy, high tech and innovative and it was going to be expected to be utilized.” Physician 
 “Make things work there and then it diffused and then once it starting to diffuse you jump on it and make it happen.” Physician and informatics professional 
 “You hit a point where you can't practice without being on line and we hit that momentum and it just started to run really.” Deputy CIO 
 “When we heard that there was peer pressure to get those few remaining progress notes into the computer we knew that we were there.” Nurse and informatics professional 
Having organizational leadership, commitment and vision “(The CEO) invested in it when there was very little reason to invest in it. He was way ahead of his time. When there was tremendous cost constraints and against all of his senior managers advice, including my own, he invested a lot of money in IS. Because it takes so much time to develop, especially when you are developing it for the first time, he was right on target.” Physician manager 
 “When there were bumps and bruises along the way and some people questioned whether they should be doing this they would get a friendly call from (the CEO) that this is the direction we are going in and everyone is going to march in this direction.” Deputy CIO 
 “I think strong physician leadership. This has grown out of the physician community, folks who are well regarded as clinicians and as part of the group and colleagues. Chief of Medicine 
 “You have to have leadership that has a vision of where you are going and you have to stay the course. Any time you are on the cutting edge there are going to be fresh starts and you have to be able to re-articulate the vision so people can see where it is going to lead.” CEO 
Improving clinical processes and patient care “My objective overall was to improve patient care. As with most facilities we struggled with having clinical information available to the physicians and practitioners at the point of contact where it was in the wards or in the clinics.” CEO 
 “We are able to standardize care around best practices in a much more effective manner when it's enforced by an order entry system. It aids the physicians in terms of not being forced to remember huge amounts of choices for drugs or other clinical scenarios.” Physician 
 “Safety is a big part of quality management and once a gap is noted that could cause errors we would like to close that gap quickly and comprehensively. With paper systems it is just impossible to do that”. Physician manager 
Involving clinicians in design and modification of system “We had over 530 people involved, and doctors hired to help us design screens and everything. The doctors were very much part of the effort.” IT Manager 
 “We developed screens and asked for their input before we settled on a certain pathway or a certain format. We involved them as much as we could without getting bogged down in all the minutiae.” Physician 
 “We said we want any suggestions, if this thing sucks and you think it does then tell us about it. Being that open minded and being willing to go back and change things and modify it was very helpful. It showed you were interested.” Physician 
 “They didn't just wait until someone got frustrated enough with a process that they sent them an email or showed up in their office. They set up regular meetings on a weekly basis so that those who were users of the system had automatic input into the development of the system. Getting that feedback has been criti-caland responding to it also.” Chief of Medicine 
Maintaining or improving clinical productivity “It's easier to get the physician to the workstation if you let them do all of their work there, all of their ordering there and that's the approach that we have taken.” Physician and informatics professional 
 “I think that training was one. They all had to be trained and we spent a lot of time training them, sometimes one on one. I can recall some of those who were not as quick as others we would send some trainers to spend one on one time with them.” Physician 
 “We would go out and hover in the areas, we made rounds frequently, often times we would be there before the physician, just as they were getting ready to call, we'd say ‘here we are.’” Nurse and informatics professional 
Building momentum and support amongst clinicians “Once he convinced us we became his disciples and then all of us went out and had a meet with all the medical staff in division meetings and department meetings. We demonstrated and talked about it and evangelized the clinical staff that this was something good, something sexy, high tech and innovative and it was going to be expected to be utilized.” Physician 
 “Make things work there and then it diffused and then once it starting to diffuse you jump on it and make it happen.” Physician and informatics professional 
 “You hit a point where you can't practice without being on line and we hit that momentum and it just started to run really.” Deputy CIO 
 “When we heard that there was peer pressure to get those few remaining progress notes into the computer we knew that we were there.” Nurse and informatics professional 

Organizational Leadership, Commitment, and Vision

The presence of high-level leadership was considered the single most important factor. It was demonstrated by the long-term commitment of resources and by providing an understandable vision for the organization. At LDSH and WMH, where development took place over a number of decades, the commitment was less important at the beginning but became crucial with the increasing number of systems and resources required over time. Leadership was provided directly by the CEO at BWH and VAPS; at the other sites it came from high-level clinicians and clinical managers. Structures were established at each site to oversee the development and/or implementation of clinical information systems. These included board-level committees to provide strategic direction, committees and departments that included both information system and clinical informatics personnel, and forums to involve informatics and clinical staff. All sites appointed people with clinical backgrounds to lead the move to computerization. Financial and human resources were committed over several years at a time with overt recognition that it would take a number of years to reap any benefits.

Improving Clinical Processes and Patient Care

At all sites, the primary goal of investing in these systems was to improve the quality of clinical care. Each site had a learning culture that sought to understand clinical processes and improve them through the use of information technology. The design of clinical information systems was dictated by clinical problems and the need to provide relevant and timely information to clinicians. They were designed to support the underlying clinical processes and were subject to ongoing evaluation and modification. All sites experimented with different features of the systems and relied on stringent verification mechanisms and regular feedback from clinicians to ensure that modifications to systems did not introduce new problems. There was considerable exchange of ideas among LDSH, WMH, BWH and other pioneering sites as each sought to learn what was working at the other sites. At QMC and VAPS the acquisition of systems was consciously directed by the research conducted at other sites. There were strong links between clinicians involved in informatics and quality improvement, with many working in both areas.

Involving Clinicians in the Design and Modification of the System

All sites undertook enormous efforts to involve physicians and other clinicians in the design and modification of these systems. This goal was achieved through a wide range of formal and informal mechanisms to capture timely feedback from clinicians. All sites recruited respected clinicians from medical, nursing, and therapy backgrounds, with informatics experience or interest, to act as the bridge between practicing clinicians and the information systems. It was considered crucial that the decision support and rules within the order entry and event monitoring systems were seen to have ownership by an expert physician or clinical group and not by the “computer.”

Maintaining or Improving Clinical Productivity

Despite the fact that computerized ordering took extra time every effort was made to maintain or improve the productivity of clinicians. As much as possible the sites wanted clinicians to be able to perform the majority of their work at the computer. There was widespread availability of computers—in inpatient units one per bed at LDSH and 3–5 per bed at other sites; in clinics and emergency departments, one or more per room. Physicians could view patient information and enter orders from their offices or homes. There were note templates and order sets for common conditions and easy ordering of tests and medications in response to alerts and reminders. Sites had to make constant trade-offs between adding new functionality and keeping the response speed of systems to a level accepted by clinicians. All sites provided massive amounts of training and support, particularly when systems were first implemented, although this was less important at LDSH, where implementation had taken place over a number of decades. All sites limited the amount of time that the systems were unavailable (range: 20 minutes to 5 hours per month). This down-time was generally scheduled on a regular basis with procedures to provide continued data access (e.g., printed patient summaries, data for current patients stored on the local hard drive of one computer per unit).

Building Momentum and Support among Clinicians

Each site built on early successes, including published research, and used respected physicians and other clinicians to lead the move to computerization. Over time, as they observed their colleagues using these systems and as more information was made available on the computer, other physicians began to use them. Typically the first systems implemented were those that provided benefits to clinicians without requiring additional effort from them, such as computerized results and computerization of dictated notes. LDSH and WMH slowly built support from clinicians over a number of decades. At BWH when computerized ordering was implemented, it was made mandatory for resident physicians to use the system. QMC and VAPS allowed 2–3 years to achieve high levels of physician ordering.

Discussion and Conclusions

This study evaluated five leading U.S. information systems, with the intent of identifying similarities and differences in what their systems included and what the underlying reasons for success appeared to be. There were many similarities: all sites had coded laboratory results and problem, allergy, and medication lists that made it possible to provide decision support in computerized ordering and to implement event monitors. The differences were also of note: only WMH and VAPS had computerized physician progress notes in inpatient units, suggesting that this is a relatively challenging application to implement. Only LDSH and VAPS had computerized inpatient nursing notes, again suggesting that this is relatively difficult. All sites had computerized ambulatory notes, suggesting that this may be a good place to begin with clinical documentation; not surprisingly, there were differences in the proportion of notes that were dictated or entered directly across sites. Each site had at least a summary document accessible on the computer for most encounters, including emergency department visits.

Computerization of ordering varied by site. In inpatient units all sites had computerized ordering, although LDSH physicians were not directly entering orders and the decision support at that site did not include alternative choices or cost, presumably because these were not relevant for nonphysicians. The three sites at which house staff physicians were responsible for most inpatient ordering had high levels of direct physician entry. In contrast to the inpatient setting, the penetration of computerized ordering in clinics and emergency departments varied substantially among sites.

The positive impact of computerized ordering and event monitors at LDSH, WMH, and BWH are consistent with reviews that have found computerized decision support for medication ordering and the provision of preventive care guidelines to be the most common systems to have a positive effect on clinician performance.4,43 Although there is little formal evidence of their benefits, it is clear that clinicians at all sites valued computerized access to results and notes; this may well be a prerequisite before physicians will begin computerized ordering.

Despite differences in the ways that each site has acquired and implemented clinical information systems, a common set of factors appears to be associated with successful implementation. These include unusually strong leadership and a clearly defined long-term commitment, which is difficult to find in health care in the U.S. today. Other keys were clear focus on improving clinical processes and gaining clinician involvement and support while maintaining or improving productivity. In a recent review, Kuhn and Giuse also identified the need for huge political investment, systems that optimize clinical workflow, and clinician involvement in the design of systems.44 They also found that physicians will support the implementation of clinical information systems when they perceive that the objective is to help them provide and improve patient care and that the push for computerization is coming from fellow physicians. In keeping with the different approaches taken at each site, Lorenzi et al. found that “The strategy that each institution develops must meets its particular needs, goals and culture.”45

This study has a number of limitations. First, one researcher conducted the site visits, although the potential for bias was minimized because all researchers had input into the design of the survey and interview questions and two researchers analyzed the data. Second, the data obtained about the use of systems were largely self-reported, although reports were checked by comparing data from multiple sources. Third, the findings from these sites may not be transferable to other sites. However, although only QMC was using a commercially available system, the other systems were widely used in sites other than those in this study.40–42,46 Finally, other factors probably are relevant to successful implementation, particularly those related to technical aspects of the systems themselves, which were not included in this research.

These sites have implemented clinical information systems that collect and communicate data to clinicians when and where they need it and provide decision support that guides clinicians to make appropriate choices. Excellent results retrieval applications and coded data make it possible to provide higher-level functionality, such as computerized order with decision support and event monitoring. Computer-ized documentation is still relatively challenging and is not necessary for providing decision support, although a good place to begin appears to be the ambulatory setting. Although there does not appear to be a single pathway to success, the experience of these sites may provide a valuable guide for others who are yet to start or are just beginning the implementation of clinical information systems.

References

1
Institute of Medicine.
Committee on improving the patient record. The computer based patient record: an essential technology for healthcare
 .
Washington, DC
,
National Academy Press
,
1991
.
2
Tang
PC
Hammond
WE
.
A progress report on computer-based patient records in the United States
. In
Dick
RS
Steen
EB
Detmer
DE
(eds):
The Computer-based Patient Record: An Essential Technology for Healthcare
 
2nd ed.
Washington, DC
,
National Academy Press
,
1997
, pp
1
20
.
3
Kohn
LT
Corrigan
JM
Donaldson
MS
(eds).
To Err is Human: Building a Safer Health System
 .
Washington, DC
,
National Academy Press
,
1999
.
4
Hunt
D
Haynes
R
Hanna
S
Smith
K
.
Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review
.
JAMA
 
1998
;
280
(
15
):
1339
1346
.
5
Tierney
WM
Miller
ME
Overhage
JM
McDonald
CJ
.
Physician inpatient order writing on microcomputer workstations: effects on resource utilization
.
JAMA
 
1993
;
269
:
379
383
.
6
Bates
DW
Leape
LL
Cullen
DJ
et al
.
Effect of computerized physician order entry and a team intervention of prevention of serious medical errors
.
JAMA
 
1998
;
28
:
1339
1346
.
7
Pestotnik
S
Classen
D
Evans
R
Burke
J
.
Implementing antibiotic practice guidelines though computer-assisted decision support: clinical and financial outcomes
.
Ann Intern Med
 
1996
;
124
(
10
):
884
890
.
8
Musen
MA
Shahar
Y
Shortliffe
EH
.
Clinical decision-support systems
. In
Shortliffe
EH
Perrault
LE
Wiederhold
G
Fagan
LM
(eds):
Medical Informatics—Computer Applications in Health Care and Biomedicine
 .
2nd ed.
New York, Springer
; 200, pp
573
609
.
9
Metzger
JB
Amatayakul
M
Simpson
N
.
Lessons learned from the Davies Program: The first four years
. In
Steen
EB
(ed):
Proceedings of the First Annual Nicholas E Davies CPR Recognition Symposium
 
April
4–6
,
1995
,
Washington, DC, New York
,
McGraw-Hill
,
1995
, pp
1
14
.
10
Keen
J
Packwood
T
.
Using case studies in health services and policy research
. In
Pope
C
Mays
N
(eds):
Qualitative Research in Health Care
 
2nd ed.
London
,
BMJ Books
,
2000
, pp
50
58
.
11
Grandia
LD
Pryor
TA
Willson
DF
et al
.
Building a computer-based patient record system in an evolving integrated health system
. In
Steen
EB
(ed):
Proceedings of the First Annual Nicholas E Davies CPR Recognition Symposium
 
April
4–6
,
1995
,
Washington, DC, New York
,
McGraw-Hill
,
1995
, pp
19
56
.
12
McDonald
CJ
Tierney
WM
Overhage
JM
Dexter
P
Takesue
B
Abernathy
G
.
The three-legged stool: Regenstrief Institute for Health Care
. In
Teich
JM
(ed):
Proceedings of the Third Annual Nicholas E Davies CPR Recognition Symposium
 
June
19–20
,
1995
,
Washington, DC, New York
,
McGraw-Hill
,
1997
, pp
131
158
.
13
Teich
JM
Glaser
JP
Beckley
RF
et al
.
Toward cost effective, quality care: the Brigham integrated computing system, Brigham and Women's Hospital
. In
Steen
EB
(ed):
Proceedings of the Second Annual Nicholas E Davies CPR Recognition Symposium
 
May
1–2
,
1996
,
Washington, DC
,
1996
, pp
19
56
.
14
Davis
DC
Moriyami
R
Tiwanak
G
Morse
L
Saito
C
.
Clinical performance improvement with advanced clinical information system at the Queen's Medical Center
. In
Metzger
JB
(ed):
Proceedings of the Fifth Annual Nicholas E Davies CPR Recognition Symposium
 
June
7–8
,
1999
,
Washington, DC
,
1999
, pp
77
120
.
15
Payne
TH
Torell
JT
Hoey
PJ
.
Implementation of the computerized patient record system and other clinical computing applications at the VA Puget Sound Health Care System
. In
Metzger
JB
(ed):
Proceedings of the Sixth Annual Nicholas E Davies CPR Recognition Symposium
 
November
4–5
,
Los Angeles
,
2000
, pp
77
120
.
16
Tang
PC
McDonald
CM
.
Computer-based patient-record systems
. In
Shortliffe
EH
Perrault
LE
Wiederhold
G
Fagan
LM
(eds).
Medical Informatics—Computer Applications in Health Care and Biomedicine
 
2nd ed
.
New York
,
Springer
,
2001
, pp
327
358
.
17
Patton
M
.
Enhancing the quality and credibility of qualitative analysis
.
Health Serv Res
 
1999
;
34
:5; Part II:
1189
1208
.
18
Mays
N
Pope
C
.
Quality in qualitative health research
. In
Pope
C
Mays
N
(eds):
Qualitative Research in Health Care
 
2nd ed.
London
,
BMJ Books
,
2000
, pp
89
110
.
19
Pope
C
Ziebland
S
Mays
N
.
Analysing qualitative data
. In
Pope
C
Mays
N
(eds):
Qualitative Research in Health Care
 
2nd ed.
London
,
BMJ Books
,
2000
, pp
75
88
.
20
Kuperman
GJ
Gardner
RM
Pryor
TA
.
HELP: A Dynamic Hospital Information System
 
New York
,
Springer-Verlag
,
1991
.
21
Gardner
RM
Lundsgaarde
HP
.
Evaluation of user acceptance of a clinical expert system
.
J Am Med Inform Assoc
 
1994
;
1
:
428
438
.
22
Overhage
JM
Perkins
S
Tierney
WM
McDonald
CJ
.
Controlled trial of direct physician order entry: effects on physicians' time utilization in ambulatory primary care internal medicine practices
.
J Am Med Inform Assoc
 
2001
;
8
:
361
371
.
23
Bates
DW
Boyle
DL
Teich
JM
.
Impact of computerized physician order entry on physician time
.
Proceedings of the 18th Symposium on Computer Applications in Medical Care
 
1994
, pp
996
.
24
Lee
F
Teich
JM
Spurr
CD
Bates
DW
.
Implementation of physician order entry: user satisfaction and self-reported usage patterns
.
J Am Med Inform Assoc
 
1996
;
3
:
42
45
.
25
Evans
RS
Pestotnik
SL
Classen
DC
et al
.
A computer-assisted management program for antibiotics and other antiinfective agents
.
N Engl J Med
 
1998
;
338
:
232
238
.
26
Overhage
JM
Tierney
WM
Xiao-Hua
Z
McDonald
CJ
.
A randomized trial of “corollary orders” to prevent errors of omission
.
J Am Med Inform Assoc
 
1997
;
4
:
364
375
.
27
Tierney
WM
Miller
ME
McDonald
CJ
.
The effect on test ordering of informing physicians on the charges for outpatient diagnostic tests
.
N Engl J Med
 
1990
;
322
:
1499
1504
.
28
Bates
DW
Teich
JM
Lee
J
et al
.
The impact of computerized physician order entry on medication error prevention
.
J Am Med Inform Assoc
 
1999
;
6
:
313
321
.
29
Teich
JM
Merchia
PR
Schimz
JL
et al
.
Effects of computerized physician order entry on prescribing practices
.
Arch Intern Med
 
2000
;
160
:
2741
2747
.
30
Chetow
GM
Lee
J
Kuperman
GJ
et al
.
Guided medication dosing for inpatients with renal insufficiency
.
JAMA
 
2001
;
286
:
2839
2844
.
31
Bates
DW
Kuperman
GJ
Rittenberg
E
et al
.
A randomized trial of a computer based intervention to reduce utilization of redundant laboratory tests
.
Am J Med
 
1999
;
106
:
144
150
.
32
Kuperman
GJ
Teich
JM
Tanasijevic
MJ
et al
.
Improving response to critical laboratory results with automation: results of a randomized control trial
.
J Am Med Inform Assoc
 
1999
;
6
:
512
522
.
33
Jha
AK
Kuperman
GJ
Teich
JM
et al
.
Identifying adverse drug events: Development of a computer-based monitor and comparison with chart review and simulated voluntary report
.
J Am Med Inform Assoc
 
1998
;
5
:
305
314
.
34
McDonald
CJ
Hui
SL
Smith
DM
et al
.
Reminders to Physicians from an introspective computer medical record
.
Ann Intern Med
 
1984
;
100
:
130
138
.
35
McDonald
CJ
Hui
SL
Tierney
WM
.
Effects on computer reminders for influenza vaccination on morbidity during influenza epidemics
.
MD Comput
 
1992
;
9
:
304
312
.
36
Dexter
PR
Perkins
S
Overhage
JM
Maharry
K
Kohler
RB
McDonald
CJ
A computerized reminder system to increase the use of preventive care for hospitalized patients
.
N Engl J Med
 
2001
;
345
:
965
970
.
37
Bates
DW
Studer
J
Reilly
CA
Cureton
EL
Spurr
CD
Kuperman
GJ
.
Evaluating the impact of a computerized ambulatory record
.
Proc AMIA Symp 2000
  p
964
.
38
Overhage
JM
Dexter
PR
Perkins
SM
et al
.
A randomized controlled trial of clinical information shared from another institution
.
Ann Emerg Med
 
2002
;
39
:
14
23
.
39
Bates
DW
Pappius
E
Kuperman
GJ
et al
.
Using information systems to measure and improve quality
.
Int J Med Inform
 
1999
;
53
:
115
124
.
40
Gardner
R
Pryor
T
Warner
H
.
The HELP hospital information system: update 1998
.
Int J Med Inform
 
1999
;
54
(
3
):
169
182
.
41
McDonald
CJ
Overhage
JM
Tierney
WM
et al
.
The Regenstrief Medical Record System: a quarter century's experience
.
Int J Med Inform
 
1999
;
54
(
3
):
67
74
.
42
Tiech
JM
Glaser
JP
Beckley
RF
et al
.
The Brigham integrated computing systems (BICS): advanced clinical systems in an academic hospital environment
.
Int J Med Inform
 
1999
;
54
(
3
):
197
208
.
43
Kaplan
B
.
Evaluating informatics applications - clinical decision support systems literature review
.
Int J Med Inform
 
2001
;
64
:
15
37
.
44
Kuhn
KA
Giuse
DA
.
From hospital information systems to health information systems: problems, challenges, perspectives
.
Method Inform Med
 
2001
;
40
:
275
287
.
45
Lorenzi
NM
Riley
RT
Blyth
AJ
Southon
G
Dixon
BJ
.
Antecedents of the people and organizational aspects of medical informatics: Review of the literature
.
J Am Med Inform Assoc
 
1997
;
4
:
79
93
.
46
Kolodner
RM
(ed):
Computerizing Large Integrated Health Networks: The VA success
 
New York
,
Springer Verlag
,
1997
.
Research was conducted while David Doolan was a Harkness Fellow in Health Care Policy. Funding for the fellowship was provided by the Commonwealth Fund, a New York City-based private independent foundation. The views presented here are those of the authors and not necessarily those of the Commonwealth Fund, its director, officers, or staff.
The authors are indebted to the many individuals at the five case sites who gave us the value of their experience and knowledge. Although it is not possible to mention every person by name, we thank the following people for facilitating the site visits: Marc Overhage MD, PhD, at Regenstrief Institute and Wishard Memorial Hospital; Dan Davis, MD, at Queens Medical Center; and Jan Torell at Veteran's Affairs Puget Sound Health Care System. We also thank the three anonymous reviewers of this paper for their valuable contributions.

Comments

0 Comments