Abstract

In this systematic review, the authors analyze the functionality provided by recent computer-based guideline implementation systems and characterize the effectiveness of the systems. Twenty-five studies published between 1992 and January 1998 were identified. Articles were included if the authors indicated an intent to implement guideline recommendations for clinicians and if the effectiveness of the system was evaluated. Provision of eight information management services and effects on guideline adherence, documentation, user satisfaction, and patient outcome were noted.

All systems provided patient-specific recommendations. In 19, recommendations were available concurrently with care. Explanation services were described for nine systems. Nine systems allowed interactive documentation, and 17 produced paper-based output. Communication services were present most often in systems integrated with electronic medical records. Registration, calculation, and aggregation services were infrequently reported. There were 10 controlled trials (9 randomized) and 10 time-series correlational studies. Guideline adherence improved in 14 of 18 systems in which it was measured. Documentation improved in 4 of 4 studies.

Despite the considerable effort and resources that have been invested in the development and dissemination of clinical practice guidelines, there continues to be considerable variation in the effectiveness of guidelines to bring about changes in the behavior of clinicians. A number of studies have found that, despite serious initiatives on the part of national organizations to develop and disseminate guidelines, practitioners may still ignore them.1–6.

Greco and Eisenberg7 devised a general taxonomy of methods that may be used to influence clinician behavior. These include education, feedback, participation by physicians in efforts to bring about change, administrative rules, financial incentives, and penalties. Several investigators have attempted to identify which factors in guideline implementation strategies are most efficacious. Davis and Taylor-Vaisey8 found that reminder systems, academic detailing, and the use of combined interventions were most effective.

Grimshaw and Russell9 found that the guideline implementation strategies most likely to be effective were those that delivered patient-specific advice at the time and place of a consultation. Computers can provide, concurrent with care, advice that is tailored to the needs of individual patients. A systematic review by Johnston et al.10 found that computer-based decision support can improve clinician performance.

Any computer-based tool is more likely to work if it is integrated with clinical activities. Elson11 pointed out the critical role of workflow integration for effective guideline implementation. To be accepted, guideline implementation applications should give back to the user something of value to offset the inconvenience of using the system.

In the course of developing a computer-based guideline implementation system to assist in the management of childhood asthma and in creating additional tools, we identified eight information management services that promote workflow integration—recommendation, documentation, registration, communication, calculation, explanation, presentation, and aggregation. Each service adds value to a computer application that should translate to an increased probability for success. In addition, the services provide a structure for comparison and evaluation of dissimilar implementations.

Because both successful and unsuccessful strategies have provided many of these services, we sought to perform a detailed analysis of the functionality delivered by current computer-based implementation systems. In this paper, we analyze which information management services have been delivered by recently described guideline implementations. We also review the effectiveness of the computer-based interventions in influencing clinicians' behavior and changing patient outcomes.

Methods

Using the OVID search engine, the MEDLINE and CINAHL databases from 1992 through January 1998 were searched. Search terms included the following MeSH headings—algorithm, computer-assisted decision making, computer-assisted therapy, consensus statement, guideline adherence, health planning guidelines, health services research, medical audit, practice guideline, process and outcome evaluation, quality assurance, quality of health care, and reference standard—and the following text words—remind$, alert$, guideline$, implement$, and computer$. We also reviewed books and bibliographies of primary and review articles.

We limited the review to papers published in or after 1992 for three reasons: 1) The U.S. Agency for Health Care Policy and Research (AHCPR) began its dissemination of evidence-based clinical practice guidelines in 1992, leading a vanguard of new interest in guideline implementation. 2) Several information management services are delivered optimally with a graphical user interface. The first broadly successful operating system that supported the interface (Windows 3.1, Microsoft Inc., Redmond, Washington) was released early in 1992. 3) Two influential systematic literature reviews on similar topics covered papers that had been published prior to 1992.9,10

We selected studies for review if computers were used as part of an implementation strategy for clinical practice guidelines; the authors specifically indicated an intent to implement guideline recommendations—not simply to provide computer-based decision support; the implementations were intended to influence health care providers (thereby excluding systems that provided recommendations directly to patients); and the studies included an evaluation component that objectively studied some aspect of the effectiveness of the system in a practice setting.

Information Management Services Model

We have devised an information management services model for the implementation of clinical practice guidelines. Briefly, the model comprises eight components, defined as follows:

  • Recommendation: the determination of appropriate, guideline-specified activities that should occur under specific clinical circumstances

  • Documentation: the collection, recording, and storage of observations, assessments, and interventions related to clinical care

  • Explanation: the provision of background information on decision variables and guideline-specified actions (e.g., definitions, measures of quality or cost) and the rationale that supports guideline recommendations, including evidence and literature citations

  • Presentation: the creation of useful output from internal data stores

  • Registration: the recording and storage of administrative and demographic data to uniquely identify the patient, provider(s), and encounter

  • Communication: the transmission and receipt of electronic messages between the clinician and other information providers

  • Calculation: the manipulation of numeric or temporal data, or both, to derive required information

  • Aggregation: the derivation of population-based information from individual patient data

▪   A variety of methods may be used to incorporate each service in a computer-based guideline implementation. Although their inclusion should result in a more comprehensive, workflow-integrated system, individual services may be excluded from any specific application.

▪   Each pertinent article was scrutinized by at least two of the authors for evidence that the system described there did or did not provide each information management service. Disagreements between the authors were settled by discussion. In addition, evidence of effectiveness was extracted and summarized. We determined the types of studies that were performed to evaluate each system (using the classification system used by Grimshaw and Russell9) and then ascertained the effectiveness of the system with regard to guideline adherence and other process measures (such as documentation and user satisfaction) and patient outcome measures when available. Because of the wide variety of study types and outcome variables, no quantitative meta-analysis of the results was possible.

Results

A total of 25 papers that described 20 discrete systems were identified. We included more than one report on a single system if descriptions and evaluations of a single system were segregated into more than one report or if the authors investigated more than one guideline implementation strategy on the same system. Features of the guideline implementation systems are summarized in Table 1.

Table 1

Computer-based Guideline Implementation Systems

Study (year) Domain; System; Site Source of Guideline Recommendation Explanation Documentation Registration Communication Calculation Presentation Aggregation 
Bouhaddou et al.12 (1994) Three procedure preauthorizations (cholecystectomy, cataract extraction, knee arthroscopy); ILIAD; IHC Health Plan and University of Utah Medical Center, Salt Lake City, Utah Locally developed Surgical preauthorization; delayed Logic and deviations from preauthorization criteria Surgical indications documented on paper and phoned or faxed to central office; prompted, noninteractive Nd None Nd Paper: status of preauthorization; full report of guideline compliance/deviations Nd 
Burack et al.13 (1994) Mammography screening; multiple practice sites in Detroit, Mich. USPSTF and other authorities Mammography reminder; concurrent Breast cancer risk factors Procedure indication (screening or diagnostic) initiator, patient response to referral; prompted, noninteractive Yes Nd Age Paper: reminder form, previous mammography results, appointment postcard Nd 
Burack and Gimotty14 (1997)           
Day et al.15 (1995) Low back pain; Emergency Department Expert Charting System; UCLA Emergency Medicine Center, Los Angeles, Calif. AHCPR with local adaptation Advice about appropriate tests, treatment, and disposition; prompts for missing history or physical exam items; concurrent Nd Core history and physical examination items; clinician's rationale for deviation from recommendations; prompted, interactive Nd None Nd Paper: after-care instructions for patients, laboratory and treatment orders, prescriptions Nd 
         On-screen: color-coded suggestion prompts and order screens  
Dexter et al.16 (1998) Advanced directives; Regenstrief Medical Records System; General Medicine Practice, Indianapolis, Ind. Nd Reminders to discuss advanced directives; concurrent Nd Paper-based encounter sheet allows choice of discussed, deferred, or rationale for deviation; prompted, noninteractive EMR EMR (Regenstrief system) Age Paper: reminders on encounter forms Nd 
Goethe and Bronzino17 (1995) Pharmacotherapy in psychiatry; Clinical Evaluation and Monitoring System; Institute of Living, Hartford, Conn. Locally developed Alerts for appropriate psychotropic medications based on diagnosis, appropriate baseline and followup laboratory testing and drug monitoring, potential for drug-drug interactions; concurrent Summary information about medications, potential for drug-drug interactions Documentation checklists for key symptoms and behavioral issues; rationale for deviation; prompted, interactive Nd Laboratory, pharmacy, and diagnostic data Nd On-screen: alerts, historical laboratory values, medications, medical problems Nd 
Goethe et al.18 (1997)           
Litzelman et al.19 (1993) Preventive care (fecal occult blood, mammography, Pap smear); Regenstrief Medical Records System; Indianapolis, Ind. Canadian Task Force on the Periodic Health Exam, Lifetime Health Monitoring, American Cancer Society Reminders about fecal occult blood testing annually if age >50, mammography for women over age 50, Pap testing based on age and previous test results; concurrent Explanation of rule that generated reminder Clinic visit notes handwritten on encounter forms; rationale for deviation; prompted, noninteractive EMR EMR (Regenstrief system) Age Paper: indicated tests on the encounter form, reminder report Nd 
Lobach and Hammond20 (1997) Diabetes; CAMP; Duke University Medical Center, Durham, N.C. American Diabetes Association with local adaptation Care recommendations regarding which studies or procedures are currently due and due at next visit; concurrent Nd Handwritten documentation that a recommended action was performed, declined, or never to be done; data entry by clinic personnel; prompted, noninteractive EMR EMR, scheduling, accounting, laboratory orders, results (TMR system) Age, intervals Paper: recommendations on encounter form, prescriptions, orders Nd 
Margolis et al.21 (1992) Six common pediatric problems; CHARTS; Community Pediatrics Clinic, Olfaqqueem, Israel Nd Advice from clinical management algorithms; concurrent Nd Clinical observations recorded on clinical algorithm serve as visit form; prompted, interactive Nd Nd Nd On-screen: clinical algorithm Paper: record of the visit Nd 
Nilasena et al.22 (1994) Diabetes; University of Utah Medical Center and the VA Medical Center, Salt Lake City, Utah American Diabetes Association with local adaptation Alerts about high-risk aspects of clinical profile; concurrent Nd Seven data entry forms, self-contained database; data entry by clerical personnel from paper forms; prompted, noninteractive Demographics None Nd Paper: health maintenance report with demographics, preventive health status, schedule of upcoming and past-due preventive activities Nd 
Nilasena and Lincoln23 (1995)           
Ornstein et al.24 (1993) 13 preventive services; Division of Family Medicine, Medical University of South Carolina, Charleston, S.C. USPSTF Reminders of deficient preventive services, e.g., dental, diet, injury prevention counseling, immunizations, screening of blood pressure, fecal occult blood, Pap smears, mammograms; concurrent Explanation and citations for each preventive service tracked Integrated with EMR (problem lists, progress notes, social history); tracks preventive services; some notes dictated and transcribed; prompted, interactive EMR EMR system, laboratory results Age, intervals Paper: reminders to physicians, annual letters to patients to alert them about preventive services Nd 
         On-screen: reminders to physicians  
Ornstein et al.25 (1995)           
Overhage et al.26 (1996) 22 adult preventive care in hospitalized patients; Regenstrief Medical Record System; Wishard Memorial Hospital, Indianapolis, Ind. USPSTF Preventive care reminders, suggested orders; concurrent Citations of literature to support recommendations Integrated with Regenstrief inpatient and outpatient EMR; prompted, interactive EMR Order entry for laboratory and pharmacy; EMR (Regenstrief system) Age, creatinine clearance Paper: reminder on daily rounds reports Nd 
         On-screen: reminders displayed during order entry  
Robbins et al.27 (1993) Lipid Management Program; private practice, Norfolk, Va. National Cholesterol Education guidelines with local adaptation Reminders about patients for whom laboratory data are due; concurrent Nd Lipid results and current therapy; prompted, noninteractive Nd Nd Nd Paper: reminders of laboratory tests due; results sent to patients and referring physicians Nd 
Rossi and Every28 (1997) Pharmacotherapy with calcium channel blockers in hypertension; DHCP and ACQUIP; Seattle VA Medical Center, Seattle, Wash. JNC V Advice regarding use of diuretic and beta blocker for at-risk patients; concurrent Cardiovascular risk associated with calcium channel blockers Paper-based form documents appropriate indications for calcium channel blocker therapy; data entry by clerical personnel; prompted, noninteractive EMR EMR (DHCP, ACQUIP systems) None Paper: reminder attached to prescription refill form Nd 
Safran et al.29 (1995) HIV management; Center for Clinical Computing, Beth Israel Hospital, Boston, Mass. Locally developed Alerts and reminders regarding HIV patients, including laboratory results, recommended medications and dosages, referrals, immunizations; concurrent Nd Integrated with EMR that includes problems, medications, preventive screenings, progress notes; prompted, interactive EMR EMR, scheduling, test ordering (Clinical Computing System) Nd On-screen: alerts, reminders (appear only when patient record is accessed) Laboratory, scheduling, demographics, dates of admission and discharge and alerts triggered 
Schriger et al.30 (1997) Occupational exposure; Emergency Department Expert Charting System; UCLA Emergency Medicine Center, Los Angeles, Calif. Locally developed Tests and treatments recommended, optional, not recommended; concurrent Computer's reasoning for each recommendation History of exposure event, exposed worker, and source; rationale for deviation; prompted, interactive Nd Nd Nd Paper: after-care instructions for patients (modifiable by clinician), prescriptions Deviation rates by decision 
Tape and Campbell31 (1993) Health maintenance; COSTAR medical record system; University of Nebraska Internal Medicine Clinic, Omaha, Neb. Nd Reminders about health maintenance deficiencies based on age, sex, chronic disease, and past health maintenance records; concurrent Nd Paper-based document; prompted, noninteractive EMR EMR (COSTAR system) Age, intervals Paper: problem list, medication list, and most recent progress note Nd 
Turner et al.32 (1994) Preventive care; private practices in eastern North Carolina Nd Reminders about influenza vaccine, Pap smears, breast exams, and mammography; concurrent Nd Nd Nd Nd Age Paper: reminder Nd 
Vincent et al.33 (1995) Disease prevention, cancer detection, immunization; Quality Care Program; Swedish Hospital, Seattle, Wash. Nd Recommended health maintenance procedures based on individual risk factors; concurrent Nd Paper-based worksheet completed at encounter and entered into system by clerical staff; prompted, noninteractive Yes Nd Age, intervals Paper: worksheet attached to medical record; health maintenance reminder letters for patients Nd 
Willson et al.34 (1995) Pressure ulcer prevention and treatment; LDS Hospital, Salt Lake City, Utah AHCPR Reminders to nurses to perform Braden assessments; stage appropriate treatment recommendations; concurrent Nd General screening and Braden scale; prompted, interactive EMR EMR (HELP system) Nd On-screen: alerts Nd 
Zielstorff et al.35 (1996) Pressure Ulcers; Pressure Ulcer Prevention & Management System; Massachusetts General Hospital, Boston, Mass. AHCPR Treatment plan, risk status; concurrent Definitions of individual data items Data entry screens provide explicit cues for assessment data; prompted, interactive Nd Nd Nd Paper: patient-specific treatment plan Nd 
Zielstorff et al.36 (1997)           
Study (year) Domain; System; Site Source of Guideline Recommendation Explanation Documentation Registration Communication Calculation Presentation Aggregation 
Bouhaddou et al.12 (1994) Three procedure preauthorizations (cholecystectomy, cataract extraction, knee arthroscopy); ILIAD; IHC Health Plan and University of Utah Medical Center, Salt Lake City, Utah Locally developed Surgical preauthorization; delayed Logic and deviations from preauthorization criteria Surgical indications documented on paper and phoned or faxed to central office; prompted, noninteractive Nd None Nd Paper: status of preauthorization; full report of guideline compliance/deviations Nd 
Burack et al.13 (1994) Mammography screening; multiple practice sites in Detroit, Mich. USPSTF and other authorities Mammography reminder; concurrent Breast cancer risk factors Procedure indication (screening or diagnostic) initiator, patient response to referral; prompted, noninteractive Yes Nd Age Paper: reminder form, previous mammography results, appointment postcard Nd 
Burack and Gimotty14 (1997)           
Day et al.15 (1995) Low back pain; Emergency Department Expert Charting System; UCLA Emergency Medicine Center, Los Angeles, Calif. AHCPR with local adaptation Advice about appropriate tests, treatment, and disposition; prompts for missing history or physical exam items; concurrent Nd Core history and physical examination items; clinician's rationale for deviation from recommendations; prompted, interactive Nd None Nd Paper: after-care instructions for patients, laboratory and treatment orders, prescriptions Nd 
         On-screen: color-coded suggestion prompts and order screens  
Dexter et al.16 (1998) Advanced directives; Regenstrief Medical Records System; General Medicine Practice, Indianapolis, Ind. Nd Reminders to discuss advanced directives; concurrent Nd Paper-based encounter sheet allows choice of discussed, deferred, or rationale for deviation; prompted, noninteractive EMR EMR (Regenstrief system) Age Paper: reminders on encounter forms Nd 
Goethe and Bronzino17 (1995) Pharmacotherapy in psychiatry; Clinical Evaluation and Monitoring System; Institute of Living, Hartford, Conn. Locally developed Alerts for appropriate psychotropic medications based on diagnosis, appropriate baseline and followup laboratory testing and drug monitoring, potential for drug-drug interactions; concurrent Summary information about medications, potential for drug-drug interactions Documentation checklists for key symptoms and behavioral issues; rationale for deviation; prompted, interactive Nd Laboratory, pharmacy, and diagnostic data Nd On-screen: alerts, historical laboratory values, medications, medical problems Nd 
Goethe et al.18 (1997)           
Litzelman et al.19 (1993) Preventive care (fecal occult blood, mammography, Pap smear); Regenstrief Medical Records System; Indianapolis, Ind. Canadian Task Force on the Periodic Health Exam, Lifetime Health Monitoring, American Cancer Society Reminders about fecal occult blood testing annually if age >50, mammography for women over age 50, Pap testing based on age and previous test results; concurrent Explanation of rule that generated reminder Clinic visit notes handwritten on encounter forms; rationale for deviation; prompted, noninteractive EMR EMR (Regenstrief system) Age Paper: indicated tests on the encounter form, reminder report Nd 
Lobach and Hammond20 (1997) Diabetes; CAMP; Duke University Medical Center, Durham, N.C. American Diabetes Association with local adaptation Care recommendations regarding which studies or procedures are currently due and due at next visit; concurrent Nd Handwritten documentation that a recommended action was performed, declined, or never to be done; data entry by clinic personnel; prompted, noninteractive EMR EMR, scheduling, accounting, laboratory orders, results (TMR system) Age, intervals Paper: recommendations on encounter form, prescriptions, orders Nd 
Margolis et al.21 (1992) Six common pediatric problems; CHARTS; Community Pediatrics Clinic, Olfaqqueem, Israel Nd Advice from clinical management algorithms; concurrent Nd Clinical observations recorded on clinical algorithm serve as visit form; prompted, interactive Nd Nd Nd On-screen: clinical algorithm Paper: record of the visit Nd 
Nilasena et al.22 (1994) Diabetes; University of Utah Medical Center and the VA Medical Center, Salt Lake City, Utah American Diabetes Association with local adaptation Alerts about high-risk aspects of clinical profile; concurrent Nd Seven data entry forms, self-contained database; data entry by clerical personnel from paper forms; prompted, noninteractive Demographics None Nd Paper: health maintenance report with demographics, preventive health status, schedule of upcoming and past-due preventive activities Nd 
Nilasena and Lincoln23 (1995)           
Ornstein et al.24 (1993) 13 preventive services; Division of Family Medicine, Medical University of South Carolina, Charleston, S.C. USPSTF Reminders of deficient preventive services, e.g., dental, diet, injury prevention counseling, immunizations, screening of blood pressure, fecal occult blood, Pap smears, mammograms; concurrent Explanation and citations for each preventive service tracked Integrated with EMR (problem lists, progress notes, social history); tracks preventive services; some notes dictated and transcribed; prompted, interactive EMR EMR system, laboratory results Age, intervals Paper: reminders to physicians, annual letters to patients to alert them about preventive services Nd 
         On-screen: reminders to physicians  
Ornstein et al.25 (1995)           
Overhage et al.26 (1996) 22 adult preventive care in hospitalized patients; Regenstrief Medical Record System; Wishard Memorial Hospital, Indianapolis, Ind. USPSTF Preventive care reminders, suggested orders; concurrent Citations of literature to support recommendations Integrated with Regenstrief inpatient and outpatient EMR; prompted, interactive EMR Order entry for laboratory and pharmacy; EMR (Regenstrief system) Age, creatinine clearance Paper: reminder on daily rounds reports Nd 
         On-screen: reminders displayed during order entry  
Robbins et al.27 (1993) Lipid Management Program; private practice, Norfolk, Va. National Cholesterol Education guidelines with local adaptation Reminders about patients for whom laboratory data are due; concurrent Nd Lipid results and current therapy; prompted, noninteractive Nd Nd Nd Paper: reminders of laboratory tests due; results sent to patients and referring physicians Nd 
Rossi and Every28 (1997) Pharmacotherapy with calcium channel blockers in hypertension; DHCP and ACQUIP; Seattle VA Medical Center, Seattle, Wash. JNC V Advice regarding use of diuretic and beta blocker for at-risk patients; concurrent Cardiovascular risk associated with calcium channel blockers Paper-based form documents appropriate indications for calcium channel blocker therapy; data entry by clerical personnel; prompted, noninteractive EMR EMR (DHCP, ACQUIP systems) None Paper: reminder attached to prescription refill form Nd 
Safran et al.29 (1995) HIV management; Center for Clinical Computing, Beth Israel Hospital, Boston, Mass. Locally developed Alerts and reminders regarding HIV patients, including laboratory results, recommended medications and dosages, referrals, immunizations; concurrent Nd Integrated with EMR that includes problems, medications, preventive screenings, progress notes; prompted, interactive EMR EMR, scheduling, test ordering (Clinical Computing System) Nd On-screen: alerts, reminders (appear only when patient record is accessed) Laboratory, scheduling, demographics, dates of admission and discharge and alerts triggered 
Schriger et al.30 (1997) Occupational exposure; Emergency Department Expert Charting System; UCLA Emergency Medicine Center, Los Angeles, Calif. Locally developed Tests and treatments recommended, optional, not recommended; concurrent Computer's reasoning for each recommendation History of exposure event, exposed worker, and source; rationale for deviation; prompted, interactive Nd Nd Nd Paper: after-care instructions for patients (modifiable by clinician), prescriptions Deviation rates by decision 
Tape and Campbell31 (1993) Health maintenance; COSTAR medical record system; University of Nebraska Internal Medicine Clinic, Omaha, Neb. Nd Reminders about health maintenance deficiencies based on age, sex, chronic disease, and past health maintenance records; concurrent Nd Paper-based document; prompted, noninteractive EMR EMR (COSTAR system) Age, intervals Paper: problem list, medication list, and most recent progress note Nd 
Turner et al.32 (1994) Preventive care; private practices in eastern North Carolina Nd Reminders about influenza vaccine, Pap smears, breast exams, and mammography; concurrent Nd Nd Nd Nd Age Paper: reminder Nd 
Vincent et al.33 (1995) Disease prevention, cancer detection, immunization; Quality Care Program; Swedish Hospital, Seattle, Wash. Nd Recommended health maintenance procedures based on individual risk factors; concurrent Nd Paper-based worksheet completed at encounter and entered into system by clerical staff; prompted, noninteractive Yes Nd Age, intervals Paper: worksheet attached to medical record; health maintenance reminder letters for patients Nd 
Willson et al.34 (1995) Pressure ulcer prevention and treatment; LDS Hospital, Salt Lake City, Utah AHCPR Reminders to nurses to perform Braden assessments; stage appropriate treatment recommendations; concurrent Nd General screening and Braden scale; prompted, interactive EMR EMR (HELP system) Nd On-screen: alerts Nd 
Zielstorff et al.35 (1996) Pressure Ulcers; Pressure Ulcer Prevention & Management System; Massachusetts General Hospital, Boston, Mass. AHCPR Treatment plan, risk status; concurrent Definitions of individual data items Data entry screens provide explicit cues for assessment data; prompted, interactive Nd Nd Nd Paper: patient-specific treatment plan Nd 
Zielstorff et al.36 (1997)           

Note: ACQUIP indicates Ambulatory Care Quality Improvement Program; AHCPR, Agency for Health Care Policy and Research; CAMP, computer-assisted management protocol; COSTAR, Computer-stored Ambulatory Record; DHCP, decentralized hospital computer system; EMR, electronic medical record; JNC V, Joint National Committee on Detection, Evaluation and Treatment of High Blood Pressure; Nd, not discussed; USPSTF, U.S. Preventive Services Task Force; VA, Veterans Administration.

Eleven systems were based on national guidelines, including those published by AHCPR; the American Diabetes Association; the National Cholesterol Education Program; the Joint National Committee on Detection, Evaluation, and Treatment of High Blood Pressure (JNC V); and the U.S. Preventative Services Task Force (USPSTF). In several cases, the authors commented on the need for local modifications of the guidelines. Four systems implemented locally developed guidelines. In five systems, the guideline source was not described.

Thirteen of the guideline implementations addressed patient management issues and therapy, including one system that provided pre-authorization for surgical procedures12 and another system that was intended to improve discussion of advance directives.16 The other seven systems provided guidance with screening and health maintenance activities.

Services

All systems provided patient-specific recommendations. The scope of the recommendations encompassed a broad range of clinical activities, including appropriate tests and treatments, alerts about at-risk states, and reminders of appropriate physical assessments and screening activities. With one exception, all systems provided recommendations concurrently with care.12 A variable number of factors were evaluated by the systems to determine appropriate intervention recommendations.

Nine reports documented that some explanation functionality was provided. These services provided background information, definitions, and risks as well as the rationale that supported specific recommendations. One system offered literature citations.

Most systems provided prompts for documentation of relevant findings that served a reminder function for the clinician-user. In many cases, these data were supplemented by complete medical record capabilities. Documentation services were provided in a variety of ways. Several systems relied on paper-based recording of clinician observations, which were later entered into the computer by clerical personnel. Others made use of online data entry, particularly those that were part of larger electronic medical record (EMR) systems. In nine systems the documentation process was interactive.*

Several of the reports described systems that were integrated with institutional EMR systems. Documentation services for these systems tended to make use of the functionality of the EMR. Other systems were essentially stand-alone systems and not integrated with an EMR.

Like the data entry services, presentation services also varied considerably. Presentation modalities included paper-based display of reminders that were attached to patient charts, on-screen reminders and alerts, onscreen display of algorithms, patient summaries, customizable after-care instructions, and annual birthday letters to patients regarding appropriate preventive services. Seventeen of the systems made use of paper-based output of some kind.

Electronic communication services most frequently provided interfaces to the EMR and to order-entry functions. Interfaces existed in some systems to pharmacy, scheduling, and laboratory results reporting. Standalone systems, by definition, offered no electronic communication services.

None of the reports on stand-alone systems described provision of registration processes. Some mechanism for identification of patient demographics, provider, and the encounter was assumed to have been present in all cases, but specific mechanisms were not described. The EMR-related systems presumably have the capability to integrate demographic and administrative data.

Calculation services were used to calculate patient ages and intervals between tests and to trigger rules related to preventive services. The Lipid Management Program calculated lipid fractions.27

Aggregation services were described for only a few systems. In many cases, it is clear that database capabilities would allow aggregation of individual patient data, but only two reports explicitly described aggregation services. Schriger et al.30 noted that the database could be used to calculate deviation (non-adherence) rates by physicians. The Beth Israel Clinical Care System captured data about laboratory tests, demographics, dates of admission and discharge, and response to alerts that were used to generate aggregate reports.29

Evaluations

The methodologies used to evaluate the effectiveness of the 20 guideline implementations included ten controlled trials (nine of which were randomized) and ten time-series studies (none of which incorporated external controls; one applied a switchback design). The outcome variables that were measured also varied considerably and are summarized in Table 2.

Table 2

Effectiveness of Guideline Implementation Systems

Study Methodology Documentation Adherence Satisfaction Outcomes 
Bouhaddou et al.12 Time series without external control Improved; indication of surgery Improved; rate of request for surgery, mixed; increased preauthorization approval rate Improved Cost increased 
Burack et al.13 RCT by patient Nd Increased use of mammography in health department setting, but not in HMO setting Nd Nd 
Burack and Gimotty14      
Day et al.15 Time series without external control Improved after-care instructions No effect Nd No effect on cost 
Dexter et al.16 RCT by provider team Nd Increased discussion with intervention (24% vs. 4% without) Nd Improved rate of advance directive completion, 4% vs. 15% 
Goethe et al.18 Time series without external control Nd Improved response to alerts Nd Nd 
Litzelman et al.19 RCT by provider team Nd Improved mammography and fecal occult blood testing; no effect on Pap testing Nd Nd 
Lobach and Hammond20 RCT by clinician Nd Two-fold increase in compliance with guidelines for 3 of 8 standards; failure for 5 of 8 Nd Nd 
Margolis et al.21 Time series without external control Improved; 3 of 6 diseases Improved for 2 of 6 diseases; decreased inappropriate use of antibiotics for 2 diseases Too tedious, physicians refused to continue Nd 
Nilasena et al.22 RCT by physician Nd Improved average total compliance score for both control and intervention 70% found forms difficult to use and did not reduce time to provide care Nd 
Nilasena and Lincoln23      
Ornstein et al.25 Time series without external control Nd Improved compliance with counseling, screening tests, breast exams, and thyroid function tests; no change for immunizations, fecal occult blood, Pap smear, mammography No improvement in patient's perceived preventive services delivery Nd 
Overhage et al.26 RCT by provider team Nd No effect: control, 24% compliance rate; intervention, 23% Nd Nd 
Robbins et al.27 Time series without external control Nd Nd Nd Cholesterol, LDL, and triglycerides levels decreased; HDL increased 
Rossi and Every28 RCT by provider Nd Improved: 11.3% of patients changed to first line antihypertensive drugs Nd No significant change in BP 
Safran et al.29 RCT by site Nd Improved: response time to alerts, 52 vs. 11 days Nd No change in admission rates, ER visits, survival or pneumocystis admissions 
Schriger et al.30 Time series with switchback Improved for 6 of 7 history items Improved: 4 laboratory tests; 4 of 5 treatments Nd Cost: charges decreased for laboratory and prescriptions 
Tape and Campbell31 Nonrandomized controlled trial by provider team Nd Improved sigmoidoscopy and immunization rates; no change in thyroxine, Pap, mammography, fecal occult blood screening Nd Nd 
Turner et al.32 RCT by physician Nd Small improvement in health maintenance activities Nd Nd 
Vincent et al.33 Time series without external control Nd Improved provider compliance; patient compliance improved for some Nd Nd 
Willson et al.34 Time series without external control Nd Improved for both prevention and treatment protocols Nd Decrease in incidence of pressure ulcers 
Zielstorff et al.36 Time series with control Nd No effect: clinical decision making Positive for instructional adequacy and user satisfaction Nd 
Study Methodology Documentation Adherence Satisfaction Outcomes 
Bouhaddou et al.12 Time series without external control Improved; indication of surgery Improved; rate of request for surgery, mixed; increased preauthorization approval rate Improved Cost increased 
Burack et al.13 RCT by patient Nd Increased use of mammography in health department setting, but not in HMO setting Nd Nd 
Burack and Gimotty14      
Day et al.15 Time series without external control Improved after-care instructions No effect Nd No effect on cost 
Dexter et al.16 RCT by provider team Nd Increased discussion with intervention (24% vs. 4% without) Nd Improved rate of advance directive completion, 4% vs. 15% 
Goethe et al.18 Time series without external control Nd Improved response to alerts Nd Nd 
Litzelman et al.19 RCT by provider team Nd Improved mammography and fecal occult blood testing; no effect on Pap testing Nd Nd 
Lobach and Hammond20 RCT by clinician Nd Two-fold increase in compliance with guidelines for 3 of 8 standards; failure for 5 of 8 Nd Nd 
Margolis et al.21 Time series without external control Improved; 3 of 6 diseases Improved for 2 of 6 diseases; decreased inappropriate use of antibiotics for 2 diseases Too tedious, physicians refused to continue Nd 
Nilasena et al.22 RCT by physician Nd Improved average total compliance score for both control and intervention 70% found forms difficult to use and did not reduce time to provide care Nd 
Nilasena and Lincoln23      
Ornstein et al.25 Time series without external control Nd Improved compliance with counseling, screening tests, breast exams, and thyroid function tests; no change for immunizations, fecal occult blood, Pap smear, mammography No improvement in patient's perceived preventive services delivery Nd 
Overhage et al.26 RCT by provider team Nd No effect: control, 24% compliance rate; intervention, 23% Nd Nd 
Robbins et al.27 Time series without external control Nd Nd Nd Cholesterol, LDL, and triglycerides levels decreased; HDL increased 
Rossi and Every28 RCT by provider Nd Improved: 11.3% of patients changed to first line antihypertensive drugs Nd No significant change in BP 
Safran et al.29 RCT by site Nd Improved: response time to alerts, 52 vs. 11 days Nd No change in admission rates, ER visits, survival or pneumocystis admissions 
Schriger et al.30 Time series with switchback Improved for 6 of 7 history items Improved: 4 laboratory tests; 4 of 5 treatments Nd Cost: charges decreased for laboratory and prescriptions 
Tape and Campbell31 Nonrandomized controlled trial by provider team Nd Improved sigmoidoscopy and immunization rates; no change in thyroxine, Pap, mammography, fecal occult blood screening Nd Nd 
Turner et al.32 RCT by physician Nd Small improvement in health maintenance activities Nd Nd 
Vincent et al.33 Time series without external control Nd Improved provider compliance; patient compliance improved for some Nd Nd 
Willson et al.34 Time series without external control Nd Improved for both prevention and treatment protocols Nd Decrease in incidence of pressure ulcers 
Zielstorff et al.36 Time series with control Nd No effect: clinical decision making Positive for instructional adequacy and user satisfaction Nd 

Note: BP indicates blood pressure; Nd, not discussed; HDL, high-density lipoprotein; LDL, low-density lipoprotein; RCT, randomized controlled trial

Four studies looked at documentation and found improvement in each case. The average number of relevant data items for surgical pre-authorization increased from 4.0 to 28.812; the mean percentage increase for documentation of common pediatric problems was 58 percent,21 for management of back pain 30.2 percent,15 and for management of exposure to body fluids 42 percent.30

Eighteen of the 20 studies evaluated provider adherence to the guidelines. In 14 of the 18, some level of improved adherence was described. In several reports, adherence improvements occurred for some of the measured outcomes but not for all.

Failure to improve adherence using computer-based strategies was reported in four studies. An attempt to improve preventive care guideline adherence for hospitalized patients failed because of functional and systemic barriers that interfered with providing preventive care to inpatients.26 One study of prevention and management of pressure ulcers was unable to show any effect of the computer-based intervention on nursing decision making.36 In that case, the authors concluded that there was not enough gain for the effort of data entry. A system designed to influence decision making in emergency room patients with back pain failed because of general confusion regarding the utility of plain x-rays in these patients and the fact that recommendations were not enforced.15 Finally, in a study of diabetes management guidelines, compliance improved to the same degree in both control and intervention groups; the authors questioned study design issues.23

Clinician satisfaction was addressed in four studies. Two investigators found that users were satisfied with computer-based guideline interventions.12,36 On the other hand, physician-users of a clinical algorithm system found data entry so tedious that they refused to continue,21 and Nilasena et al.22 found that 70 percent of users complained that data entry forms were difficult to use and inefficient.

Eight studies examined patient outcomes. A study of an intervention for low-back pain found no effect on cost,15 whereas costs increased in both a system for management of health care workers exposed to body fluids and another that pre-authorized surgery.12,30 Use of a lipid tracking system was associated with improvements in patients' cholesterol and lipid fractions.27 A system for prevention of pressure ulcers was associated with a decreased incidence of decubiti,34 and Dexter et al.16 reported a significant improvement in the completion of advance directives (15 percent vs 4 percent for a control group) using a computer-based reminder system. An intervention to substitute appropriate antihypertensives for calcium channel blockers did not have any effect on patients' blood pressure,28 and alerts about appropriate HIV management did not change admission rates, emergency department visits, survival, or pneumocystis admissions.29

Discussion

To better understand the design factors responsible for the success or failure of computer-based guideline intervention strategies, we analyzed reports on 20 systems that were intended to implement guideline recommendations in clinical practice. Specifically, we assessed the use of eight information management services, which we believe may be useful in integrating computerized systems into clinical workflow. Many reports failed to describe the systems in sufficient detail to ascertain the presence or absence of some of these services. Therefore, we were unable to create meaningful summary ratings of individual systems that might correlate with the outcomes described. However, we were able to describe qualitatively many aspects of the reported design of current computer applications used as guideline intervention tools and to summarize measures of their effectiveness.

All systems delivered patient-specific recommendations, and in most cases the advice was made available concurrently with care, thus meeting Grimshaw and Russell's criteria for implementations with a high probability of success.9 However, providing recommendations in this manner was neither necessary nor sufficient to ensure adherence. Several authors were unable to influence guideline adherence with concurrent reminders. Even providing delayed feedback was associated in one case with increased procedure authorization rates, although this system's influence may have been related to financial incentives and disincentives.12

The level of specificity of the advice varied considerably, as evidenced by the number of factors that were weighed by the programs to trigger relevant recommendations. Some systems simply checked a patient's age and gender to discern appropriate preventive interventions, whereas others monitored ongoing clinical transactions and considered multiple factors (e.g., diagnoses, laboratory results, and medications) in arriving at recommendations for changing medications or dosages and for planning treatment.

Somewhat surprisingly, fewer than half the reports documented provision of explanation services. More than 15 years ago, Teach and Shortliffe37 showed the importance of providing explanation for computer-based advisories.37 One noteworthy benefit of the use of computers for implementation of guideline recommendations is their capability to link recommendations dynamically to the evidence that supports them.

Most reports described the use of on-screen and paper-based prompts to remind users of critical information that should be documented. Clinicians entered data into computers directly and interactively in fewer than half the systems. Even some long-established EMR systems depended on completion of paper forms with subsequent data entry by clerical personnel. Likewise, paper-based output was described for 17 of the 20 systems. It seems clear that the paperless office remains a vision of the future.

Registration, calculation, communication, and aggregation services were infrequently described. These components offer tremendous potential benefit for well-designed computer-based guideline implementation. Providing communication services requires networked systems. Registration services may seem mundane, but an interface to an administrative database that contains this information may be vital to the success of a computer-based initiative by diminishing the clerical workload for clinicians. Calculation and aggregation services are basic functions of many computer systems that were rarely reported in these guideline implementation systems.

The evaluations of system effectiveness varied markedly in design, implementation, and level of description. In many, the evaluations of effectiveness were methodologically weak. In addition, the guidelines that were implemented differed considerably in content, from health maintenance reminders to alerts for active management of specific disease states. There were also notable variations in clinical settings—inpatient, emergency room, ambulatory clinic, private office, and public health department—and in evaluation methodology.

Fourteen studies reported some improvement in adherence to guidelines, seemingly independent of the information management services provided. Clearly, adherence to guideline recommendations can be improved in many cases using computer-based interventions. Likewise, documentation is regularly assisted with computers, but user satisfaction may be affected adversely by tedious data entry requirements in the absence of offsetting system benefits. In both studies with negative evaluations of user satisfaction,21,23 arduous data entry was suggested as a reason for poor system acceptance. Few studies examined patient outcomes to validate the effectiveness of the systems.

Conclusions

Many factors influence the success or failure of guideline implementation systems. While provision of a wide array of information management services may be important, it may not be sufficient to ensure success. To adequately evaluate the effect of those services on the success or failure of a computer-based guideline implementation, more of the confounding variables need to be controlled. In the studies described here, different types of guidelines, different settings, and different system implementations make conclusions about the relationship between information management services and outcomes difficult. In addition, a component of publication bias is likely to be present, in that the generally favorable results may represent a biased subset of system implementations.

Our information management services model was designed to provide a checklist for providing solutions that maximize workflow integration. Although this model may not cover exhaustively all factors responsible for implementation acceptance, we believe that it can be used profitably for the design of computer-based guideline implementation strategies and can serve as a framework for system evaluation. Future system developers should learn from the successes and failures of past systems.

The authors thank the members of the Guidelines Review Group at the Yale Center for Medical Informatics, who were instrumental in the conceptualization of the information management services model.

References

1
Lomas
J
Anderson
GM
Domnick-Pierre
K
Vayda
E
Enkin
MW
Hannah
WJ
.
Do practice guidelines guide practice? The effect of a consensus statement on the practice of physicians
.
N Engl J Med
 
1989
;
321
:
1306
11
.
2
Bezerra
JA
Stathos
TH
Duncan
B
Gaines
JA
Udall
JN
.
Treatment of infants with acute diarrhea: what's recommended and what's practiced
.
Pediatrics
 
1992
;
90
:
1
4
.
3
Siegel
D
Lopez
J
.
Trends in antihypertensive drug use in the United States: do the JNC V recommendations affect prescribing?
.
JAMA
 
1997
;
278
:
1745
8
.
4
Lenfant
C
.
JNC Guidelines: Is the message getting through?
.
JAMA
 
1997
;
278
:
1778
9
.
5
Soumerai
SB
McLaughlin
TJ
Spiegelman
D
Hertzmark
E
Thibault
G
Goldman
L
.
Adverse outcomes of underuse of β-blockers in elderly survivors of acute myocardial infarction
.
JAMA
 
1997
;
277
:
115
21
.
6
Rockall
TA
.
Influencing the practice and outcome in acute upper gastrointestinal haemorrhage
.
Gut
 
1997
;
41
:
606
11
.
7
Greco
P
Eisenberg
J
.
Changing physicians' practices
.
N Engl J Med
 
1993
;
329
:
1271
74
.
8
Davis
DA
Taylor-Vaisey
A
.
Translating guidelines into practice: a systematic review of theoretic concepts, practical experience and research evidence in the adoption of clinical practice guidelines
.
Can Med Assoc J
 
1997
;
157
:
408
16
.
9
Grimshaw
JM
Russell
IT
.
Effect of clinical guidelines on medical practice: a systematic review of rigorous evaluations
.
Lancet
 
1993
;
342
:
1317
22
.
10
Johnston
ME
Langton
KB
Haynes
RB
Mathieu
A
.
Effects of computer-based clinical decision support systems on clinical performance and patient outcome: a critical appraisal of research
.
Ann Intern Med
 
1994
;
120
:
135
42
.
11
Elson
RB
.
Clinical practice guidelines
.
Dis Manage Health Outcomes
 
1997
;
1
:
63
74
.
12
Bouhaddou
O
Frucci
L
Cofrin
K
et al
.
Implementation of practice guidelines in a clinical setting using a computerized knowledge base (Iliad)
.
Proc. 18th Annu Symp Comput Appl Med Care
 
1994
:
258
62
.
13
Burack
RC
Gimotty
PA
George
J
Stengle
W
Warbasse
L
Moncrease
A
.
Promoting screening mammography in inner-city settings: a randomized controlled trial of computerized reminders as a component of a program to facilitate mammography
.
Med Care
 
1994
;
32
:
609
24
.
14
Burack
RC
Gimotty
PA
.
Promoting screening mammography in inner-city settings: the sustained effectiveness of computerized reminders in a randomized controlled trial
.
Med Care
 
1997
;
35
:
921
31
.
15
Day
F
Hoang
LP
Ouk
S
Nagda
S
Schriger
DL
.
The impact of a guideline-driven computer charting system on the emergency care of patients with acute low back pain
.
Proc 19th Annu Symp Comput Appl Med Care
 
1995
:
576
80
.
16
Dexter
PR
Wolinsky
FD
Gramelspacher
GP
et al
.
Effectiveness of computer-generated reminders for increasing discussions about advance directives and completion of advance directive forms
.
Ann Intern Med
 
1998
;
128
:
102
10
.
17
Goethe
JW
Bronzino
JD
.
Design issues in developing clinical decision support and monitoring systems
. In:
Bronzino
JD
(ed).
The Biomedical Engineering Handbook
 .
Boca Raton, Fla
:
CRC Press
,
1995
:
2668
75
.
18
Goethe
JW
Schwartz
HI
Szarek
BL
.
Physician compliance with practice guidelines
.
Conn Med
 
1997
;
61
:
553
8
.
19
Litzelman
DK
Dittus
RS
Miller
ME
Tierney
WM
.
Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols
.
J Gen Intern Med
 
1993
;
8
:
311
7
.
20
Lobach
DF
Hammond
WE
.
Computerized decision support based on a clinical practice guideline improves compliance with care standards
.
Am J Med
 
1997
;
102
:
89
98
.
21
Margolis
CZ
Warshawsky
SS
Goldman
L
Dagan
O
Wirtschafter
D
Pliskin
JS
.
Computerized algorithms and pediatricians' management of common problems in a community clinic
.
Acad Med
 
1992
;
67
:
282
4
.
22
Nilasena
DS
Lincoln
MJ
Turner
CW
et al
.
Development and implementation of a computer-generated reminder system for diabetes preventive care
.
Proc. 18th Annu Symp Comput Appl Med Care
 
1994
:
831
5
.
23
Nilasena
DS
Lincoln
MJ
.
A computer-generated reminder system improves physician compliance with diabetes preventive care guidelines
.
Proc 19th Annu Symp Comput Appl Med Care
 
1995
:
640
5
.
24
Ornstein
SM
Garr
DR
Jenkins
RG
.
A comprehensive microcomputer-based medical records system with sophisticated preventive services features for the family physician
.
J Am Board Fam Pract
 
1993
;
6
:
55
60
.
25
Ornstein
SM
Garr
DR
Jenkins
RG
Musham
C
Hamadeh
G
Lancaster
C
.
Implementation and evaluation of a computer-based preventive services system
.
Fam Med
 
1995
;
277
:
260
6
.
26
Overhage
JM
Tierney
WM
McDonald
CJ
.
Computer reminders to implement preventive care guidelines for hospitalized patients
.
Arch Intern Med
 
1996
;
156
:
1551
6
.
27
Robbins
JA
Dickinson
WA
Bartel
AG
Hartman
CW
.
Lipid Management Program: results of applying national guidelines in a private practice
.
South Med J
 
1993
;
86
:
289
92
.
28
Rossi
RA
Every
NR
.
A computerized intervention to decrease the use of calcium channel blockers in hypertension
.
J Gen Intern Med
 
1997
;
12
:
672
8
.
29
Safran
C
Rind
DM
Davis
RB
et al
.
Guidelines for management of HIV infection with computer-based patient's record
.
Lancet
 
1995
;
346
:
341
6
.
30
Schriger
DL
Baraff
LJ
Rogers
WH
Cretin
S
.
Implementation of clinical guidelines using a computer charting system: effect on the initial care of health care workers exposed to body fluids
.
JAMA
 
1997
;
278
:
1585
90
.
31
Tape
TG
Campbell
JR
.
Computerized medical records and preventive health care: success depends on many factors
.
Am J Med
 
1993
;
94
:
619
25
.
32
Turner
RC
Peden
JG
O'Brien
K
.
Patient-carried card prompts vs computer-generated prompts to remind private practice physicians to perform health maintenance measures
.
Arch Intern Med
 
1994
;
154
:
1957
60
.
33
Vincent
EC
Hardin
PA
Norman
LA
Lester
EA
Stinton
SH
.
The effects of a computer-assisted reminder system on patient compliance with recommended health maintenance procedures
.
Proc 19th Annu Symp Comput Appl Med Care
 
1995
:
656
60
.
34
Willson
D
Ashton
C
Wingate
N
et al
.
Computerized support of pressure ulcer prevention and treatment protocols
.
Proc 19th Annu Symp Comput Appl Med Care
 
1995
:
646
50
.
35
Zielstorff
RD
Barnett
GO
Fitzmaurice
JB
et al
.
A decision support system for prevention and treatment of pressure ulcers based on AHCPR guidelines
.
Proc AMIA Annu Fall Symp
 
1996
:
562
6
.
36
Zielstorff
RD
Estey
G
Vickery
A
Hamilton
G
Fitzmaurice
JB
Barnett
GO
.
Evaluation of a decision support system for pressure ulcer prevention and management: preliminary findings
.
Proc. AMIA Annu Fall Symp
 
1997
:
248
52
.
37
Teach
RL
Shortliffe
EH
.
An analysis of physician attitudes regarding computer-based clinical consultation systems
.
Comput Biomed Res
 
1982
;
14
:
542
58
.
*
References 15, 17, 21, 24, 26, 29, 30, 34, 35.
References 20, 22, 25, 26, 28, 29, 31, 34.
This work was supported in part by the National Library of Medicine through grants 1-R29-LM-05552 and T-15-LM-07056. Dr. Shiffman is a Robert Wood Johnson Generalist Physician Faculty Scholar.

Comments

0 Comments