Implementation and evaluation of a prototype consumer reporting system for patient safety events

Abstract

Objective

No methodologically robust system exists for capturing consumer-generated patient safety reports. To address this challenge, we developed and pilot-tested a prototype consumer reporting system for patient safety, the Health Care Safety Hotline.

Design

Mixed methods evaluation.

Setting

The Hotline was implemented in two US healthcare systems from 1 February 2014 through 30 June 2015.

Participants

Patients, family members and caregivers associated with two US healthcare systems.

Intervention

A consumer-oriented incident reporting system for telephone or web-based administration was developed to elicit medical mistakes and care-related injuries.

Main Outcomes Measures

Key informant interviews, measurement of website traffic and analysis of completed reports.

Results

Key informants indicated that Hotline participation was motivated by senior leaders’ support and alignment with existing quality and safety initiatives. During the measurement period from 1 October 2014 through 30 June 2015, the home page had 1530 visitors with a unique IP address. During its 17 months of operation, the Hotline received 37 completed reports including 20 mistakes without harm and 15 mistakes with injury. The largest category of mistake concerned problems with diagnosis or advice from a health practitioner. Hotline reports prompted quality reviews, an education intervention, and patient follow-ups.

Conclusion

While generating fewer reports than its capacity to manage, the Health Care Safety Hotline demonstrated the feasibility of consumer-oriented patient safety reporting. Further research is needed to understand how to increase consumers’ use of these systems.

adverse event reporting, patient safety, patient engagement, consumer health informatics

With first-hand knowledge of the patient's medical history and direct observation of the patient's status and care experience, patients and their families often are able to identify care-related errors and injuries. Many are able to relay their concerns and observations to health professionals in ways that prevent or mitigate harm, and that inform improvements [1]. However, it has been difficult to elicit this information reliably outside of research studies and other limited settings [2–7].

There are several reasons that prior reporting systems might have yielded fewer reports than would be expected based on the prevalence of medical error. Patients and caregivers may be unaware of the opportunity to report to freestanding systems that are not well known locally. They may see limited value in reporting if it is unclear whether the information will be used to make care safer, or if they worry about alienating their usual caregivers. They may find systems difficult to use or may need assistance in conveying their observations or concerns.

To address this challenge and opportunity, the US Agency for Healthcare Research and Quality (AHRQ) solicited proposals to develop and test a prototype consumer reporting system for patient safety events, to collect and analyze reports, and to compare those events reported by consumers with reports by healthcare professionals [8, 9].

Under contract with AHRQ, a research team led by RAND Corporation investigators designed and developed a prototype consumer reporting system (from 1 September 2011 to 31 January 2014), and then implemented and tested the system (from 1 February  2014 to 30 June 2015). This report summarizes the findings from the implementation and evaluation process, lessons learned and implications for future efforts to support consumer reporting of patient safety events.

Methods

Design and development

The design and development of the prototype consumer reporting system for patient safety events is described in detail elsewhere [6]. Briefly, we performed a literature review and environmental scan to understand the experience of previous healthcare consumer reporting systems. We developed a reporting tool suitable for telephone or web-based administration by drawing on existing instruments, and then refined the structure and content using focus groups, cognitive testing, by eliciting public comment and by regular guidance from an expert panel that included patient and consumer advocates. We developed both web and telephone platforms for administration and called our prototype consumer reporting system the ‘Health Care Safety Hotline’.

We structured the reporting tool to elicit both medical mistakes (errors) and injuries related to care, offering reporters the option to address either or both domains. ‘Mistake’ was defined as ‘something that was done (or not done) by a healthcare provider that would be considered incorrect at the time that it happened.’ We elicited information about injuries by asking reporters about ‘negative effects’, a phrase that elicited relevant reports in prior research [10], focus groups and cognitive interviews. We indicated that ‘negative effects can be physical or emotional and they may include infections, drug reactions or other complications’ [11]. We also elicited reports of conditions that consumers perceived to be ‘unsafe’, even if no error or injury occurred.

We developed protocols for processing telephone and web-based reports that included physician review of the reported information by a member of the research team within 48 h of receipt to identify potentially actionable reports (when the reporter consented to share information with care providers), clarification of ambiguous reports (with the reporter's prior consent), and classification of reports by research staff based on type of event, preventability and severity and contributing factors.

Our system enabled consumers to allow their report to be shared with specific providers. We created procedures to ensure the confidentiality of reports under protections afforded by the Healthcare Research and Quality Act of 1999 and to Patient Safety Organizations (PSOs) under the Patient Safety and Quality Improvement Act of 2005 [12, 13]. ECRI Institute hosted the telephone and web platforms. The project was reviewed and approved in advance by the RAND Corporation institutional review board.

Site selection

To pilot test the program, the research team sought candidate healthcare organizations that were committed to patient safety [14], located in communities with a diverse patient population, and amenable to local marketing efforts, and participants in a federally recognized PSO in order to facilitate confidential and protected data exchange and analysis. Of 18 candidate organizations, two large systems in one metropolitan area were selected and agreed to participate as partner sites. The systems provide comprehensive inpatient and ambulatory services for adult and pediatric patients in more than two million encounters annually.

Preparations at partner sites

Research team members conferred with liaisons at the partner systems to secure the support of clinical leaders and other stakeholders, and to work with the research team to review operational procedures. Reports that consumers agreed to share with the healthcare organization would be transmitted securely within 72 h, in time for the organization to investigate and respond in a timely manner to reports that met the criteria for grievances under the CMS Conditions of Participation for Medicare (42 C.F.R. 482.13).

Guided by the terms of the AHRQ solicitation, we designed the program to receive ‘unsolicited’ reports. As a result, the research team and study sites focused primarily on consumer education and outreach. Posters and pamphlets described the Hotline as ‘a research project… to make care better by making it safer.’ Documents were co-branded with the name and logo of the health system, RAND and ECRI Institute. Posters were distributed to hospitals for hallway display, brochures were available in lobbies, waiting rooms and registration desks, and business cards were included in admission packets. Promotional materials were also displayed in pharmacies, outpatient clinics and affiliated healthcare facilities. Outreach efforts intensified toward the end of the first year of Hotline implementation, informed by discussions with consumer members of the organization's Patient/Family Advisory Council, to engage case managers, nurses and interpreters in providing Hotline information during discharge planning and follow-up.

Evaluation design

The Hotline implementation evaluation used a ‘logic model’ based on the theory that resources in the form of community outreach and a user-friendly reporting platform would result in activities such as consumer reporting and events reviews. These activities would lead to outputs such as event reports and summaries that could, in turn, affect care delivery outcomes [15]. Accordingly, we used interviews to gain information from key informants about each partner organization's preparation, implementation process and operational experience. A subset of key informants included the risk management, patient safety and patient relations staff members who were directly responsible for Hotline implementation. Additional informants included their department supervisors as well as senior managers and clinical leaders from areas including medicine, nursing, legal and public affairs. We sought detailed information about preparations, obstacles, concerns, resource requirements, as well as early and subsequent experience. We followed up with conference calls with key informants throughout the pilot and at its conclusion.

Second, we measured traffic to the Hotline website in order to assess respondents’ safety reporting activities and potential difficulties navigating the site. We began to monitor web traffic nine months into the project through to the conclusion (from 1 October 2014 to 30 June 2015). We were able to attribute page views to unique IP addresses without unblinding the reporter's identity in order to examine user navigation of the website.

Third, we characterized completed Hotline reports. We examined consumers’ narrative reports and their characterization of the type of event and contributing factors, relationship between the reporter and the patient, and the reporter's willingness to share information with the partner organization involved in the event. Physician reviewers on the research team classified consumer reports using ‘AHRQ Common Formats’ [16]. We sought to determine the novelty and impact of consumer reports by asking partner organizations to match the shared Hotline reports with existing health system incident reports, to track the handling of consumer reports, and to describe any resulting changes in care processes.

Results

Partner site interviews

At each site, participation in the Hotline pilot was motivated by senior clinical and patient safety leader support for an initiative that was closely aligned with existing commitments to patient safety, patient engagement and transparency. Consumer reporting was attractive because it reinforced existing initiatives and messaging, building on the prior commitments of internal stakeholders.

Public relations staff members described the challenge of articulating a message that promoted the virtue of the healthcare organization as a transparent and consumer-friendly entity, while at the same time requesting information about lapses in care. They anticipated consumer ambivalence about the appropriateness of error reporting to an external entity rather than to the provider organization directly. Risk managers and patient relations staff had been concerned about the additional workload associated with a possible influx of new reports, although they later found that the initiative could be accommodated readily within existing resources. Patient relations staff at each organization took a lead role in managing the response to shared Hotline reports. This process at each organization replicated the established method for handling patient grievances, including acknowledgment, formal investigation, timely response and documentation required by the CMS Conditions of Participation in Medicare.

Web traffic

From 1 October 2014 to 30 June 2015, the Hotline home page received visits from 1530 visitors with a unique IP address, or an average of 170 unique visitors per month (Table 1). Seventy percent accessed the Hotline through Google or another search engine. Visitors commonly accessed the Frequently Asked Questions (FAQ) page (n = 647) and the first page of the event reporting form, a welcome and introduction page (n = 405). However, only 20 visitors completed a report via the website; 17 additional respondents completed reports using a toll-free telephone number. Seven visitors were under 18 years old and not permitted to complete a report. Seventeen additional reports were started but incomplete.

Table 1

Patient Safety Hotline website and telephone contacts

Activity	N
Total visitors with unique IP address	1530
Source of access
Search engine	382
Other	166
FAQ page	647
Event reporting form page 1—welcome and introduction	405
Event reporting form page 2—screening and consent	396
Completed online reports	20
Complete telephone Hotline reports	17
Total completed reports	37
Consented to clarification call	34
Completed clarification call*	22
Agreed to share report and patient identification with clinician or facility	34

*The remaining 34–22 = 12 reports did not require clarification.

Reporter characteristics

During its 17 months of operation (February 2014 through June 2015), the Hotline received 37 distinct reports, or 2.3 per month—all in English. Patients completed 14 reports and caregivers completed 23 additional reports (11 by a parent and 12 by a spouse, domestic partner or other family member). Four individuals entered a total of 12 reports. Most (n = 26 of 29) reporters agreed to participate in a follow-up clarification call and to share the report and the patient's identifying information with the physician or facility where the event occurred. The demographic profile of the patients identified in completed reports is shown in Table 2.

Table 2

Characteristics of patients identified in Hotline reports (N = 29)

Patient characteristic
Age, mean (range), SD	33.8 (1–90), 22.1
Gender, n (%)
Male	10 (34.5)
Female	18 (62.1)
Did not report	1 (3.4)
Race/ethnicity, n (%)
White	20 (69.0)
Black or African American	4 (13.8)
Hispanic, Latino or Spanish origin	1 (3.4)
Did not report	4 (13.8)
Insurance, n (%)
Private insurance	16 (55.2)
Medicare	3 (10.3)
Medicaid	3 (10.3)
Other	1 (3.4)
Did not report	6 (20.8)

Hotline reports

Report characteristics are displayed in Table 3. Of the 37 completed reports, 20 identified mistakes without harm and 15 reported mistakes with injury. The largest category of mistakes concerned problems with diagnosis or advice from a health practitioner. For example, one reporter stated, ‘no one seemed to be immediately trying to relieve the [patient's] discomfort or to assess the causes’. Reporters described operative delays, a cursory examination, inadequate staffing and a medication error. Most events occurred in the hospital or hospital emergency department.

Table 3

Patient Safety Hotline reports, summary (N = 37)

Report characteristic	N
Total reports	37
Reporter
Patient	14
Caregiver	23
Type of report
Mistake only	5
Negative effect only	17
Both	15
Mistake
Prescription drugs	1
Tests, procedures, or surgery	1
Pregnancy or childbirth	1
Diagnosis or advice from a health practitioner	5
Poor cleanliness or hygiene	2
Something else or multiple mistakes	10
Total	20
Negative effect
Physical only	3
Emotional only	5
Both	14
Total	22
Location of event
Hospital	18
Emergency department	7
Doctor's office or clinic	3
Contributing factors*
Communication
Staff did not listen to patient	17
Staff ignored what patient told them	17
Staff did not spend enough time with the patient	12
Other communication problems	15
Any communication problem	21
Care coordination
Staff did not seem to work well together as a team	12
Staff lacked follow up	11
Other care coordination problem	2
Any care coordination problem	19
Access
Patient did not get help or advice they needed	13
Patient did not get tests or treatments patient believed necessary	6
Patient did not get tests or treatments provided believed necessary	2
Any access problem	14
Responsiveness of staff
Patient did not get help as soon as the patient needed it	11
Patient did not get care as soon as soon as the patient needed it	9
Any responsiveness problem	13
Verification
Lack of up-to-date information about the patient	4
Someone did not correctly identify the patient	3
Any verification problem	7
Information shared with…
Healthcare practitioner	23
Family member or friend	17
Healthcare administrator or manager	15
Lawyer	5
Someone else	4

*Totals may exceed 100% because multiple responses were allowed.

The most commonly identified contributing factors included problems with communication, care coordination, access and staff responsiveness. Reporters complained that staff did not listen to the patient, ignored what the patient said, spent too little time with the patient or worked together poorly as a team.

Physician classification of Hotline reports

Physician reviewers classified the initial report before a clarification call was performed and again after Hotline staff elicited additional information. In no case did the clarification call change the initial classification.

As shown in Table 4, reviewers classified 23 of the 37 reported events as patient safety incidents that reached the patient. Reviewers judged that five events resulted in moderate harm. Examples included a uterine inversion; bruising, pain and bone fracture; and an allergic reaction to a medication. In total, 12 incidents involved mild harm, and six incidents reached the patient but resulted in no harm. A single incident was classified as a close call that did not reach the patient.

Table 4

Physician reviewer classification of Patient Safety Hotline reports (N = 37)

Report classification	Number of reports	Duration of harm
Incident
No harm	6
Mild harm	12	<1 year: 8
		≥1 year: 0
		Unknown: 4
Moderate harm	5	<1 year: 3
		≥1 year: 1
		Unknown: 1
Severe harm	0
Death	0
Near miss	1
Unsafe condition	8
Service complaint	4
Unclassified	1
Total	37

Eight additional incidents were classified as unsafe conditions that increased the probability of a patient safety event. Examples included a patient who was disconnected from a sequential compression device while recovering from orthopedic surgery and a clinician who failed to disinfect their hands. Four of the remaining reports were classified as service complaints such as concerns about food, parking and long wait times.

Healthcare organization use of reported events

When consumers consented to share a Hotline report with a named healthcare organization, the research team followed up with the organization within 45 days to discover how the organization had used the Hotline report. The health systems identified only 6 of 34 Hotline-reported events in their internal databases. The six ‘matched’ events provided information that augmented the existing information. In addition, the organizations were able to identify retrospectively and address each of the remaining consumer-reported incidents with a high degree of confidence.

The Hotline reports generated activity within each organization, including department-level quality reviews (n = 21), follow-up contact with the patient (n = 16) and clinician (n = 5), or other institutional reviews (n = 5). One report led to a staff education intervention and plans to perform simulation training and develop a rapid response protocol.

Discussion

We describe the deployment and evaluation of a web- and telephone-based prototype consumer reporting system for patient safety events at two health systems located in a northeastern US community. Providers had expressed concerns about the feasibility, usability and healthcare organization's ability to manage a large volume of reports, but there were relatively few reports during the first year of operation. The reports received by the system yielded new and actionable events that patients or their caregivers perceived to be errors and physically and emotionally injurious, or both. These reports augmented existing staff-oriented reporting systems. Those who reported events were generally willing to share their identifying information with the Hotline and with the healthcare organizations where the events occurred. This project confirmed prior research that demonstrated the capacity of consumers to identify patient safety events that were unknown to healthcare organizations, and provided valuable lessons and critical research opportunities for the development of future consumer reporting systems.

The research team developed questions that were easily understandable to potential reporters with limited health literacy, including concepts such as medical mistakes and ‘negative effects’. Free text and open-ended statements provided key information about the nature of the event and extent of harm [17, 18]. Follow-up structured response elements or follow-up calls by the research team provided little additional information beyond the narrative, suggesting that reporters generally present the most relevant information in narrative form when given that opportunity.

Our design may offer insights about the willingness of consumers to report to an independent organization (in this case RAND/ECRI) co-branded by the partner healthcare organizations. Consumers who fear that providers and healthcare organizations will retaliate against individuals who submit critical reports may prefer reporting to an independent entity. On the other hand, consumers with a close relationship to their providers and healthcare organizations may feel disloyal reporting to outsiders. It is also likely that the outreach and education effort to patients, caregivers and providers may have been insufficient to generate significant public awareness, even with intensified outreach as the project progressed. Outreach and education activities would require significant and sustained attention in future consumer reporting projects, ideally informed by early engagement of consumers in both planning and evaluation.

In contrast to the Hotline model we evaluated, research studies actively solicit reports from patients and caregivers. Research studies demonstrate high rates of patient-reported adverse events and near misses across a variety of inpatient and ambulatory care settings, with adverse event rates of 8–23% [10, 19–27]. Rather than utilizing the voluntary ‘opt-in’ approach taken by the Hotline, prior researchers have generally elicited reports from specific individuals, using standard survey methods, typically after an episode of care. Reporting thus may be most likely when specifically solicited, or when outreach efforts are tied temporally to a clinical episode or a specific trigger (e.g. a new medication or procedure).

Ultimately, the success of a consumer-oriented patient safety reporting system depends on when and how queries are made, and whether potential reporters have the motivation and opportunity to say something about their care. This observation suggests that there may be a narrow window for eliciting reports from individuals whose care is episodic. It may be felicitous to incorporate patient safety reporting in the context of urgent-care follow-up calls or post-discharge patient satisfaction surveys. Patients with chronic illnesses may be receptive to regular queries as part of a series of office encounters or via a patient Internet portal. In contrast, caregivers of patients who require an extended period of care (e.g. premature infants in the neonatal intensive care unit or patients undergoing a course of chemotherapy) may be available and receptive to in-person interviews and may provide real time feedback to managers about remediable interventions to enhance care delivery. The use of multiple modalities for reaching potential reporters calls for examination of various means of using in-person, telephone, paper and electronic modalities, including setting up clinic kiosks and using mobile applications on tablets or smart phones, especially at care transitions [28, 29].

While the project yielded fewer reports than it had the capacity to manage, the Hotline itself demonstrated that web- and telephone-based consumer-oriented patient safety reporting is feasible, yielding new and actionable information for healthcare organizations and stakeholders. Further research is needed to understand why visitors to the Hotline website failed to submit more completed reports. We suspect that the Hotline's multi-layer reporting interface—designed to address both privacy concerns and research requirements—may have fatigued or alarmed potential reporters. It is also possible that some visitors to the website were correctly directed to report a service complaint, for example, to a more appropriate venue. An active outreach approach to potential reporters is an important technique that has served researchers well and might help to attain the promise of this sort of approach. The research literature suggests that there is much to be learned from patients and caregivers about the failures of our healthcare systems and how to make them better [30], if we can figure out how and when to ask the questions.

Acknowledgments

The findings and conclusions expressed are those of the authors and do not necessarily represent the views of AHRQ or the U.S. Department of Health and Human Services.

Funding

The research reported in this article was funded under contract #HHSA2902010000171 from the Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services.