The adaptation of the 'Healthcare Complaints Analysis Tool' for general practice.

BACKGROUND
Patient complaints about care in general practice are underutilized as a source of safety improvement data.


OBJECTIVE
This study aimed to adapt a secondary care complaints analysis tool for use in general practice contexts and assess the validity, reliability and usability of the adapted tool.


METHODS
The study was conducted in two phases. Phase A: The Healthcare Complaints Analysis Tool (HCAT) designed for use in secondary care was adapted for use in general practice using an iterative six-stage process. Phase B: Participants from key stakeholder groups [General practitioners (n = 5), complaints managers (n = 9), health service researchers (n = 4)]. Participants completed an online survey and analysed 20 fictionalized patient complaints using the adapted tool. Inter-rater reliability and agreement with a referent standard were analysed using Gwet's AC1 statistic.


RESULTS
Phase A: The HCAT was adapted to the Healthcare Complaints Analysis Tool (General Practice) [HCAT(GP)]. The HCAT(GP) tool consists of three domains (clinical, management and relationship problems), and seven categories. The HCAT(GP) had both content and face validity. Phase B: Inter-rater reliability was substantial for the HCAT(GP) categories (Gwet's AC1 = 0.65). Within-group agreement on the seven HCAT(GP) categories was substantial to perfect (AC1 0.61-0.85). Participants had substantial to perfect agreement with the referent standard across the survey with a mean AC1 of 0.899 (Range 0.76-0.97).


CONCLUSIONS
This study reports the adaptation of the HCAT(GP) and has established that the tool has sufficient validity, reliability and usability. This adapted tool can be applied to general practice complaints to identify areas for improvement.


Introduction
Health care complaints made by patients are an important source of data to inform quality and safety improvement (1). Complaints offer privileged insights into the patient journey, and help identify issues in safety and quality of care, that may not be visible to health care workers (2). Analysis of complaints can therefore identify issues in care that otherwise might have been missed (3).
While research on complaints is advancing in secondary care (4)(5)(6), there is a lack of research and understanding pertaining to the benefits of analysing complaints in general practice (7). The Healthcare Complaints Analysis Tool (HCAT) (8), developed for the analysis of hospital complaints, has been vital in efforts to tap into the potential of patient complaints as a data source. The HCAT allows researchers and practitioners alike to categorize complaints based on their content, severity and harm caused to patients (8). This reliable and valid tool has been used to identify hotspots (an area of high risk within care) and blind-spots (an area of care that is either unobservable or incorrectly observed) in hospital care, and suggest Family Practice, 2021, 1-6 doi:10.1093/fampra/cmab040 areas for safety improvement (3). While the HCAT has been applied to out-of-hours general practice complaints (9), the tool was specifically designed for use in secondary care. Therefore, it is unclear whether the HCAT is reliable and appropriate for use in general practice contexts. Some aspects of the HCAT in particular may not be appropriate for use in analysing complaints about general practice as they refer to parts of the patient journey exclusively present in secondary care (e.g. discharge, admissions, care on ward) (8). This paper aims to explore this across two phases by: (i) adapting and testing the validity of a version of HCAT specific to general practice; and (ii) examining the reliability and usability of this adapted tool.

Methods and Results
Design This is a cross-sectional study, with two phases: (i) the adaptation and validity assessment of a version of the HCAT designed to classify complaints about general practice, and (ii) an online survey of key stakeholders to evaluate the reliability and usability of the adapted tool.

Phase A Method
The adaption of HCAT for general practice was carried out in six stages.

Stage 1: Reviewing the original HCAT
The starting framework for this study was the original HCAT (8). The HCAT consists of several parts, which are applied sequentially to a complaint: • Firstly, the content of the complaint is classified using one of the seven HCAT problem categories (quality, safety, environment, institutional processes, listening, communication and respect and patient rights). • Next, the severity of the complaint is identified, ranging from 1 (Low) to 3 (High). • Third, the stage of care at which an issue occurred is recorded (admissions, examination and diagnosis, care on the ward, operation and procedures, or discharge and transfer). • Finally the overall harm to the patient reported in the complaint is identified from 1 (minimal harm) to 5 (catastrophic harm).
In order to assess content validity (10) of the HCAT in general practice contexts, the seven HCAT problem categories were used to synthesize data in a systematic review of 21 studies on patient complaints about general practice (7). The categories were found to be appropriate for capturing all of the issues emerging from the complaints in the included studies, which gave initial indications that the HCAT could be used to categorize complaints made about general practice (7). As a result of this, researchers decided to focus on the HCAT for classifying general practice complaints.

Stage 2: Testing HCAT categories
The HCAT problem categories were used to analyse a sample of 30 real general practice complaints by the authors (two experienced PhD level patient safety researchers, and a doctoral student undertaking a PhD in patient complaints). The HCAT categories were found to suitably classify all issues occurring in this sample of general practice complaints, thus providing evidence of face validity.

Stage 3: Testing HCAT Severity classification
The HCAT severity classification was applied by the authors to the same sample of complaints as in stage two. It proved difficult to apply the HCAT examples for ascertaining severity of an issue to general practice complaints. Following discussion, the hospital-specific severity examples were replaced with general practice-appropriate examples. The revised severity scales were applied to a different sample of 30 complaints by the authors and found to appropriately capture complaint severity in general practice.

Stage 4: Testing HCAT Stages of care
The sample of complaints used in stage two was analysed in terms of stages of care by the authors. It was found that the HCAT stages of care required adaptation for general practice. The authors decided to align the stages of care to the patient journey in general practice: accessing care, while in the practice, during the consultation, referral/ follow-up and other/unspecified. These categories were tested these on the second sample of complaints used in stage three.

Stage 5: Testing HCAT harm ratings
The definitions used for the harm ratings in the original HCAT were not entirely clear when applied to general practice. Therefore, following a review of the original risk matrix used to develop the HCAT harm ratings (11), the wording of the harm ratings was changed slightly to reflect general practice settings. The revised harm levels were then applied by the authors to the second sample of complaints used in stage three and found to be appropriate.

Stage 6: Expert consultation
Following the minor changes made to the HCAT, the adapted version of the tool was designated the Healthcare Complaints Analysis Tool (General Practice) [HCAT(GP)], and shown to three members of the Irish National Complaints Governance and Learning Team. These experts work directly with health care complaints daily, and have oversight on the complaints process within the Irish health service. They were satisfied that the adapted tool would be suitable for use with general practice complaints. This expert review also indicated that the tool had content validity (10).

Key Messages
• Patient complaints have the potential for use in safety and quality improvement. • There is a need for a reliable tool to analyse complaints about general practice.
• The adaptation of the Healthcare Complaints Analysis Tool (HCAT) is reliable.
• The adapted HCAT(GP) has the potential for quality improvement in general practice.
• Future research should apply the HCAT(GP) to real general practice complaints.

Phase A results
A summarised version of HCAT(GP) domains and categories, with sample severity ratings, is provided in Figure 1. The complete HCAT(GP), including stages of care, severity ratings and harm, is shown in Supplementary Data 1.

Phase B: reliability and usability testing
Phase B method

Participants
Eligible participants were General Practitioners (GPs), health care policymakers/complaints managers and health service researchers in Ireland. Participants were recruited using purposive sampling (12), between September and December 2020, with one author (EOD) emailing stakeholders directly. Efforts were made to ensure stakeholder groups were proportionally represented.

Development of survey
An online survey was developed (see Supplementary Data 2). This survey included 20 fictionalized complaints, which were created by the researchers based on real general practice complaints. Real complaints were not used due to ethical concerns. Researchers ensured that issues within the complaints spanned the range of the HCAT(GP) problem categories, severity ratings, stages of care and harm. Of the complaints, 14 contained only one issue, and six contained multiple issues (n issues per complaint: 2-4). For each of the complaints, researchers set a 'correct' response based on their experience of using the HCAT, and set this as a referent standard against which the participants would be compared. Following the analysis of the fictionalized complaints, participants were asked to rate the tool on usability, defined as whether they found it easy to use, easy to learn, user-friendly, and whether they would recommend it to a colleague. These questions can be found in Supplementary Data 2, and were based on the literature on technology acceptance (13)(14)(15). Usability is important to ascertain, as technology is more likely to be accepted if it fulfils the above criteria, and this is true also of complaints analysis tools (13)(14)(15).

Online training
Participants were sent a link to the HCAT(GP) which can be found in Supplementary Data 2. Prior to the survey, the participants watched a 13-minute video explaining the purpose of the HCAT(GP), with worked examples of the application of the tool. The video can be found at this link https://youtu.be/vo8ckQ8Gh48.

Analysis
Responses were analysed by: (i) assessing the inter-rater reliability between the participants both within their stakeholder groups and across the sample at large; and (ii) comparing participants' responses to the referent standard. Inter-rater reliability is defined as multiple coders observing the same issues within a complaint, and this is important for complaints analysis to ensure confidence in the tool's findings (8). Gwet's AC1 was used to calculate both the inter-rater reliability and the comparison of coder responses to the referent standard (8,16). Agreement data from Gwet's AC1 were interpreted as: 0.01-0.2 = poor/slight, 0.21-0.4 = fair, 0.41-0.6 = moderate, 0.61-0.8 substantial, 0.81-1 = perfect (17). Descriptive statistics on participants' ratings of the usability of the tool were also analysed. The analysis was conducted using R statistical software (18).

Demographics
A total of 18 individuals participated in Phase B of the study -four were health service researchers, nine were experts working in complaints analysis, and five were GPs. A further 10 GPs were contacted and declined to participate due to other commitments.

Inter-rater reliability
There were no missing data for the variables of interest. Inter-rater reliability was substantial for HCAT(GP) categories across all complaints (Gwet′s AC1 = 0.65). Reliability on categories was greater for single issue complaints than multiple issue complaints (See Table 1). Agreement across different aspects of the HCAT(GP) was also calculated, and these results are presented in Table 1. Within-group agreement on HCAT(GP) categories was substantial to perfect (Gwet's AC1 0.61-0.85), indicating that the adapted tool can be used reliably by individuals across professions to classify complaints made about general practice.

Comparison to referent standard
Participants had substantial to perfect agreement with the referent standard across the survey with a mean Gwet's AC1 of 0.899 (range 0.76-0.97).

Usability
Participants indicated that they found the HCAT(GP) to be useful, user friendly, and that they would recommend it to their colleagues (see Table 2). However, one participant did not find the HCAT-GP easy to learn, and this should be considered when rolling out training to wider audiences.

Discussion
This paper described the adaptation of the HCAT for use in general practice, and established content and face validity, inter-rater reliability and usability. General practice complaints are underutilized as a source of data on patient experiences, and lacking a tool for their systematic analysis (7). The HCAT(GP) has great potential to support the use of patient complaints for safety and quality improvement. The HCAT(GP) provides a reliable means through which GPs, health researchers and complaints managers alike can analyse patient complaints about general practice. The reliability of this tool was comparable to that of the original HCAT, with participants achieving substantial to perfect scores against a referent standard (8). Across all problem categories, the overall reliability of the HCAT(GP) was found to be sufficient. The reliability testing in our study also indicated that participants with different professional backgrounds were able to successfully apply the tool to general practice complaints. GP participants and complaints managers both achieved substantial reliability when rating the complaints categories, while researchers had perfect reliability. It is particularly important that the HCAT(GP) can be used reliably by all stakeholders as general practice is a disjointed system, with lots of boundaries between different aspects of the patient journey (19)(20)(21). Table 2. Participant responses to usability questions on HCAT(GP) from survey, 2020

Prompt
Strongly disagree/ disagree n (%) Neither agree nor disagree n (%) Agree/strongly agree n (%) The HCAT(GP) was easy to use 0 (0%) 0 (0%) 14 (100%) The HCAT(GP) was useful 0 (0%) 2 (14%) 12 (86%) The HCAT(GP) does everything I would expect it to 0 (0%) 1 (7%) 13 (93%) The HCAT(GP) was user friendly 0 (0%) 0 (0%) 14 (100%) I learned to use the HCAT(GP) quickly 1 (7%) 1 (7%) 12 (86%) I would recommend the HCAT(GP) to colleagues 0 (0%) 1 (7%) 13 (93%) Participants considered the HCAT(GP) to be a useful and usable tool. This is an important aspect of this study, as acceptance by stakeholders is vital for a tool or intervention to be successfully introduced (22). This is particularly the case in a context as busy and complex as general practice, and GPs indicating that the HCAT(GP) was useful and usable bodes well for its introduction in the 'real world'. With GPs often working in silos (23), having the ability to conduct complaints analysis at a local level and feeding up to national level in a standardized manner will be of huge benefit (24). Individual care teams can learn from complaints within their own practice and others, and can apply this learning to in-house processes in order to prevent future harm to patients. Moreover, using the HCAT(GP), in conjunction with the HCAT for hospital care, health care systems can ensure that the entire patient journey is represented and captured for safety and quality improvement at both national and local levels.

Limitations
There are a number of limitations inherent in our study. First, the tool was not applied to real complaints, but to fictionalized complaints based on real-life general practice complaints. This was necessary to address ethical concerns with using real patient data for research without their consent (25). Second, the tool was adapted and tested on complaints based on one health care system, which differs in structure and funding to other countries (26). This may have limited the generalisability of the tool to international systems. Third, the HCAT(GP) was not tested for criterion or construct validity. Future work should explore construct validity by conducting a factor analysis of the HCAT(GP) (10). Finally, there were only a small number of GP participants in this study, which may limit our understanding of how acceptable and reliable this tool maybe when used by this population.

Implications for research and practice
The HCAT(GP) should be used to analyse a larger sample of real general practice complaints, in order to explore the benefits to practice of analysing general practice complaints using a reliable tool. The original HCAT has been applied in a similar manner to secondary care contexts internationally (3,(27)(28)(29), with the data from the HCAT categorizations and further analysis of these showing promise for contributing to safety improvement (3,28,29). Extending this systematic analysis of patient complaints to general practice will complement the work being done in secondary care, and capture more learning from patient complaints across the health care system. The HCAT(GP) allows researchers, practitioners and policymakers to identify issues that are occurring most frequently within general practice settings, and provide an evidence base for the implementation of quality and safety improvement measures. It can also be used across different levels of the health care system, as while there may only be a small number of complaints at the individual practice level, the tool could be used to first categorize complaints at a local level and the categorized data could be shared regionally or nationally in order to conduct higher-level analysis of complaints trends. The standardization of complaints categorization through using this tool would support this higher-level analysis. It therefore has potential both as a tool for categorization, and as a first step in the analysis of trends in health care complaints.
Future research should also explore the reliability of certain aspects of the tool, in particular the 'harm' and 'severity' ratings. The HCAT(GP) also does not include subcategories, in contrast to the original HCAT. This decision was made as the subcategories in the original HCAT were not tested for reliability (8). Future research should explore the reliability of subcategories for the HCAT, and explore the potential of subcategories for the HCAT(GP). Multiple-issue complaints were also less reliably analysed than single-issue complaints, which should be interrogated by future research as complaints often have more than one key issue within them (8). It is possible that since the participants were presented with the multiple issue complaints towards the end of the survey that fatigue played a role in the loss of reliability (30). One means of improvement could be the organisation of stakeholder consultations or think-aloud sessions, which would afford the opportunity to stakeholders in GP complaints to determine how they can be aided in utilizing the tool for safety improvement.
Finally, the HCAT(GP) was found to be mostly moderately to substantially reliable with minimal training. This is an important finding as extensive training would be a barrier to the widespread adoption and use of the tool. The survey used in Phase B of this study to train the participants could be used in a similar manner to the IT-based training of the original HCAT (8) and allow for feedback to be provided to participants on their performance. Further work could also explore any improvements to reliability or usability through the delivery of face-to-face training.

Conclusion
Patient complaints about general practice are underutilized for safety improvement. The HCAT(GP) has the potential for use as a reliable tool for analysing health care complaints about general practice. This has implications for the systematic analysis of these complaints, and could provide a means through which patient safety in general practice can be improved.

Supplementary material
Supplementary material is available at Family Practice online.

Declarations
Ethical approval: This study was granted ethical approval by the National University of Ireland Galway Research Ethics committee, reference number 20-Jan-14. Source of funding: This research was supported by the Hardiman PhD scholarship from the National University of Ireland Galway.