- Split View
-
Views
-
Cite
Cite
Jayne L Wheway, Gyuchan Thomas Jun, Adopting systems models for multiple incident analysis: utility and usability, International Journal for Quality in Health Care, Volume 33, Issue 4, 2021, mzab135, https://doi.org/10.1093/intqhc/mzab135
- Share Icon Share
Abstract
This study aims to present two system models widely used in Human Factors and Ergonomics (HF/E) and evaluate whether the models are adoptable to England’s national patient safety team in improving the exploration and understanding of multiple incident reports of an active patient safety issue and the development of the remedial actions for a potential National Patient Safety Alert. The existing process of examining multiple incidents is based on inductive thematic analysis and forming the remedial actions is based on barrier analysis of intelligence on potential solutions. However, no formal systems models evaluated in this study have been used.
AcciMap and Systems Engineering Initiative for Patient Safety (SEIPS) were selected, applied and evaluated to the analysis of two different sets of patient safety incidents: (i) incidents concerning ingestion of superabsorbent polymer granules and (ii) incidents concerning the interruption in use of High Nasal Flow Oxygen. The first set was analysed by the first author and the utility and usability were reflected. The second set was analysed collectively by a purposeful sample of patient safety team members, who create the National Patient Safety Alerts from incident-level data and information. All of them attended a 30-min video-based training and a 1.5 h case-based online workshop. Post-workshop individual interviews were conducted to evaluate their perceived utility and usability of each model.
The patient safety team showed overwhelming support for the utility of the system models as a ‘framework’ that provides a systematic, structured way of looking at an issue and examining the causes, whilst also sharing concerns regarding their usability. AcciMap was viewed useful particularly in providing a visual comprehensive overview of the issue but considered chaotic by some participants due to many arrows between factors. SEIPS was perceived easier to understand due to the familiarity of the structure (Donbedian’s model), but the non-hierarchical format of SEIPS was considered less useful.
The participants of the study agreed with the high level of utility of both models for their unique strengths, but shared some concern for the usability of them in terms of complexity and further training/coaching time would be required to adopt these models in their daily practices. It is recommended that the gap between HF/E practitioners and patient safety practitioners can be narrowed by strengthening education, and coaching and mentoring relationships between the two groups, led by the increasing number of healthcare practitioners who embrace their membership to HF/E practice.
Introduction
The National Reporting and Learning System (NRLS) collects patient safety incident data that could have or did lead to patient harm, reported by healthcare staff in National Health Service (NHS) organizations in England and Wales [1]. The national patient safety team review and analyse these multiple incidents to develop risk reduction and prevention strategies [2, 3]. National Patient Safety Alerts are issued to the NHS when specific actions to reduce risk of death or disability can be taken by healthcare providers [4, 5].
The process of examining multiple incidents is conducted via ‘iterative theme-based analyses of the narratives’ of incident reports [2], by a team of clinical, subject matter and human factor experts working within the team. The process of conducting the analysis of incident reports is based on grounded theory or thematic analysis (depending on the specificity of the issue under exploration), and forming the remedial actions is based on barrier analysis of the intelligence on potential solutions, with tailoring to error types identified within the incident analysis [6, 7]. However, no formal systems modelling tools are used.
Various generic system models and approaches, by way of systemic accident analysis, have been developed in the fields of Human Factor and Ergonomics (HF/E) and used to analyse accidents in many safety-critical industries [8]. The review of the literature found a lack of applications of the system models to healthcare incident analysis, although a system approach has long been advocated within the patient safety team and more recently sanctioned by HF/E and NHS leaders for use in healthcare [9–11]. However, most healthcare examples in the literature were applied by researchers rather than healthcare practitioners, which could have contributed to the slower uptake of these approaches in healthcare [8].
AcciMap is one of the widely used general systemic accident analysis methods whose utility and benefits have been proven through frequent application in a broad range of fields [12]. It represents multiple contributing factors and their relationships onto the levels of the socio-technical system. The diagram also depicts the context within which an accident occurred and the interactions that resulted in the event [13]. Systems Engineering Initiative for Patient Safety (SEIPS) is one of the most widely used healthcare HF/E system models. The SEIPS model is based on Donbedian’s structure-process-outcome model of healthcare quality and the feedback loop concept of systems theory [14]. The general composition of this model is that the sociotechnical work system consisting of five interacting components (person(s), tasks, tools and technologies, organization and environment) produces work processes, which shape outcomes.
This study evaluated the utility and usability of these two system models (AcciMap and SEIPS) and their applications to multiple healthcare incident analysis at the national level. Case-based analyses were used to test the examination of an active patient safety issue for a greater exploration and understanding of the system to create actions for a potential National Patient Safety Alert to improve safety across the NHS.
Methods
Setting
The use of case studies by HF/E researchers is an established method and also enabled analysis of the models by the authors of this paper with HF/E background and significant experience in both AcciMap and SEIPS [15–19]. A second case study was used for workshops conducted with the study participants, the results of which were collected via interview following the workshops.
The first case, which was the subject of a National Patient Safety Alert in 2019, focused on 27 incidents where superabsorbent polymer granules were ingested [20, 21]. This case was selected for system analysis conducted by the first author of this study. The incidents, including three fatalities, were collected from a free text search of healthcare incident reports to the NRLS described within the 2019 National Patient Safety Alert [20], and within an earlier 2017 Warning Alert [21]. The authors carried out system analysis of this case.
The second case was incidents concerning the interruption in use of High Nasal Flow Oxygen, collected from a free text search of 17 healthcare incident reports to the NRLS described within a National Patient Safety Alert issued in 2020 [22]. System analysis was conducted for this case using both system models by the 10 participants, working collectively in two groups with five people in each.
Participants
The participants were selected as a purposeful sample, hand-picked to represent the population of relevance in the study [23]. The national patient safety team contains a small team of 12 patient safety experts with clinical backgrounds (including in critical care, theatre, emergency care/trauma, medical equipment, older people, medical specialties, pharmacy and midwifery) who lead the review and analysis of patient safety issues and development of National Patient Safety Alerts. Participation from all team members was invited and 10 of the 12 members of the clinical patient safety team took part. Of the two not taking part, one had to drop out due to an urgent assignment just prior to the workshops and the other is the first author. None of the participants had previous knowledge of system models and had served in the team between 8 months and 12 years.
Procedure
Analysis of case one
The analyses involved the construction of the AcciMap diagram and the completion of the SEIPS model by the authors and validation by the participants who were all familiar with the case (ingestion of superabsorbent polymer gel granules). The process involves clinical review of the incident reports and other intelligence, such as discussion with subject-matter experts, organizational and national policies, to iteratively examine the system within and around the incidents according to the system model utilized. The process for the creation of an AcciMap develops from the outcomes, upwards and outwards across the sociotechnical system and hierarchy (such as individual, team, management and organizational behaviour) to explore and understand the causes and contributing factors. SEIPS builds up the elements of the work system (tasks, tools and technologies, organization and environment) of the incidents around the people involved, which produces the process and results in the outcomes of the incidents. The outcomes of the analysis were reported.
Analysis of case two
Due to the participants’ lack of experience with the two system models a 30-min PowerPoint with video narration created by the first author was made accessible to the participants outlining systems theory and describing the concepts of the system models with case examples. This pre-workshop video enabled a flexible viewing of this introductory tutorial up to 2 weeks prior to the next step. A 1.5-h case-study-based online workshop was then conducted with the participants, followed by individual interviews.
The workshop provided further training on the models and presented the SEIPS and AcciMap-based system analysis of the first case as an illustration for discussion and learning. It then set to facilitate the creation of an AcciMap and SEIPS model of the second case (interruption in use of High Nasal Flow Oxygen) by the participants. Relevant data on this case collated from the incident reports were presented to each group for the formation of both SEIPS and AcciMap models. This in-depth case-based application of the models by patient safety experts provided a ‘real world’ opportunity to experience the usability and utility of the system models.
The analysis was conducted separately in two small groups (five people in each group) and facilitated by the first author. The structure of both workshops was same except the order of the application of the two models; one workshop applied the SEIPS model first and the other looked at AcciMap first to provide a more balanced test and reduce any bias on the order of models presented.
Interviews
Participants were given an aid memoir consisting of open questions around the utility and usability of the models, to capture immediate thoughts following the workshop and prior to the interviews. Semi-structured individual interviews were conducted online within 10 days of the workshop by the first author (the interview schedule can be seen in supplementary material). The interviews were transcribed and coded (selecting significant sections from participant statements) by the first author using NVivo analysis software to iteratively develop the themes further and analyse by a process of continual reflection and search for meaning within the themes to explore the research question [24]. The process of coding, and the derivation and identification of themes to develop understanding and analysis were corroborated with the second author.
Findings
System analysis of case one by a HF/E expert
AcciMap
Figure 1 shows the aggregated AcciMap of the ingestion incidents of superabsorbent polymer granules. The gels are used to rapidly solidify unwanted/hazardous liquid such as vomit and urine to reduce falls and infection risk, but there is a risk of suffocation and death if ingested. The construction of the AcciMap using multiple incident reports involved the review of the evidence within each report and placing this within the hierarchical aspects of the AcciMap. Once in place, the causal links were made. So whilst the reports did not necessarily contain all the information for a detailed investigation for each single incident, the collective data were used to explore and understand the multiple incidents and causes representing the current risks and consequences in the system and to develop remedial actions.
The aggregated AcciMap shows immediate multiple patient factors for ingesting the polymer gels—confusion with sugar or salt, confusion due to mental health difficulties, etc. The AcciMap helped to connect these factors with other conditions raised in other reports such as the availability of the polymer gel granules loose in tubs and sachets in clinical areas, the similarity to drinks thickener and the risk had been neither identified, nor documented for patients with dementia who may present a greater risk of accidental ingestion. Further causal or contributory factors for these elements could than be described as they were discovered, and placed within the Accimap, higher up the hierarchy. The identification of issues such as the drive for stronger infection prevention and control measures regarding spillages and the reduction in patient falls in in-patient settings are significant connections to be made.
This method created a clear and visual representation of the interconnections and links for further analysis. The map also provided a focal point for team discussion about what action might be taken and at which level within the system to create the strongest barrier for harm reduction. Further review then enabled the addition of a colour-coded theming around the sphere of influence and action, such as those that might be covered by a potential National Patient Safety Alert or in other ways, perhaps around procurement or influencing national policies.
SEIPS
Figure 2 shows the application of the SEIPS model of the ingestion incidents of superabsorbent polymer granules. Initially, the most significant aspects of the incidents were identified and populated to the work-process-outcomes model. The elements of each of these components were then described in detail in the second section of the model to explore and create a holistic understanding of the issues within the system.
This model was able to pick out the detail of the reports and identified themes which can guide a plan for action. In this example, two main themes brought out were those of direct patient care (colour-coded in blue) and organizational decisions (colour-coded in yellow). The relevant issues identified include lack of nursing staff’s awareness, lack of risk assessment, lack of clinical leadership involvement in purchasing and lack of education and training.
System analysis of case two by the participants
The 10 participants were asked to consider and validate the completed case analysis of case one by the HF/E expert and to collectively create both an AcciMap and a SEIPS model from a blank template, for case two. Both patient safety issues were known to them.
Interviews
The findings from the participants are taken from the individual interviews following the participatory workshops and qualitatively analysed, as described.
There was overwhelming support for the utility (or usefulness) of both system models for the patient safety team as a ‘framework’—a systematic, structured way of looking at an issue and examining the causes. Their comments on the general utility of both models include capturing the brainstorming or thinking, enabling thematic analysis and prioritization come to the fore and ‘adding more disciplined thinking’.
Half of the participants commented that the models would take time to construct and may need further training or coaching but all of them said the investment ‘might’ or would be of benefit.
AcciMap
Six participants across both groups had a clear preference for this model. The utility of the model was described as a useful ‘overview’ or ‘comprehensive road map’. The model was considered useful by enabling representation of the hierarchy in which the issue sits and enabling the framing of it to see what information is missing, what aspect needs attention and where actions need to be targeted.
Two participants thought it would provide a tool for brainstorming and there were several comments describing it as a model to show ‘the workings’, expand the scoping or enable a ‘filter down into manageable chunks’.
However, although the model was described many times as being visual and clear, the usability was a concern for half of the participants. This was described in terms of viewing and understanding the arrows connecting the elements of the issue, depicting their relationships and associations, although there was also acknowledgement that understanding the interactions between the elements of the system is at the heart of systems analysis. This confusion regarding the visual links included those who preferred AcciMap as a model.
SEIPS
Four participants preferred SEIPS because they felt more familiar with its clinically focussed construction (structure, process and outcome) and liked the model’s emphasis on being patient pathway-based and patient-centred. The participants thought that SEIPS, in parallel to the views on the AcciMap, could help with recording the contributory factors in an objective way and providing an audit trail or assurance framework for the analysis.
Three participants held the view of the SEIPS model as ‘straight-forward’, ‘brief’ and ‘not as busy’, with another three recognizing that this model enabled them to add details, annotate the issue, show a snapshot and map the issue in a logical order.
The detail required in the structure, however, was an area of concern. Several participants wondered how it might fit with the current development of alerts, usually aimed at an organizational level, and whether completing the various sections might require duplicate information and be too complex and time consuming.
Comparisons between AcciMap and SEIPS
Comparisons between the two models do not indicate that one model was better in terms of preference, utility or usability for the participants. The views of each system models’ utility and usability had consistent threads for many participants across several themes and are summarized in Table 1.
System models . | Participants’ views (perceived utility and usability) . |
---|---|
AcciMap |
|
SEIPS |
|
System models . | Participants’ views (perceived utility and usability) . |
---|---|
AcciMap |
|
SEIPS |
|
System models . | Participants’ views (perceived utility and usability) . |
---|---|
AcciMap |
|
SEIPS |
|
System models . | Participants’ views (perceived utility and usability) . |
---|---|
AcciMap |
|
SEIPS |
|
Six out of the ten participants preferred AcciMap, although two of them also liked SEIPS almost equally. Of the four who expressed a preference for SEIPS, three participants suggested one of the models could be used to brainstorm the issue and then the results placed into the other model. Two suggested that AcciMap is used for the initial brainstorming before using SEIPS as an opportunity to draw out themes and develop the alert; another thought SEIPS might be used as a brainstorming tool and the data ‘transferred’ on AcciMap to form a ‘simplified visual’.
There is no correlation between the participants’ preferences (one model over another, equally both, or combined application) in connection to which workshop they attended. There is also no correlation between the various views and the participants’ professional background or time spent in the patient safety team. Several mentioned being a ‘detailed person’ and favouring SEIPS or being a ‘visual learner’ or ‘graphic type’ person and so finding AcciMap ‘more appealing’.
The participant’s role in the national team may have influenced their views of the models’ utility and usability. Four out of the five who were keen to adopt at least one model had played a key role in the most recently issued National Patient Safety Alerts.
Of these participants there were slightly more (three of the five) preferring AcciMap to those preferring SEIPS. There was a leaning towards the SEIPS model among those currently working in frontline clinical roles, or those most recently left. The use of a familiar healthcare model, although flexibly adding or deleting structural components and connecting work system structure with a complex care pathway, has been acknowledged as valuable for those using SEIPS [14, 25].
Discussion
Statement of principal findings
The need for a systems approach in understanding and learning from errors in healthcare has been recognized as essential and ‘underpins the NHS Patient Safety Strategy’ [9]. Therefore the results of the evaluation is valuable research; the testing and synthesis of two systems models with the national patient safety team responsible for exploring and understanding multiple incident reports for national learning and action. Few examples of systems models in healthcare were found in the literature, and of those most are led by HF/E researchers and academics rather than healthcare practitioners [8].
This research–practice gap suggests that the methods may not meet the requirements of practice in healthcare or are inaccessible to practitioners [8]. There are some examples of successful pieces of work applied with healthcare practitioners, although they were shown to require extensive human factors expert’s time and facilitation skills [26, 27]. This aligns with concerns expressed by the participants in this study that system model applications in healthcare practice require further education and training, coaching and practice; multiple team members are undertaking Masters’ courses in human factors which provide an opportunity for shared learning with HF/E practitioners rather than a traditional helicopter in-and-out specialist approach.
Interpretation within the context of the wider literature
The review of literature also highlighted a dearth of evidence on multiple incident investigation for any system model, including those designed for healthcare [17, 28]. This research therefore takes forward new knowledge into uncharted waters. The national patient safety team has always had the systems approach as core but had no formal systems models for the point of combining the exploration of the multiple incidents and creation of remedial actions. This study suggests that those with more recent hands-on experience of multiple incident analysis were more willing to use one of these models. Previous research has found that there was a difference in requirements in choosing a system model between investigators and researchers in conducting a ‘live’ accident analysis [19]. This adds strength to using these models for multiple incident analysis conducted by experienced patient safety practitioners.
In this study, many participants found the visual representation of AcciMap useful in developing an initial overview of the issues and decisions and providing a visual record to be kept and revisited. However, the ‘chaotic’ arrows may require a rethink for practical application in this team [24].
The participants’ familiarity with the SEIPS structure made it to be perceived very usable, but the structure may be missing the hierarchical points required to be addressed in the creation of National Patient Safety Alerts which frequently aim actions at the organizational level.
The comments on the utility of both models extended beyond the literature. Many of the participants felt the models could provide a record of decisions made, showing the ‘workings’, a reminder of what was done, an audit trail or assurance model.
Strengths and limitations
A limitation of the study is that the role of the main researcher, a senior team member by experience, may have influenced the research outcomes. However, using accountable and transparent methods, being open with participants about any views around the topic, reflecting on roles in the findings and critically evaluating the interpretation with literature enables the study to be seen with ‘relevance and confidence’ [29].
There were strengths in checking and challenging the data with the participants, who were familiar with the incident reports and with the inductive coding (selecting significant sections from interview data) and identification of themes corroborated by the second author. Clear interview transcriptions and notes from the workshops including aid memoir notes from participants and the HF/E evaluation enabled triangulation for further depth and breadth of the participants’ expressed perspectives and interpretations. The use of the NVivo software package facilitated storage, searching and coding of qualitative data, and the use of two case studies, two system models and two methods of analysis (HF/E expert and patient safety expert practitioners) ensured a multi-faceted examination of the models.
The study was limited to the national team’s work on learning from incidents. However, this is a role to support frontline healthcare for patients’ safe care and treatment in the NHS across England and thus may be applicable for patient safety specialists and leads for incident investigations across the sector.
Implications for policy, practice and research
The focus of this study is timely; the national drive for patient safety in embedding ‘systems thinking, human factors and just culture principles’ [9] and the HF/E community’s ambitions for healthcare in ‘embedding’ the systems approach as a ‘fundamental human factors concept’ [10] and as an approach in learning from adverse events [11] is happening now. This study contributes to these desires by finding out if systems models might be useful and usable in supporting this ambition for patient safety. It also shows that HF/E practitioners and researchers are key to embedding systems thinking into healthcare, both theoretically and practically, and the use of system models may thus support both professional groups.
One of the questions raised by this study is how to use both the AcciMap and SEIPS together in an integrated/combined model and make use of the overlay of the strengths of both models, and for other model components to be considered, such as variable performance and an error type taxonomy. A further study could develop and evaluate this integrated/combined model and explore how to train the team to become familiar with both and choose which model may suit a particular issue when necessary.
Future applications in practice may be to test AcciMap and SEIPS in local NHS organizations for multiple incidents issues, such as specific medication errors or pressure ulcers. There is also increasing experience in the Safety II approach, or learning from successful events [30], and the place of systems models should be explored within this field.
Conclusions
This study set out to evaluate whether two system models are easy to use and useful to the national patient safety team in improving the scrutiny of an active patient safety issue and related evidence for a greater exploration and understanding of the system in order to create the remedial actions for a potential National Patient Safety Alert. There was overwhelming support for the utility (or usefulness) of the system models for the patient safety team as a ‘framework’—a systematic, structured way of looking at an issue and examining the causes. The participants were divided based on their preferred model, recognizing the different qualities each of the models could bring in their utility, whilst also bringing concerns regarding their usability.
It is recommended that the gap between HF/E practitioners and healthcare practitioners can be narrowed by strengthening education and coaching and mentoring relationships between the two groups, led by the increasing number of healthcare practitioners who embrace their membership to HF/E practice.
Supplementary material
Supplementary material is available at International Journal for Quality in Health Care online.
Acknowledgements
We acknowledge with thanks the participants of the study from the Patient Safety Team of NHS England and NHS Improvement. We would also like to acknowledge the support of Dr Frances Healey.
Funding
There are no sources of funding to declare.
Contributorship
The study was conceived by J.W. with input from G.T.J. It was carried out and analysed by J.W., with inductive coding support from G.T.J. The manuscript was drafted by J.W. with critical revisions by G.T.J. Both authors have agreed to the final version.
Ethics and other permissions
The research was carried out with permission from the Design School, Loughborough University. No separate ethics committee approval was necessary.
Data availability statement
Details of the third-party data analysed in the article are in the public domain via the references provided [20–22]. The figures provide the qualitative data findings from the case studies generated in the course of the study. Other qualitative findings are drawn from the interviews described in the Methods section.