Review of simulation studies in anaesthesia journals, 2001–2010: mapping and content analysis

Summary

Despite widespread adoption of simulation-based training in medical education, there remains scepticism about its cost-effectiveness and long-term impact on patient outcomes. Medical simulation is well established in anaesthesia where it is considered an important educational tool. This review of key clinical anaesthesia literature is used as a case study of clinician uptake within a specialty and to investigate evidence for translational impact using both qualitative and quantitative data. We examined high-impact journal publications from 2001 to 2010 and extracted data covering authors, institutions, simulation modality, purposes of simulation, and various aspects of study design/methodology used. A total of 320 papers containing primary data were included. We found broad acceptance and uptake in anaesthesia with an increase in publications over the time period, mainly attributable to a steady increase in manikin studies. Studies using manikin technology (130/320; 41%) are distinguished as skills/performance studies (76; 58%) and studies focused on the use, testing, and validation of equipment (52; 40%). A total of 110 papers (34%) assessed the performance of technical and non-technical skills (68% and 32%, respectively). Growth in the use of structured checklists/validated tools to assess performance is mainly observed in the non-technical domain. Only 10% of these papers include follow-up data from the clinical environment. There is a lack of research examining performance transfer, sustainability, and direct patient outcomes and experiences. These publication patterns are instructive for those involved in medical educational and for other clinical specialties developing simulation.

anaesthesia, patient simulation, reviews, task performance and analysis

Editor's key points

• Simulation is increasingly used in anaesthesia training.

• The type and purpose of simulation studies published between 2001 and 2010 was assessed.

• Studies included technical and non-technical skills and equipment validation.

• While widely accepted in anaesthesia, the evidence for benefits transferred to clinical practice is still limited.

Simulation in healthcare

Simulation in healthcare can be variously defined as a technique, technology, or process.¹ For example, ‘a technique to replace or amplify real-patient experiences with guided experiences, artificially contrived, that evokes or replicates substantial aspects of the real world in a fully interactive manner’;² ‘any technology or process that re-creates a contextual background that allows a learner to experience success, mistakes, receive feedback, and gain confidence in a safe environment’.³

The use of a range of devices, including manikins, actors, virtual reality, and part-task trainers, for various aims and purposes, such as training, problem solving, and psychomotor skill rehearsal, has been described.⁴ After early development of simulation in the areas of cardiopulmonary resuscitation, cardiology, and anaesthesia,^5–9 application has been extended to areas as diverse as community nursing and psychiatry.^10,11

Evidence for effectiveness

Simulation is now gaining acceptance in principle as a viable teaching and learning method in healthcare.^1,2,12 Despite significant set up and maintenance costs, more than 1500 manikin-equipped simulation centres now exist worldwide.¹³ A number of drivers for simulation as a teaching and learning tool have been outlined, including fiscal and time constraints on physician-teachers, the fast evolution of technology, the opportunity for standardized teaching and reduced training duration, and compatibility with the patient safety aim of practising with no danger to patients.¹⁴

As stated recently, ‘It is time to put see one, do one, teach one behind us. Procedure training should involve a combination of didactics and simulation, with objective evidence of technical competency before exposing patients to the risk of procedures performed by novice operators […] we should not wait to broadly adopt this tool in our teaching and in our assessment of competency […] we have reached the tipping point’ (where simulation is fully integrated and accepted into medical education)’.¹⁵

Despite this acceptance in principle, the quality of evidence showing that patient care quality increases as a direct consequence of simulation-based learning is disputed. Recent reviews^16–19 have outlined a lack of outcomes data and limited evidence for the transfer of skills into practice. However, these papers do not provide numerical evidence, and therefore, it is hard to be specific about the published evidence on transferability and outcomes data, patterns of simulation applications, and distributions of particular methods and modalities. The importance of a more rigorous approach to simulation-based medical education and evaluation based on concepts from translational science has been stressed.²⁰

Simulation in anaesthesia

Although many other clinical, specialties, including nursing and surgery, also have a long history of simulation-based education and training, it has been reported that simulation in healthcare has its ‘roots in anaesthesia’²¹ driven by an interest in resuscitation from the 1960s.²² Early work on technical skills such as intubation⁵ has developed into work aimed at the combined performance of the operating team in a realistic clinical environment.²³

Thus, simulation in anaesthesia is arguably at a stage of development where it can inform those working in other specialties.

We were interested in the evolution of simulation in anaesthesia over the last 10 yr period and in describing clinician uptake through clinical literature and reviewing the evidence for its translation into practice. We aimed to map and describe the recent simulation literature in anaesthesia journals in order to examine the acceptance and uptake of simulation by practitioners in this particular clinical domain. We wished to track changes over time and look at interactions, purposes of simulation, study design, and modality. In addition, we reviewed the evidence for the effectiveness of simulation, particularly in skill acquisition and transference to patient outcomes. The evidence base in clinical anaesthesia will be of interest to those working in other specialties. Specific objectives were: The review was designed to describe a large body of work using rigorous definitions, data extraction, and coding. This method allowed us to quantify data and use inferential statistics where appropriate. Reviews and discussion papers on medical simulation^1,24,25 have tended to be qualitative/thematic. However, quantification has some advantages, chiefly the ability to describe large bodies of evidence and to find interactions, for example, between study origin and substantive or methodological aspects. The quantitative review has been described as an efficient way to summarize large literatures.²⁶ Our definition of simulation is broad and we have precluded formal meta-analysis,²⁷ chiefly because of heterogeneity in the purpose, modality, unit of analysis, and research design of included studies.²⁸

• to investigate patterns in simulation across anaesthesia journals, methods, authors;

• to describe types of simulation (modalities) and their applications (purposes/aims of studies) over the 10 yr period;

• to examine the evidence for the effectiveness of simulation (particularly in skill acquisition and transference to real patient outcomes).

Definitions

Drawing on discussions of the definition of simulation,^2,24 our general definition of simulation has three components: Further codes for analysis drew upon and expanded various taxonomies and review tools from the simulation literature. We used some of the dimensions and distinctions proposed by others as important to extract data from a large cohort of papers. Specifically, we have referred to the following literature.

• a device for simulating a patient or part of a patient;

• used for technical and/or non-technical skills training or validation of equipment or technique;

• interacts appropriately with actions taken by the clinician.

• Applications of simulation: We drew extensively from Gaba² who outlines 11 dimensions of simulation applications including but not limited to: purpose and aims; participants; knowledge, skills attitudes, or behaviour addressed; feedback methods applied (some are derived in part from Miller).²⁹

• Modalities: Reznick and MacRae¹² propose five types of simulation namely bench models, live animals, cadavers, human performance simulators, and virtual reality surgical simulators.^12,30 To this framework, we added simulation using actors/volunteers. Animal models are part of the simulation landscape,¹⁴ but most animal studies we accessed involved drug testing and were excluded, but papers examining techniques or anaesthetists' performance using animal models are incorporated into our study.

• Transferability: Questions of transferability such as ‘Does practice with the simulator enable better performance on the simulator task itself?’ have been outlined;³¹ and the crucial step in validating a simulator: Does practice with the simulator task lead to an increase in proficiency on the real task?³² There is a similarity between ‘level one’ translational science concerned with results achieved in the educational laboratory and ‘level two’ translational science which ‘aims to produce evidence of clinical effectiveness at the level of the patient’ and leads to ‘improved downstream patient care practices’.²⁰

Search strategy

Citation report

A bibliographic database search was conducted in May–June 2011. First, we extracted a subject-specific Journal Citation Report from ISI Web of knowledge (on May 4, 2011) (2009 JCR Science edition), requesting journals categorized under the heading anaesthesiology and sorted by the impact factor.^33,34

Within the subject of anaesthesiology, we selected the top 10 journals by the impact factor from Anaesthesiology (impact factor 5.486; 5 yr 5.264) to the Canadian Journal of Anaesthesia (impact factor 2.306; 5 yr 2.15). After initial scoping to estimate the volume of published papers, the final search syntax used was: simulation/simulate/simulated/simulator or virtual reality/VR or manikin/mannequin or haptic/bench-model/task-trainer/phantom.

The search was limited to the last full 10 yr period, January 2001 to December 2010. Thus, the final inclusion criteria were: published in top 10 anaesthesiology category journals by impact factor; published in last 10 complete years before December 2010; and use of simulation or related terms.

Exclusion criteria

• Non-relevant papers: We excluded papers where simulation was only mentioned in passing or in reference. In studies where patients were involved, inclusion required some discrete simulated activity (e.g. practice on a manikin followed by observed performance on patients). We excluded cases where the study involved patients undergoing surgery (e.g. using a rigid cervical collar to simulate restricted neck mobility).³⁵

• Review papers: As we wished to examine primary evidence and analyse the research methodologies used, we drew up a list of non-empirical papers to be excluded which covered: reviews synthesizing evidence; editorials and commentaries; correspondence; discussion papers; opinions, essays, and critiques; meeting and conference reports.

• Data simulation: Using our general definition that patients must be simulated in some way, we excluded papers where the ‘simulation’ referred to the computer and mathematical modelling, use of software for pharmacokinetics, and stochastic techniques. We define this type of simulation as primarily a substitute for repeated trials. If the specific ‘simulation’ text obtained by the search syntax referred to a patient-substitution device rather than mathematical or statistical modelling, the paper was included.

• Drug and equipment testing: Our definition required that simulation be a technique for training or performance. We thus excluded papers where the sole purpose of simulation was to test the effects of anaesthetic drugs. Where the main purpose was to test equipment (e.g. different laryngoscopes) or techniques (e.g. modified retrograde tracheal intubation), we have incorporated this as broadly performance-related and have included these papers.

Data extraction

A pro-forma was drawn up to code papers. Two researchers, working independently, used the tool to assess papers. Ten per cent were cross-coded to ensure coding reliability. Coding took place within SPSS v19.0 (IBM, New York, NY, USA) using drop-down menus with code labels attached. Codes were either: numerical (e.g. year of study, number of participating physicians), ordinal (e.g. seniority of staff involved, unit of analysis), or categorical (e.g. location, study design, statistics used, simulation device, clinical domain, purpose of simulation) (Table 1).

Table 1

Dimensions and codes used for data extraction

Dimension	Codes	Adapted from
Purpose	Education/training; technical skills/performance; clinical rehearsal; testing and validation of equipment/techniques; addressing non-technical skills/human factors/attitudes (e.g. teamwork; multi-disciplinary aspects)	Anderson and Leflore;4 Ennen and Satin;32 Gaba2 (see Fig. 1)
Type of simulation	Virtual reality/augmented reality (VR/AR); bench models; cadavers; animals; manikins; part-task trainers; actors; volunteers (incl. role play)	Ahmed and colleagues;56 Gaba;2 Goff;14 Hallikainen and colleagues;57 Reznick and MacRae12
Clinical domain	e.g. intubation; ventilation; bronchoscopy; laryngoscopy	Cooper and Taqueti;24 Gaba2
Participants	Students; physicians; nurses; AHPs; managers/legislators; clerks	Gaba2
Design/analysis	Prospective or retrospective; randomized, controlled, blinded; qualitative; descriptive; correlational; within groups; between groups; modelling	Tsuda and colleagues58
Outcomes	Performance improvement; patient outcomes	Vankipuram and colleagues;31 McGaghie and colleagues20
Unit of analysis	Individual/team/organization	Ennen and Satin;32 Gaba2
Clinical environment	Clinical environment simulated or not	Gaba;2 Hallikainen and colleagues57
Feedback and assessment	Peer/expert/researcher critique; use of debrief; use of validated checklists	Anderson and Leflore;4 Byrne and Greaves;59 Gaba;2 Tsuda and colleagues58

Results

A total of 3966 papers were retrieved via the search strategy. These were all examined via the inclusion and exclusion criteria (Fig. 1). This resulted in 320 papers being included for further analysis (8% of those retrieved). As might be predicted given journal location and language, papers from the USA (83; 26%) and the UK (60; 19%) made up nearly half (45%) of papers included. While 27 countries are represented, the only countries outside the USA/UK accounting for more than 5% of the database are Canada (42 papers; 13%), Germany (19 papers; 6%), and Austria (16 papers; 5%).

Fig 1

Papers retrieved (n=3966), excluded (n=3646), and included in mapping exercise (n=320).

Type of simulation (modality)

Some journals accounted for a larger proportion of included papers than others, and there was a variation in (i) included papers by journal and (ii) type of simulation (Table 2).

Table 2

Simulation modality by journal (n=320)

Journal (2009 JCR impact factor)	Type of simulation							Journal totals (%)
	Volunteer	Cadaver	Animal	Manikin/part-task trainer	Bench model	Virtual/augmented reality	Mix
Pain (5.37)	1	0	4	0	0	0	0	5 (2)
Anesthesiology (5.35)	0	5	18	18/1	5	1	1	49 (15)
Reg Anesth Pain Med (4.16)	0	6	5	0	6	0	1	18 (6)
Br J Anaesth (3.83)	1	2	2	18/3	12	6	2	47 (15)
Eur J Pain (3.61)	0	0	1	0	0	0	2	3 (1)
Anesth Analg (3.08)	6	6	17	48/3	16	0	11	107 (33)
Clin J Pain (3.01)	0	1	1	0	0	0	0	2 (1)
Anaesthesia (2.86)	1	2	1	40/15	9	1	4	73 (23)
J Neurosurg Anesth (2.41)	0	0	2	0	0	0	0	2 (1)
Can J Anaesth (2.31)	0	1	2	6/0	3	1	1	14 (4)
Simulation totals (%; n=319, 1 missing)	9 (3)	23 (7)	53 (17)	140/22 (48)	51 (16)	9 (3)	22 (7)	320 (100)

As expected, the distribution of papers is biased towards journals with an anaesthetic rather than analgesic focus. The mean number of papers per journal is 32 (range 2–107; sd 36). Manikin/part-task trainer studies account for 48% (152/320) of papers overall. Animal studies make up 17% (53/320) of papers overall and bench models 16% (51/320). Other types of simulation are relatively rare.

From the 3 yr moving averages in publication volume for the six coded types of simulation (excluding papers with multiple modalities; n=300), it can be seen (Fig. 2) that the volume of published simulation papers across the 10 journals has increased by 126% from a 3 yr moving average in 2001–3 of 19.33 to a 7 yr high figure of 43.67 in 2008–2010. This can mainly be attributed to a steady increase in manikin studies (n=130) over the time period studied from a 3 yr average of 8.67 in 2001–3 to a 3 yr average of 19.67 in 2008–2010 (an increase of 127%). Bench-model studies (n=51) have also increased (by 150%) to a 3 yr moving average high of 6.67 in 2008–2010. Animal models have decreased by 29% from a high around 2003–5.

Fig 2

Volume of simulation modalities from 2001–2010 (three year moving average; n=300).

Purpose of simulation

The purpose of simulation in the 320 studies was: practising with equipment (142, 44%), training or evaluating performance on technical skills (75, 23%), validation or evaluation of anaesthetic techniques (68, 21%), and higher level performance, including human factors and non-technical skills (35, 11%).

The last two 3 yr periods since 2005–7 have seen a 150% increase in papers with a non-technical/ human factors focus to a 7 yr high 3 yr average of 6.67 in 2008–2010 (Fig. 3). The largest observable increase is in the use of simulation for equipment testing/validation (up 215% from a 3 yr moving average for publication volume of 6.67 for 2001–3 to 21 for 2008–2010).

Fig 3

Volume of simulation purposes from 2001–2010 (three year moving average; n=320).

Study designs

Most papers described single site, cross-sectional studies. There were just 14 (4%) longitudinal designs and 10 (3%) multi-site studies. Around half used randomization techniques (148; 46%) and a slightly smaller number reported control groups (128, 40%). One-quarter (74; 23%) used blinding (usually investigators blinded to study design). Forty papers (13%) reported a randomized, controlled, blinded trial.

With regard to statistical testing, 260 (81%) used an inferential statistical approach with 47 (15%) using qualitative and/or descriptive analysis only. Between-groups analyses (usually using controls) were more prevalent than within-groups analysis (30% and 21%, respectively) with a further 59 papers (18%) using a mixed/crossover design.

Performance and transferability

Participants in the 110 (34%) studies with human participants (excluding equipment testing and technique validation) were mostly doctors (76; 86%) or doctors and others (17; 15%). Seventy-five papers (23%) assessed technical skills. A wide range of skills was practised or assessed, the most common being intubation, ventilation, needle guidance, and laryngoscopy. A further 35 studies (11%) focused on non-technical skills (e.g. anaesthetists' non-technical skills).³⁶

There are differences in approach when technical and non-technical skills are being addressed (Table 3). Non-technical skills studies are significantly more likely to use participant debriefing, expert critique of performance, validated checklists, manikins, actors, and scripted scenarios. Proportions of randomized, controlled designs are similar.

Table 3

Design of papers addressing technical and non-technical skills (n=110)

Design aspects	Technical skills papers (n=75)	Non-technical skills papers (n=35)	χ2 analysis (odds ratio; 95% CI)
Manikin studies (n=81)	50 papers (67%)	31 papers (89%)	χ2=5.9; P<0.05 (3.9; 1.2–12.2)
Use of written scenarios (n=73)	43 papers (57%)	30 papers (86%)	χ2=9; P<0.01 (4.5; 1.56–12.78)
Studies using actors/‘plants’ (n=32)	17 papers (23%)	15 papers (43%)	χ2=4.7; P<0.05 (2.56; 1.1–6)
Focus on combined performance/teamwork (n=14)	0 papers (0%)	14 papers (40%)	n/a (1.7; 1.3–2.2)
Use of debrief (n=38)	18 papers (24%)	20 papers (57%)	χ2=11.6; P<0.001 (4.2; 1.8–9.9)
Use of expert critique (n=48)	27 papers (36%)	21 papers (60%)	χ2=5.6; P<0.05 (2.7; 1.2–6.1)
Use of validated checklist (n=40)	21 papers (28%)	19 papers (54%)	χ2=7.1; P<0.01 (3; 1.3–7)
Randomized, controlled, blinded designs (RCTs; n=20)	13 papers (17%)	7 papers (20%)	χ2=0.1; NS (1.2; 0.43–3.3)
Reported data from real clinical environment (n=12)	8 papers (11%); see Table 4	3 papers (9%); see Table 4	n/a (1; 0.3–3.9)

There were 11 papers (10% of those addressing skills training) that contained data from clinical performance in the non-simulated environment (i.e. transfer to the clinical practice of anaesthesia). As these papers constitute the best available evidence for transferability, we were interested in (i) study design and (ii) whether or not patient outcome data were included in analysis (Table 4).

Table 4

Studies including evidence from performance in the clinical (non-simulated) environment (n=11)

Paper	Modality and domain	Design and participants	Brief description and outcomes
Naik and colleagues38	Bench model	Randomized controlled trial; blinded assessors	Compared didactic training with simulation using a simple model to train fibreoptic orotracheal intubation (FOI) skills. Subjects randomized to training conditions. Skills evaluated pre- and post-training. Performance evaluated 10 days later in the operating theatre with a global rating scale and checklist administered by blinded evaluators. Training with the model led to faster intubation and higher success rates than didactic training
	Intubation	Residents; n=24
Crabtree and colleagues40	VR	Randomized controlled trial; blinded assessors	Compared high- and low-fidelity models for training FOI skills. Subjects randomized to training conditions. Performance in clinical practice (1 week post-training) evaluated with a global rating scale and checklist administered by blinded evaluators. There was no difference between fidelity groups on the performance measures. There was no correlation between the time to complete a task during training and the clinical FOI skill
	Intubation	Respiratory therapists; n=30
Smith and colleagues41	Manikin	Case study	Report of successful resolution of a clinical emergency involving resuscitation of bupivacaine-induced cardiac arrest treated with i.v. lipid emulsion. The clinicians involved had recently received simulation training involving a scenario nearly identical to this case
	Resuscitation	Anaesthesia fellows
Berkow and colleagues43	Manikin	Retrospective review	Investigated changes in the number of emergency surgical interventions for inability to intubate and ventilate before (4 yr) and after (11 yr) the introduction of a difficult airway management programme. Emergency surgical interventions were significantly lower after the introduction of the programme; however, the programme was multi-faceted with simulation training being just one component
	Intubation	Residents
Friedman and colleagues60	High- and low-fidelity simulators	Randomized controlled trial; blinded assessors	Study compared high- and low-fidelity simulation training for epidural needle insertion. Subjects randomized to condition. Performance evaluated by blinded examination of videotaped clinical performance using a global rating scale and a manual skill checklist at three points: performance of first epidural; 31–90 epidurals; >90 epidurals. No significant differences were found between high- and low-fidelity training at any time. Low-fidelity training was as effective as high-fidelity training
	Epidural	Residents; n=24
Howes and colleagues61	Manikin	Observational study	Study evaluated the ability of novices to use a supraglottic airway device. Novices used the device with a manikin and after adequate performance was demonstrated, they used the device with 50 anaesthetized patients under supervision. Performance by novices was assessed by time, pharyngeal seal pressure, and complications and was similar to performance by experts. The conclusion was that the mask is suitable for use by airway novices
	Intubation	Doctors, students, and nurses
Marsland and colleagues37	Manikin	Observational study; blinded assessors	Study investigated acquisition of skill in manipulating fibreoptic bronchoscope during simulation training and transfer of skill to clinical practice. Ninety-six per cent of participants achieved fibreoptic bronchoscope proficiency on the manikin within 4 h of practice. Ninety-three per cent then showed proficient performance at first attempt with clinical volunteers. Performance was evaluated using a global rating scale by blinded assessors
	Bronchoscopy	Novice endoscopists; n=29
Khawaja and colleagues42	Manikin	Case study	Report of the management of a patient with a rare condition causing a difficult airway. A multi-member team rehearsed predefined roles and then managed the airway via inhaled induction of anaesthesia, followed by flexible fibreoptic intubation. The planning and decision-making that led to a successful outcome for the case are described and attributed to knowledge of anaesthesia crisis resource management methods rehearsed in a simulator. Non-technical skills included
	Intubation	Residents
Bruppacher and colleagues39	Manikin	Randomized controlled trial; blinded assessors	Compared the effectiveness of simulation training and an interactive seminar for training weaning patients from cardiopulmonary bypass. Subjects randomized to conditions. Skills were assessed at pre-test and 2 and 5 weeks post-training. Performance was evaluated by a blinded rater using Anaesthetists' Non-technical Skills checklist and a global rating scale. Pre-test scores were similar for both groups, but the simulation group scored higher at 2 and 5 weeks post-training. Non-technical skills included
	Cardiology	Post-graduates; n=20
Gale and colleagues45	Manikin	Validation study	Study examined the use of ratings of non-technical skills exhibited in simulated scenarios as selection criteria for specialist anaesthetists. Simulation was not used for training, but moderate correlations were found between selection criteria scores (including performance on simulated scenarios) and workplace performance scores after 1 yr. Non-technical skills included
	Non-technical	Specialist anaesthetists; n=224/n=68 at follow-up
Nishisaki and colleagues44	Manikin	Observational study with retrospective review	Assessed the effectiveness of just in time simulation training in tracheal intubation and general resident skill training at the beginning of each ‘on call’ period. Retrospective review was also performed comparing performance data for the previous 3 yr and the intervention period. There was no difference in success at first attempt and overall tracheal intubation success between those who had received the training and those who had not, and no difference between the study period and historical data
	Intubation	Residents and nurses

There was a variation in purpose, modality, and evidence for studies that investigated transfer of skill to the clinical environment (Table 4). It is notable that outcome data in this set of papers are still mainly performance assessment data, rather than patient outcome data. Skills training studies tended to use checklist/rating methods and some infer improved clinical outcomes from evidence that skills had transferred to the real world.^37,38

The strength of evidence for the effectiveness of simulation in training skills and improving patient outcomes is mixed. Of the 11 studies, there were only four randomized controlled trials, two of which measured pre-training skill levels and two of which did not. The best evidence may come from two studies using controlled trials and blinded assessors that suggest simulation training leads to higher clinical and non-technical skill levels than didactic methods of teaching.^38,39 However, no correlation between performance in the simulator and clinical practice was found using time as a dependent measure.⁴⁰ Other dependent measures might be more sensitive to performance differences.⁴⁰

Two case studies reported an inferred link between simulation training and successful resolution of real clinical cases. In one,⁴¹ the clinicians involved had recently practised a similar crisis in simulation training; in the other,⁴² the clinicians involved in resolving a crisis had implemented the non-technical skills taught in simulation training.

Retrospective reviews, in which performance and safety data for different time periods are compared based on when simulation training was introduced, found mixed evidence. One⁴³ found that performance did improve after an airway management programme, including simulation training was introduced. Simulation was only one part of a multi-faceted programme and it is difficult to know what contribution simulation alone made to improved performance. An observational study³⁷ found that training in a simulator led to competent clinical performance, but did not compare simulation with other training methods. However, no increase in intubation success was found after ‘just in time’ simulator training.⁴⁴ Finally, a study of the use of simulation scores as selection criteria for anaesthesia specialists found moderate correlations between simulation scores and performance during the first year, providing some evidence of the predictive validity of simulated performance data.⁴⁵ However, only those already achieving high performance on simulation were studied and moreover simulation score was only part of the selection criteria.

We used a network-model methodology using MS Node XL statistical software (Microsoft, Seattle, WA, USA) to map collaboration; 343 institutions collaborated at least once (there are 517 discrete links between co-publishing institutions) (Supplementary Fig. S1).

The network analysis shows that published research is fairly widely distributed across many relatively autonomous institutions co-publishing with a small group of partners. However, there is some variance. There are some key ‘hubs’ in the anaesthetic simulation network which have high collaborative output and may foster further collaboration by linking their various partners. The widest range of collaboration was by Harvard University, with 19 separate collaborations.

Discussion

This study has drawn on the literature using definitions that are fairly inclusive, and thus we have described a field with a wide range of modalities (type of simulation), used to address a wide range of issues (purpose of simulation), and incorporating a range of methodologies (study designs, participants). Based on quantitative and qualitative analyses, the results clearly show that simulation in anaesthesia is now widely accepted and used for a broad range of purposes. There is a substantial body of evidence describing experience with simulation, but limited evidence showing transfer of trained skills into practice or positive impact on the safety and quality of care. Although previous authors have reported a dearth of evidence for impact on clinical practice and patient outcomes,^16–19,25 this paper provides quantitative data carefully extracted from recent literature showing little improvement in this regard.

There are significant methodological challenges for researchers investigating these questions. The evidence for direct patient benefit is hampered by simulation being in effect a complex intervention.^46,47 Ethical control groups, robust and validated instruments, and data on patient outcomes are difficult to come by, and there are many potential confounding variables. It is recognized that a methodological ‘gold standard’ may be elusive, but recommend ‘thematic, sustained, and cumulative research programs’.²⁰

Innovative methods are needed. These might, for example, involve creative use of ethnographic and qualitative methods to investigate in depth the role of simulation in the process of clinical skill acquisition. These methods could provide insights into the effect of simulation on human problem solving, decision-making, attention, and performance that would be valuable in designing simulation scenarios and embedding simulation into comprehensive training programmes that also incorporate other methods. Further assessment of the benefits of simulation using well designed, controlled trials is needed, but this should not obscure the need to investigate at a finer grain the effects of simulation on clinical reasoning and performance. Reliable and valid evaluation tools and metrics are also required to assess performance and behavioural change.^24,48,49

It is of interest that non-technical skills studies are significantly more likely to use tools such as post-training debriefs and validated checklists than those addressing technical skills. This suggests that assessment of technical skills acquired during training could follow validation techniques developed for non-technical skills evaluation. Checklists can assist with feedback and debriefing by providing a scaffold for the facilitators. The post-training debrief is important in order to maximize the learning that occurs as a result of the simulation experience with the aim of enabling change on both an individual and system level.⁵⁰ For non-technical skills training, it provides an opportunity to explore the behavioural aspects of healthcare such as teamwork and communication, but allowing trainee reflection on the experience could also be relevant for technical skills training. For many, the key to effective simulation-based training is not so much in the simulation scenario now as in the subsequent debrief. However, this is an area of research with simulation that is still lacking.⁵¹

As the number of papers increases over time, there is a noticeable increase in the number of manikin-based studies. It is likely that, as a realistic manikin becomes a component of anaesthesia simulation programmes,⁵² manikin-based simulation is likely to be applied even more widely in the future.

In addition to studying anaesthetists' performance, anaesthesia simulators can be ‘used to evaluate the ergonomics and performance of equipment, especially during the development phase’.⁵³ This is a developing field and simulation is increasingly linked with the evaluation, testing, and validation of anaesthetic techniques and equipment.

This review aimed to fulfil a particular purpose. It is inclusive of modalities and purposes of simulation but carefully defined in literature scope. Recent anaesthetic simulation reviews⁵⁴ cite studies from the journals we have included; however, during our study period, papers on simulation and anaesthetic performance will have been published outside the anaesthetic/analgesic field⁵⁵ or in lower impact journals. Further mapping could extend this study by investigating (i) the extent of this literature and (ii) whether our data are representative of this wider field or whether, for example, papers with particular applications, modalities, and designs around anaesthetic performance tend to be published in the non-clinical/simulation/medical education literature.

This study fills a gap in the anaesthetic and analgesic literature by mapping, describing, and quantifying recent patterns in anaesthetic simulation literature published in clinical journals. We found an increasing evidence-base and acceptance of simulation in anaesthesia into the clinical landscape. The patterns described are likely to be of interest in other clinical specialties with a shorter history of simulation training, especially for planning further applications of simulation and evaluating long-term impacts on practice and the quality of care. Longitudinal work with evidence of transferability to clinical practice remains elusive and this is likely to be the case for other specialties. Concerted efforts are required to develop innovative methods, both qualitative and quantitative, to address complexity and fill this evidence gap.

Supplementary material

Supplementary material is available at British Journal of Anaesthesia online.

Declaration of interest

None declared.

Funding

This work was supported by the UK National Institute for Health Research (NIHR). The views expressed in this publication are those of the authors and not necessarily those of the UK National Health Service, NIHR, or the UK Department of Health.