Summary
The assessment of surgical competency has become a priority for both surgical educators and licensing boards. Surgical educators must incorporate rigorous, reliable, and valid means of assessment into residency programs. Objective evaluation of technical skills has been extensively explored in various surgical specialties, but its role in cardiac surgery has not been well studied and there is limited experience with integration into the educational curricula. Several cardiac and vascular surgery simulation models have been designed and evaluated, ranging from simple low-fidelity models using inert materials to a complex, computer-controlled, high-fidelity simulator using biological tissues to practice entire surgical cases. Most of the available models have not been well validated or integrated into educational curricula. The cardiac surgery simulation tools in development need validation and incorporation into structured, competency-based training curricula. The ongoing development of surgical simulators and educational curricula will enable a transition from the century-old graded responsibility training program to a competency-based program, where trainees must demonstrate technical competence to progress to the next level of training and gain certification and re-certification – ultimately ensuring better and faster technical skill acquisition as well as improved quality of care and patient safety.
1 Background
Surgical educators and licensing boards have a responsibility to ensure public safety through adequate training of surgeons. Surgical competency has become a priority, and there is a consensus among both educators and society that a trainee must demonstrate acceptable proficiency with complex tasks, prior to starting independent practice. A surgeon’s technical skills are of crucial importance for patient safety; for many patients, a successful clinical outcome depends on having a well-performed technical procedure [1].
Traditionally, surgeons have developed their technical skills by a process of protracted exposure to supervised, graded operative experience. A skill is the result of applying a specific combination of abilities to a given task, and performance implies the overall efficiency of completing a task [2]. Procedural training in current practice is often unsystematic and unstructured, and validated teaching methods have not been well integrated into clinical education [1]. Furthermore, objective assessment of technical competence has historically been difficult and often based on subjective observation and logbooks [3]. In the past few years, there has been an increasing interest among educational researchers to design and validate tools for safe and efficient training, and objective and valid assessment in cardiac surgery. However, this emerging evidence has not been translated into clinical practice.
Coronary artery bypass surgery is one of the most commonly performed cardiac surgical operations worldwide, and is the most studied, documented, evaluated, and audited treatment in the history of medicine [4]. The recent interest in surgical simulation, combined with advances in computer and materials technology, has enabled the rapid development of realistic surgical simulation models. However, the role of simulation in cardiac surgery has not been well established.
Surgical educators will need to incorporate meaningful assessment into residency programs, using rigorous, reliable, and valid tools [5], as defined in Table 1 . Reliability is the reproducibility and precision of a test or testing device. Validity measures whether the simulator is actually teaching or measuring what it is designed to teach or measure. ‘Face validity’ is a measure of the realism of the simulator, specifically how closely the assessment resembles the ‘real’ task. ‘Content validity’ considers the extent to which the model measures technical skill as opposed to knowledge, and assesses the appropriateness of the simulator as a technical skill teaching modality. Both face validity and content validity can be met by expert consensus. ‘Construct validity’ indicates if a simulator effectively distinguishes an inexperienced surgeon from an experienced one. ‘Criterion validity’ compares the evaluation results from a new simulator with the results from the old method of evaluation. The two forms of criterion validity are ‘concurrent validity’ – the extent of a simulator’s correlation with the ‘gold standard’ – and ‘predictive validity’ – the extent to which the simulator predicts future performance. To be an effective tool to assess ‘competency’, performance on a simulation model should predict, or at least correlate with, performance in the operating room (OR) [6,7].
Definitions of reliability, validity and competency.
Definitions of reliability, validity and competency.
The aim of this article is to review the literature on the training and assessment of procedural skills in cardiac surgery, and objectively evaluate simulation models as valid educational and examination tools. The cardiac surgery simulation models discussed in this article are objectively assessed on the basis of validity and competency, as summarized in Table 3.
2 Current practice in cardiac surgery training
To become a safe and effective cardiac surgeon, a trainee must achieve proficiency with the complex psychomotor skills necessary to perform sophisticated technical tasks. Currently, technical skills and surgical decision making are usually taught in the OR.
Cardiac surgery requires a number of specific skills that may pose a significant challenge to the novice surgeon. Many cardiac surgery procedures consist of creating and remodeling anatomical structures, rather than ligation and removal of structures as in many other surgical procedures. The small size of the cardiac anatomical elements requires very fine sutures and instruments, and surgeries are performed with the magnification of high-power loupes. Fluoroscopy and wire-handling skills are required to manipulate devices placed in the heart from outside the chest cavity.
A variety of difficulties associated with the current patient-centered teaching methodology may limit completion of the educational objectives: exposure to technical procedures is based on patient availability; logistical issues such as pressure for maximum speed and efficiency; a hierarchical system where trainees are taught surgical skills based on seniority rather than adequate skills and knowledge; and concerns about the safety of trainees performing advanced procedures on patients [1].
Much of the pioneering cardiac surgery research was performed on live dogs. Contemporary surgical skills training includes live animals, human cadavers, and ex-vivo animal tissues. The limitations of live-animal and cadaver surgery have been part of the drive to develop bench-top simulation models. Live-animal surgery requires highly skilled personnel and expenditure for the animal, and provides inexact replication of human anatomy. Human cadavers are useful simulation models when the fidelity of tissue is required and the anatomical differences of animals are unacceptable; however, cadaveric tissue may not provide as realistic an experience as does living, pulsating tissue [6].
Studies support the notion that beginner surgeons can benefit from practicing their surgical skills on simplified bench models, suggesting that low-fidelity models are sufficient for motor learning to occur. The learning specificity theory, proposed in the motor learning literature, states that the closer the practice conditions are to the test or real-life condition, the better the learning. Trainees learning on high-fidelity models (resembling real-life OR situations) should therefore demonstrate more motor learning than trainees learning on low-fidelity models [8].
The porcine heart is similar in size and anatomy to the human heart; the bovine heart, although larger, is also similar to the human heart. Live animals are used to teach cardiac surgery technical skills when a living, beating heart is required – a common example is off-pump coronary artery bypass surgery performed on live anesthetized pigs. Human cadavers are used when a highly accurate depiction of human anatomy is required; common examples are mitral valve exposure and minimally invasive surgical procedures. A combination of human and animal tissue may be used – a common example is surplus pieces of human saphenous vein or internal mammary artery may be saved and trainees practice anastomoses onto porcine hearts in the laboratory.
The monetary cost of laboratory training is often an issue for training programs. Live animals and cadavers tend to be expensive due to the ethical issues and the required support staff. Porcine hearts are usually not destined for human consumption and are readily obtainable from commercial slaughterhouses at low cost. Surplus human vein or artery is readily obtainable at no cost; however, surgical consent forms should include a clause permitting research use of surplus materials, obviating ethical concerns.
3 Simulation models in cardiac surgery
Some of the basic cardiac surgery technical skills include sternotomy and wound closure, arterial and venous cannulation to institute cardiopulmonary bypass, valve implantation, anastomosis of coronary artery bypass grafts, and replacement of sections of the aorta with synthetic material.
The effectiveness of simulation training has been demonstrated primarily for junior-level learners, where fidelity of the models may be less important [5]. A variety of surgical models may used for practicing surgical procedures, including simulated reality training, bench models composed of inert and biological material, live-animal models, and cadavers [1]. The cardiac surgery technical skill assessment models vary from a simple low-fidelity coronary anastomosis model using inert materials to a complex, computer-controlled, high-fidelity simulator using biological tissues to practice entire surgical cases. The cardiac surgery simulation models in the published literature are summarized in Table 2 , and the effectiveness of each simulation model as a teaching tool is evaluated in Table 3 .
Simulation models of cardiac and vascular surgery.
Simulation models of cardiac and vascular surgery.
Comparison of reported validity and competency assessments.
Comparison of reported validity and competency assessments.
The published literature on cardiac surgery simulation primarily relates to coronary artery bypass grafting. However, the full range of cardiac surgery procedures have been modeled and are regularly practiced in resident wet labs without formal assessment; these models can be formally evaluated in the same manner as the coronary anastomosis simulation models.
In addition to simulation models developed in academic institutions for residency training, the medical-device industry has developed high-fidelity electronic simulation suites to train physicians and surgeons in the use of their devices. Medtronic (Minneapolis, MN, USA) has developed a simulation suite that accurately models percutaneous coronary intervention (PCI), trans-femoral and trans-apical aortic valve replacement, and endovascular stent grafting. The Medtronic simulator is available at corporate training facilities and university laboratories upon physician request, and also at major cardiac surgery conferences.
4 Bench-top models
Tokuda and Song described a simple model of a coronary anastomosis created by stretching medical gloves over a towel. The model is constructed by wadding a towel to form a ball and stretching a glove over it, followed by stretching a second towel over the first; the entire model is placed inside a box representing the chest cavity. The two gloves represent the anterior and posterior walls of the coronary artery, while the finger of another glove simulates a graft. The model was modified to simulate a beating-heart anastomosis by wrapping the towel and gloves around an intra-aortic balloon pump [9]. The authors have not reported evaluation of the model as a valid training tool.
Izzat and colleagues developed a mechanical off-pump coronary anastomosis simulator. The model consisted of a floating platform resembling the epicardial surface and a series of wheels below the platform connected to an electric motor. With rotation of the motor, the wheels intermittently elevate the platform, simulating heartbeats. Adjustment of motor speed altered the perceived heart rate, and adjustment of wheel locations simulated dysrhythmias. Trainees placed a porcine mammary artery on the platform and practiced coronary anastomoses [10]. The validity of this tool in cardiac surgery training has not been evaluated yet.
Fann and colleagues used synthetic silicone models to simulate both on- and off-pump coronary artery bypass grafting. Trainees anastomosed a 3-mm silicone vein graft to a 3-mm silicone coronary artery mounted on a silicone heart. The synthetic heart is used as a stationary model, or beating in a fashion similar to a human heart [11].
The same group developed an educational curriculum incorporating the synthetic coronary artery bypass model and evaluated its use as a training device with eight cardiothoracic surgery trainees. At the first laboratory session, the trainees viewed a 5-min instructional video on the use of the Castroviejo needle driver, and then performed two coronary anastomoses on a stationary heart model, and two on a beating-heart model. Trainees practiced at home on a portable task station for 1 week, recording their practice time in minutes, and returned to the simulation laboratory to repeat anastomoses on each of the two models. The simulation laboratory sessions were videotaped and scored using the objective structured assessment of technical skill (OSATS) method by two experienced surgeons. The analysis showed some (but not all) residents improved, and demonstrated a ‘ceiling effect’ with the simulator and/or a ‘plateau effect’ with the trainee [11]. The ‘ceiling’ and ‘plateau’ effects likely demonstrate the limitations of the model as a training tool – some trainees cannot become more proficient at a task because they are already experts, or the model does not provide enough fidelity to allow the trainee to improve.
Fann and colleagues subsequently developed a coronary artery anastomosis ‘Boot Camp’ experience for 33 first-year cardiothoracic surgical residents; each resident participated in a 4-h coronary anastomosis session and their performance was evaluated with live and blinded video scores based on the OSATS methodology. The ‘Boot Camp’ included a high-fidelity tissue-based model (composed of a porcine heart and cryopreserved human saphenous veins) and a low-fidelity synthetic model (composed of 4-mm synthetic target vessels and 4-mm synthetic graft vessels). The residents first received 20 min of didactic instruction, followed by an anastomosis on the high-fidelity model, then an anastomosis on the low-fidelity model, followed by a second anastomosis on the high-fidelity model. There was a live assessment of the demonstrated statistically significant improvement in skill with each successive anastomosis. Blinded video assessment by three surgeons confirmed the statistical significance of the improvement with each task station. The residents completed an exit questionnaire in which all participants thought an anastomosis on the porcine model was realistic, and all participants felt more confident in their coronary anastomosis skills [12].
Hance and colleagues developed a cardiac surgery simulator consisting of four simulated tasks, three of which were porcine tissue based and one latex based: (1) aortic root cannulation; (2) vein graft to aorta anastomosis; (3) vein graft to left anterior descending (LAD) coronary artery anastomosis; and (4) femoral triangle dissection. The models were composed of: (1) a length of free porcine descending thoracic aorta, pressurized with simulated blood by a pulsatile pump to provide realistic feel – the trainee placed an aortic cannula in the usual fashion with purse-string sutures, then decannulated and obtained hemostasis; (2) porcine ureter was used to simulate human saphenous vein and was sutured to the LAD of a porcine heart to simulate a distal coronary bypass graft anastomosis; (3) porcine ureter was sutured to the aorta of a porcine heart to simulate a proximal coronary bypass graft anastomosis; and (4) trainees dissected the femoral triangle in a latex model of the groin and isolated the femoral artery [7].
Hance and colleagues studied 40 cardiac surgeons of varying skill levels who each completed the four task stations described earlier. Each surgeon’s performance was scored using the OSATS methodology by a live examiner and three blinded video examiners. Evaluation of the simulation model established construct validity of all four workstations with significant discrimination of surgeon skill level. The saphenous vein to LAD anastomosis model – the most complex workstation – was demonstrated to be the best discriminator of surgeon skill. Inter-rater reliability was found to be acceptable using this simulation model, and inter-station reliability was high [7].
Wilasrusmee and colleagues evaluated technical skill with a laboratory arterio-venous (AV) fistula creation laboratory model, and then evaluated performance of AV fistula creation in the OR on human subjects. Subjects using the laboratory model were asked to: (1) close the end of a 6-mm polytetrafluoroethylene (PTFE) graft using a continuous suturing technique with 6/0 polypropylene; (2) perform end-to-end and (3) end-to-side anastomosis using the same materials and techniques [13]. PTFE has the characteristic of having a needle hole that is inversely proportional to the skill of passing a needle through the material, and hence has less needle-hole leakage when sutured by an expert [2]. The AV fistula model is intended for vascular surgery, and is also applicable to cardiac surgery, particularly the end-to-side anastomosis.
Wilasrusmee subsequently evaluated trainees in the laboratory and OR using a modified OSATS checklist. General surgery residents at the institution had never performed vascular anastomoses themselves; however, the simulation model provided the trainees with a chance to use the instruments, understand the techniques, and gain technical proficiency. The best predictors of surgical competence in the OR were the level of competence measured in the simulation laboratory and the year of residency training. Outcome parameters included anastomotic leakage, completion time, and OSATS score [14].
Sidhu and colleagues evaluated a simulator consisting of high- and low-fidelity vascular anastomosis models in the laboratory, and correlated laboratory performance with performance on a vascular anastomosis in a live pig. Senior and junior residents were divided into two groups, a low-fidelity group which anastomosed a 4-mm plastic tube graft to a stationary plastic tube graft and a high-fidelity group which anastomosed a 4-mm plastic tube graft to the brachial artery of a human cadaver. One week after the training phase, trainees performed an end-to-side anastomosis of a plastic graft to the femoral artery of a live anesthetized pig. Trainees were assessed using OSATS procedural and global rating checklists, final product analysis, anastomosis completion time, and hand-motion analysis using electromagnetic tracking sensors [8].
The study demonstrated that both junior and senior residents who trained on the high-fidelity model achieved higher final product scores on the live pig model. Time to completion and hand-motion analysis was significantly different between senior and junior residents, but no difference was found between the low- and high-fidelity training groups. Junior residents who trained on the high-fidelity model achieved higher procedural checklist scores on the live pig model, but no similar correlation was found for senior residents. Global rating scales were not significantly different between low- and high-fidelity training for junior or senior residents. Junior and senior residents who trained on the high-fidelity model achieved significantly higher scores on final product analysis. The authors concluded that for vascular anastomosis it is important to provide appropriate model fidelity trainees of all levels to optimize the effectiveness of bench-top model training [8].
5 Simulated reality
Ramphal and colleagues developed a complex, computer-controlled, high-fidelity cardiac surgery training system using biological tissues with the ability to simulate entire surgical cases. In addition to the required technical skills, the simulations include adverse clinical scenarios requiring clinical judgment. The simulated OR includes a humanoid mannequin, draped in the usual fashion, simulated real-time hemodynamic monitoring, and an ‘anesthetist’ who directs the simulation. The surgical model is a porcine heart instrumented to beat or fibrillate as desired, simulated coronary blood flow, and sensors to quantify the physical forces placed on the heart when handled by the trainee. Standard and beating-heart coronary artery bypass, aortic valve replacement, aortic homograft replacement, and pulmonary autograft procedures can be simulated with high degrees of realism [15]. The authors have not reported evaluation of the model as a valid training tool.
Ramphal and colleagues developed their simulator largely due to the low volume of cardiac surgery cases in Jamaica; the aim was not only resident education, but also education of the entire surgical team, including OR nurses. The simulator has been further developed at the University of North Carolina through sponsorship by the American Board of Thoracic Surgery. Assessment of the face validity of the model demonstrated excellent results [16]. This simulator relies heavily on clinical knowledge and judgment rather than pure technical skill; therefore, the content validity may not be as strong as a single-skill bench-top station.
6 Technical skill assessment methods
Cardiac surgery trainees currently maintain a record of the procedures performed in the OR. Some jurisdictions require a specified number of procedures for certification to independent practice, as recorded in a case log. Trainees must discuss with the supervising surgeon their role in the operation, and which components of the operation they can record as being the primary surgeon. These case logs are based on sheer numbers rather than technical ability, and do not account for the variance in the aptitude between trainees. It is well known that logbooks lack validity and more reliable assessment tools are needed to ensure the technically proficiency of a graduating cardiac surgeon.
Surgical simulation provides trainees an opportunity to practice their surgical skills, but practice alone is not sufficient to assess competence. To assess technical proficiency, monitor progress, and provide reliable and structured feedback, a trainee’s technical skill and performance of a simulated surgical task must be formally assessed using valid tools. A variety of metrics to assess technical skills in surgical simulation have been proposed and evaluated; the metrics used in the published cardiac surgery simulators are detailed in Table 4 .
Evaluation methods used by cardiac and vascular surgery simulation models.
Evaluation methods used by cardiac and vascular surgery simulation models.
6.1 Global rating scales vs procedure-specific checklists
A trainee’s technical skill can be assessed using the OSATS methodology. OSATS was initially designed and validated for bench model tasks and has been extensively validated in general surgery. The assessment technique consists of a task-specific checklist and a global rating scale [17]. The technical skill checklist is a useful part of the OSATS technique, but it has been shown to have less power and reliability than the global rating scale [18,19].
6.2 OSATS – live and video assessment
Live scoring provides near-instantaneous scoring and feedback to trainees, but requires the presence of a trained examiner at all assessment sessions. Video scoring eliminates the need for examiners to be on site during simulation sessions; examiners can evaluate the relevant portions of each simulation session on their own schedule. Hance and colleagues assessed trainee performance with a live rating by a single expert surgeon, and a retrospective video rating by three blinded expert surgeons. There was strong correlation between live and blinded scores; however, unblinded live scores were consistently higher than blinded video scores. The authors concluded that retrospective scoring of each video by three examiners was very labor intensive and unnecessary [7].
6.3 Dexterity-based assessment
Hand-motion analysis has been validated using the Imperial College Surgical Assessment Device (ICSAD), a computerized electromagnetic tracking system. The ICSAD system collects Cartesian coordinate information at a resolution of 1 mm and a frequency of 20 Hz from electromagnetic trackers attached to the dorsum of each hand (inside a sterile latex glove in the operating room). The hand-motion analysis has been shown to be an effective method to measure open surgical skill in the laboratory-based model [20]. Furthermore, a strong correlation between hand-motion analysis and OSATS assessment has been demonstrated [18].
6.4 Time-based assessment
Time to complete a task has traditionally been used for assessment of technical performance. However, faster performance is not always associated with better quality and improved outcomes. Sidhu and colleagues found that increasing expertise significantly predicted a lower completion time for vascular anastomoses, but did not report correlation of completion time with surgical outcome [8]. Wilasrusmee’s vascular anastomosis model demonstrated that longer completion time (and perhaps more diligence) in the laboratory predicted a longer completion time and lower severity of anastomotic leakage in the OR [14].
6.5 Quality-based assessment
Trainees may be assessed on the quality of their surgical tasks by functional performance of the completed surgical task. Wilasrusmee and colleagues quantified the number and severity of leaks in their laboratory and demonstrated that the number and severity of leaks in the laboratory were highly predictive of leakage grade in the OR [14].
7 Future research and implementation in practice
The purpose of surgical simulation is to teach technical skills and assess proficiency before a trainee performs procedures on patients in the OR. A validated educational curriculum is required to first teach the technical skills, assess and confirm proficiency, and then transfer the skills from the simulated environment to the OR. Although several training and assessment tools have been designed and validated, translation into clinical practice has been suboptimal [1].
Grantcharov and Reznick described an educational curriculum to teach technical skills and transfer the acquired technical skills from the simulated environment to the OR – a pre-patient training phase followed by training in a clinical situation. The pre-patient training phase should be taught outside the clinical setting and consists of: (1) cognitive knowledge surrounding the procedure; (2) instruction in basic enabling skills; and (3) opportunity to perform a procedure on a variety of models. The clinical training phase consists of a graduated training program – observation of a procedure, followed by performing components of a procedure, and finally performance of an entire procedure [1].
Residency training in Europe and North America is on the verge of a paradigm shift as strict limits are placed on trainee working hours due to concerns about patient safety and resident well-being. A major concern with a reduction in working hours is the perception that trainees will require longer residencies to allow adequate clinical training; residency training must become more efficient if trainees are to acquire clinical competency within a time frame similar to the current system.
Residents will need to practice technical skills in a highly deliberate and time-efficient manner to achieve surgical competence. Fidelity may be less important at relatively junior levels of training; simulators will likely be used to demonstrate proficiency in basic techniques [5], followed by honing surgical skills in the OR on patients.
Many similarities exist between the aviation industry and the practice of medicine, particularly in the OR. Pilots learn to fly basic airplanes and progress to larger and more complex aircraft as their flying skills progress. Advances in computing technology have revolutionized training in the aviation industry, and the primary training tool for advanced aircraft is now high-fidelity flight simulators. Pilots no longer fly empty airliners to acquire skill in that aircraft – they learn to fly an aircraft in a simulator, and the pilot’s first flight in an airliner is often on a scheduled route with a full load of passengers. Airline pilots must undergo flight simulator training on a regular and recurring basis to maintain certification; they practice uncommon scenarios and emergency procedures to develop and maintain their skills in case of a rare event of a real emergency. The highly structured system of pilot training and re-certification may provide valuable insight into teaching and evaluating surgical skills.
A competency-based orthopedic surgery curriculum has been developed at the University of Toronto and approved by the Royal College of Physicians and Surgeons of Canada. This new curriculum commenced on 1 July 2009, and three of the 12 incoming postgraduate trainees each year (immediately upon completion of medical school) enter the competency-based training program. The curriculum consists of modules covering all aspects of orthopedic surgery, and residents move from one module to the next when they have achieved the goals of that module, rather than traditional time-based rotations. The competency-based orthopedic surgery curriculum is anticipated to provide a significant opportunity for the trainees to complete the training program and take the Royal College examinations in less than the usual 5 years.
The competency-based orthopedic training program is operational at the University of Toronto and will serve as a model for other specialties and other residency training programs. The use of simulation to teach and assess procedural skills will become an integral component of cardiac surgery training and certification as resident work hours are restricted. The ongoing development of surgical simulators and educational curricula will enable a transition from the century-old graded responsibility training program to a competency-based program where trainees must demonstrate technical competence to progress to the next level of training.
Individual residency training programs will incorporate simulation into competency-based education curricula; however, the greatest advantage may arise in certification and re-certification by national and international accreditation bodies. Implementation of national and international simulator-based training standards will allow standardized objective assessment of technical skill for all trainees, and will become an integral component of certification. Cardiac surgeons must demonstrate technical competence, in addition to oral and written examinations, to enter and continue practice.
Simulation models must be developed and formally evaluated for the full range of cardiac surgery procedures to allow a transition to competency-based residency training. Academic institutions, program directors, and accreditation boards must anticipate and actively prepare for the inevitable paradigm shift in residency training already occurring in other procedural specialties. The ultimate goal of this approach is to ensure better and faster competency acquisition as well as improved quality of care and patient safety.
