Validation of a novel hip arthroscopy simulator: establishing construct validity

Abstract

Hip arthroscopy (HA) is technically demanding and associated with a prolonged learning curve. Recently, arthroscopic simulators have been developed to anatomically model various joints including the knee, shoulder and hip. The purpose of this study is to validate a novel HA simulator. Twenty trainees and one sports medicine fellowship-trained orthopaedic surgeon at a single academic institution were recruited to perform a diagnostic HA procedure using the VirtaMed ArthroS hip simulator. Trainee characteristics, including level of training, general arthroscopy experience and hip specific arthroscopy experience, were gathered via questionnaire. For the purpose of this study, participants were categorized as novice (<25), intermediate (25–74) or experienced (≥75) based on the number of prior arthroscopies performed. Various performance metrics, including composite score, time and camera path length were recorded for each attempt. Metrics were analyzed categorically using ANOVA tests with significance set to P < 0.05. Composite performance score in the novice cohort was 114.5 compared with 146.4 and 151.5 in the intermediate and experienced cohorts (P = 0.0019), respectively. Novice arthroscopists performed the simulated diagnostic arthroscopy procedure in an average time of 321 s compared with 202 s and 181 s in the intermediate and experienced cohorts (P < 0.002), respectively. Cartilage damage and simulator safety score did not differ significantly between groups (P = 0.775). Simulator composite score and procedure time showed strong correlation with year of training (r = 0.65 and −0.70, respectively) and number of arthroscopies performed (r = 0.65 and −0.72). The ArthroS hip simulator shows good construct validity and performance correlates highly with total number of arthroscopic cases reported during training.

INTRODUCTION

Arthroscopic surgery is one of the most rapidly growing areas within orthopaedics, with surgical utilization reflecting this growth [1–3]. Hip arthroscopy (HA) has most recently seen a marked increase in utilization. HA offers surgeon a minimally invasive approach for the diagnosis and treatment of a myriad of hip pathology, including femoroacetabular impingement syndrome. Despite the proposed advantages of the minimally invasive technique, HA is technically challenging and associated with a prolonged learning curve before excellent outcomes and minimal morbidity can be consistently achieved [4].

Given the technical challenges associated with HA, there has been increasing focus on how to best prepare orthopaedic trainees interested in HA [5, 6]. Recent changes in resident work hour restrictions in the United States have resulted in a perceived decline in operative experience [7–9]. This is particularly concerning when considering technically demanding procedures with a prolonged learning curve, such as HA. To this point, Mehta et al. found that surgeons needed to perform at least 388 HA for their patients to have ≤10% chance of requiring subsequent hip surgery within 5 years of the index HA [10]. However, Westermann et al. found that residents in the United States graduate with, on average, only five HA surgical experiences during their 5 years of training (podium presentation at International Society for Hip Arthroscopy, 2018). As a result, arthroscopic simulators have been increasingly utilized and studied as an alternative modality to aid trainees in recent years given growing concerns with limited operative experience for these technically demanding procedures [11–13]. Virtual reality (VR)-based simulators provide several advantages for trainees, including a risk-free environment and time flexibility for skill development. Prior studies have validated the use of simulators for both knee and shoulder arthroscopy skill acquisition along with transfer validity from a simulated environment to the operating room [14–16]. However, there has been a lack of evidence regarding the validity of simulated HA and its potential use for skill development [17, 18].

The purpose of this study was to establish the construct validity of a novel VR simulation system, the VirtaMed ArthroS for HA by comparing various parameters between groups of novice, intermediate and experienced arthroscopists. We hypothesized that more experienced arthroscopists would outperform less experienced arthroscopists for all measured metrics, thus supporting the novel simulator’s construct validity.

MATERIALS AND METHODS

Sixteen orthopaedic surgery residents, four senior medical students and one orthopaedic sports medicine fellowship-trained attending at a single academic tertiary institution participated in this prospective study in June 2018. For the purpose of this study, three cohorts were identified a priori by level of experience with respect to prior arthroscopy volume. The three cohorts were stratified as novice (<25 arthroscopic procedures performed), intermediate (25–74 arthroscopic procedures performed) and experienced (≥75 arthroscopic procedures performed). The novice cohort was comprised of four post-graduate year 1 (PGY 1) orthopaedic surgery residents and four medical students interested in orthopaedic surgery. The intermediate group comprised of eight junior residents (PGY 2/3) and one senior resident (PGY 4/5). The experienced cohort comprised of three senior residents and one fellowship-trained orthopaedic sports medicine attending. Participants were excluded if they had previous experience with the VR HA simulator. Demographic data for study participants are shown in Table I. Subjects filled out a questionnaire prior to participating in the surgical tasks, including hand dominance, sub-specialty interest, total number of previous arthroscopic procedures performed and total number of previous HA procedures performed. In addition to categorizing trainees based on number of procedures performed, participants were also categorized level of training (PGY 1 through 5, fellow, staff) for Pearson correlation analysis.

Table I.

Participant demographics

Participant characteristics	Novice (<25 arthroscopies)	Intermediate (25–74 arthroscopies)	Experienced (>74 arthroscopies)
Participants (n)	8	9	4
Mean number of arthroscopies (n, ±SD)	0.25 ± 0.66	36 ± 9.6	395 ± 465.64
Mean number of hip arthroscopies (n, ±SD)	0	4 ± 5.68	52 ± 85.74
Handedness (right:left)	7:1	7:2	4:0
Year in traininga	4:4 (MS4: PGY-1)	8:1 (PGY2/3: PGY4/5)	3:1 (PGY4/5: Prof.)

^a

MS4, fourth-year medical student; PGY-1, orthopaedic intern; PGY2/3, junior orthopaedic resident; PGY4/5, senior orthopaedic resident; Prof., fellowship-trained sports medicine surgeon.

SD, standard deviation.

Simulation testing was performed in a supervised environment using the ArthroS Hip module by VirtaMed (Zurich, Switzerland). A standardized introduction and orientation to the simulator was given to all participants by the same individual not affiliated with the current study. This simulation exercise used pre-established mid-anterior and anterolateral portals. Five anatomic structures were visualized through the anterolateral portal and six through an anterior portal (Table II). Composite score, procedure time, camera path length, iatrogenic femoral head cartilage injury, iatrogenic acetabular cartilage injury and number of fluoroscopic images taken were recorded by the simulator software. Composite score was calculated using a proprietary scoring formula by VirtaMed.

Table II.

Anatomic objectives for diagnostic HA simulation

Anterolateral portal	Anterior portal
Acetabular fossa	Acetabular fossa
Ligamentum teres	Ligamentum teres
Posterior medial acetabulum and labrum	Posterior transverse ligament
Anterior acetabulum and labrum	Anterior transverse ligament
Anterolateral acetabulum and labrum	Superior articular cartilage
	Lateral labrum

Statistical analysis was performed using Microsoft Excel (Redmond, WA, USA). Descriptive statistics included participant factors such as level of experience with arthroscopy, hand dominance and level of training. Categorical analysis using the one-way ANOVA test was used to compare composite performance score, time to simulation completion, camera path length and safety score between the three cohorts, with statistical significance set at P < 0.05.

RESULTS

On an average, the novice cohort performed 0.25 arthroscopic procedures (standard deviation [SD]: 0.66, range: 0–2), the intermediate cohort 36 (SD: 9.6, range: 28–50) and the experienced cohort performed 395 (SD: 465.64, range 80–1200) prior to participation in the study. On an average, the intermediate cohort performed four hip arthroscopies (SD: 5.68, range: 0–15) and the experienced cohort performed 52 hip arthroscopies (SD: 85.74, range: 1–200), whereas the novice cohort did not have any prior experience with HA.

Composite performance score was significantly lower in the novice cohort (114.5 ± 14.91) compared with the intermediate (146.4 ± 17.17) and experienced cohort (151.5 ± 17.18) (P < 0.002). Novice arthroscopists performed the module in an average time of 321 s (SD: 67.04) compared with 202 s (SD: 36.26) and 181 s (SD: 24.52) in the intermediate and experienced cohort, respectively (P < 0.002). Camera path length did not differ significantly between groups with an average of 202 ± 42.2 cm, 172 ± 74.5 cm, and 147 ± 59.4 cm in the novice, intermediate and experienced cohorts (P = 0.3804), respectively. Additionally, cartilage damage and safety score did not differ significantly between groups with a mean safety score of 78.75 ± 10.44, 82.11 ± 9.81 and 82 ± 6.52 in the novice, intermediate and experienced cohorts (P = 0.775), respectively (Table III).

Table III.

Comparison of objective measures of hip arthroscopy simulator

	Novice (<25 arthroscopies)	Intermediate (25–74 arthroscopies)	Experienced (>74 arthroscopies)	P-value
Composite performance score (score, ±SD)	114.5 ± 14.91	146.44 ± 17.17	151.5 ± 17.18	P < 0.002
Procedure time (s, ±SD)	321.75 ± 67	202.44 ± 36.3	181.69 ± 24.5	P < 0.002
Camera path length (cm, ±SD)	202.63 ± 42.21	172.26 ± 74.51	147.36 ± 59.41	P = 0.3804
Safety score (score, ±SD)	78.75 ± 10.44	82.11 ± 9.81	82 ± 6.52	P = 0.7749

Significant findings are indicated in bold (P < 0.05).

Pearson correlation coefficients were utilized to assess for significant correlations between different testing variables (Table IV). Simulator composite score and time to task completion showed strong correlation with year of training (r = 0.65 and −0.70, respectively). With respect to prior arthroscopic experience (novice, intermediate or experienced), there was a strong correlation with total score (r = 0.65) and time (r = −0.72), but a poor correlation with path length (r = −0.32) and acetabulum or femoral head cartilage damage (r = 0.01 and 0.18, respectively).

Table IV.

Pearson’s correlation (r) between participant characteristics and simulator metrics

Participant characteristics	Total score	Procedure time	Camera path length	Safety score	Scratching of the femoral cartilage	Scratching of acetabulum
HA	0.3815	−0.2759	−0.3918	0.2075	−0.1178	−0.0813
Total arthroscopy	0.4008	−0.3296	−0.3857	0.1775	−0.1052	−0.0468
Year of training	0.6496	−0.7036	−0.3806	0.1490	−0.2372	0.1444
Simulator use	0.4604	−0.4776	−0.3037	0.1608	−0.1052	0.2420
Experience (novice, intermediate, experienced)	0.6497	−0.7220	−0.3185	0.1430	0.1778	0.0141

DISCUSSION

Virtual reality HA simulators provide a risk-free environment for orthopaedic trainees to improve their skills for a technically demanding surgical procedure. In this study, we found that the VirtaMed ArthoS HA simulator performance correlates highly with previous arthroscopic experience and level of training, with composite performance score, procedure time and camera path length showing the highest correlation with prior arthroscopic experience. However, procedural safety, including iatrogenic femoral head and acetabular cartilage injury, did not differ between the various skill levels.

Currently, there is a paucity of data on HA simulators. A 2018 cross-sectional study by Erturan et al. recruited 52 participants to perform a standardized bench-top simulated HA task using a HA simulator (Sawbones Europe) [19]. Participants were divided into expert (four fellowship-trained staff), trainee (28 residents and fellows) or novice (20 interns and medical students). The study noted a significant difference between the arthroscopic ability of all groups when analyzing performance based on time and motion analysis as well as the Basic Arthroscopic Knee Skill Scoring System (BAKSSS) Global Rating Scale [20]. Recently, Bartlett et al. assessed the face validity of a VR HA simulator, the Simbionix ArthoMentor [21] (Littleton, CO, USA). They recruited 18 surgical residents and 7 faculty hip arthroscopists from a HA training course without previous experience using VR simulators. All participants in their study performed a basic diagnostic HA locating 12 anatomical targets within the joint. Upon completion, they completed a questionnaire using a 10-point Likert scale evaluating the simulator on realism as well as its use in a learning environment. The simulator used in that study was found to have a high degree of realism with respect to visual representation, instrumentation and procedure itself. However, less than 50% reported tactile feedback received from the soft tissues to be realistic. They found that 84% of participants reported the simulator to be a useful tool for intern level trainees, 88% reported useful for resident level and 80% said it would be useful for attending level hip arthroscopists.

The present study showed a greater difference in nearly all outcome measures except iatrogenic cartilage injury when comparing novice to intermediate skill cohorts rather than intermediate to experienced cohorts. The findings of the study may suggest that participants in the intermediate cohort performed comparably to those in the experienced arthroscopists cohort. Therefore, the findings of this study should be understood in the context of total arthroscopic experience as opposed to HA experience. Intuitively, simulators are likely to provide the most benefit for novice or intermediate trainees as opposed to experienced, fellowship-trained hip arthroscopists. We found that composite performance score and time to complete simulation correlated strongly with experience. This is in agreement with a recent Level II randomized trial using an arthroscopic knee simulator which showed shorter time to completion and a trend toward improved performance with training compared with an untrained cohort [22].

Previous literature has shown training on VR simulators to translate into improved arthroscopic proficiency on both cadaveric specimens as well as in vivo. In 2015, Rebolledo et al. showed that training junior orthopaedic residents with knee and shoulder arthroscopic simulators resulted in improved performance in cadaveric models when compared with those that received didactic training only [23]. Watermann et al. demonstrated significant improvement in task time, probe distance and ASSET score in vivo in a cohort of orthopaedic trainees who underwent training using a VR shoulder simulator compared with a control group [24]. The present study corroborates these findings; the Arthro-S simulator, especially when using the composite score, may be useful for evaluating the progress of mid-level trainees interested in learning HA. However, further research on how improvements in these measured and scored tasks translated to an in vivo operative scenario are warranted.

LIMITATIONS

There are several limitations to this study. First, there is no control group in this study, and therefore, we are unable to speculate whether or not the simulator training is superior to other modalities, including didactic or cadaver-based training. In addition, the number of previous total arthroscopic procedures and hip arthroscopies were obtained using a questionnaire, relying on accurate procedure reporting prior to enrolment in the study. The simulated procedure was not comprised of certain key components of in vivo HA such as hip distraction, bleeding/visibility issues, joint access or portal establishment. Lastly, the study included one fellowship-trained hip arthroscopists. Therefore, conclusions regarding construct validity of the VirtaMed ArthroS hip simulator in relation to experience with HA could not be established in this study. Therefore, the findings of this study should be understood in the context of total arthroscopic experience as opposed to HA experience.

CONCLUSIONS

The ArthroS hip simulator shows good construct validity and performance correlates highly with total number of arthroscopic cases reported during training. Certain metrics, such as simulated iatrogenic cartilage injury, cannot distinguish between different levels of surgical experience. The ArthoS composite score may be a useful metric to assess progress for trainees learning HA.

ACKNOWLEDGEMENTS

We acknowledge Catherine Fruehling for her help with setting the simulation laboratory for this study.

FUNDING

No outside financial funding was utilized for this study.

CONFLICT OF INTEREST STATEMENT

Dr. Karam has stocks or stock options with Iowa Simulation Solutions LLC. Dr. Westermann receives research support from Smith and Nephew. Dr. Willey receives research support from Biomet. All other authors declare no conflict of interest.