Introduction

The training paradigm of surgery residents changes secondary to patient volume, complexity of care provided, concerns for proper supervision, patient safety, and duty hour restrictions to surgeons in training [1]. Residents need to develop operative technical skill leading to autonomy for long term success [2]. In the current model, self-logged procedural numbers are used to document operative competence and end-of-rotation aggregate faculty assessments are based on re-call. Documentation of operative proficiency is becoming a more important issue as demonstrated by the recent increase of required cases for graduation by the ACGME/ABS. Most surgery training programs formally evaluate a resident’s operative performance intermittently based on faculty’s collective memory of their operative experiences. Some groups have developed smart phone applications to allow for real time, high frequency assessment of resident’s operative ability [3]. This data leads to easier monitoring of resident progress, teaching behaviors of faculty, and dynamics of OR environment [2].

Endoscopy is a principal component of general surgery practice. The ABS recently increased minimum endoscopy numbers and formalized a flexible endoscopy curriculum [4,5]. The ABS Flexible Endoscopy Curriculum (FEC) is a stepwise instructional program that provides cognitive and technical milestones for each of the five surgical postgraduate years [6].

Beginning with the 2018 graduating class, all general surgery residents are required to successfully navigate the FEC prior to completing residency and taking the ABS Qualifying Examination [7,8]. An important milestone within the FEC is passing the Fundamentals of Endoscopic Surgery (FES™) exam [6,9]. FES is a test of knowledge and skill in flexible gastrointestinal endoscopy and consists of a comprehensive, web-based didactic component, a multiple-choice knowledge test, and a five-module virtual reality skills examination. GAGES (Global Assessment of Gastrointestinal Endoscopic Skills) was developed as a tool to measure endoscopic skills [10]. It was also used as a guide to the development of modules and metrics in the simulator skills component of FES. GAGES can be used to measure the effectiveness of simulator training or skills curricula designed to teach the basic technical skills required for flexible endoscopy. A reasonable clinical performance using the GAGES tool has been shown to correlate with passing the FES hands-on test.

Our institution has developed a smart phone application that allows faculty to provide real time GAGES scores of residents performing flexible endoscopy. The objective of this study is to establish the feasibility of integrating the usage of this innovation application into the workflow of busy, academic clinician educator surgeons.

Methods

This is a prospective, pilot study designed to describe the feasibility of obtaining real time GAGES scores of residents performing flexible endoscopy using a smart phone application. This study was deemed exempt from IRB approval by the IRB office. This study occurred over a 6-month period from February 2017 through August 2017. Four attending surgeons participated. They were asked to complete as many evaluations as possible. After initial training on how to use the smart phone application, no additional reminders were given to the faculty. The Global Assessment of Gastrointestinal Endoscopic Skills (GAGES) tool was developed by expert endoscopists and educators through a multicentre, multidisciplinary trial designed to demonstrate the reliability and validity of this instrument for evaluating basic endoscopic skills [4]. The fundamental skills required for flexible endoscopy were identified and then concentrated into two global assessments: GAGES Upper Endoscopy (GAGES-UE) (Figure 1) and GAGES Colonoscopy (GAGES-C) (Figure 2). The assessments were developed after the Objective Structured Assessment of Technical Skills (OSATS) for open surgery and the Global Operative Assessment of Laparoscopic Skills (GOALS) for laparoscopic surgery. GAGES is easy to administer and can be used to measure the effectiveness of skills training or as a formative tool to provide trainees with specific, objective feedback [11].

The evaluation system was developed and implemented as an additional feature of the Wayne State University Surgery departmental application, a mobile-based application that streamlines intra-departmental communication (Figure 3). The application was developed natively and is maintained for both the ios platform, using X-Code IDE and the Swift programming language, and the Android platform, using Android Studio IDE and the Java programming language. The application is available for ios via the Apple App Store and for Android from the Google Play Store. Data is accessed via a web API access layer hosted as a Microsoft Azure Web App. All data is stored in a Microsoft Azure cloud based SQL database. Utilizing the Microsoft Azure platform for backend services ensures maximum uptime, portability, and scalability. The GAGES score was modified in the phone application from the original description. We added a free text comment section at the end of the form to allow the attending surgeon to provide additional feedback. The date and time stamp were automatically created at the time of the evaluation. We did not allow the attending to fill in the date or time in order to help encourage real time usage of the phone application.

The data collected was the number of evaluations completed by each faculty member, the date and time of each evaluation, and the actual GAGE score for each resident. Descriptive statistics were used to describe the distribution of faculty evaluations and the ratings for each PGY level resident. Ease of use was evaluated by conducting unstructured interviews to the participating faculty.

Results

During the study period, the four attending surgeons completed 239 evaluations on 21 residents. All four attendings (100%) participated in the study. All residents (100%) on service were included. The 239 completed evaluations represented 43% of the total number of endoscopy procedures (n=558) performed by the surgeons. Of the evaluations completed, 163 used the GAGES upper endoscopy form, and 76 used the GAGES lower endoscopy form. Table 1 describes the usage of the application. Attending 1 completed 78 GAGES-UE and 37 GAGES-C; Attending 2 completed 46 GAGES-UE and 16 GAGES-C; Attending 3 completed 22 GAGES-UE and 14 GAGES-C; and Attending 4 completed 17 GAGES-UE and 9 GAGES-C.

Of the 21 residents that received evaluations, 7 of them were PGY-5s, 4 were PGY-4s, 3 were PGY-3s, and 7 were PGY-2s (Table 2). The PGY-5s were evaluated 63 times using GAGES-UE and 45 times using GAGES-C; the PGY-4s were evaluated 28 times using GAGES-UE and 18 times using GAGES-C; the PGY-3s were evaluated 35 times using GAGES-UE and 3 times using GAGES-C; and the PGY-2s were evaluated 37 times using the GAGES-UE and 10 times using GAGES-C. On average, each PGY-5 was evaluated 9 (range 5-16) times using GAGES-UE and 6 (range 4-8) times using GAGES-C; each PGY-4 was evaluated 7 (range 5-8) times using GAGES-UE and 5 (range 1-7) times using GAGES-C; each PGY-3 was evaluated 12 (range 8-14) times using GAGES-UE and 1 (range 1) time using GAGES-C; and each PGY-2 was evaluated 5 (range 3-7) times using GAGES-UE and 1 (range 1-2) time using GAGES-C. Of the 239 evaluations, additional comments were made on 91 (38%) of the evaluations.

Discussion

In summary, this study shows the feasibility of using our real-time smartphone application to evaluate residents on endoscopy skill using GAGES scores. The four attending surgeons in this study were able to complete over 225 evaluations in 6 months on 21 residents. The FES exam was created through a task force formed by SAGES to address the need for consistency in endoscopic training and to measure fundamental endoscopic knowledge and skills [9]. The FES program consists of online materials and didactics; the exam consists of a cognitive component and skills component. All components were designed to measure the skills and knowledge required to perform safe flexible endoscopy [12]. Concern exists that the current ACGME/ABS required case volume is not adequate to guarantee trainees pass the FES manual skills exam, with a reported 25% failure rate for general surgery chief residents who have already completed their required clinical endoscopy experience [6]. That first time FES failure rates holds true among graduating general surgery residents who are pursuing careers specifically in GI surgery and plan to use endoscopy in practice [8]. A clinical experience of at least 105 total endoscopies predicted a passing score on the FES manual skills exam with a sensitivity of 0.92 and specificity of 0.40. A different group has shown that achieving a Global Assessment of Gastrointestinal Endoscopic Skills-Colonoscopy (GAGES-C) score >15/20 is also strongly predictive of passing the FES manual skills exam [13].

Ensuring endoscopic skill proficiency is a major challenge, and currently, there is no standardized method for assessing and monitoring resident’s flexible endoscopy experience. There is also no standardized way of evaluating a resident’s intraoperative experience. The Zwisch scale is being increasingly recognized as an important simple model for teaching and assessing residents in the operating room [1]. The Zwish Model for teaching and evaluating residents in the OR is a standardized approach and nomenclature that informs teaching and codifies performance. The model is comprised of 4 general stages of operative development: show and tell, active help, passive help, and supervision only [1]. The Zwisch scale has been used on an automated smartphone based system [3]. During a 7-month period, 27 faculty completed 1490 operative performance assessments on 31 residents using a smartphone system. The authors concluded that a 1-dimensional global rating scale could be used to collect faculty guidance data that accurately and reliably measure operative performance. Additionally, the integrated smartphone-based method makes it feasible to perform continuous resident intraoperative performance evaluations. Other studies have shown that the use of a smartphone based system for assessment of operative performance does not have a major negative influence on operating room times or operating room satisfaction [14]. The Zwisch scale has also been used to demonstrate that an autonomy gap exists between expectations of procedural autonomy for residents and what actually occurs in the operating room [15].

The Zwisch scale has been incorporated into other innovative, smartphone based tools such as SIMPL (System for Improving and Measuring Procedural Learning) [16]. SIMPL has been shown to easily integrate into surgical training programs to enhance the frequency and timeliness of intraoperative performance assessment. The SIMPL smartphone application has been used by 444 attending surgeons to rate 536 categorical residents in 10,130 procedures in an effort to document the readiness of US general surgery residents for independent practice [17]. The attending surgeons found the use of the application easy, and not that time consuming. The current phone application requires active Internet connection to work. It was noted that in some locations in the hospital there was no Internet available – and thus the scores could not be entered. It was recommended in future iterations of the application that the phone locally stores the scores, and then uploads to central database when a good Internet connection is available.

The current workflow was set-up such that each resident would receive an email every two weeks telling them that they have new GAGES scores available to view. The residents would then click on a link in the email that would lead them to the new scores. Future studies would have to survey and study how frequently the residents check their scores, and how they incorporate this enhanced feedback into their training. The limitation of this study is that it was a small pilot study with only four motivated attending surgeons participating and there was no control to evaluate the advantage of real-time GAGES.

Conclusion

In summary, this study shows the feasibility of using our realtime smartphone application to evaluate residents on endoscopy skill using GAGES scores. Future steps include opening this smart phone application to all members of the department of surgery and use it throughout the residency. Eventually, incorporation of the real time GAGES scores into the 6-month summative evaluation that the Clinical Competency Committee performs would be ideal.

Figure 1

Figure 2

Figure 3

References

1. Da Rosa DA, Zwischenberger JB, Meyerson SL, George BC, Teitelbaum EN, et al. (2013) A theory-based model for teaching and assessing residents in the operating room. J Surg Educ 70: 24-30.
2. Fryer JP, Teitelbaum EN, George BC, Schuller MC, Meyerson SL, et al. (2017) Effect of Ongoing Assessment of Resident Operative Autonomy on the Operating Room Environment. J Surg Educ 2017.
3. George BC, Teitelbaum EN, Meyerson SL, Schuller MC, DaRosa DA, et al. (2014) Reliability, validity, and feasibility of the Zwisch scale for the assessment of intraoperative performance. J Surg Educ 71: 90-96.
4. Vassiliou MC, Kaneva PA, Poulose BK, Dunkin BJ, Marks JM, et al. (2010) Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surgical Endoscopy 24: 1834-1841.
5. Bittner JG, Marks JM, Dunkin BJ, Richards WO, Onders RP, et al. (2007) Resident Training in Flexible Gastrointestinal Endoscopy: A Review of Current Issues and Options. Journal of Surgical Education 64: 399-409.
6. Gardner AK, Scott DJ, Willis RE, Van Sickle K, Truitt MS, et al. (2017) Is current surgery resident and GI fellow training adequate to pass FES? Surgical Endoscopy 31: 352-358.
7. Ritter EM, Taylor ZA, Wolf KR, Franklin BR, Placek SB, et al. (2017) Simulation-based mastery learning for endoscopy using the endoscopy training system: a strategy to improve endoscopic skills and prepare for the fundamentals of endoscopic surgery (FES) manual skills exam. Surgical Endoscopy.
8. Gardner AK, Ujiki MB, Dunkin BJ (2017) Passing the fundamentals of endoscopic surgery (FES) exam: linking specialty choice and attitudes about endoscopic surgery to success. Surgical Endoscopy 2017.
9. Poulose BK, Vassiliou MC, Dunkin BJ, Mellinger JD, Fanelli RD, et al. (2014) Fundamentals of Endoscopic Surgery cognitive examination: development and validity evidence. Surgical Endoscopy 28: 631-638.
10. Vassiliou MC, Kaneva PA, Poulose BK, Dunkin BJ, Marks JM, et al. (2010) Global Assessment of Gastrointestinal Endoscopic Skills (GAGES): a valid measurement tool for technical skills in flexible endoscopy. Surg Endosc 24: 1834-1841.
11. Mueller CL, Kaneva P, Fried GM, Feldman LS, Vassiliou MC, et al. (2014) Colonoscopy performance correlates with scores on the FES manual skills test. Surg Endosc 28: 3081-3085.
12. Vassiliou MC, Dunkin BJ, Fried GM, Mellinger JD, Trus T, et al. (2014) Fundamentals of endoscopic surgery: creation and validation of the hands-on test. Surgical Endoscopy 28: 704-711.
13. Mueller CL, Kaneva P, Fried GM, Feldman LS, Vassiliou MC (2014) Colonoscopy performance correlates with scores on the FES™ manual skills test. Surgical Endoscopy 28: 3081-3085.
14. Fryer JP, Teitelbaum EN, George BC, Schuller MC, Meyerson SL, et al. (2017) Effect of Ongoing Assessment of Resident Operative Autonomy on the Operating Room Environment. Journal of Surgical Education 2017.
15. Meyerson SL, Teitelbaum EN, George BC, Schuller MC, Da Rosa DA, et al. (2014) Defining the autonomy gap: when expectations do not meet reality in the operating room. Journal of surgical education 71: e64-e72.
16. Bohnen JD, George BC, Williams RG, Schuller MC, Da Rosa DA, et al. (2016) The feasibility of real-time intraoperative performance assessment with SIMPL (system for improving and measuring procedural learning): early experience from a multi-institutional trial. Journal of surgical education 73: e118-e130.
17. George BC, Bohnen JD, Williams RG, et al. (2017) Readiness of US General Surgery Residents for Independent Practice. Annals of Surgery 2017.