The Effect of Clinical Rotation Difficulty on American Board of Surgery In-Training Examination (ABSITE) Performance

Introduction

The American Board of Surgery In-Training Examination (ABSITE) is given to general surgery residents throughout the United States. It is administered every year at the end of January or early February. ABSITE consists of 250 multiple-choice questions given over 5 hours. Per the American Board of Surgery website, the stated purpose of ABSITE is a formative, not summative, evaluation to monitor resident progress in knowledge of applied science and management of clinical problems related to surgery. It, however, was later demonstrated that residents who score below the 30-35^th percentile are at an increased risk for failing both the American Board of Surgery (ABS) Qualifying Exam (QE) and Certifying Exam (CE).^1,2 Some programs now use the 30th-percentile as a “pass/fail” mark in determining resident promotion and graduation, despite being outside of its intended purpose.³ It is also used as a critical factor during many fellowship applications.^4,5 With the change in United States Medical Licensing Examination (USMLE) Step 1 to “pass/fail” some have discussed the possibility of ABSITE following suit.⁶

There have been few papers to date that evaluate the effects of clinical rotations on ABSITE performance. Marcadis et al. found a significantly positive correlation of total operative cases logged to ABSITE performance.⁷ Cassidy et al. then demonstrated that prior clinical rotations correlated to performance on those relevant sections for interns; such that if an intern had a colorectal rotation they scored better on colorectal questions.⁸ Aljamal et al. found that two-months of acute care surgery prior to ABSITE was associated with significantly improved performance.⁹ Elkbilu et al. found significant improvement for interns having a critical care rotation prior to ABSITE, but not for acute care surgery or trauma.¹⁰ Kandagatla et al. saw no improvement based on having a dedicated breast surgery rotation.¹¹ Residents in a flexible subspecialty focused training program saw no significant difference.¹² It has repeatedly been shown that there is no significant difference based on protected research time.^13–16

The literature to date is heterogenous, although it does appear that the education from exposure during clinical rotations may correlate with ABSITE performance in those fields, particularly in interns who have only had seven rotations prior to their first exam. The exact effect, however, is not entirely understood. To date, there have been no studies to evaluate the effect of rotation difficulty on ABSITE performance.

Materials & Methods

Our program is a hybrid community-academic program that accepts five residents per year. Residents travel on rotations throughout the state of North Dakota, but the primary facility is a Level I trauma center. We performed a 5-year retrospective review of all general surgery interns in our program. The study received IRB approval prior to data collection. Using official residency rotation schedules, it was determined which rotations each intern had completed during the first seven months of the academic year, July-January, prior to ABSITE. These rotations were then compared to ABSITE performance by percentile score. For residents that served a preliminary year and then joined the program as a categorical resident the following year, only their first ABSITE performance as a preliminary resident was included. Interns were excluded if they first completed a preliminary year at another institution prior to starting as an intern here. Only interns taking ABSITE for the first time were included.

Rotation “difficulty” was defined as “hard” or “easy”. All residents were surveyed asking their subjective opinion of each rotation. “Hard” rotations on the survey were defined as having “more demanding clinical duties while at work or more frequently required to stay late” while “easy” rotations were defined as having “less demanding clinical duties while at work and less frequently required to stay late”. Results were determined by simple majority. “Hard” rotations were defined on the survey as having at least 50% of responses grading the rotation as hard. “Easy” rotations were defined on the survey as having over 50% of responses grading the rotation as easy.

ABSITE performance was compared to all rotations completed throughout the first seven months as well as the defined “difficulty” of the rotation. We also compared ABSITE performance based on sex and categorical standing. For categorical residents, regular track vs rural track residents were also compared. ABSITE “Failure” was defined as scoring below the 30^th percentile. Due to the small sample size, a Shapiro-Wilk test was performed to ensure a normal distribution. All quantitative variables were compared using an independent T-test and all qualitative variables were compared using a chi-squared test.

Results

Over the 5-year period, a total of 26 interns took ABSITE for the first time. A Shapiro-Wilk test did not show any evidence of non-normality (W=0.96, p=0.37). Based on this outcome, and after visual confirmation by a Q-Q plot, we decided to use a parametric independent T-test for quantitative variables. Demographics are included in Table 1. The median percentile score was 63%. Average percentile score for males (n=17) was 68.2% and females (n=9) was 51.9% (p=0.089). Categorical residents (n=22) had an average 62.3% while preliminary residents (n=4) had an average 64.0% (p=0.445). Of the categorical residents, interns on a regular track (n=16) had an average percentile score 62.2% compared to rural track interns (n=6) with an average 63.6% (p=0.457).

“Hard” rotations in our institution were defined as Trauma and Acute Care Surgery (TRACS), Surgical Critical Care (ICU) and Cardiothoracic-Vascular Surgery (CT-Vascular). “Easy” rotations were defined as Fargo-General Surgery, Grand Forks-General Surgery, Grand Forks-Acute Care Surgery, Surgical Oncology/Colorectal Surgery and Veteran’s Association (VA). Resident survey results found that for rotations defined as “hard” 80.3% of responses rated “hard” while for rotations defined as “easy” only 13.8% of responses rated “hard” (p < 0.001).

There was no significant difference based on any individual rotation during those first seven months. The effects of “hard” rotations during the first seven months of the academic year is demonstrated in Table 2. There was no difference based on having 1-4 months of “hard” rotations prior to ABSITE. Residents, however, who had five or more months of “hard” rotations prior to ABSITE only scored an average percentile of 33.5% compared to 65.0% for those that did not (p=0.004).

Many residents use a dedicated study period of increased focus in December and January to prepare for the exam. A comparison of ABSITE performance to “hard” rotations during the dedicated study period (December and January) is given in Table 3. Having a “hard” rotation in January alone was not associated with significant change in ABSITE performance, nor was having a “Hard” rotation in January or December. Residents having “hard” rotations in both December and January, however, had significantly lower ABSITE performances. These residents scored an average percentile of 37.3% compared to 65.9% for those that did not (p=0.009). Residents with “hard” rotations in both December and January had a 66.7% risk for “failing” ABSITE compared to only 4.3% risk for those that did not, odds ratio 15.3 (p=0.001).

Discussion

For interns taking their first ABSITE examination, there does appear to be correlation to the difficulty of clinical rotations completed in the first seven months of residency. In our institution, the “hard” rotations are defined as TRACS, ICU and CT-Vascular. During these rotations, residents have more demanding clinical duties and are more frequently required to stay late. They, therefore, have less free time to allow studying outside of clinical duties.

No individual rotation determined performance. Residents having up to half of their initial clinical rotations being “hard” saw no significant difference in ABSITE performance. But once residents started having more than four “hard” rotations of the first seven, a significant decline in performance was seen. These residents obtained on average half the percentile of those that did not.

Many residents use December and January as a dedicated study period in which studying is intensified with increased focus. It is during these months that there seems to be the largest impact on ABSITE performance. There was no significant difference if residents had a single “hard” rotation in either December or January of the dedicated study period. However, residents that had a “hard” rotation in both December and January had significantly worse performances with a nearly 30-percentile average decrease. These residents had a 66.7% risk for “failing” ABSITE, 15 times greater than those who had at least one “easy” rotation during those months.

Programs could consider the time intensity of specific clinical rotations when creating intern schedules. Each program will have different “hard” rotations that require more time and energy. Each rotation will have different allowances for study outside of clinical duties. Interns still need the “hard” rotations, although they could be limited to no more than four of the first seven months, with other less difficult rotations as well. During the dedicated study period of December and January, interns could be scheduled to have only one “hard” rotation with at least one “easy” rotation to permit focused study.

Although these considerations could be implemented to possibly improve performance, the question should be asked if they should be. As stated, ABSITE is designed as a formative, not summative, evaluation and should be considered as such. Although clinical rotations could be optimized for ABSITE performance, a program should primarily determine schedules with the goal of producing excellent trained surgeons, not simply excellent test-takers. If schedule making allows for such provisions it would be reasonable to permit, although performance on ABSITE should not be the primary focus.

An unanticipated finding was the difference between male and female performance. Although it was not to the level of significance (p=0.089), males scored on average 16.3 percentile points higher than females. This difference primarily seemed to be at the upper-end of performers. The top 26% of scores were all from males while the remaining were more evenly distributed. Only one female “failed” during this time and females were not disproportionately among the lowest scorers. For the remaining analysis, there was no significant difference between males and females in regards to “hard” rotations. Although it was not significant, this finding should warrant further study.

There are some limitations to this review. First, our program is a hybrid program with many surgical specialties combined into the majority of rotations. This is opposed to some academic institutions that may have a wider variety of more specialized rotations. Second, our program as a whole performed well over this time period with an average percentile of 62.6%. A program with lower-performing residents may see different results. Third, every program will have different “hard” rotations which may have a different effect on performance. Fourth, just because a resident was on a rotation that is generally busier does not necessarily mean that it was busy during the month that they were on rotation. Similarly, just because a rotation generally allows more free time does not automatically ensure that the residents actually used that free time for studying.

The next step is to evaluate the effects of clinical rotations to residents of all post-graduate years, not just interns. We believe that the limited study time from “hard” rotations would have a lesser effect in more senior residents who have already prepared for ABSITE in prior years, although data would be needed to support the claim. Another step is to compare the effects of all clinical rotations throughout residency to the American Board of Surgery (ABS) Certification examination. With the formative intent of ABSITE, ABS certification may serve a more clinically significant outcome.

Conclusions

There appears to be a negative correlation to ABSITE performance based on the “difficulty” of rotations with less free time outside of clinical duties. Much of the information on ABSITE is not often seen in clinical practice. The education gained through clinical rotation appears to have a smaller effect on ABSITE performance than does individual study. Clinical rotations may make more of an impact indirectly based on the amount of time permitted to study than the actual education they obtain in the rotation. Programs may be able to optimize their intern rotation schedules to improve ABSITE performance.

Declaration of Conflicting Interests

The authors declare that they have no conflicts of interest.