Predicting academic performance in surgical training

Michael J. Yost, Jeffery Gardner, Richard Mc Murtry Bell, Stephen A. Fann, John R. Lisk, William G. Cheadle, Mitchell Goldman, Susan Rawn, John A. Weigelt, Paula M. Termuhlen, Randy J. Woods, Erick D. Endean, Joy Kimbrough, Michael Hulme

Research output: Contribution to journalArticle

9 Citations (Scopus)

Abstract

Introduction During surgical residency, trainees are expected to master all the 6 competencies specified by the ACGME. Surgical training programs are also evaluated, in part, by the residency review committee based on the percentage of graduates of the program who successfully complete the qualifying examination and the certification examination of the American Board of Surgery in the first attempt. Many program directors (PDs) use the American Board of Surgery In-Training Examination (ABSITE) as an indicator of future performance on the qualifying examination. Failure to meet an individual program's standard may result in remediation or a delay in promotion to the next level of training. Remediation is expensive in terms of not only dollars but also resources, faculty time, and potential program disruptions. We embarked on an exploratory study to determine if residents who might be at risk for substandard performance on the ABSITE could be identified based on the individual resident's behavior and motivational characteristics. If such were possible, then PDs would have the opportunity to be proactive in developing a curriculum tailored to an individual resident, providing a greater opportunity for success in meeting the program's standards. Methods Overall, 7 surgical training programs agreed to participate in this initial study and residents were recruited to voluntarily participate. Each participant completed an online assessment that characterizes an individual's behavioral style, motivators, and Acumen Index. Residents completed the assessment using a code name assigned by each individual PD or their designee. Assessments and the residents' 2013 ABSITE scores were forwarded for analysis using only the code name, thus insuring anonymity. Residents were grouped into those who took the junior examination, senior examination, and pass/fail categories. A passing score of 70% correct was chosen a priori. Correlations were performed using logistic regression and data were also entered into a neural network (NN) to develop a model that would explain performance based on data obtained from the TriMetrix assessments. Results A total of 117 residents' TriMetrix and ABSITE scores were available for analysis. They were divided into 2 groups of 64 senior residents and 53 junior residents. For each group, the pass/fail criteria for the ABSITE were set at 70 and greater as passing and 69 and lower as failing. Multiple logistic regression analysis was complete for pass/fail vs the TriMetrix assessments. For the senior data group, it was found that the parameter Theoretical correlates with pass rate (p < 0.043, B =-0.513, exp(B) = 0.599), which means increasing theoretical scores yields a decreasing likelihood of passing in the examination. For the junior data, the parameter Internal Role Awareness correlated with pass/fail rate (p < 0.004, B = 0.66, exp(B) = 1.935), which means that an increasing Internal Role Awareness score increases the likelihood of a passing score. The NN was able to be trained to predict ABSITE performance with surprising accuracy for both junior and senior residents. Conclusion Behavioral, motivational, and acumen characteristics can be useful to identify residents "at risk" for substandard performance on the ABSITE. Armed with this information, PDs have the opportunity to intervene proactively to offer these residents a greater chance for success. The NN was capable of developing a model that explained performance on the examination for both the junior and the senior examinations. Subsequent testing is needed to determine if the NN is a good predictive tool for performance on this examination.

Original languageEnglish (US)
Pages (from-to)491-499
Number of pages9
JournalJournal of Surgical Education
Volume72
Issue number3
DOIs
StatePublished - Jan 1 2015

Fingerprint

examination
resident
surgery
performance
Internship and Residency
neural network
Names
director
Logistic Models
Education
Certification
Advisory Committees
Curriculum
training program
logistics
Regression Analysis
Group
anonymity
trainee
dollar

All Science Journal Classification (ASJC) codes

  • Surgery
  • Education

Cite this

Yost, M. J., Gardner, J., Bell, R. M. M., Fann, S. A., Lisk, J. R., Cheadle, W. G., ... Hulme, M. (2015). Predicting academic performance in surgical training. Journal of Surgical Education, 72(3), 491-499. https://doi.org/10.1016/j.jsurg.2014.11.013

Predicting academic performance in surgical training. / Yost, Michael J.; Gardner, Jeffery; Bell, Richard Mc Murtry; Fann, Stephen A.; Lisk, John R.; Cheadle, William G.; Goldman, Mitchell; Rawn, Susan; Weigelt, John A.; Termuhlen, Paula M.; Woods, Randy J.; Endean, Erick D.; Kimbrough, Joy; Hulme, Michael.

In: Journal of Surgical Education, Vol. 72, No. 3, 01.01.2015, p. 491-499.

Research output: Contribution to journalArticle

Yost, MJ, Gardner, J, Bell, RMM, Fann, SA, Lisk, JR, Cheadle, WG, Goldman, M, Rawn, S, Weigelt, JA, Termuhlen, PM, Woods, RJ, Endean, ED, Kimbrough, J & Hulme, M 2015, 'Predicting academic performance in surgical training', Journal of Surgical Education, vol. 72, no. 3, pp. 491-499. https://doi.org/10.1016/j.jsurg.2014.11.013
Yost MJ, Gardner J, Bell RMM, Fann SA, Lisk JR, Cheadle WG et al. Predicting academic performance in surgical training. Journal of Surgical Education. 2015 Jan 1;72(3):491-499. https://doi.org/10.1016/j.jsurg.2014.11.013
Yost, Michael J. ; Gardner, Jeffery ; Bell, Richard Mc Murtry ; Fann, Stephen A. ; Lisk, John R. ; Cheadle, William G. ; Goldman, Mitchell ; Rawn, Susan ; Weigelt, John A. ; Termuhlen, Paula M. ; Woods, Randy J. ; Endean, Erick D. ; Kimbrough, Joy ; Hulme, Michael. / Predicting academic performance in surgical training. In: Journal of Surgical Education. 2015 ; Vol. 72, No. 3. pp. 491-499.
@article{59c07d2989f244448145347f78d41cb5,
title = "Predicting academic performance in surgical training",
abstract = "Introduction During surgical residency, trainees are expected to master all the 6 competencies specified by the ACGME. Surgical training programs are also evaluated, in part, by the residency review committee based on the percentage of graduates of the program who successfully complete the qualifying examination and the certification examination of the American Board of Surgery in the first attempt. Many program directors (PDs) use the American Board of Surgery In-Training Examination (ABSITE) as an indicator of future performance on the qualifying examination. Failure to meet an individual program's standard may result in remediation or a delay in promotion to the next level of training. Remediation is expensive in terms of not only dollars but also resources, faculty time, and potential program disruptions. We embarked on an exploratory study to determine if residents who might be at risk for substandard performance on the ABSITE could be identified based on the individual resident's behavior and motivational characteristics. If such were possible, then PDs would have the opportunity to be proactive in developing a curriculum tailored to an individual resident, providing a greater opportunity for success in meeting the program's standards. Methods Overall, 7 surgical training programs agreed to participate in this initial study and residents were recruited to voluntarily participate. Each participant completed an online assessment that characterizes an individual's behavioral style, motivators, and Acumen Index. Residents completed the assessment using a code name assigned by each individual PD or their designee. Assessments and the residents' 2013 ABSITE scores were forwarded for analysis using only the code name, thus insuring anonymity. Residents were grouped into those who took the junior examination, senior examination, and pass/fail categories. A passing score of 70{\%} correct was chosen a priori. Correlations were performed using logistic regression and data were also entered into a neural network (NN) to develop a model that would explain performance based on data obtained from the TriMetrix assessments. Results A total of 117 residents' TriMetrix and ABSITE scores were available for analysis. They were divided into 2 groups of 64 senior residents and 53 junior residents. For each group, the pass/fail criteria for the ABSITE were set at 70 and greater as passing and 69 and lower as failing. Multiple logistic regression analysis was complete for pass/fail vs the TriMetrix assessments. For the senior data group, it was found that the parameter Theoretical correlates with pass rate (p < 0.043, B =-0.513, exp(B) = 0.599), which means increasing theoretical scores yields a decreasing likelihood of passing in the examination. For the junior data, the parameter Internal Role Awareness correlated with pass/fail rate (p < 0.004, B = 0.66, exp(B) = 1.935), which means that an increasing Internal Role Awareness score increases the likelihood of a passing score. The NN was able to be trained to predict ABSITE performance with surprising accuracy for both junior and senior residents. Conclusion Behavioral, motivational, and acumen characteristics can be useful to identify residents {"}at risk{"} for substandard performance on the ABSITE. Armed with this information, PDs have the opportunity to intervene proactively to offer these residents a greater chance for success. The NN was capable of developing a model that explained performance on the examination for both the junior and the senior examinations. Subsequent testing is needed to determine if the NN is a good predictive tool for performance on this examination.",
author = "Yost, {Michael J.} and Jeffery Gardner and Bell, {Richard Mc Murtry} and Fann, {Stephen A.} and Lisk, {John R.} and Cheadle, {William G.} and Mitchell Goldman and Susan Rawn and Weigelt, {John A.} and Termuhlen, {Paula M.} and Woods, {Randy J.} and Endean, {Erick D.} and Joy Kimbrough and Michael Hulme",
year = "2015",
month = "1",
day = "1",
doi = "10.1016/j.jsurg.2014.11.013",
language = "English (US)",
volume = "72",
pages = "491--499",
journal = "Journal of Surgical Education",
issn = "1931-7204",
publisher = "Elsevier Inc.",
number = "3",

}

TY - JOUR

T1 - Predicting academic performance in surgical training

AU - Yost, Michael J.

AU - Gardner, Jeffery

AU - Bell, Richard Mc Murtry

AU - Fann, Stephen A.

AU - Lisk, John R.

AU - Cheadle, William G.

AU - Goldman, Mitchell

AU - Rawn, Susan

AU - Weigelt, John A.

AU - Termuhlen, Paula M.

AU - Woods, Randy J.

AU - Endean, Erick D.

AU - Kimbrough, Joy

AU - Hulme, Michael

PY - 2015/1/1

Y1 - 2015/1/1

N2 - Introduction During surgical residency, trainees are expected to master all the 6 competencies specified by the ACGME. Surgical training programs are also evaluated, in part, by the residency review committee based on the percentage of graduates of the program who successfully complete the qualifying examination and the certification examination of the American Board of Surgery in the first attempt. Many program directors (PDs) use the American Board of Surgery In-Training Examination (ABSITE) as an indicator of future performance on the qualifying examination. Failure to meet an individual program's standard may result in remediation or a delay in promotion to the next level of training. Remediation is expensive in terms of not only dollars but also resources, faculty time, and potential program disruptions. We embarked on an exploratory study to determine if residents who might be at risk for substandard performance on the ABSITE could be identified based on the individual resident's behavior and motivational characteristics. If such were possible, then PDs would have the opportunity to be proactive in developing a curriculum tailored to an individual resident, providing a greater opportunity for success in meeting the program's standards. Methods Overall, 7 surgical training programs agreed to participate in this initial study and residents were recruited to voluntarily participate. Each participant completed an online assessment that characterizes an individual's behavioral style, motivators, and Acumen Index. Residents completed the assessment using a code name assigned by each individual PD or their designee. Assessments and the residents' 2013 ABSITE scores were forwarded for analysis using only the code name, thus insuring anonymity. Residents were grouped into those who took the junior examination, senior examination, and pass/fail categories. A passing score of 70% correct was chosen a priori. Correlations were performed using logistic regression and data were also entered into a neural network (NN) to develop a model that would explain performance based on data obtained from the TriMetrix assessments. Results A total of 117 residents' TriMetrix and ABSITE scores were available for analysis. They were divided into 2 groups of 64 senior residents and 53 junior residents. For each group, the pass/fail criteria for the ABSITE were set at 70 and greater as passing and 69 and lower as failing. Multiple logistic regression analysis was complete for pass/fail vs the TriMetrix assessments. For the senior data group, it was found that the parameter Theoretical correlates with pass rate (p < 0.043, B =-0.513, exp(B) = 0.599), which means increasing theoretical scores yields a decreasing likelihood of passing in the examination. For the junior data, the parameter Internal Role Awareness correlated with pass/fail rate (p < 0.004, B = 0.66, exp(B) = 1.935), which means that an increasing Internal Role Awareness score increases the likelihood of a passing score. The NN was able to be trained to predict ABSITE performance with surprising accuracy for both junior and senior residents. Conclusion Behavioral, motivational, and acumen characteristics can be useful to identify residents "at risk" for substandard performance on the ABSITE. Armed with this information, PDs have the opportunity to intervene proactively to offer these residents a greater chance for success. The NN was capable of developing a model that explained performance on the examination for both the junior and the senior examinations. Subsequent testing is needed to determine if the NN is a good predictive tool for performance on this examination.

AB - Introduction During surgical residency, trainees are expected to master all the 6 competencies specified by the ACGME. Surgical training programs are also evaluated, in part, by the residency review committee based on the percentage of graduates of the program who successfully complete the qualifying examination and the certification examination of the American Board of Surgery in the first attempt. Many program directors (PDs) use the American Board of Surgery In-Training Examination (ABSITE) as an indicator of future performance on the qualifying examination. Failure to meet an individual program's standard may result in remediation or a delay in promotion to the next level of training. Remediation is expensive in terms of not only dollars but also resources, faculty time, and potential program disruptions. We embarked on an exploratory study to determine if residents who might be at risk for substandard performance on the ABSITE could be identified based on the individual resident's behavior and motivational characteristics. If such were possible, then PDs would have the opportunity to be proactive in developing a curriculum tailored to an individual resident, providing a greater opportunity for success in meeting the program's standards. Methods Overall, 7 surgical training programs agreed to participate in this initial study and residents were recruited to voluntarily participate. Each participant completed an online assessment that characterizes an individual's behavioral style, motivators, and Acumen Index. Residents completed the assessment using a code name assigned by each individual PD or their designee. Assessments and the residents' 2013 ABSITE scores were forwarded for analysis using only the code name, thus insuring anonymity. Residents were grouped into those who took the junior examination, senior examination, and pass/fail categories. A passing score of 70% correct was chosen a priori. Correlations were performed using logistic regression and data were also entered into a neural network (NN) to develop a model that would explain performance based on data obtained from the TriMetrix assessments. Results A total of 117 residents' TriMetrix and ABSITE scores were available for analysis. They were divided into 2 groups of 64 senior residents and 53 junior residents. For each group, the pass/fail criteria for the ABSITE were set at 70 and greater as passing and 69 and lower as failing. Multiple logistic regression analysis was complete for pass/fail vs the TriMetrix assessments. For the senior data group, it was found that the parameter Theoretical correlates with pass rate (p < 0.043, B =-0.513, exp(B) = 0.599), which means increasing theoretical scores yields a decreasing likelihood of passing in the examination. For the junior data, the parameter Internal Role Awareness correlated with pass/fail rate (p < 0.004, B = 0.66, exp(B) = 1.935), which means that an increasing Internal Role Awareness score increases the likelihood of a passing score. The NN was able to be trained to predict ABSITE performance with surprising accuracy for both junior and senior residents. Conclusion Behavioral, motivational, and acumen characteristics can be useful to identify residents "at risk" for substandard performance on the ABSITE. Armed with this information, PDs have the opportunity to intervene proactively to offer these residents a greater chance for success. The NN was capable of developing a model that explained performance on the examination for both the junior and the senior examinations. Subsequent testing is needed to determine if the NN is a good predictive tool for performance on this examination.

UR - http://www.scopus.com/inward/record.url?scp=84927121754&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84927121754&partnerID=8YFLogxK

U2 - 10.1016/j.jsurg.2014.11.013

DO - 10.1016/j.jsurg.2014.11.013

M3 - Article

VL - 72

SP - 491

EP - 499

JO - Journal of Surgical Education

JF - Journal of Surgical Education

SN - 1931-7204

IS - 3

ER -