Modernizing assessment of surgical trainees in the workplace

Published:November 11, 2021DOI:https://doi.org/10.1016/j.mpsur.2021.10.004

      Abstract

      Surgical training has evolved significantly over the last few decades. The old model of an informal apprenticeship and ad-hoc mentoring by a single or small team of consultant supervisors has been slowly replaced with formal assessment against a standardized specialty curriculum and annual review of competence by a panel of trainers. The introduction of new surgical curricula from August 2021 has continued this modernization of surgical assessment with greater emphasis on quality of interactions between trainer and trainee. The requirement of a fixed number of work-place based assessments to be completed per year has been dropped. There is a move to competency-based assessment rather than time based. Competencies have been divided into core competencies (Generic Professional Capabilities) and key competencies of a speciality (Capabilities in Practice). These will be assessed by a group of consultant supervisors, and structured and constructive feedback will be provided using the multi-consultant report, a tool designed to give defined feedback that can be actioned by trainees. Trainees will be able to measure their performance against tangible goals and see their progressions towards acquiring the skills required of a day one consultant.

      Keywords

      Introduction

      This article aims to explain how the assessment of surgical trainees has evolved over the past decade and how the new curricula seek to further improve the experience of surgical training for modern trainees.

      What is assessment?

      Assessment is a method used to evaluate, measure and document learning progress, skill acquisition and the educational development needs of a trainee against a defined reference. The reference can be against what is ‘average’ (norm-referenced), for example rating a trainee as ‘satisfactory’ compared to their peers on multi-source feedback.
      • Beard J.D.
      Assessment of surgical skills of trainees in the UK.
      An alternative is that the assessment is criterion-referenced against a set of statements, for example ‘able to perform the procedure under supervision.
      • Beard J.D.
      Assessment of surgical skills of trainees in the UK.
      Assessment can be either summative or formative based on its purpose. Formative or low-stakes assessments are used to provide feedback and aid development. Summative or high-stakes assessments are used for benchmarking or certification.
      For surgical trainees the development of clinical and operative skills is critical to progression towards consultancy. Modern surgical training has been based on Miller's triangle, which defines a simple hierarchy for the acquisition of clinical skills and their assessment.
      • Miller G.E.
      The assessment of clinical skills/competence/performance.
      It describes the process by which a trainee develops from knowledge of a procedure to the ability to perform a procedure (Figure 1).
      • Miller G.E.
      The assessment of clinical skills/competence/performance.
      The highest level of the pyramid is deemed as ‘competence’, which is defined as having sufficient knowledge, judgement, skill and strength for a particular duty.
      • Miller G.E.
      The assessment of clinical skills/competence/performance.
      Figure 1
      Figure 1Millers triangle as it relates to modern surgical assessment.
      • Beard J.D.
      Assessment of surgical skills of trainees in the UK.
      ,
      • Miller G.E.
      The assessment of clinical skills/competence/performance.
      Over time, modern surgical training has moved towards a competence-based assessment. However, current methods of competence-based assessment measure what a surgical trainee can do mostly in controlled aspects of clinical practice, for example in an operating theatre with a supervisor or in skills laboratories. In order to have accurate assessments, we must be sure that our assessment tools are both reliable, valid, acceptable and feasible. For an assessment to be valid, it must test what it claims to test, it must be reproducible between different trainers, it must not harm patients or trainees and it must be feasible within the context of NHS service provision. Validity remains one of the most challenging aspects of surgical assessment; how can we objectively prove that a trainee is ready for consultancy?

      Current modes of assessment

       The Annual Review of Competency Progression (ARCP)

      As part of their training all trainees are assessed against competencies, capabilities, knowledge, skills and behaviours listed in the specialty curricula approved by the General Medical Council (GMC) specific to their specialty training programme.
      The Conference of Postgraduate Medical Deans
      The Gold Guide 8th Edition: a reference guide for postgraduate foundation and specialty training in the UK.
      The overall assessment of a trainee is a combination of supervised learning events, summative assessments by clinical supervisors and a triangulated judgement made by an educational supervisor.
      The Conference of Postgraduate Medical Deans
      The Gold Guide 8th Edition: a reference guide for postgraduate foundation and specialty training in the UK.
      This is formally appraised at an Annual Review of Competency Progression (ARCP).
      An ARCP is designed to provide a mechanism for the submitted evidence of a trainee's performance to be reviewed and for their progress towards the completion of their training programme to be recorded, described as an ‘outcome’ (Box 1). ARCPs usually consist of a panel of trainers, including educational and clinical supervisors, the Postgraduate Dean or representative, and the Training Programme Director (TPD).
      The Conference of Postgraduate Medical Deans
      The Gold Guide 8th Edition: a reference guide for postgraduate foundation and specialty training in the UK.
      To ensure external scrutiny, ARCP panels should also include an external advisor of the same speciality but from another region, the Specialty Advisory Committee (SAC) Liaison member as well as a lay advisor.
      The Conference of Postgraduate Medical Deans
      The Gold Guide 8th Edition: a reference guide for postgraduate foundation and specialty training in the UK.

       Outcomes

      • Outcome 1: Satisfactory progress
      • Achieving the competencies and capabilities in the curriculum at the appropriate rate
      • Outcome 2: Development required, additional training time not required
      • Overall acceptable performance but there are some competencies that have not been fully achieved
      • Outcome 3: Additional training time required
      • The ARCP panel have identified that additional training is required that will extend the duration of the training programme
      • Outcome 4: Released from training programme
      • The panel recommends that the trainee is released from the training programme due to insufficient and sustained lacked of progress despite additional training
      • Outcome 5: Incomplete evidence presented
      • There is outstanding evidence preventing the ARCP panel making a statement about progress
      • Outcome 6: Gained all required competencies to complete the training programme
      • The ARCP panel will recommend either completion of training (for core) or CCT (for higher speciality training) to be awarded
      The ARCP process should be fair and transparent.
      • Hope C.
      • Reilly J.-J.
      • Griffiths G.
      • Lund J.
      • Humes D.
      Factors associated with attrition and performance throughout surgical training: a systematic review and meta-analysis.
      Female surgical trainees in the UK are 11% more likely to receive a non-standard outcome at ARCP than males.
      • Hope C.
      • Lund J.
      • Griffiths G.
      • Humes D.J.
      Differences in progression by surgical specialty: a national cohort study.
      Age was also found to be associated with a higher risk of non-standard outcomes, with the risk increasing by 4% with each year.
      • Hope C.
      • Lund J.
      • Griffiths G.
      • Humes D.J.
      Differences in progression by surgical specialty: a national cohort study.
      It is vital therefore to minimize bias, whether unconscious or not, in the assessment process and by extension that ARCP panels are diverse and represent the population they are assessing. Trainees at risk of a non-standard outcome should have been informed of this prior to the ARCP and should have had the opportunity to discuss this with their educational supervisor and/or TPD.
      The Conference of Postgraduate Medical Deans
      The Gold Guide 8th Edition: a reference guide for postgraduate foundation and specialty training in the UK.
      Trainees have the right to request a review or appeal of the ARCP panel's decision.
      The Conference of Postgraduate Medical Deans
      The Gold Guide 8th Edition: a reference guide for postgraduate foundation and specialty training in the UK.

       Work-based assessments (WBAs)

      Workplace based assessments for surgical trainees include case-based discussions, clinical evaluation exercises, observation of teaching, direct observation of procedural skills, procedure-based assessments and multi-source feedback. These workplace-based assessments serve to assess many of the Generic Professional Capabilities (GPC) expected by the GMC including clinical judgement, communication, team-working, professionalism, compassion and probity.
      • Prakash J.
      • Chatterjee K.
      • Srivastava K.
      • Chauhan V.
      • Sharma R.
      Workplace based assessment: a review of available tools and their relevance.
      In addition, they are intended to assess surgical skills, clinical examination and assessment.
      • Prakash J.
      • Chatterjee K.
      • Srivastava K.
      • Chauhan V.
      • Sharma R.
      Workplace based assessment: a review of available tools and their relevance.
      Case-based discussions (CBD) are designed for a trainer to assess the clinical judgement of a trainee, their application of surgical knowledge and their decision-making process.
      The Intercollegiate Surgical Curriculum Programme
      A CBD is intended to allow the trainee and trainer to critically assess a case for which the trainee has been directly responsible and allow for reflection on the reasoning behind choices that the trainee has made.
      The Intercollegiate Surgical Curriculum Programme
      Clinical evaluation exercises (mini-CEX)is an observational tool for the assessment of an aspect of a clinical encounter in order to facilitate the formal assessment of core clinical competencies.
      • Prakash J.
      • Chatterjee K.
      • Srivastava K.
      • Chauhan V.
      • Sharma R.
      Workplace based assessment: a review of available tools and their relevance.
      It is designed to allow an expert to directly observe and rate the performance of a trainee during a 10–20 minute real clinical encounter.
      The Intercollegiate Surgical Curriculum Programme
      Aspects of a clinical encounter that can be assessed include, but are not limited to, history-taking, clinical examination, communicating with patients and professional and interpersonal skills.
      The Intercollegiate Surgical Curriculum Programme
      Observation of teaching (OOT): A core competency of surgical training is the ability to teach. A good grasp of teaching methods is required by every consultant in order to train the next generation of surgeons. An observation of teaching is intended for an observer to assess formal teaching sessions delivered by a trainee.
      The Intercollegiate Surgical Curriculum Programme
      It does not include ad-hoc teaching on the ward round or bedside teaching sessions on the ward.
      The Intercollegiate Surgical Curriculum Programme
      Direct observation of procedural skills (DOPS) forms the building blocks of early surgical training. DOPS are set at the standard for Core Surgical Training and cover a range of basic diagnostic or interventional procedures or parts of procedures.
      The Intercollegiate Surgical Curriculum Programme
      The skills assessed in DOPS are skills that may be performed in environments outside the operating theatre such as on the ward, acute assessment units or in outpatient clinics as well as inside the operating theatre.
      The Intercollegiate Surgical Curriculum Programme
      Procedure-based assessment (PBA) assess a trainee's operative technical skills combined with their professional skills in the theatre setting. Each operative procedure is divided into a series of competencies within five domains.
      The Intercollegiate Surgical Curriculum Programme
      For the most part, many of the competencies will be common to all procedures, but some competencies will be specific to that procedure.
      The Intercollegiate Surgical Curriculum Programme
      The trainee is also given a global rating from observed procedure only up to the ability to perform the procedure to the standard expected of a consultant.
      The Intercollegiate Surgical Curriculum Programme
      Multisource feedback (MSF), sometimes described as 360-degree feedback, is an anonymized structured feedback of a trainee's performance by their peers and multidisciplinary team-mates. The purpose is to assess many of the GPCs expected by the GMC including clinical judgement, communication, team-working, professionalism, compassion and probity.
      • Prakash J.
      • Chatterjee K.
      • Srivastava K.
      • Chauhan V.
      • Sharma R.
      Workplace based assessment: a review of available tools and their relevance.
      An MSF comprises a self-assessment and a minimum of 12 assessments of the trainee's performance from a variety of professions and the trainee's educational supervisor.
      The Intercollegiate Surgical Curriculum Programme
      Trainees select the colleagues and then are given a feedback report on a 3-point scale including the colleagues' anonymized written comments.
      The Intercollegiate Surgical Curriculum Programme
      The trainee meets with their educational supervisor to discuss their MSF performance and then makes comments and signs off the MSF assessment.
      The Intercollegiate Surgical Curriculum Programme
      The educational supervisor can also recommend a repeat MSF later in the trainee's placement.
      The evidence for WBAs: Proponents of WBAs argue that they display a high level of validity, as the individual is assessed in their true working environment rather than a simulated one.
      • Prakash J.
      • Chatterjee K.
      • Srivastava K.
      • Chauhan V.
      • Sharma R.
      Workplace based assessment: a review of available tools and their relevance.
      It can also can be woven into routine working practice with minimal disruption to service.
      • Prakash J.
      • Chatterjee K.
      • Srivastava K.
      • Chauhan V.
      • Sharma R.
      Workplace based assessment: a review of available tools and their relevance.
      WBAs can be completed by multiple assessors and give structure to ‘naturally occurring learning’ that was sorely lacking in the old apprenticeship model.
      • Prakash J.
      • Chatterjee K.
      • Srivastava K.
      • Chauhan V.
      • Sharma R.
      Workplace based assessment: a review of available tools and their relevance.
      For general surgery the current requirement is 45 validated WBAs per year, an equivalent of one per week. This number has increased exponentially over time, from trainees completing an average of 6 per annum in 2007.
      • Shalhoub J.
      • Santos C.
      • Bussey M.
      • et al.
      Descriptive analysis of the use of workplace-based assessments in UK surgical training.
      Opponents of WBAs point to this as an indication that in their current form they are treated as a ‘tick-box exercise’. A survey of surgical trainees in the UK demonstrated evidence of this; 20% of trainees reported getting an unsatisfactory outcome at ARCP as a sole result of insufficient WBA numbers.
      • Beamish A.J.
      • Johnston M.J.
      • Harries R.L.
      • et al.
      Real-world use of workplace-based assessments in surgical training: a UK nationwide cross-sectional exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training.
      In addition, the methods for completing WBAs can be flawed as a third of WBAs were completed weeks to months after the learning event, and two-thirds of trainees were aware of a trainer allowing a trainee to populate the feedback on their behalf.
      • Beamish A.J.
      • Johnston M.J.
      • Harries R.L.
      • et al.
      Real-world use of workplace-based assessments in surgical training: a UK nationwide cross-sectional exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training.
      In their feedback on the utility of WBAs, 44% of respondents highlighted the volume of WBAs as a critical factor in poor engagement.
      • Beamish A.J.
      • Johnston M.J.
      • Harries R.L.
      • et al.
      Real-world use of workplace-based assessments in surgical training: a UK nationwide cross-sectional exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training.
      A high-quality WBA requires that the learning event is identified as such in advance, that the feedback is given in a timely fashion with engagement of both trainee and trainer and that both are given opportunity to reflect on the event. Recommendations by both trainees and trainers to refine the WBA process have included ensuring adequate IT facilities and dedicated time in the trainer schedule to allow for engagement with WBAs.
      • Beamish A.J.
      • Johnston M.J.
      • Harries R.L.
      • et al.
      Real-world use of workplace-based assessments in surgical training: a UK nationwide cross-sectional exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training.
      ,
      • Phillips A.W.
      • Madhavan A.
      • Bookless L.R.
      • et al.
      Surgical trainers' experience and perspectives on workplace-based assessments.
      Additionally, streamlining the format of the WBA is recommended to reduce generic ‘tickboxes’ and increase free text comments.
      • Beamish A.J.
      • Johnston M.J.
      • Harries R.L.
      • et al.
      Real-world use of workplace-based assessments in surgical training: a UK nationwide cross-sectional exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training.
      As the focus of the topic changes with greater clinical experience, the type of WBA evolves; feedback from senior trainees and trainers indicated that CBDs and PBAs are considered most useful and mini-CEX the least useful over time and that the requirement for a mandatory number of each should change over time.
      • Beamish A.J.
      • Johnston M.J.
      • Harries R.L.
      • et al.
      Real-world use of workplace-based assessments in surgical training: a UK nationwide cross-sectional exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training.
      ,
      • Phillips A.W.
      • Madhavan A.
      • Bookless L.R.
      • et al.
      Surgical trainers' experience and perspectives on workplace-based assessments.

      The new curriculum

      The new surgical curricula which were introduced in August 2021 aim to make surgical training outcomes-based rather than time-based (Figure 2).
      The Intercollegiate Surgical Curriculum Programme
      This means that all teaching and assessment is based around the concept of the trainee becoming ‘the day-1 consultant’, i.e. that all surgical training should prepare the trainee to be able to manage all duties expected for the first day of consultant practice, and training finishes only when they are ready to do so.
      The Intercollegiate Surgical Curriculum Programme
      These duties include managing the unselected take, providing clinical care in the outpatient or ward setting, running operating lists and general multidisciplinary team working.
      The Intercollegiate Surgical Curriculum Programme
      Training will now be arranged into three phases. The aim of phase 1 is to gain the competencies that are currently laid out in the core surgical training curriculum and the indicative time remains 2 years. Phase 2 is to gain experience in the breadth of a speciality and in managing the unselected emergency take.
      The Intercollegiate Surgical Curriculum Programme
      At the end of phase 2, trainees will be eligible to sit the Intercollegiate Specialty Examination of their speciality and the indicative time for this is 4 years.
      The Intercollegiate Surgical Curriculum Programme
      The outcome of phase 3 is to have gained all the capabilities necessary for safe practice as a day-1 consultant.
      The Intercollegiate Surgical Curriculum Programme
      When this outcome is reached a trainee is awarded an Outcome 6 at ARCP.
      The Intercollegiate Surgical Curriculum Programme
      In addition to new phased arrangement of training, the core skills required to be a specialist have been divided into GPCs and Capabilities in Practice (CiPs).
      The Intercollegiate Surgical Curriculum Programme
      GPCs are a framework that set out the essential generic capabilities that are required of clinicians to provide safe and high-quality medical care. CiPs are high-level outcomes that are required by a day-1 consultant.
      The Intercollegiate Surgical Curriculum Programme
      The idea underpinning this division is that the GMC recognized that WBAs were too granular, too detailed and too centred on individual tasks.
      The Intercollegiate Surgical Curriculum Programme
      The five CiPs generic to all surgical specialties are:
      • managing an outpatient clinic
      • managing the unselected emergency take
      • manages ward rounds and the on-going care of in-patients
      • manages an operating list
      • manages multi-disciplinary working.
      Some specialities like paediatric and cardiothoracic surgery have specific CiPs such as assessing and managing an infant.
      The Intercollegiate Surgical Curriculum Programme
      Each CiP integrates many different essential tasks and requires a combination of knowledge, clinical skills, professional skills, technical skills and decision making. When all CiPs are achieved to the level expected of a day 1 consultant, training can be concluded and the trainee will achieve an Outcome 6 at ARCP.
      The Intercollegiate Surgical Curriculum Programme
      To achieve a thorough assessment of CiPs, trainees will now mainly be assessed in the workplace through the multi-consultant report (MCR).
      The Intercollegiate Surgical Curriculum Programme
      The MCR involves clinical supervisors meeting as a group at the mid-point and end of placements to evaluate a trainee's level of required supervision in a number of generic and speciality-specific domains.
      The Intercollegiate Surgical Curriculum Programme
      The aim is to identify if a trainee is requiring an appropriate level of supervision for their stage in training and, if not reaching a supervision level of a day-1 consultant or beyond, which areas need development over the next 6 months of training.
      The Intercollegiate Surgical Curriculum Programme
      There is formatted structured feedback pre-populated which the clinical supervisors can select in order to give trainees specific and actionable feedback.
      The Intercollegiate Surgical Curriculum Programme
      In addition, trainees are required to perform a self-assessment as reflective practice and to identify their own areas for development. Once completed, the MCR is sent to the trainee's educational supervisor and the results are discussed with the trainee. As part of this, the requirement to complete certain numbers of WBAs per year will be dropped and instead trainees will be encouraged to complete them to record experience of critical conditions and index procedures.
      The Intercollegiate Surgical Curriculum Programme
      • Assessment is a method used by trainers to evaluate, measure and document learning progress, skill acquisition and the educational development needs of their trainees
      • An Annual Review of Competency Progression is a formative method of assessment which includes a review of supervised learning events, summative assessments by clinical supervisors and a triangulated judgement made by an educational supervisor
      • Workplace-based assessments include case-based discussions, clinical evaluation exercise, observations of teaching, direct observation of procedural skills, procedure-based assessments and multisource feedback. Each is designed to assess a different aspect of clinical competency
      • The new general surgical curricula reduce the number of workplace-based assessments required by trainees and emphasizes quality learning encounters
      • Trainees are assessed against the Generic Professional Capabilities and Capabilities in Practice that are required of a ‘day-1 consultant’
      • The multi-consultant report aims to provide objective structured feedback against the required Capabilities in Practice of their speciality for a trainee to action

      References

        • Beard J.D.
        Assessment of surgical skills of trainees in the UK.
        Ann Royal Coll Surgeons Engl. 2008; 90: 282-285
        • Miller G.E.
        The assessment of clinical skills/competence/performance.
        Acad Med. 1990; 65: 63-67
        • The Conference of Postgraduate Medical Deans
        The Gold Guide 8th Edition: a reference guide for postgraduate foundation and specialty training in the UK.
        2020 (London)
        • Hope C.
        • Reilly J.-J.
        • Griffiths G.
        • Lund J.
        • Humes D.
        Factors associated with attrition and performance throughout surgical training: a systematic review and meta-analysis.
        World J Surg. 2021; 45: 429-442
        • Hope C.
        • Lund J.
        • Griffiths G.
        • Humes D.J.
        Differences in progression by surgical specialty: a national cohort study.
        medRxiv [Internet]. 2021; ([cited 2021-07-26]. Available from:)
        • Prakash J.
        • Chatterjee K.
        • Srivastava K.
        • Chauhan V.
        • Sharma R.
        Workplace based assessment: a review of available tools and their relevance.
        Ind Psychiatry J. 2020; 29: 200
        • The Intercollegiate Surgical Curriculum Programme
        The new surgical curriculum for August. 2021; ([Internet]. London. [cited 2021 October 1] Available from: https://www.iscp.ac.uk/iscp/curriculum-2021/)
        • Shalhoub J.
        • Santos C.
        • Bussey M.
        • et al.
        Descriptive analysis of the use of workplace-based assessments in UK surgical training.
        J Surg Educ. 2015; 72: 786-794
        • Beamish A.J.
        • Johnston M.J.
        • Harries R.L.
        • et al.
        Real-world use of workplace-based assessments in surgical training: a UK nationwide cross-sectional exploration of trainee perspectives and consensus recommendations from the Association of Surgeons in Training.
        Int J Surg. 2020; 84: 212-218
        • Phillips A.W.
        • Madhavan A.
        • Bookless L.R.
        • et al.
        Surgical trainers' experience and perspectives on workplace-based assessments.
        J Surg Educ. 2015; 72: 979-984