Journal Information
Visits
12
Vol. 49. Issue 6.
(June 2025)
Original article
Full text access
Effect of a competence based medical education program on training quality in Intensive Care Medicine. COBALIDATION TRIAL
Visits
12
Álvaro Castellanos-Ortegaa,
, María Jesús Broch Porcara, Diego Palacios-Castañedab, Vicente Gómez-Telloc, Miguel Valdiviab, Carlos Vicenta, Isabel Madrida, Nuria Martinezb, Manuel José Párragad, Elena Sanchoa, María del Castillo Fuentes-Durae, Rafael García-Rosf
a Intensive Care Department. Hospital Universitario y Politécnico La Fe, Valencia, Spain
b Intensive Care Department, Hospital Puerta de Hierro-Majadahonda, Spain
c Intensive Care Department, University Hospital Moncloa, Madrid, Spain
d Intensive Care Department, Hospital Universitario Morales Meseguer, Murcia, Spain
e Department of Methodology of the Behavioral Sciences, University of Valencia, Spain
f Department of Developmental and Educational Psychology, University of Valencia, Spain
This item has received
Article information
Abstract
Full Text
Bibliography
Download PDF
Statistics
Tables (5)
Table 1. Criteria to determine de levels of competence considered in the study.
Tables
Table 2. Hospitals participants (ICU Departments), number of residents/year accredited, number of residents participants in the baseline and in the final OSCE, number of formative assessments performed and number of CoBaTrICE competences assessed in each resident of the CoBaTrICE group.
Tables
Table 3. Final OSCE. Inter-rater reliability: Intraclass Correlation Coefficient for the total scores given by the raters, Kappa for the Competence Level achieved.
Tables
Table 4. Number of performances, total scoring, percentage of CEPE and CNEPE completed, and competency level achieved in both groups CoBaTrICE (experimental group) and traditional (control group) in the five final OSCE scenarios.
Tables
Table 5. Distribution of residents by the levels of competence achieved in the OSCE: CoBaTrICE group, control group, and total participants.
Tables
Show moreShow less
Additional material (1)
Abstract
Objectives

The main objective of this study was to evaluate whether the implementation of CoBaTrICE (Competency-Based Training in Intensive Care Medicine in Europe) provides higher levels of competency in comparison with the current official time-based program in Intensive Care Medicine in Spain. Secondary objectives were: 1) To determine the percentage of critical essential performance elements (CEPE) accomplished, 2) To determine compliance with workplace-based assessments (wba).

Design

Multicenter cluster randomized trial.

Setting

Thirteen Spanish ICU Departments.

Participants

Thirty-six residents

Intervention

The implementation of CoBaTrICE included: (1) Training the trainers; (2) Wba; (3) The use of an electronic portfolio. The level of competency achieved by each participant was determined by a simulation-based Objective Structured Clinical Exam (OSCE) performed at the end of the 5th year of training period.

Main variables of interest

Total scoring in the five scenarios, CEPE completed, level of competency (1–5) achieved.

Results

A total of 119 performances from 26 residents (17 from CoBaTrICE group and 9 from control group) were analyzed in the OSCE. CoBaTrICE residents´ achieved higher levels of competency [2 (1−5) vs. 2 (1−3), p = 0.07) and higher percentages of CEPE´s accomplishment than the control group (78% vs. 71%, p = 0.09).

Conclusions

The CoBaTrICE group showed a better performance trend in comparison to the control group, but the differences were not statistically significant. Since the number of Wba performed was low, additional research is needed to determine the potential superiority of CoBaTrICE.

Keywords:
Training
Assessment
Simulation
Competency-based education
Resumen
Objetivos

El objetivo principal del estudio fue evaluar si la implementación de CoBaTrICE (Competency-Based Training in Intensive Care Medicine in Europe) proporciona niveles de competencia superiores al programa oficial español actual de medicina intensiva basado en rotaciones. Objetivos secundarios: 1) determinar la proporción de elementos críticos de desempeño esenciales (CEPE) completados, 2) determinar el cumplimiento de CoBaTrICE.

Diseño

Ensayo multicéntrico aleatorio de tipo conglomerado.

Ámbito

Trece departamentos de medicina intensiva españoles.

Participantes

Treinta y seis residentes.

Intervención

La implementación de CoBaTrICE incluyó: (1) curso de feedback para tutores; (2) realización de evaluaciones formativas en la UCI; (3) uso de un portafolio electrónico. El nivel de competencia de cada participante se determinó mediante un examen clínico objetivo estructurado (ECOE) basado en simulación realizado al final de la residencia.

Principales variables de interés

Puntuación total en los cinco escenarios, CEPE completadas, nivel de competencia alcanzado (1 a 5).

Resultados

Se analizaron un total de 119 actuaciones de 26 residentes (17 del grupo CoBaTrICE y 9 del grupo control). Los residentes de CoBaTrICE consiguieron niveles más altos de competencia [2 (1–5) vs. 2 (1–3), p = 0,07] y mayores porcentajes de CEPE que el grupo control (78% vs. 71%, p = 0,09).

Conclusiones

El grupo CoBaTrICE mostró una tendencia a un mejor desempeño que el grupo control, pero las diferencias no fueron estadísticamente significativas. Dado que el número de evaluaciones realizadas fue muy bajo, se necesita investigación adicional para determinar la posible superioridad de CoBaTrICE.

Palabras clave:
Formación
Evaluación
Simulación
Formación en competencias
Full Text
Introduction

The current postgraduate medical training model in most European countries is the so called “time-based training”.1–3 This paradigm assumes that mere exposure to clinical experiences based on temporary rotations through different Departments suffices to acquire the necessary professional competencies. Opportunistic learning and volume of practice rather than learning guided by objectives defines competency. Certification depends on a logbook of cases, and a generic, subjective report about knowledge, technical and non-technical skills acquired by the resident, which is performed after every rotation and yearly by the tutors and the head of the ICU Department. The effectiveness of this model of training to provide the adequate level of competency to the trainees in intensive care medicine (ICM) has not been determined. To overcome these limitations, a new model based on the gradual acquisition of clearly defined, observable and measurable competencies is gaining acceptance.4 Competency based medical education (CBME) proposes more solid principles such as: (a) defining the learning outcomes that must be shown by residents at the end of their training period; (b) focusing attention upon the development and demonstration of skills, attitudes and knowledge acquired by residents in the course of the training process;(c) prioritizing training evaluation and constructive feedback centered on the performance levels of the residents in the real-life working context; and (d) using a broad range of evaluation tools and methods.5 The CBME model for ICM in Europe is called CoBaTrICE (Competency Based Training in Intensive Care Medicine in Europe) (https://www.esicm.org/education/cobatrice/) an international partnership of professional organizations and critical care clinicians whose ultimate aims are to assure a high-quality level education in ICM, to harmonize training in ICM without interfering with national specific regulations and to allow for free movement of ICM professionals across Europe.6–10 The competences of CoBaTrICE have been adopted by Union Européenne des Médecins Spécialistes (UEMS) as Common Training Framework10 as well as into the training programs of several European countries.1 Spain is planning to do so.11,12 Research in this field is still limited because the model has been applied in a partial and scantly structured manner13 therefore, whether CBME programs provide better outcomes than the traditional ones is unknown. The main objective of this study was to evaluate whether the implementation CoBaTrICE provides higher levels of competency at the end of the training period in comparison with the current official time-based program in ICM in Spain. Secondary objectives were: (1) To determine the percentage of critical essential performance elements (CEPE) accomplished; (2) To determine compliance with CoBaTrICE and barriers to an effective implementation. Since the traditional examinations typically measure knowledge and performance rather than competency in the complex situations of ICM, we chose a simulation-based objective structured clinical examination (OSCE) to assess non-technical and therapeutic management skills as well as the ability to integrate knowledge, judgment, communication, and teamwork into the simulated practice setting.14–18

Materials and methodsDesign and setting

ICM in Spain is a five–year primary specialty divided in two stages: Stage 1 consists of an initial two-year block (R1-R2) of training that is spent in anaesthesia and medicine; Stage 2 consists of a three-year block (R3-R5) that covers general and specific ICM training in a variety of “special” areas including coronary care, polytrauma, paediatric, neurosurgical, post-transplant and cardiothoracic ICM.19 There is an official program that specifies the clinical rotations to be carried out each year of training. The transition from one year to the next one of training is held every May 31.

We conducted a multicenter cluster randomized trial of 14 ICU Departments from 14 academic referral hospitals located in Spain.

Participants

There were 36 R3 consent participants belonged to 13 ICU Department´s from 13 academic referral hospitals located in Spain, one hospital eventually declined to participate in the OSCE and was excluded from the study. They were followed during two years of their specific training period in ICM, from the end of their third year (R3) to the end of the fifth year (R5). CoBaTrICE (7 hospitals, 20 residents) were compared with the current time-based model of training in ICM in Spain (6 hospitals, 16 residents). Enrolment started on February 1, 2019, and the follow-up ended on May 31, 2021. The participating ICUs are general medical and surgical ICU accredited to train 2−3 new residents in ICM per year, being 3 residents the highest number per year allowed in each hospital in Spain.

ICU Departments where CoBaTrICE was being implemented using the three core elements of the intervention (see below) were excluded.

The study was approved by the ethics committee of the Instituto de Investigación Sanitaria La Fe. After obtaining informed consent, participants who volunteered for the study were allocated to the simulation scenarios.

Intervention

The implementation of CoBaTrICE included the three following essential elements20: (1) Training the trainers, tutors received a 12 -h course in formative assessment, debriefing techniques, and effective feedback. (2) Workplace based assessments (Wba) to promote learning and to guarantee that the predefined competences and skills are effectively acquired. Current Wb observation methods such as mini-clinical examination exercise, direct observation of procedural skills and acute care assessment tool were used. (3) The use of an electronic portfolio specifically created as a personal collection of training documents that includes the record of formative assessments, activities, and the levels of competence achieved (http://estudioportres.i3net.es/cms/home_tutor.php).

The level of competency achieved by each participant was determined by a simulation based OSCE performed at the end of the third year of training (baseline) and at the end of the fifth year of the training period. The OSCEs were performed in April 2019 and May 2021 respectively, at four simulation centers geographically close to participant hospitals: Hospital la Fe, Valencia; IAVANTE, Granada; Hospital Clinic, Barcelona; University Francisco de Vitoria, Madrid. Each participant performed in five 15-min, standardized patient, or high-fidelity simulated clinical crisis scenarios.

Designing five standardized scenarios and rating instruments

A more detailed description can be found in reference number.21 In summary: via a Delphi technique, an independent panel of 10 intensivists subject matter experts performed the following tasks: (1) To select the CoBaTrICE competences to be assessed. (2) To design the five scenarios. (3) To define the items of the checklist for each scenario that were classified as follows: a) the critical essential performance elements (CEPE), and b) the critical non-essential performance elements (CNEPE) that must be observed and scored in a yes/no format. CEPE are defined as essential steps or actions in the management of the patient which if missed could have an immediate significant impact on morbidity and mortality. CNEPE are also important for the adequate management of the patient, but they don´t have an immediate influence on the outcome. There were 7–12 CEPE and 13–15 CNEPE in each scenario (see Table 1S Supplemental Data).

The performances were video recorded.22 All the video recordings were uploaded and hosted in the Cobalidation web-based platform. The videos were randomly assigned and then rated by two blinded raters, members of the experts panel using the specific checklists with a detailed description of the competencies technical (diagnosis and treatment) and non-technical (communication, team leadership, resource management) associated with each item, as well as specific information about what is expected to be done by the trainee. After each video-assessment, the performance of the resident was classified in five levels of competency (Table 1).

Table 1.

Criteria to determine de levels of competence considered in the study.

Level  CEPE/CNEPE performed appropiately  Level of autonomy/support needs 
Less than 60% of the CEPE  The participant needs guidance and direct supervision in all cases. 
Equal or more than 60% but less than 80% of the CEPE  The participant needs guidance and supervision in most of situations 
Equal or more than 80% but less than 100% of CEPE  The participant needs some guidance and supervision in complex situations 
All CEPE (100%) but less than 60% of CNEPE  The participant can perform the activity under indirect supervision. 
All CEPE (100%) and ≥ 80% of the CNEPE  The participant is independent to perform the activity. 

CEPE: critical essential performance elements; CNEPE: critical non-essential performance elements.

Measures included: (1) the percentage of CEPE completed; (2) the percentage of CNEPE completed; (3) the competency level achieved in each scenario; (4) the total scoring achieved in each scenario which was calculated as follows:

Total score (range 0–100) = (number of CEPE completed × 2 points + number of CNEPE completed × 1 point) × 100/potential maximum score achievable.

Standardization of Scenario Delivery.15–17,23

The design of the scenarios involved the use of “high fidelity mannekins” (Meti HPS® and iStan®), and “standardized patient actors” along with airway task trainer to perform intubation when indicated.

Main outcomesPrimary outcome

The competency level achieved on a descriptive scale of 1–5 (novice to independent practitioner) in each scenario at the end of the 5th year.

Secondary outcomes

  • 1

    Percentage of CEPE completed in the OSCE.

  • 2

    Percentage CNEPE completed in the OSCE.

  • 3

    The total score achieved in each scenario.

  • 4

    Compliance with CoBaTrICE implementation in the intervention group defined as the number of Wba performed and the number of CoBaTrICE competences assessed throughout the two-years of follow-up.

Statistical analysis

It was estimated that 115 performances would have 80% power (2-sided α = 0.05) to detect a difference in the level of competence achieved of one level on the 5-level scale. Whilst there is no information available on this issue, we postulate that one level would represent a minimally important difference.

Results are shown as median and maximum-minimum range. To compare continuous variables, the Mann–Whitney U test was used. All tests were two-tailed, and p < .05 was predetermined to define statistical significance. Inter-rater reliability for the OSCE scenarios at item level was estimated with Fleiss kappa coefficient and their 95% confidence intervals, and for the scores, with the Intraclass Correlation Coefficient (ICC) and their 95% confident intervals.

All Analysis were performed using SPSS statistical package version 23.0 (SPSS Inc., Chicago, IL).

Results

A total of 295 performances were analysed [176 in the baseline OSCE (36 residents)] and 119 in the final OSCE [26 residents (17 residents of the CoBaTrICE group and 9 of the control group)]; 3 R5 of the CoBaTrICE group and 7 R5 of the control group did not perform the final OSCE. In the final OSCE, the participants were 11 male and 15 female, mean age 31.5 ± 2.8 (range 29–40) years from 11 ICU Departments of 11 Spanish teaching hospitals (Table 2). Eleven video-records of scenarios 2 (6), 3 (4), and 4 (1) were either lost or discarded for low quality. The distribution of the participants in the OSCE in each of the four simulation centers was as follows: Barcelona, 7; Madrid, 5; Granada, 4; and Valencia, 10.

Table 2.

Hospitals participants (ICU Departments), number of residents/year accredited, number of residents participants in the baseline and in the final OSCE, number of formative assessments performed and number of CoBaTrICE competences assessed in each resident of the CoBaTrICE group.

Hospitals participants  Residents/year N  Residents participants in the baseline OSCE N  Residents participants in the final OSCE N  Residents, number of formative assessments (number of CoBaTrICE competences assessed)  Group 
Consorci Corporació Sanitária Parc Taulí, Sabadell, Barcelona.  R1 = 2 (27)R2 = 13 (70)R3 = 7 (26)  CoBaTrICE 
Hospital Clínico Universitario de Valencia, Valencia.  R1 = 3 (34)R2 = 8 (32)  CoBaTrICE 
Hospital Clínico San Carlos, Madrid.  R1 = 1 (1)R2 = 0 (0)R3 = 0 (0)  CoBaTrICE 
Hospital Clínico Universitario Virgen de la Arrixaca, Murcia.    Control 
Hospital General Universitario de Alicante, Alicante.  R1 = 2 (23)R2 = 0 (0)R3 = 6 (51)  CoBaTrICE 
Hospital Universitario Doctor Peset, Valencia    Control 
Hospital Universitario 12 de Octubre, Madrid.    Control 
Hospital Universitario Germans Trias i Pujol, Badalona, Barcelona    Control 
Hospital Universitario Doctor Negrín. Las Palmas de Gran Canaria.  R1 = 1 (3)R2 = 13 (74)R3 = 6 (43)  CoBaTrICE 
Hospital Universitario La Paz, Madrid.    Control 
Hospital Universitario Virgen de la Nieves, Granada.    Control 
Hospital Universitario Virgen del Rocío, Sevilla.  R1 = 0 (0)R2 = 0 (0)R3 = 0 (0)  CoBaTrICE 
Hospital Universitario Vall d´Hebron, Barcelona.  R1 = 1 (3)R2 = 1 (3)R3 = 1 (3)  CoBaTrICE 

CoBaTrICE: Competency Based Training in Intensive Care Medicine in Europe.

Inter-rater reliability

Intraclass Correlation Coefficient (ICC) for the total scores given by the raters, and Fleiss’ Kappa for the competence level achieved are shown in Table 3. Fleiss’ Kappa across scenarios ranged between 0.278 (scenario 4) and 0.432 (scenario 2). Median two-way random effect model ICC ranged from 0.422 (scenario 1) to 0.862 (scenario 2).

Table 3.

Final OSCE. Inter-rater reliability: Intraclass Correlation Coefficient for the total scores given by the raters, Kappa for the Competence Level achieved.

Scenario  ICC (95 CI)  Kappa 
1.High risk postoperative management.  .422 [-.392, 760]  .280 [.022, .538] * 
2.Management of septic shock, ARDS, and endotracheal intubation.  .862 [.651, .945]  .432 [.117, .747] ** 
3.Initial assessment and management of the multiple-trauma patient.  .788 [.489, .912]  .307 [.042, .571] * 
4.Cardiogenic shock and cardiopulmonary resuscitation.  .550 [-.023, .800]  .278 [.021, .535] * 
5.Neurocritical care.  .699 [.327, .865]  .302 [.066, .538] * 
Total  .748 [.636, .825]  .346 [.225, .467] *** 

*p < .05; **p < .01; *** p < .001***.

ARDS: Acute Respiratory Distress Syndrome; ICC: Intraclass Correlation Coefficient; CI: confidence interval; OSCE: simulation-based objective structured clinical examination.

Compliance with the implementation of the program in the intervention group. During the time frame (two years of CoBaTrICE implementation), the number of formative assessments performed in the intervention group was 62 [median 2, range (0–13)], the number of CoBaTrICE competencies assessed was 392 [median 24 (0–74)]. Only in 6 out 20 residents of the CoBaTrICE group were assessed at least 30% of the competences of the CoBaTrICE program (Table 2).

Residents’ performance

The performance was video recorded for later scoring by blinded trained raters. Eight performances (6,7%) needed a third rater due to a discrepancy in two or more CEPE completed between the two raters. The overall results of the 119 performances in the five crisis management scenarios are shown in table S2. The comparison between groups is shown in Table 4. Residents of CoBaTrICE group achieved higher levels of competency in comparison to the control group [level 2 (1–5) vs. level 2 (1–3), p = 0.07)], levels higher than 2 were achieved in 43.4% (33/76) of the CoBaTrICE group vs. 27.9% (12/43) in the control group (Table 5). The median score of the CoBaTrICE group was 74 points out of 100 (range 40–94) vs. 69 points (range 49–92) in the control group, p = 0.13. The median of CEPE completed in the CoBaTrICE group was 78% (range 45–100) vs. 71% (range 56–95) in the control group (p = 0.09). The residents of the CoBaTrICE group completed more CEPE than the residents of the control group in all scenarios, being the differences in the scenario 2 almost statistically significant (p = 0.06). CEPE were more frequently completed in the scenario 1 (86%, range 64–100) and less frequently completed in the scenario 2 (66%, range 50−100).

Table 4.

Number of performances, total scoring, percentage of CEPE and CNEPE completed, and competency level achieved in both groups CoBaTrICE (experimental group) and traditional (control group) in the five final OSCE scenarios.

  Performances NTotal ScoreCEPE completed (%)CNEPE completed (%)Competency Level achieved
Scenarios  CoBaTrICE  Control  CoBaTrICE  Control  CoBaTrICE  Control  CoBaTrICE  Control  CoBaTrICE  Control 
1.High risk postoperative management,  17  78 (56−85)  78 (60−84)  .71  86 (64−100)  79 (71−86)  .82  70 (36−85)  77 (42−89)  .28  3 (2−4)  2 (2−3)  .49 
2.Management of septic shock, ARDS, and endotracheal intubation  12  66.5 (40−86)  57.5 (49−68)  .13  75.5 (50−100)  61.5 (57−69)  .06  57.5 (24−79)  51 (41−72)  .67  2 (1−4)  2 (1−2)  .16 
3.Initial assessment and management of the multiple-trauma patient.  14  70.5 (44−94)  67.5 (57−91)  .97  73 (45−100)  72.5 (62−95)  .76  60 (35−85)  63 (50−85)  .76  2 (1−5)  2 (2−3)  .87 
4.Cardiogenic shock and cardiopulmonary resuscitation.  16  75.5 (58−89)  76 (61−83)  .51  78 (56−95)  73 (56−89)  .42  73.5 (47−87)  73 (60−92)  .89  2 (1−3)  2 (1−3)  .63 
5.Neurocritical care  17  80 (59−92)  71 (54−92)  .20  76 (63−100)  69 (57−94)  .29  82 (50−100)  76 (48−100)  .26  2 (2−4)  2 (1−3)  .52 
Overall ratings  76  43  74 (40−94)  69 (49−92)  .13  78 (45−100)  71 (56−95)  .09  70 (24−100)  69 (41−100)  .59  2 (1−5)  2 (1−3)  .07 

CEPE: Critical essential performance elements; CNEPE: Critical non-essential performance elements. CoBaTrICE: Competency Based Training in Intensive Care Medicine in Europe; ARDS: Acute Respiratory Distress Syndrome; OSCE: simulation-based objective structured clinical examination; ARDS: Acute Respiratory Distress Syndrome.

Results are shown as median and range (in brackets).

*p < .05.

Table 5.

Distribution of residents by the levels of competence achieved in the OSCE: CoBaTrICE group, control group, and total participants.

Scenarios OSCE  Competency Level achieved
  CoBaTrICE group  Control group  Total participants
Competency Level  II  III  IV    II  III  IV    II  III  IV 
1.High risk postoperative management,      12  13 
2.Management of septic shock, ARDS and endotracheal intubation     
3.Initial assessment and management of the multiple-trauma patient      14 
4.Cardiogenic shock and cardiopulmonary resuscitation.      13  10 
5.Neurocritical care  10      15 
Total  37  29    26  12    11  63  41 

OSCE: simulation-based objective structured clinical examination; CoBaTrICE: Competency Based Training in Intensive Care Medicine in Europe; ARDS: Acute Respiratory Distress Syndrome.

Discussion

In this multicentric cluster-based randomized trial which main objective was to evaluate whether the implementation of CoBaTrICE provides higher levels of competency in comparison to the current official time-based program in ICM in Spain, there were not significant differences between both models in the level of competency achieved by the R5 participants in the simulated-based OSCE. Nevertheless, despite the low number of Wba performed, residents from the CoBaTrICE group showed a better trend in performance, they achieved a higher level of competency in comparison to the control group, levels higher than 2 were achieved in 43.4% of the performances in the CoBaTrICE group vs. 27.9% in the control group (p = 0.07), and they completed more CEPE than residents of the control group in all scenarios (p = 0.09). Results of the baseline OSCE did not show differences between both groups regarding the primary and secondary objectives of the study.21

To the best of our knowledge, no publications have been made involving the comparison of the effectiveness of the two current training models in postgraduate medicine. CBME stands for a shift in emphasis away from the more traditional time-based programs, based solely on exposure to experiences such as clinical rotations, in favor of a model which is a learner-centered approach that emphasizes achieving specific outcomes. The implementation of CBME is challenging because requires organizational and cultural changes, resources, particularly more dedication of teaching time, as well as the training of tutors and staff members in formative assessment and feedback techniques.20,24–26 Frequent formative Wba as well as the record of the learning experiences in an electronic portfolio are the essential elements to promote learning, self-reflection, progression, and ultimately to guarantee that the predefined competences and skills are effectively acquired.20,25–27 Reliability in achievement of competencies correlates directly with the number of observations in as many different contexts as possible, these frequent assessments have been described as a burden on residents and trainers alike, being the main barrier to the implementation of CBME.28 The COVID-19 pandemic put a huge strain on critical care resources worldwide, as systems struggled to provide high-quality care for a surge of critically ill patients. This exceptional situation that occurred just during the study period (2020–2022) might have influenced the low number of Wba performed in the CoBaTrICE group preventing the trial from showing a significant difference between both groups.

Regarding the performance of the residents, our data raise concerns and considerations around the development for some CEPE. Only 38% of the participants achieved level 3 or higher, and, as previously seen in the baseline OSCE there was a great variability, many residents failed to apply a complete protocol of endotracheal intubation in a high-risk critical patient, which can be explained by the findings of the recently published prospective study of tracheal intubation in critically ill adults in Spain.29 Although there are not similar studies performed in ICM, our results are not very different from those obtained in studies conducted to assess performance of anesthesia professionals using simulation.14,17,30,31 Weinger et al.17 in their study of 268 board-certified anesthesiologists found that CEPE were commonly omitted. Schumacher et al.32 in their study performed at the end of the Pediatrics residency found that the level of “practice without supervision” was achieved only in 47% of the competencies.

Our findings suggest that more work may be needed to ensure that graduates of intensive care residency training can be trusted for independent clinical practice. In our view, CBME is the way, as it focusses on clinical excellence translating the needs of contemporary society for improved health care into competencies that must be mastered by motivated well-trained intensivists.33,34

Limitations

Unfortunately, the study coincided with the COVID 19 pandemic what might have conditioned the low number of Wba performed in the CoBaTrICE group and the reduction of participants in the final OSCE, which in turn may have underpowered the trial to achieve significant results.

Simulation is being more and more used as a tool for high-stakes assessments,35–37 however, there is little documentation of a relationship between simulation performance and performance in the clinical setting.38–40 It has been shown that those with more training and experience perform better in the scenarios, suggesting that simulation-based assessments may ultimately prove useful as an indicator that they are ready for unsupervised practice in the real world.14,18,30

The definitions of the levels of competency were stablished by a consensus of a panel of experts in ICM. We expected residents to meet the standard of unsupervised practice at the completion of training in all 5 scenarios, surprisingly, only 5 performances achieved levels 4 or 5, suggesting that either training needs to be enhanced significantly in these areas or the OSCE scenarios were excessively demanding. It should be also noted that inter-rater reliability was adequate for the total scoring (ICC = .748) but lower than expected for the competency levels (Kappa = .346), which can be explained by the fact that the difference between two consecutive levels was based on achieving just 1or 2 additional CEPEs out of 7–12 of each scenario. Overall, the methodology used was acceptable considering the high number of items used to assess the complexity of “competency”. In addition, the study provides a pathway to identify gaps in performance in common problem areas and offer insight for other programs and specialties on how simulation for competencies assessment can best be utilized.

Conclusions

There was a trend to better performance and higher levels of competency in the CoBaTrICE group in comparison to the time-based group but did not reach statiscal significance. Since the number of Wba performed in the CoBaTrICE group was low, additional research is needed to determine whether the implementation of CoBaTrICE is superior to the current official time-based program.

CRediT authorship contribution statement

ACO and RGR had full access to all the data in the study and took responsibility for the integrity of the data and the accuracy of the data analysis. ACO and RGR were responsible for the study design, data analysis, and interpretation, and the writing of the manuscript. MCFD and MDSB were responsible for the statistical analysis. ACO, MJB, DPC, VGT, MV, CV, IM, NM, ES, MJP were responsible for selection the CoBaTrICE competences to be assessed, design the scenarios, define the items of the checklist for each scenario, organization and implementation of the OSCE in the four simulation centers, assessment and rating the performances. ACO was responsible for participants recruitment. All authors read and approved the final manuscript.

Funding

The study has received funding from:

ESICM Trials Group Award 2018. Trials Group Awards. European Society of Intensive Care Medicine.

Conselleria de Educación, Investigación, Cultura y Deporte. Generalitat Valenciana. Código proyecto AICO/2018/126

Declaration of competing interest

The authors declare that they have no competing interests.

Acknowledgements

The tutors and heads of the ICU Departments from: Hospital Universitario la Paz, Madrid. Abelardo García de Lorenzo, María José Asensio. Hospital Clínico San Carlos, Madrid: Miguel Sánchez, Manuel Álvarez. Hospital Universitario 12 de octubre, Madrid: Juan Carlos Montejo, José Luis Pérez Vela. Hospital Clínico Universitario Virgen de la Arrixaca, Murcia: Rubén Jara, Carlos Luis Albacete. Hospital Universitario Virgen de las Nieves, Granada: José Miguel Pérez Villares, María Redondo. Hospital Universitario Virgen del Rocío, Sevilla: Rosario Amaya, Yael Corcia. Hospital Universitario de Gran Canaria doctor Negrín, Las Palmas De Gran Canaria: Sergio Ruiz, Catalina Sánchez. Hospital Clínico Universitario, Valencia: Marisa Blasco, Ángela Jordá. Hospital Universitario Doctor Peset, Valencia: Rafael Zaragoza, Santiago Borrás. Hospital General Universitario, Alicante: Francisco Jaime, José Luis Antón. Consorci Corporació Sanitária Parc Taulí, Sabadell: Ana Ochagavía, Ana Navas. Hospital Vall D´hebron, Barcelona: Ricard Ferrer, Marcos Pérez. Hospital Universitari Germans Trias i Pujol Badalona: Pilar Ricart, Fernando Armestar.

Simulation centers: Hospital la Fe, Valencia; Hospital Clinic, Barcelona; IAVANTE, Granada; University Francisco de Vitoria, Madrid.

Appendix A
Supplementary data

The following is Supplementary data to this article:

References
[1]
J. Bion, H.U. Rothen.
Models for intensive care training. A European perspective.
Am J Respir Crit Care Med, 189 (2014), pp. 256-262
[2]
G. Jonker, L.A. Manders, A.P. Marty, C.J. Kalkman, T.J. Ten Cate, E.F. van Gessel, et al.
Variations in assessment and certification in postgraduate anaesthesia training: a European survey.
Br J Anaesth, 119 (2017), pp. 10
[3]
N.K. Kjaer, T. Kodal, A.F. Shaughnessy, D. Qvesel.
Introducing competency-based postgraduate medical training: gains and losses.
Int J Med Educ, 2 (2011), pp. 110-115
[4]
C. Carraccio, R. Englander, E. Van Melle, O. Ten Cate, J. Lockyer, M.K. Chan, International Competency-Based Medical Education Collaborators, et al.
Advancing competency-based medical education: a charter for clinician-educators.
Acad Med., 91 (2016), pp. 645-649
[5]
J.R. Frank, L. Snell, R. Englander, E.S. Holmboe, Collaborators I..
Implementing competency-based medical education: moving forward.
Med Teach., 39 (2017), pp. 568-573
[6]
J.F. Bion, H. Barrett.
Development of core competencies for an international training programme in intensive care medicine.
Intensive Care Med, 32 (2006), pp. 1371-1383
[7]
CoBaTrICE Collaboration..
The views of patients and relatives of what makes a good intensivist: a Euro- pean survey.
Intensive Care Med, 33 (2007), pp. 1913-1920
[8]
CoBaTrICE Collaboration..
The educational environment for training in intensive care medicine: structures, processes, outcomes and challenges in the European region.
Intensive Care Med, 35 (2009), pp. 1575-1583
[9]
CoBaTrICE Collaboration..
International standards for programmes of training in intensive care medicine in Europe.
Intensive Care Med, 37 (2011), pp. 385-393
[10]
P. Póvoa, I. Martin-Loeches, F. Duska, CoBaTrICe Collaboration.
Intensive Care Med, 48 (2022), pp. 1093-1094
[11]
J. Roca, J.M. Pérez, M. Colmenero, H. Muñoz, L. Alarcón, G. Vázquez.
Professional competence for the care of critical patients: beyond specialities.
Med Intensiva, 31 (2007), pp. 473-484
[12]
A. Castellanos-Ortega, H.U. Rothen, N. Franco, L.A. Rayo, I. Martin-Loeches, P. Ramírez, et al.
Training in intensive care medicine. A challenge within reach.
Med Intensiva, 38 (2014), pp. 305-310
[13]
L.D. Gruppen, J.C. Burkhardt, J.T. Fitzgerald, M. Funnell, H.M. Haftel, M.L. Lypson, et al.
Competency-based education: programme design and challenges to implementation.
Med Educ, 50 (2016), pp. 532-539
[14]
D.J. Murray, J.R. Boulet, W.A. Boyle, M.B. Beyatte, J. Woodhouse.
Competence in decision making: setting performance standards for critical care.
Anesth Analg, 133 (2021), pp. 142-150
[15]
M.J. Hastie, J.L. Spellman, P.P. Pagano, J. Hastie, B.J. Egan.
Designing and implementing the objective structured clinical examination in anesthesiology.
Anesthesiology, 120 (2014), pp. 196-203
[16]
M. Chiu, J. Tarshis, A. Antoniou, T.L. Bosma, J.E. Burjorjee, N. Cowie, et al.
Simulation-based assessment of anesthesiology residents’ competence: development and implementation of the Canadian National Anesthesiology Simulation Curriculum (CanNASC).
Can J Anaesth, 63 (2016), pp. 1357-1363
[17]
M.B. Weinger, A. Banerjee, A.R. Burden, W.R. McIvor, J. Boulet, J.B. Cooper, et al.
Simulation-based assessment of the management of critical events by board-certified anesthesiologists.
Anesthesiology, 127 (2017), pp. 475-489
[18]
W.A. Boyle, D.J. Murray, M.B. Beyatte, J.G. Knittel, P.W. Kerby, J. Woodhouse, et al.
Simulation-based assessment of critical care "Front-Line" providers.
Crit Care Med., 46 (2018), pp. e516-e522
[19]
Junta Directiva de la Sociedad Española de Medicina Intensiva, Crítica y Unidades Coronarias (SEMICYUC).
Medicina intensiva en España [Intensive medicine in Spain].
Med Intensiva., 35 (2011), pp. 92-101
[20]
A. Castellanos-Ortega, M.J. Broch, M. Barrios, M.C. Fuentes-Dura, M.D. Sancerni-Beitia, C. Vicent, et al.
Acceptance and validity of the methods used to implement a competency based medical education programme in an Intensive Care Department of a teaching referral center.
Med Intensiva, S0210-5691 (2020),
[21]
Á Castellanos-Ortega, M.J. Broch, D. Palacios-Castañeda, V. Gómez-Tello, M. Valdivia, C. Vicent, et al.
Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study.
Med Intensiva (Engl Ed), 46 (2022), pp. 491-500
[22]
L.J. Easdown, M.L. Wakefield, M.S. Shotwell, M.R. Sandison.
A checklist to help faculty assess ACGME milestones in a video-recorded OSCE.
J Grad Med Educ., 9 (2017), pp. 605-610
[23]
W.R. McIvor, A. Banerjee, J.R. Boulet, T. Bekhuis, E. Tseytlin, L. Torsher, et al.
A taxonomy of delivery and documentation deviations during delivery of high-fidelity simulations.
Simul Healthc, 12 (2017), pp. 1-8
[24]
M.T. Nousiainen, K.J. Caverzagie, P.C. Ferguson, J.R. Frank.
Implementing competency-based medical education: what changes in curricular structure processes are needed?.
Med Teach, 39 (2017), pp. 594-598
[25]
K.J. Caverzagie, M.T. Nousiainen, P.C. Ferguson, O. Ten Cate, S. Ross, K.A. Harris, et al.
Overarching challenges to the implementation of competency-based medical educa- tion.
Med Teach, 39 (2017), pp. 588-593
[26]
J. Lockyer, C. Carraccio, M.K. Chan, D. Hart, S. Smee, C. Touchie, ICBME Collaborators, et al.
Core principles of assessment in competency-based medical education.
Med Teach, 39 (2017), pp. 609-616
[27]
Jayson M. Stoffman.
Overcoming the barriers to implementation of competence-based medical education in post-graduate medical education: a narrative literature review.
[28]
A. Szulewski, H. Braund, D.J. Dagnone, L. McEwen, N. Dalgarno, K.W. Schultz, et al.
The assessment burden in competency-based medical education: how programs are adapting.
Acad Med, 98 (2023), pp. 1261-1267
[29]
J. Garnacho-Montero, E. Gordillo-Escobar, J. Trenado, F. Gordo, L. Fisac, E. García-Prieto, Intubation Prospective (INTUPROS) Study Investigators, et al.
A nationwide, prospective study of tracheal intubation in critically ill adults in Spain: management, associated complications, and outcomes.
Crit Care Med, 52 (2024), pp. 786-797
[30]
R.H. Blum, J.R. Boulet, J.B. Cooper, S.L. Muret-Wagstaff, Harvard Assessment of Anesthesia Resident Performance Research Group.
Simulation-based assessment to identify critical gaps in safe anesthesia resident performance.
Anesthesiology, 120 (2014), pp. 129-141
[31]
R.H. Blum, S.L. Muret-Wagstaff, J.R. Boulet, J.B. Cooper, E.R. Petrusa, K.H. Baker, Harvard Assessment of Anesthesia Resident Performance Research Group, et al.
Simulation-based assessment to reliably identify key resident performance attributes..
Anesthesiology, 128 (2018), pp. 821-831
[32]
D.J. Schumacher, D.C. West, A. Schwartz, S.T. Li, L. Millstein, E.C. Griego, et al.
Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network General Pediatrics Entrustable Professional Activities Study Group. Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities.
[33]
C.R. Lucey, G.E. Thibault, O. Ten Cate.
Competency-based, time-variable education in the health professions: crossroads.
[34]
J.R. Frank, J. Karpinski, J. Sherbino, L.S. Snell, A. Atkinson, A. Oswald, et al.
Competence by design: a transformational national model of time-variable competency-based postgraduate medical education.
Perspect Med Educ, 13 (2024), pp. 201-223
[35]
J. Dupre, V.N. Naik.
The role of simulation in high-stakes assessment.
BJA Educ, 21 (2021), pp. 148-153
[36]
M.I. Soffler, D.D. Claar, J.I. McSparron, D.N. Ricotta, M.M. Hayes.
Raising the stakes: assessing competency with simulation in pulmonary and critical care medicine.
Ann Am Thorac Soc., 15 (2018), pp. 1024-1026
[37]
E. Sinz, A. Banerjee, R. Steadman, M.S. Shotwell, J. Slagle, W.R. McIvor, et al.
Reliability of simulation-based assessment for practicing physicians: performance is context-specific.
BMC Med Educ, 21 (2021), pp. 207
[38]
R. Brydges, R. Hatala, B. Zendejas, P.J. Erwin, D.A. Cook.
Linking simulation-based educational assessments and patient-related outcomes: a systematic review and meta-analysis.
Acad Med., 90 (2015), pp. 246-256
[39]
S. Boet, M.D. Bould, L. Fung, H. Qosa, L. Perrier, W. Tavares, et al.
Transfer of learning and patient outcome in simulated crisis resource management: a systematic review.
Can J Anaesth, 61 (2014), pp. 571-582
[40]
M.J. Broch Porcar, Á Castellanos-Ortega.
Patient safety, what does clinical simulation and teaching innovation contribute?.
Med Intensiva (Engl Ed), S2173-5727 (2024), pp. 00101-00102
Copyright © 2025. The Authors
Download PDF
Idiomas
Medicina Intensiva (English Edition)
Article options
Tools
Supplemental materials
es en

¿Es usted profesional sanitario apto para prescribir o dispensar medicamentos?

Are you a health professional able to prescribe or dispense drugs?