The current postgraduate medical training model in most European countries is the so called “time-based training”.1–3 This paradigm assumes that mere exposure to clinical experiences based on temporary rotations through different Departments suffices to acquire the necessary professional competencies. Opportunistic learning and volume of practice rather than learning guided by objectives defines competency. Certification depends on a logbook of cases, and a generic, subjective report about knowledge, technical and non-technical skills acquired by the resident, which is performed after every rotation and yearly by the tutors and the head of the ICU Department. The effectiveness of this model of training to provide the adequate level of competency to the trainees in intensive care medicine (ICM) has not been determined. To overcome these limitations, a new model based on the gradual acquisition of clearly defined, observable and measurable competencies is gaining acceptance.4 Competency based medical education (CBME) proposes more solid principles such as: (a) defining the learning outcomes that must be shown by residents at the end of their training period; (b) focusing attention upon the development and demonstration of skills, attitudes and knowledge acquired by residents in the course of the training process;(c) prioritizing training evaluation and constructive feedback centered on the performance levels of the residents in the real-life working context; and (d) using a broad range of evaluation tools and methods.5 The CBME model for ICM in Europe is called CoBaTrICE (Competency Based Training in Intensive Care Medicine in Europe) (https://www.esicm.org/education/cobatrice/) an international partnership of professional organizations and critical care clinicians whose ultimate aims are to assure a high-quality level education in ICM, to harmonize training in ICM without interfering with national specific regulations and to allow for free movement of ICM professionals across Europe.6–10 The competences of CoBaTrICE have been adopted by Union Européenne des Médecins Spécialistes (UEMS) as Common Training Framework10 as well as into the training programs of several European countries.1 Spain is planning to do so.11,12 Research in this field is still limited because the model has been applied in a partial and scantly structured manner13 therefore, whether CBME programs provide better outcomes than the traditional ones is unknown. The main objective of this study was to evaluate whether the implementation CoBaTrICE provides higher levels of competency at the end of the training period in comparison with the current official time-based program in ICM in Spain. Secondary objectives were: (1) To determine the percentage of critical essential performance elements (CEPE) accomplished; (2) To determine compliance with CoBaTrICE and barriers to an effective implementation. Since the traditional examinations typically measure knowledge and performance rather than competency in the complex situations of ICM, we chose a simulation-based objective structured clinical examination (OSCE) to assess non-technical and therapeutic management skills as well as the ability to integrate knowledge, judgment, communication, and teamwork into the simulated practice setting.14–18

It was estimated that 115 performances would have 80% power (2-sided α = 0.05) to detect a difference in the level of competence achieved of one level on the 5-level scale. Whilst there is no information available on this issue, we postulate that one level would represent a minimally important difference.

Results are shown as median and maximum-minimum range. To compare continuous variables, the Mann–Whitney U test was used. All tests were two-tailed, and p < .05 was predetermined to define statistical significance. Inter-rater reliability for the OSCE scenarios at item level was estimated with Fleiss kappa coefficient and their 95% confidence intervals, and for the scores, with the Intraclass Correlation Coefficient (ICC) and their 95% confident intervals.

All Analysis were performed using SPSS statistical package version 23.0 (SPSS Inc., Chicago, IL).

A total of 295 performances were analysed [176 in the baseline OSCE (36 residents)] and 119 in the final OSCE [26 residents (17 residents of the CoBaTrICE group and 9 of the control group)]; 3 R5 of the CoBaTrICE group and 7 R5 of the control group did not perform the final OSCE. In the final OSCE, the participants were 11 male and 15 female, mean age 31.5 ± 2.8 (range 29–40) years from 11 ICU Departments of 11 Spanish teaching hospitals (Table 2). Eleven video-records of scenarios 2 (6), 3 (4), and 4 (1) were either lost or discarded for low quality. The distribution of the participants in the OSCE in each of the four simulation centers was as follows: Barcelona, 7; Madrid, 5; Granada, 4; and Valencia, 10.

In this multicentric cluster-based randomized trial which main objective was to evaluate whether the implementation of CoBaTrICE provides higher levels of competency in comparison to the current official time-based program in ICM in Spain, there were not significant differences between both models in the level of competency achieved by the R5 participants in the simulated-based OSCE. Nevertheless, despite the low number of Wba performed, residents from the CoBaTrICE group showed a better trend in performance, they achieved a higher level of competency in comparison to the control group, levels higher than 2 were achieved in 43.4% of the performances in the CoBaTrICE group vs. 27.9% in the control group (p = 0.07), and they completed more CEPE than residents of the control group in all scenarios (p = 0.09). Results of the baseline OSCE did not show differences between both groups regarding the primary and secondary objectives of the study.21

To the best of our knowledge, no publications have been made involving the comparison of the effectiveness of the two current training models in postgraduate medicine. CBME stands for a shift in emphasis away from the more traditional time-based programs, based solely on exposure to experiences such as clinical rotations, in favor of a model which is a learner-centered approach that emphasizes achieving specific outcomes. The implementation of CBME is challenging because requires organizational and cultural changes, resources, particularly more dedication of teaching time, as well as the training of tutors and staff members in formative assessment and feedback techniques.20,24–26 Frequent formative Wba as well as the record of the learning experiences in an electronic portfolio are the essential elements to promote learning, self-reflection, progression, and ultimately to guarantee that the predefined competences and skills are effectively acquired.20,25–27 Reliability in achievement of competencies correlates directly with the number of observations in as many different contexts as possible, these frequent assessments have been described as a burden on residents and trainers alike, being the main barrier to the implementation of CBME.28 The COVID-19 pandemic put a huge strain on critical care resources worldwide, as systems struggled to provide high-quality care for a surge of critically ill patients. This exceptional situation that occurred just during the study period (2020–2022) might have influenced the low number of Wba performed in the CoBaTrICE group preventing the trial from showing a significant difference between both groups.

Regarding the performance of the residents, our data raise concerns and considerations around the development for some CEPE. Only 38% of the participants achieved level 3 or higher, and, as previously seen in the baseline OSCE there was a great variability, many residents failed to apply a complete protocol of endotracheal intubation in a high-risk critical patient, which can be explained by the findings of the recently published prospective study of tracheal intubation in critically ill adults in Spain.29 Although there are not similar studies performed in ICM, our results are not very different from those obtained in studies conducted to assess performance of anesthesia professionals using simulation.14,17,30,31 Weinger et al.17 in their study of 268 board-certified anesthesiologists found that CEPE were commonly omitted. Schumacher et al.32 in their study performed at the end of the Pediatrics residency found that the level of “practice without supervision” was achieved only in 47% of the competencies.

Our findings suggest that more work may be needed to ensure that graduates of intensive care residency training can be trusted for independent clinical practice. In our view, CBME is the way, as it focusses on clinical excellence translating the needs of contemporary society for improved health care into competencies that must be mastered by motivated well-trained intensivists.33,34

Unfortunately, the study coincided with the COVID 19 pandemic what might have conditioned the low number of Wba performed in the CoBaTrICE group and the reduction of participants in the final OSCE, which in turn may have underpowered the trial to achieve significant results.

Simulation is being more and more used as a tool for high-stakes assessments,35–37 however, there is little documentation of a relationship between simulation performance and performance in the clinical setting.38–40 It has been shown that those with more training and experience perform better in the scenarios, suggesting that simulation-based assessments may ultimately prove useful as an indicator that they are ready for unsupervised practice in the real world.14,18,30

The definitions of the levels of competency were stablished by a consensus of a panel of experts in ICM. We expected residents to meet the standard of unsupervised practice at the completion of training in all 5 scenarios, surprisingly, only 5 performances achieved levels 4 or 5, suggesting that either training needs to be enhanced significantly in these areas or the OSCE scenarios were excessively demanding. It should be also noted that inter-rater reliability was adequate for the total scoring (ICC = .748) but lower than expected for the competency levels (Kappa = .346), which can be explained by the fact that the difference between two consecutive levels was based on achieving just 1or 2 additional CEPEs out of 7–12 of each scenario. Overall, the methodology used was acceptable considering the high number of items used to assess the complexity of “competency”. In addition, the study provides a pathway to identify gaps in performance in common problem areas and offer insight for other programs and specialties on how simulation for competencies assessment can best be utilized.

There was a trend to better performance and higher levels of competency in the CoBaTrICE group in comparison to the time-based group but did not reach statiscal significance. Since the number of Wba performed in the CoBaTrICE group was low, additional research is needed to determine whether the implementation of CoBaTrICE is superior to the current official time-based program.

ACO and RGR had full access to all the data in the study and took responsibility for the integrity of the data and the accuracy of the data analysis. ACO and RGR were responsible for the study design, data analysis, and interpretation, and the writing of the manuscript. MCFD and MDSB were responsible for the statistical analysis. ACO, MJB, DPC, VGT, MV, CV, IM, NM, ES, MJP were responsible for selection the CoBaTrICE competences to be assessed, design the scenarios, define the items of the checklist for each scenario, organization and implementation of the OSCE in the four simulation centers, assessment and rating the performances. ACO was responsible for participants recruitment. All authors read and approved the final manuscript.

The study has received funding from:

ESICM Trials Group Award 2018. Trials Group Awards. European Society of Intensive Care Medicine.

Conselleria de Educación, Investigación, Cultura y Deporte. Generalitat Valenciana. Código proyecto AICO/2018/126