To develop and evaluate several predictive models, based on ML algorithms, that estimate the risk of ICA 24 hours before the event, by using variables registered the hospital’s Electronic Health Record (EHR) that belonged to a cohort of both ICA and non-ICA patient. To validate the hypothesis that ML models may allow to predict ICA with a significantly higher accuracy than traditional score systems that employ a smaller set of features. To establish which demographic, clinical, laboratory and antecedent variables are most powerful predictors of ICA.

A retrospective, observational, based on real-world data, cohort study was performed. Two cohorts of patients belonging to the Son Llàtzer University Hospital (HUSLL), a Spanish second regional level hospital, were used1: all patients who had suffered an ICA (positive cases)2; a sample of patients with no ICA (control patients), randomly selected among hospitalisations where no cardiac arrests occurred. All patients had been hospitalised through the Emergency Department, and the distribution of patients among hospital services was similar in the two cohorts. For data collection of positive cases, the Utstein style guidelines18 were followed, where the inclusion criteria were: all patients over 18 years of age who suffered a cardiorespiratory arrest in the hospital wards from September 2009 to december 2021 and treated by the ICU, with cardiac arrest being defined as loss of pulse with attempted resuscitation.19 The total number of admissions in HUSLL was 1,065,722 during this period, with a prevalence of around 0.64 ICAs per 1,000 admitted patients. The exclusion criteria of ICAs for the study were: cardiac arrests where the patients refused resuscitation manoeuvres despite medical recommendation. Besides, ICAs that followed another cardiac arrest from the same patient in a less-than-7-days period were also excluded, to avoid analysis alterations. ICAs from the same patient, separated in time by more than 7 days, were considered separately. As the goal of the study was to build a model to predict ICA 24 hours before the event, the predictor variables were filtered to focus on the time of the event, analysing the measurement of each variable that was closest in time to 24 hours before the episode. It was made sure that the measurement selected as the value of each variable had been recorded in a time interval of 72 to 12 hours before the ICA. Furthermore, if no measurements of the variables had been recorded for the patient between 36 and 12 hours before an arrest, the arrest itself was discarded and not used as a data point, as it was considered that the predictor information could not represent the patient’s state one day before the event.

In healthy patients, measurements of clinical and laboratory variables that had been recorded during their hospitalisations were extracted. Then, preliminary data points were constructed every time a variable received a new measurement, with these points being composed of the last measurement of each variable in a 60 -h window. Hence, there was initially a great number of data points per hospitalisation per patient. Those data points that minimised the number of missing values for each patient were selected, considering that at least one record for each control must be selected so as not to lose patients in the sampling. Once this set of 250 points was obtained, other 125 data points were sampled randomly from the remaining ones, to complete the control cohort. In addition, it was forced that the 60 -h-window data points chosen for one same patient were more than 6 days apart, so that there was variability in the measurements. The resulting database contained a total of 750 data points (a data point consists in a row of the processed dataset that was built for the analysis of the study, extracted from the Electronic Health Records (EHR) of patients that were hospitalised during a certain period in HUSLL) corresponding to 620 different patients, 370 patients with ICA and 250 controls, where 375 records were extracted from each cohort, obtaining a balanced dataset.

This retrospective cohort study was approved by Research Ethics Committee of Illes Balears (Code: IB 4951/22 PI). The use and processing of the collected data was ensured to comply with the Organic Law 3/2018, of 5 December, on the Protection of Personal Data and the Guarantee of Digital Rights. Patient information was anonymised prior to the study.

A total of 28 variables were collected from the EHR. These variables were classified in the following types of predictors: demographic (sex and age at the time of hospitalisation), vital signs (respiratory rate (RR), systolic blood pressure (SBP), heart rate (HR), body temperature, saturation oxygen (SO2) and low level of consciousness (LC) measured with Glasgow scale), laboratory biomarker analyses (bicarbonate, blood creatinine, blood glucose, blood urea, haemoglobin, lactate, magnesium, O2 saturation, potassium) and comorbidities (arterial hypertension (ATH), chronic obstructive pulmonary disease (COPD), diabetes, dialysis, dyslipidaemia, smoking, ischemic heart disease (IHD), liver failure, neurological disease, renal failure, cancer). Continuous variables were represented as numbers (Table 1), categorical variables were classified in binary categories (Table 2). The comorbidities were identified using ICD-9/10 code.

The missing values were filled with the median value for numerical variables and mode for binary ones.

Several ML algorithms were used to address the problem of predicting which patients would suffer an ICA. These methods employ mathematical models to capture complex associations between predictor and target variables. The target variable was the presence or absence of ICA around 24 hours after the measurement of their predictor variables.

Such algorithms were the following: K Nearest Neighbours (KNN),20,21 Support Vector Machine (SVM),22 Multilayer Perceptron (MLP),23 Random Forest (RF),24,25 Gradient boosting (GB),26 Custom Ensemble of Gradient Boosting estimators (CEGB). The latter, as its name implies, was a self-made algorithm based on combining other several GB algorithms. The ML model training is furtherly explained in Annex I in the Supplementary material.

Furthermore, MEWS6 was also evaluated on the dataset as a reference system to compare ML-based models with.

The metrics of discrimination obtained by the different ML models are shown in Table 3. All the studied ML models correctly classified most of the test data points, although the extremely wide CI in the SVM’s metrics (reaching 0 for specificity and 100 for sensitivity) indicate that said model always predicts as positive in some CV iterations. The RF, GB and CEGB algorithms correctly predict 4 out of 5 cases, with balance between the two classes. Furthermore, their CI imply that none of these tree-based models can be said to outperform the others at 95% confidence. AUC also showed a higher general predictive capacity for the tree-based algorithms, and a more limited one for the KNN and SVM classifiers. The ML model that offered an overall optimal performance was the CEGB estimator, which showed the highest accuracy, AUC, specificity, PPV and NPV, while still classifying correctly 4 out 5 positive data points (sensitivity of 0.81). It also tended to produce the narrowest CI among the tree-based models, which entails that its performance across the CV splits was the most stable.

MEWS also showed narrow CI, which suggests robustness, but its overall performance was poorer than most of the ML-based models. In general, it only classified correctly slightly more than a half of the samples. The lower limits of the KNN, MLP, RF, GB and CEGB were seldomly lower than the upper limits of MEWS’s metrics, implicating a lower performance in the true population with a 95% confidence.

SHAP analysis was carried out for the best overall performing ML model: the CEGB estimator. A summary plot displaying the features’ influences on the model’s predictions is shown in Fig. 1, whose interpretation is: first, the most important variable for the model is a low level of consciousness, which means in Glasgow scale a value less than 13. A value of 1 (presence) for said variable greatly influences the model to predict ICA. Second, the most influential quantitative variables are haemoglobin, blood glucose, blood urea, SBP, heart rate, blood creatinine and age. In the case of haemoglobin and SBP, low values tend to be predictive of ICA and high values tend to be predictive of no ICA. The association is opposite for blood glucose, blood urea, heart rate, blood creatinine and age. Third, regarding the antecedents, the most influential are hypertension, ischemic heart disease, smoking and chronic obstructive pulmonary disease. The presence of these features is predictive of ICA, although their absence is not particularly predictive of no ICA. Fourth, the least influential variables in the dataset are antecedents of dialysis, liver failure, haematological cancer and solid cancer. Note, however, that the presence of such conditions is infrequent in the dataset.

Figure 1.

SHAP summary plot with the aggregated SHAP values generated in each iteration of the cross-validation. SHAP analysis was carried out for the CEGB estimator. Each cloud of points represents the data points of the aggregated test splits among the iterations of the cross-validation. The horizontal axis shows the SHAP values assigned to every data point. If the magnitude is higher, the influence is greater. If the value is positive, the influence is positive (induces to predict cardiac arrest). If it is negative, such is the influence (induces to predict no cardiac arrest). As there is a SHAP value assigned to every variable of every data point, the clouds are grouped by variable. The variables are sorted in a descending overall importance ranking. The overall importance of a variable is established by the average magnitude of its SHAP values among all the data points. Finally, the colour of each point in a cloud represents the original value of the variable of such point. If the variable has a low value in a data point, such point will be coloured in blue in the cloud of the variable in question. Inversely, if the original value is high, the colour will be red.

(0.38MB).