The IRCU of University Hospital fundacion Jimenez Diaz is a high complexity unit with 15% of patients requiring complicated postoperative weaning procedures, while vasoactive treatment prevalence is approximately 26%. The unit has 5 beds (1:5 patient nurse ratio), being able to reach 8 in the periods of greatest need for care (1:4 patient nurse ratio in this scenario). Our unit is an open room to guarantee the global supervision of the patients that are completely telemonitorized. Supplementary Figures 1–3 describe the material and human resources, as well as its complexity according to the average GRD value of the unit. The study was aproved by our Ethic Committee. De-identified medical records were confidentially collected from a total of 1966 patients admitted to the FJD-Hospital IRCU (Madrid, Spain) from January 2007 to December 2017. A second cohort of 230 new patients with confidential records recruited from October 2018 to September 2019 in the same IRCU, were used to measure the performance of the new IRCU protocol based on the conclusions of this work. This is an observational and retrospective study. All the patients admitted to our IRCU were selected for the study and no exclusión criteria was imposed.

Vital signs of patients were recorded upon arrival at the hospital, before being admitted to the IRCU: respiratory rate (RR), temperature (T), systolic blood pressure (SBP), and diastolic blood pressure (DBP). Laboratory findings including hemoglobin (Hb), platelets (PL), leucocytes (Leucos), the International Normalized Ratio (INR), blood glucose (GLU), blood potassium (K), sodium (NA) and creatinine (CREAT) were also registered. Blood gases at the time of hospital admission were recorded (Fi02 21%), including partial oxygen pressure (P02), partial carbon dioxide pressure (PC02), PH, bicarbonates (SBC) and base excess (BE). Age and individual mortality or survival upon discharge from the IRCU was extracted from medical histories. The data was collected following standard protocols by the same personnel over the whole time period and in the two cohorts included in the study.

A criteria for variable exclusion was defined in order to avoid introduction of statistical noise that may introduce bias in final results. Parameters with a high percentage (>25%) of missing values were excluded from the analysis. For the rest of the parameters we perform the K-nearest neighbors (KNN) imputation method implemented in the DMwR R package.

Principal component analysis (PCA) was performed on the matrix with missing data imputation, using the pca3d R library.

Using ROC curves and Youden's J statistic we obtained the cut-off point that best discriminates the mortality status for each of the parameters. Based on these cut-off points we defined a binary classification for each parameter (P-value <.2) to feed a multivariate logistic regression model. Area Under the Curve (AUC), sensitivity and specificity were calculated using a ROC curve. An internal validation of the model was performed by bootstrap method to correct for optimistic prediction. A total of 1000 bootstrap samples with replacement were generated, and the differences in AUC on the bootstrap samples and the original sample were calculated. The goodness of fit of the model was evaluated by the Hosmer and Lemeshow test.

A NN model was applied using the caret R package. Data matrix was normalized using the min-max scale method. Ten re-sampled matrices were calculated having samples with mortality=1 and a random selection of the same number of samples with mortality=0. Every re-sampled matrix is then subject to a 10-fold cross validation process (90% training and 10% testing). Thus, a total of 100 neural network models are performed using “nnet” method with automatic selection of: (i) the optimal number of units per hidden layer between 1 and 5, and (ii) the optimal value for the regularization parameter (between 0.1 and 0.5 with increments of 0.1). The “twoClassSummary” method was used to compute sensitivity, specificity and the AUC. A ROC curve is calculated for the set of predictions and real values within the loop, using the 10-fold cross validation process. The final AUC, accuracy, sensitivity and specificity were calculated as the mean of the 100 neural network models performed in total. Our R script is available at https://github.com/pminguez/MachineLearning4UnbalancedData.

Using the KNN data matrix, we calculate the Spearman correlation between every pair of variables. In order to build the network, we select the pairs that have a correlation coefficient >±0.3 and a P-value <.01. The selected pairs of variables represent the edges of the correlation network.

In order to compare the failure before and after the new criteria to detect high risk patients was implemented, we fit Poisson regression models to calculate the annual trend of failure risk in the two periods. The relatives risks (RR) were compared using the Wald test, taking the coefficients of the models (logarithm of RR) and its standard errors.

A total of 1966 patients were included in the analysis having a global local IRCU mortality rate of 1.68% (33 patients) and a global failure (IRCU mortality+ICU transfer+outside IRCU delayed mortality) of about 5.39% (106 patients). Table 1 describes the study population with epidemiologic data and treatment used and Table 2 shows mean and standard deviation values for each of the variable used (INR was excluded). A PCA of the patients’ features did not show observable differences between patient's classifications according to mortality status (Fig. 1).

Fig. 1.

Principal component analysis of the dataset after missing value imputation.

(0.06MB).

To understand how the selected variables are physiologically linked, we calculated their paired correlations between all pairs of variables. Fig. 2 shows the pairs of variables with a significant correlation score (rho <0.3 and P-value <.01). Interestingly, we found that potassium levels (K) and acidosis (PH) were closely associated, but not with other parameters related to renal failure. The correlation network shows consistency between acid base variables (PC02, SBC and BE). It also presents independent relationships in hemodynamics and blood cell series.

Fig. 2.

Correlation between binary variables affecting an internal physiological relationship. Significant correlations are established when rho<0.3 and P-value<0.01.

(0.06MB).

Our dataset has a mortality ratio below 2%. In Fig. 3 we show the performance of our NN algorithm over the dataset. Despite an acceptable area under the curve (AUC) and the extremely high accuracy value (0.98), it displays a very low specificity (0.5) which reflects that working with the whole dataset introduces a strong bias toward the learning of features that favor alive over death. In order to avoid the imbalanced class distribution bias, we introduced a re-sampling step (see Methods) where the dataset is down-sampled to have the same number of dead and alive events. Fig. 4A shows the NN performance over a set of re-sampled matrices that have a balanced class distribution. Although the behavior of the model in terms of accuracy does not improve the performance using the unbalanced dataset, we observe and increase in AUC (0.71), sensitivity (0.59) and specificity (0.59) values. This model was employed for further analysis.

Fig. 3.

A. Shows the ROC curve and AUC, accuracy, sensitivity and specificity metrics for the NN model with unbalanced classes. B. Shows the importance of the variables in that NN model.

(0.12MB).

Fig. 4.

A. shows the ROC curve and AUC, accuracy, sensitivity and specificity metrics for the NN model with balanced classes. B. Shows the importance of the variables in that NN model.

(0.15MB).

The NN classification using the balanced data strategy shows that thrombocytopenia (PL), metabolic acidosis (SBC and PH), anemia (Hb), tachypnea (RR), age (AGE), sodium levels (NA), hypoxemia (PO2), leukocytopenia (LEUCOS), and hyperkalemia (K) are the most relevant parameters associated with mortality (Fig. 4B). The combination of these parameters reflects potential respiratory sepsis associated with hypoxemia and secondary pancytopenia or a primary oncohaemathologic disorder with associated septic respiratory distress. These results agree with our own experience when we have reviewed our IRCU's behavior in previous years.

In order to provide a complementary decision tool for risk prediction, we calculated the cut-off value (Youden J index) that best discriminated between dead and alive cases for every variable considered in the NN model (Supplementary Table 1) and a univariate logistic regression models was performed (Supplementary Table 2). We observed that from the top 10 variables most weighted in the NN model, six were a P-value <.05 and eight <0.1. In a multivariate logistic regression analysis, we found RR, LEUCOS, Hb, K and PH as the best combination of variables that best discriminate between classes (Table 3). This whole set of predictors is also found amongst the top 10 variables when sorted by importance in the NN model, again suggesting an overlapping signal coming from independent methods. The AUC for the logistic model was 0.75 (Fig. 5), and the bootstrap internal validation indicated an optimism of 0.016, which reflects the robustness of the model.

Fig. 5.

Uni and Multi-variate logistic regression model, ROC curve, AUC, sensitivity and specificity values are shown.

(0.05MB).

By combining both analyses, we built a decision algorithm based on the variable classification provided. In order of importance the algorithm ranks the variables as: (1) PL<266, (2) SBC<26.4, (3) PH<7.33, (4) Hb<10.4, (5) RR>24.5, (6) age>63, (7) NA>140, (8) P02<68mm, (9) LEUCOS<3.230, and (10) K>3.68.

The new criteria for identification of high risk patients consider the following decisions: (1) Patients with more than 63 years, more than two cytopenias, metabolic acidosis, hypoxemia and hypernatremia with a background of infectious disease received a vigorous antibiotic treatment, and early and aggressive resuscitation of hemodynamic, respiratory and internal environment at the time of arrival regardless of the location of the patient before entering the unit. Management then would be the same but with precocity, on arrival (2) Patients with hypoxemia and metabolic acidosis received enhanced high-flow oxygen therapy either tan NIV immediately after admission in the emergency department to improve their P02/FI02>150mmHg after one hour. Metabolic acidosis was urgently counterbalanced with bicarbonates and/or haemodinamic early ressucitation procedures. If they failed to achieve this oxygenation in one hour they should be referred to the intensive care unit. (3) Oncohematological patients who met 80% or more of the criteria and for which we were consulted, were advised to enter directly into the intensive care unit for invasive management.

The algorithm was implemented in our IRCU from October 2018 to September 2019 (230 patients, features shown in Supplementary Tables 3 and 4). In September 2019 we retrospectively analyzed this cohort. A total of 23 patients received vigorous treatment (decision 1), one of them not recovering and transferred to the ICU surviving. Decision 2 was applied to 33 patients, no patient required transfer to the ICU and there was no deceased. Five patients were subject of decision 3 and 1 was referred to the ICU in such conditions and survived. In total, 61 patients of the cohort were handled differently from how we previously did (26.52%).

In this second cohort we had 3 deceased and a mortality rate of 1.30%. The previous historical average being 1.68% (33patients), which shows more than a 20% of decrease (50% considering only last 7 years). In a wider perspective, the rate of failures (deaths and transfers to ICU) after the algorithm implementation was 3.0% (4 transfers, 3 of them for reasons related to the NN variables selected and the fourth due to refractory hypercapnic ventilatory failure). In order to compare this ratio with the whole series and avoiding the time bias we used Poisson regression models to calculate the annual trend of failure risk in the two periods. The relative risks (RR) were 0.94 and 0.41 from 2004–2017 and 2018–2019 respectively. Both RR being statistically different (Wald test, P-value=.065), see Fig. 6 to see global trend over the years.

Fig. 6.

Failure rate in our IRCU over the years with available records.

(0.11MB).