Skip to main content
  • More from ADA
    • Diabetes
    • Clinical Diabetes
    • Diabetes Spectrum
    • ADA Standards of Medical Care
    • ADA Scientific Sessions Abstracts
    • BMJ Open Diabetes Research & Care
  • Subscribe
  • Log in
  • My Cart
  • Follow ada on Twitter
  • RSS
  • Visit ada on Facebook
Diabetes Care

Advanced Search

Main menu

  • Home
  • Current
    • Current Issue
    • Online Ahead of Print
    • Special Article Collections
    • ADA Standards of Medical Care
  • Browse
    • By Topic
    • Issue Archive
    • Saved Searches
    • Special Article Collections
    • ADA Standards of Medical Care
  • Info
    • About the Journal
    • About the Editors
    • ADA Journal Policies
    • Instructions for Authors
    • Guidance for Reviewers
  • Reprints/Reuse
  • Advertising
  • Subscriptions
    • Individual Subscriptions
    • Institutional Subscriptions and Site Licenses
    • Access Institutional Usage Reports
    • Purchase Single Issues
  • Alerts
    • E­mail Alerts
    • RSS Feeds
  • Podcasts
    • Diabetes Core Update
    • Special Podcast Series: Therapeutic Inertia
    • Special Podcast Series: Influenza Podcasts
    • Special Podcast Series: SGLT2 Inhibitors
    • Special Podcast Series: COVID-19
  • Submit
    • Submit a Manuscript
    • Journal Policies
    • Instructions for Authors
    • ADA Peer Review
  • More from ADA
    • Diabetes
    • Clinical Diabetes
    • Diabetes Spectrum
    • ADA Standards of Medical Care
    • ADA Scientific Sessions Abstracts
    • BMJ Open Diabetes Research & Care

User menu

  • Subscribe
  • Log in
  • My Cart

Search

  • Advanced search
Diabetes Care
  • Home
  • Current
    • Current Issue
    • Online Ahead of Print
    • Special Article Collections
    • ADA Standards of Medical Care
  • Browse
    • By Topic
    • Issue Archive
    • Saved Searches
    • Special Article Collections
    • ADA Standards of Medical Care
  • Info
    • About the Journal
    • About the Editors
    • ADA Journal Policies
    • Instructions for Authors
    • Guidance for Reviewers
  • Reprints/Reuse
  • Advertising
  • Subscriptions
    • Individual Subscriptions
    • Institutional Subscriptions and Site Licenses
    • Access Institutional Usage Reports
    • Purchase Single Issues
  • Alerts
    • E­mail Alerts
    • RSS Feeds
  • Podcasts
    • Diabetes Core Update
    • Special Podcast Series: Therapeutic Inertia
    • Special Podcast Series: Influenza Podcasts
    • Special Podcast Series: SGLT2 Inhibitors
    • Special Podcast Series: COVID-19
  • Submit
    • Submit a Manuscript
    • Journal Policies
    • Instructions for Authors
    • ADA Peer Review
Emerging Technologies: Data Systems and Devices

Predicting the Risk of Inpatient Hypoglycemia With Machine Learning Using Electronic Health Records

  1. Yue Ruan1,2,3,
  2. Alexis Bellot4,5,
  3. Zuzana Moysova6,
  4. Garry D. Tan1,2,
  5. Alistair Lumb1,2,
  6. Jim Davies6,
  7. Mihaela van der Schaar4,5 and
  8. Rustam Rea1,2⇑
  1. 1Oxford Centre for Diabetes, Endocrinology and Metabolism, Oxford University Hospitals National Health Service Foundation Trust, Oxford, U.K.
  2. 2Oxford National Institute for Health Research Biomedical Research Centre, Oxford, U.K.
  3. 3Oxford Centre for Diabetes, Endocrinology and Metabolism, University of Oxford, Oxford, U.K.
  4. 4Department of Mathematics, University of Cambridge, Cambridge, U.K.
  5. 5Alan Turing Institute, London, U.K.
  6. 6Big Data Institute, University of Oxford Li Ka Shing Centre for Health Information and Discovery, University of Oxford, Oxford, U.K.
  1. Corresponding author: Rustam Rea, rustam.rea{at}nhs.net
  1. Y.R. and A.B. contributed equally to data analysis.

Diabetes Care 2020 Jul; 43(7): 1504-1511. https://doi.org/10.2337/dc19-1743
PreviousNext
  • Article
  • Figures & Tables
  • Suppl Material
  • Info & Metrics
  • PDF
Loading

Abstract

OBJECTIVE We analyzed data from inpatients with diabetes admitted to a large university hospital to predict the risk of hypoglycemia through the use of machine learning algorithms.

RESEARCH DESIGN AND METHODS Four years of data were extracted from a hospital electronic health record system. This included laboratory and point-of-care blood glucose (BG) values to identify biochemical and clinically significant hypoglycemic episodes (BG ≤3.9 and ≤2.9 mmol/L, respectively). We used patient demographics, administered medications, vital signs, laboratory results, and procedures performed during the hospital stays to inform the model. Two iterations of the data set included the doses of insulin administered and the past history of inpatient hypoglycemia. Eighteen different prediction models were compared using the area under the receiver operating characteristic curve (AUROC) through a 10-fold cross validation.

RESULTS We analyzed data obtained from 17,658 inpatients with diabetes who underwent 32,758 admissions between July 2014 and August 2018. The predictive factors from the logistic regression model included people undergoing procedures, weight, type of diabetes, oxygen saturation level, use of medications (insulin, sulfonylurea, and metformin), and albumin levels. The machine learning model with the best performance was the XGBoost model (AUROC 0.96). This outperformed the logistic regression model, which had an AUROC of 0.75 for the estimation of the risk of clinically significant hypoglycemia.

CONCLUSIONS Advanced machine learning models are superior to logistic regression models in predicting the risk of hypoglycemia in inpatients with diabetes. Trials of such models should be conducted in real time to evaluate their utility to reduce inpatient hypoglycemia.

Introduction

Hypoglycemia is a common and serious complication affecting people with diabetes (1). It is an inappropriately low blood glucose (BG) that results in significant morbidity in people with type 1 diabetes and in many people with type 2 diabetes (2). A BG level of ≤3.9 mmol/L is defined as level 1 hypoglycemia. A BG level of 2.9 mmol/L and lower is defined as level 2 hypoglycemia requiring immediate action, as at that level, neurogenic and neuroglycopenic symptoms begin to occur (3). Hypoglycemia can lead to permanent neurological damage if not treated promptly and can ultimately be fatal (1).

Hypoglycemia is an important and common clinical problem under inpatient settings. Retrospective analysis of a U.S. electronic medical records database showed a 20% incidence of hypoglycemia and a 7% incidence of severe hypoglycemia (4). There was an associated 66% increase in adjusted inpatient mortality risk and greater than 50% increase in length of hospitalization stay. In a cross-sectional national audit of over 200 hospitals in the U.K., the 2017 National Diabetes Inpatient Audit showed that almost one in five people with diabetes experience hypoglycemia during their hospital stay. Although only 7% experience a severe (level 2) hypoglycemic episode, this rises to 26.9% of all patients with type 1 diabetes with 185 people over the course of 1 week requiring injectable rescue treatment for their hypoglycemia. Inpatient hypoglycemia has been implicated in the development of adverse clinical and economic outcomes, including increased mortality (5–7), adverse cardiovascular outcomes (8,9), and increased duration of hospital stay (6,10,11).

A recent review article has highlighted the urgent need for evidence-based methodologies to reduce inpatient hypoglycemia. Several strategies have already been developed to predict and prevent the occurrence of inpatient hypoglycemia (12). One approach to reducing inpatient hypoglycemia is to retrospectively analyze historical clinical data and develop a prediction tool to determine the individualized risk of hypoglycemia during an inpatient admission. With such a prediction tool, prevention measures can be tested in inpatients with high hypoglycemia risk. The possibility of developing such a prediction tool lies in the growing availability of rich clinical data sets stored in a hospital’s electronic patient records (EPR) system.

Previous studies have used clinical information from local healthcare systems to develop inpatient hypoglycemia risk prediction tools (13–15). In one study, the prediction model developed by the researchers was tested in a clinical trial. This demonstrated the feasibility of using such a model to reduce severe hypoglycemia (glucose <40 mg/dL or <2.2 mmol/L) in inpatients with diabetes (14).

However, previous studies have only applied multilinear or logistic regression models on these data sets resulting in only a modest predictive capability of the models. Over the last few years, a number of advanced machine learning techniques have been developed within the field of biomedical engineering (16). These can be used to create predictive models which can be tested and compared with traditional logistic regression models in order to determine the model with the best predictive performance. This is the first study to assess the performance of novel machine learning models in predicting the risk of inpatient hypoglycemia.

Research Design and Methods

We compared the performance of 18 different machine learning algorithms in predicting the risk of hypoglycemia in inpatients with diabetes.

Data Set

The study was approved by the Oxford University Hospitals National Health Service Foundation Trust Clinical Data Warehouse Program Board following completion of a Data Protection Impact Assessment. Data from Oxford University Hospitals National Health Service Foundation Trust was used which included the Cerner EPR system, the laboratory information management system and the point-of-care testing system. All the data used was collected for routine patient management with no additional data input required for the modeling. The data set contains hospital admission data from 1 September 2014 to 30 June 2018 for qualified patients with diabetes. The qualified patients are defined as meeting the following criteria: 1) being an inpatient as coded in the EPR; 2) having one diagnosis code among E10 (type 1 diabetes), E11 (non–type 1 diabetes), E13 (other specified diabetes), E14 (unspecified diabetes), or O24 (diabetes in pregnancy) as defined in the World Health Organization ICD-10 (17); and 3) having at least one BG test performed during the hospital admissions. Hospital admission data for qualified patients, including patient demographics, procedures undertaken, diagnosis, laboratory tests, medication administration details, and vital signs, were extracted from different source data systems and pooled into a final data table for use by the machine learning prediction models. A schematic representation of the data flow from the source data systems to the final data set used in the current study is depicted in Supplementary Fig. 1.

Predictors and Outcome

The outcome of interest in the current study is the risk of inpatient hypoglycemia during a hospital admission. We prepared two binary outcome variables Hypo<4.0 and Hypo<3.0 for each hospital admission representing two different severities of the potential hypoglycemic episodes since the degree of hypoglycemia may be influenced by different clinical predictors. We put value 1 to Hypo<4.0 for any level 1 hypoglycemic episode (any BG measurement <4 mmol/L) and value 1 to Hypo<3.0 for any level 2 hypoglycemic episode (any BG measurement <3 mmol/L) detected during the hospital admission, and value 0 to the two variables if no hypoglycemic episode detected (all BG measurements >4 mmol/L).

We preprocessed the integrated data set from the EPR and prepared 42 candidate predictors of interest based on clinical knowledge and previous studies. The predictors cover patient demographics (age, sex, and so on), procedures (value 1 for at least one procedure undertaken), laboratory test results (sodium, potassium levels, and so on), medication administration details (names and doses of medication delivered including different types of insulin), and vital signs (temperature, heart rate, and so on). Additional variables were added to the data set to improve the performance of the prediction algorithm. These included an episode of hypoglycemia in a previous admission within 6 months. Table 1 shows the full list of predictors and how they were represented in the source data systems and in the prediction models with units of the predictors provided.

View this table:
  • View inline
  • View popup
Table 1

Potential predictors and how they are represented in the EPR and in the prediction models

Prediction Models

We evaluated the prediction performance of 18 different machine learning models on the data set. The models were used to predict the risk of hypoglycemia (either BG <4.0 mmol/L [Hypo<4.0] or <3.0 mmol/L [Hypo<3.0]). A total of 42 different variables were used as inputs into the prediction model. The models cover a wide range of commonly used classification algorithms including logistic regression, random forests, and artificial neural networks that have been previously demonstrated to be robust and applicable to big data sets.

Model Validation and Comparison

For internal model validation, we used a 10-fold cross-validation. We randomly selected nine-tenths of the data set to be the training data set (developing the model) and the remaining one-tenth to be used to validate the model.

The outcome variables indicate whether or not a hypoglycemic episode occurred during an admission. The model was constructed to predict the probability of at least one hypoglycemic episode occurring. We measured the area under the receiver operating characteristic curve (AUROC), which shows the probability that the model correctly ranks the risk of hypoglycemia) higher no hypoglycemia. The AUROC is not sufficient on its own to use as a hypoglycemia prognostic model as it does not take into account the prevalence of hypoglycemia in the population. It assumes that positive and negative predictions are equally important. A detailed technical analysis of the shortcomings of the AUROC was recently conducted by Saito et al. (18). To ensure a comprehensive assessment of predictive performance, we used additional metrics. We used the terminology TP to represent the number of true positive predictions; FP to represent the number of false positives; TN to represent the number of true negatives; and FN to represent the number of false negatives. We defined precision (positive predictive value) as the ratio TP/(TP + FP). This is a measure of the ability of the model to correctly predict a patient as having hypoglycemia. We defined recall (sensitivity) as the ratio TP/(TP + FN). This is a measure of the ability of the model to label as hypoglycemic all of patients who did indeed develop hypoglycemia. The precision and the recall were calculated for each machine learning model (Table 3).

Model Development

Following the development of the initial model (inpatient hypo [IH]), two further iterations of the data set were carried out. In the first change to the data set, the dose of insulin was added and the variable of intravenous insulin was indicated separately (IH+). In the second change, any previous admission in the last months containing a low glucose (<3 and <4 mmol/L) were added (previous hypo [PH]). The 18 different machine learning models were then rerun on each of these new data sets, and the model outputs were compared with the original data set. The last column in Table 1 shows which variable is used in each data set.

Variable Ranking

We sought to understand how the different variables contributed to the predictions by the XGBoost model (the best predictive model among the 18 tested models). We evaluated the predictive power of each individual variable by providing XGBoost with one variable at a time and assessing the diagnostic accuracy of the model that it constructed using only that variable. We evaluated the AUROC (using 10-fold cross-validation) to get a full picture of each variable’s predictive power.

Statistical Analysis

All statistical analyses were performed using Python 3.6 and R version 3.3. All algorithms were implemented with the machine learning library “sklearn” (19) that contains all the algorithms and data science utilities used in this report. Internal validation was obtained via 10-fold stratified cross-validation. Performance comparisons were made with t tests from which we specified P values <0.001 to be considered statistically significant.

Results

Baseline Characteristics

We analyzed data obtained from 17,658 inpatients with diabetes (9,277 males, mean [SD], age 66 [18] years) who underwent 32,758 hospital admissions between July 2014 and August 2018. We identified all the biochemical (level 1) and clinically significant (level 2) hypoglycemic episodes during these admissions. The incidence of biochemical hypoglycemia during a hospital admission was 21.5% and that of clinically significant hypoglycemia was 9.6%. This is in keeping with data from the National Diabetes Inpatient Audit (20).

A selection of the baseline characteristics of the inpatient cohort and the glycemic outcomes are reported in Table 2.

View this table:
  • View inline
  • View popup
Table 2

Baseline characteristics and glycemic outcomes of the inpatients cohorts

Model Performance

The performance metrics of the machine learning models tested on the PH data set are presented in Table 3. The AUROC varied between 0.62 and 0.96 for different machine learning models. The estimation performance was better when predicting the risk of Hypo<3.0 compared with predicting that of Hypo<4.0. The AUROC for the logistic regression model was acceptable with 0.73 and 0.75 for estimation of the risk of Hypo<4.0 and Hypo<3.0, respectively. However, the best performing model for predicting the risk of Hypo<4.0 and Hypo<3.0 was the XGBoost model, which had the highest AUROC (0.96 for both), the highest precision value (0.88), as well as a high recall value (0.70) among all the models. Figure 1 shows the ROC curves contrasting the logistic regression, gradient boosting, decision tree, and XGBoost models. Supplementary Table 4 shows the normalized confusion matrix for these four models with the true positive and true negative rates on the upper left and lower right in the matrices, respectively. The XGBoost model was again the best performing model with a true positive rate of 0.98 and a true negative rate of 0.71 (Supplementary Table 4).

View this table:
  • View inline
  • View popup
Table 3

Performance metrics of the machine learning models based on the PH data set

Figure 1
  • Download figure
  • Open in new tab
  • Download powerpoint
Figure 1

ROC curves for logistic regression, XGBoost, and decision tree model when predicting biochemical hypoglycemia.

Logistic Regression Model

Estimated regression coefficients with SEs and P values from the logistic regression model with the PH data set are presented in Supplementary Table 3. The variables that are significant predictors of hypoglycemia are shown in Table 4. Similar predictors for Hypo<4.0 and Hypo<3.0 were found. Significant predictors for both Hypo<4.0 and Hypo<3.0 included weight, type of diabetes, oxygen saturation, albumin level, sulfonylurea use, metformin use, intravenous insulin titration, long-acting human insulin use, procedures undertaken, and previous hypoglycemic episode.

View this table:
  • View inline
  • View popup
Table 4

Most significant predictors from the logistic regression model

Model Development

Two iterations of the original data set were carried out during the development of the prediction model. Supplementary Table 1 shows that the addition of a variable to discriminate patients on intravenous insulin compared with those on subcutaneous insulin and a variable for the dose of subcutaneous insulin (IH + data set) increased the best performing model (XGBoost) AUROC by 3 percentage points for Hypo<4.0. When the PH data set was used, all models showed higher AUROC values while the XGBoost and gradient boosting models stood out with a significant increase of AUROC by 15 percentage points.

Variable Ranking

Supplementary Table 2 demonstrates the relative importance of the variables with the top three most important variables being previous hypoglycemic episodes, albumin levels, and type 2 diabetes. A number of novel predictive variables were identified from the machine learning method. These include several vital signs and medications that have logical clinical rationale underlying their importance to hypoglycemia. Further studies will be required to confirm their importance in the development of hypoglycemia.

Conclusions

To our knowledge, this is the first study comparing the performance of advanced machine learning models in predicting the risk of inpatient hypoglycemia. With the rich inpatient data set collected from the EPR system in a large university hospital, the 18 machine learning models showed high predicting power with an average AUROC at 0.85 for the detection of clinically significant hypoglycemia. The model with the highest AUROC (0.96) was the XGBoost model, which outperformed the logistic regression model. This model performed equally well in predicting both level 1 hypoglycemia (BG <4 mmol/L) and level 2 hypoglycemia (BG <3 mmol/L).

Among the 18 evaluated machine learning models, most machine learning prognostic models performed markedly better than the predictions of the linear regression model. XGBoost is a highly flexible nonparametric model that integrates a large number of other machine learning models (decision trees). It was consistently the best performer with the highest AUROC, the highest precision, and good recall. It is important to note that XGBoost performed significantly better than the linear regression model, which has been used to predict inpatient hypoglycemia. There was an improvement of over 20 percentage points in terms of AUROC for prediction of both biochemical and clinically significant hypoglycemia. The high predictive capability of the XGBoost model also came with a high precision and recall showing low levels of overestimation of risk and low levels of missed events.

With the stepwise iteration of the data set, the predictive ability of the XGBoost model and some other models improved significantly. This was in contrast to the logistic regression model, which showed no significant improvement. This emphasizes the importance of developing a clinically relevant and comprehensive data set on which to base any machine learning in order to optimize the capability of the learning algorithm.

Our data set covers a wide range of potential predictors of inpatient hypoglycemia. The logistic regression model detected a number of significant predictors for clinically significant hypoglycemia including weight, type of diabetes, diastolic blood pressure, oxygen saturation, temperature, albumin levels, sulfonylurea use, metformin use, intravenous insulin use, high dose of long-acting human insulin, and people undertaking procedures (see Supplementary Table 3). Previous studies also found similar predictors, such as the albumin levels and glucose-lowering drugs (13,15,21). In our data set, undertaking any kind of procedure was also found to be a significant predictor. This is clinically understandable as procedures may disrupt the daily hospital routine of food intake and drug administration and, thereby, cause increased variability in glycemic levels.

Our machine learning models outperform other inpatient hypoglycemia prediction models that have been published using logistic or multivariate regression techniques. These have shown a discrimination of between 0.70 and 0.80 (15,21). Our model development also includes a variable for previous hypoglycemia, which has not previously been included in other prediction models.

Machine learning models have been widely used within hospital information systems to predict the risk of emergency admission, sepsis in the intensive care unit, and identifying type 2 diabetes using electronic health records (22–24). The performance of the current study also compares favorably to these other predictive models that have an AUROC of between 0.75 and 0.85.

There are several key strengths of the current study. First, we evaluated a wide range of machine learning models and compared their predicting ability against the most commonly used statistical model, which we used as a benchmark model. Second, we used an iterative approach to develop the model with additional variables that revealed how significant improvements to the model could be achieved. Third, this was the largest data set used to predict inpatient hypoglycemia containing the data for 32,758 hospital admissions. It also integrated clinical information that previous studies have not considered before, such as the medication dosage information and hypoglycemia in previous admissions.

However, as with all modeling efforts, there are also limitations. One limitation with the data set is the unavailability of carbohydrate intake/meal content information from the EPR during the hospital admissions. Carbohydrate intake has a direct impact on the postprandial glucose excursions and, consequently, the prandial insulin doses titrated to individuals and could be an important predictor for hypoglycemic events. This information is unavailable because it is not routinely recorded electronically in hospitals. Second, the current electronic health record does not record the level of hypoglycemia unawareness, prior continuous glucose monitoring (CGM) data, or prior self-monitoring of BG. These data are likely to be an important factor in developing hypoglycemia while in hospital, although for acutely unwell patients, these data may not be directly applicable. Third, our data set is derived from a single organization, and the generalizability of our best performing machine learning model needs to be tested in other data sets and evaluated in other centers. Finally, although we have developed a highly predictive model for inpatient hypoglycemia, the feasibility of using this model needs to be tested within a live EPR to confirm the ability of the model to receive data in real time and ensure that the model performs as strongly as the current data suggests.

One of the advantages of the current prediction model is that it uses variables that are readily accessible within the EPR. As a result, the model can be integrated into a decision support system under the EPR framework. In practice, the decision support system would access the clinical information of a new inpatient and feed the required information to the prediction model, which would then calculate the risk of the patient experiencing either biochemical or clinically significant hypoglycemia during his or her hospital admission. This would enable the decision support system to suggest appropriate treatment options based on individual risk levels, thereby reducing hypoglycemia and its consequent associated morbidity and potentially reduce the economic burden of prolonged hospital stay due to hypoglycemia. In a previous single-center study, a linear regression prediction model with a sensitivity of 50% and specificity of 71% was used to detect patients at risk for hypoglycemia. Clinician education resulted in medication change in 40% of patients and a reduction of 68% in the rate of severe hypoglycemia in alerted high-risk patients versus nonalerted high-risk patients (14). The benefits of a significantly more powerful prediction model based on machine learning need to be evaluated in a large multicenter randomized controlled trial.

Another potential application of the prediction model is the selection of high-risk patients who may benefit from an advanced treatment option, such as CGM or closed-loop insulin delivery. The use of CGM in the inpatient setting was considered at a recent symposium where the trials in both the intensive care unit and non–intensive care unit settings were reviewed (25). While there was some evidence that CGM may reduce the rates of severe hypoglycemia, it was recognized that there were limited data on clinical outcomes and that such technology may be most suitable for “populations.. at high risk for glucose variability and hypoglycemia.” Closed-loop insulin delivery, or artificial pancreas, is a novel treatment option for people with diabetes who require exogenous insulin administration (26). The system titrates insulin based on real-time glucose monitoring and a titration algorithm. Previous clinical studies have shown promising glycemic results of the closed-loop systems under inpatient settings (27,28). However, the system is costly and cannot be applied in every inpatient with diabetes. Prediction models, such as the one described in this study, could be used as a preselection tool to determine which patients would benefit the most from CGM or automated insulin delivery.

In conclusion, this study demonstrates for the first time, the utility of advanced machine learning models in predicting the risk of hypoglycemia for inpatients with diabetes. We have shown that these models are significantly better in predicting inpatient hypoglycemia than the traditional logistic regression model. However further trials are needed to determine if this prediction model provides a significant clinical advantage over traditional logistic regression analysis or more simple risk factor prediction models, e.g., insulin use alone. Such machine learning models need to be evaluated within a real-time clinical setting to demonstrate their ability to predict hypoglycemia following admission. The use of new technological methods, such as machine learning and artificial intelligence, are not a substitute for clinicians, but they should be used to enhance clinical judgement and support everyday decisions. Multicenter clinical trials are now needed to evaluate their utility within a clinical decision support system and reduce the burden of hypoglycemia in hospital.

Article Information

Funding. Y.R. received salary funding from a Novo Nordisk Postdoctoral Fellowship run in partnership with the University of Oxford. G.D.T., A.L., J.D., and R.R. are partly funded by the National Institute for Health Research Oxford Biomedical Research Centre.

Duality of Interest. No potential conflicts of interest relevant to this article were reported.

Author Contributions. Y.R. and A.B. carried out the data analysis. Y.R. and R.R. designed the study analysis and drafted the manuscript. All authors contributed to the interpretation of the results and critical review of the manuscript. Y.R. and R.R. are the guarantors of this work and, as such, had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Footnotes

  • The views expressed are those of the author(s) and not necessarily those of the National Health Service, the National Institute for Health Research, or the Department of Health.

  • This article contains supplementary material online at https://doi.org/10.2337/figshare.12091953.

  • Received August 30, 2019.
  • Accepted April 4, 2020.
  • © 2020 by the American Diabetes Association
https://www.diabetesjournals.org/content/license

Readers may use this article as long as the work is properly cited, the use is educational and not for profit, and the work is not altered. More information is available at https://www.diabetesjournals.org/content/license.

References

  1. ↵
    1. Cryer PE,
    2. Davis SN,
    3. Shamoon H
    . Hypoglycemia in diabetes. Diabetes Care 2003;26:1902–1912
    OpenUrlAbstract/FREE Full Text
  2. ↵
    1. UK Hypoglycaemia Study Group
    . Risk of hypoglycaemia in types 1 and 2 diabetes: effects of treatment modalities and their duration. Diabetologia 2007;50:1140–1147
    OpenUrlCrossRefPubMedWeb of Science
  3. ↵
    1. Agiostratidou G,
    2. Anhalt H,
    3. Ball D, et al
    . Standardizing clinically meaningful outcome measures beyond HbA1c for type 1 diabetes: a consensus report of the American Association of Clinical Endocrinologists, the American Association of Diabetes Educators, the American Diabetes Association, the Endocrine Society, JDRF International, The Leona M. and Harry B. Helmsley Charitable Trust, the Pediatric Endocrine Society, and the T1D Exchange. Diabetes Care 2017;40:1622–1630
    OpenUrlAbstract/FREE Full Text
  4. ↵
    1. Brodovicz KG,
    2. Mehta V,
    3. Zhang Q, et al
    . Association between hypoglycemia and inpatient mortality and length of hospital stay in hospitalized, insulin-treated patients. Curr Med Res Opin 2013;29:101–107
    OpenUrlCrossRefPubMed
  5. ↵
    1. Borzi V,
    2. Fontanella A
    . The clinical impact of hypoglycemia in hospitalized patients. Ital J Med 2015;9:11–19
    OpenUrl
  6. ↵
    1. Gómez-Huelgas R,
    2. Guijarro-Merino R,
    3. Zapatero A, et al
    . The frequency and impact of hypoglycemia among hospitalized patients with diabetes: a population-based study. J Diabetes Complications 2015;29:1050–1055
    OpenUrl
  7. ↵
    1. Turchin A,
    2. Matheny ME,
    3. Shubina M,
    4. Scanlon JV,
    5. Greenwood B,
    6. Pendergrass ML
    . Hypoglycemia and clinical outcomes in patients with diabetes hospitalized in the general ward. Diabetes Care 2009;32:1153–1157
    OpenUrlAbstract/FREE Full Text
  8. ↵
    1. Akhavan P,
    2. Aghili R,
    3. Malek M,
    4. Ebrahim Valojerdi A,
    5. Khamseh ME
    . Hypoglycemia: adverse cardiovascular outcomes in non-critically ill people with type 2 diabetes. Arch Iran Med 2016;19:82–86
    OpenUrl
  9. ↵
    1. Carey M,
    2. Boucai L,
    3. Zonszein J
    . Impact of hypoglycemia in hospitalized patients. Curr Diab Rep 2013;13:107–113
    OpenUrl
  10. ↵
    1. Evans M,
    2. Wolden ML,
    3. Thorsted BL,
    4. McEwan PC,
    5. Jacobsen JL
    . Inpatient hypoglycaemia increases length of hospital stay and all-cause mortality risk. Diabet Med 2015;32:23
    OpenUrl
  11. ↵
    1. Nirantharakumar K,
    2. Marshall T,
    3. Kennedy A,
    4. Narendran P,
    5. Hemming K,
    6. Coleman JJ
    . Hypoglycaemia is associated with increased length of stay and mortality in people with diabetes who are hospitalized. Diabet Med 2012;29:e445–e448
    OpenUrlCrossRefPubMed
  12. ↵
    1. Ruan Y,
    2. Tan GD,
    3. Lumb A,
    4. Rea RD
    . Importance of inpatient hypoglycaemia: impact, prediction and prevention. Diabet Med 2019;36:434–443
    OpenUrl
  13. ↵
    1. Stuart K,
    2. Adderley NJ,
    3. Marshall T, et al
    . Predicting inpatient hypoglycaemia in hospitalized patients with diabetes: a retrospective analysis of 9584 admissions with diabetes. Diabet Med 2017;34:1385–1391
    OpenUrl
  14. ↵
    1. Kilpatrick CR,
    2. Elliott MB,
    3. Pratt E, et al
    . Prevention of inpatient hypoglycemia with a real-time informatics alert. J Hosp Med 2014;9:621–626
    OpenUrlCrossRefPubMed
  15. ↵
    1. Mathioudakis NN,
    2. Everett E,
    3. Routh S, et al
    . Development and validation of a prediction model for insulin-associated hypoglycemia in non-critically ill hospitalized adults. BMJ Open Diabetes Res Care 2018;6:e000499
    OpenUrlAbstract/FREE Full Text
  16. ↵
    1. Park C,
    2. Took CC,
    3. Seong JK
    . Machine learning in biomedical engineering. Biomed Eng Lett 2018;8:1–3
    OpenUrl
  17. ↵
    1. WHO
    . World Health Organization. ICD-10 version:2010 [Internet], 2010. Available from https://icd.who.int/browse10/2019/en. Accessed 20 March 2019
  18. ↵
    1. Saito T,
    2. Rehmsmiere M
    . The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. Plos One 2015;10:e0118432
    OpenUrlCrossRefPubMed
  19. ↵
    1. Pedregosa F,
    2. Varoquaux G,
    3. Gramfort A, et al
    . Scikit-learn: machine learning in Python. J Mach Learn Res 2011;12:2825–2830
    OpenUrlCrossRefPubMed
  20. ↵
    1. NHS Digital
    . National Diabetes Inpatient Audit England and Wales, 2017 [Internet], 2017. Available from https://files.digital.nhs.uk/pdf/s/7/nadia-17-rep.pdf
  21. ↵
    1. Elliott MB,
    2. Schafers SJ,
    3. McGill JB,
    4. Tobin GS
    . Prediction and prevention of treatment-related inpatient hypoglycemia. J Diabetes Sci Technol 2012;6:302–309
    OpenUrlCrossRefPubMed
  22. ↵
    1. Rahimian F,
    2. Salimi-Khorshidi G,
    3. Payberah AH, et al
    . Predicting the risk of emergency admission with machine learning: development and validation using linked electronic health records. PLoS Med 2018;15:e1002695
    OpenUrl
    1. Desautels T,
    2. Calvert J,
    3. Hoffman J, et al
    . Prediction of sepsis in the intensive care unit with minimal electronic Health record data: a machine learning approach. JMIR Med Inform 2016;4:e28
    OpenUrlCrossRefPubMed
  23. ↵
    1. Zheng T,
    2. Xie W,
    3. Xu L, et al
    . A machine learning-based framework to identify type 2 diabetes through electronic health records. Int J Med Inform 2017;97:120–127
    OpenUrl
  24. ↵
    1. Wallia A,
    2. Umpierrez GE,
    3. Rushakoff RJ, et al
    . Consensus statement on inpatient use of continuous glucose monitoring. J Diabetes Sci Technol 2017;11:1036–1044
    OpenUrl
  25. ↵
    1. Boughton CK,
    2. Hovorka R
    . Advances in artificial pancreas systems. Sci Transl Med 2019;11:eaaw4949
    OpenUrlFREE Full Text
  26. ↵
    1. Bally L,
    2. Thabit H,
    3. Hartnell S, et al
    . Closed-loop insulin delivery for glycemic control in noncritical care. N Engl J Med 2018;379:547–556
    OpenUrlCrossRefPubMed
  27. ↵
    1. Thabit H,
    2. Hartnell S,
    3. Allen JM, et al
    . Closed-loop insulin delivery in inpatients with type 2 diabetes: a randomised, parallel-group trial. Lancet Diabetes Endocrinol 2017;5:117–124
    OpenUrl
PreviousNext
Back to top
Diabetes Care: 43 (7)

In this Issue

July 2020, 43(7)
  • Table of Contents
  • Table of Contents (PDF)
  • About the Cover
  • Index by Author
  • Masthead (PDF)
Sign up to receive current issue alerts
View Selected Citations (0)
Print
Article Alerts
Sign In to Email Alerts with your Email Address
Email Article

Thank you for your interest in spreading the word about Diabetes Care.

NOTE: We only request your email address so that the person you are recommending the page to knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Enter multiple addresses on separate lines or separate them with commas.
Predicting the Risk of Inpatient Hypoglycemia With Machine Learning Using Electronic Health Records
(Your Name) has forwarded a page to you from Diabetes Care
(Your Name) thought you would like to see this page from the Diabetes Care web site.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Citation Tools
Predicting the Risk of Inpatient Hypoglycemia With Machine Learning Using Electronic Health Records
Yue Ruan, Alexis Bellot, Zuzana Moysova, Garry D. Tan, Alistair Lumb, Jim Davies, Mihaela van der Schaar, Rustam Rea
Diabetes Care Jul 2020, 43 (7) 1504-1511; DOI: 10.2337/dc19-1743

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
Add to Selected Citations
Share

Predicting the Risk of Inpatient Hypoglycemia With Machine Learning Using Electronic Health Records
Yue Ruan, Alexis Bellot, Zuzana Moysova, Garry D. Tan, Alistair Lumb, Jim Davies, Mihaela van der Schaar, Rustam Rea
Diabetes Care Jul 2020, 43 (7) 1504-1511; DOI: 10.2337/dc19-1743
del.icio.us logo Digg logo Reddit logo Twitter logo CiteULike logo Facebook logo Google logo Mendeley logo
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Jump to section

  • Article
    • Abstract
    • Introduction
    • Research Design and Methods
    • Results
    • Conclusions
    • Article Information
    • Footnotes
    • References
  • Figures & Tables
  • Suppl Material
  • Info & Metrics
  • PDF

Related Articles

Cited By...

More in this TOC Section

  • The SEE Study: Safety, Efficacy, and Equity of Implementing Autonomous Artificial Intelligence for Diagnosing Diabetic Retinopathy in Youth
  • Extended Use of the Control-IQ Closed-Loop Control System in Children With Type 1 Diabetes
  • A Randomized Clinical Trial Assessing Continuous Glucose Monitoring (CGM) Use With Standardized Education With or Without a Family Behavioral Intervention Compared With Fingerstick Blood Glucose Monitoring in Very Young Children With Type 1 Diabetes
Show more Emerging Technologies: Data Systems and Devices

Similar Articles

Navigate

  • Current Issue
  • Standards of Care Guidelines
  • Online Ahead of Print
  • Archives
  • Submit
  • Subscribe
  • Email Alerts
  • RSS Feeds

More Information

  • About the Journal
  • Instructions for Authors
  • Journal Policies
  • Reprints and Permissions
  • Advertising
  • Privacy Policy: ADA Journals
  • Copyright Notice/Public Access Policy
  • Contact Us

Other ADA Resources

  • Diabetes
  • Clinical Diabetes
  • Diabetes Spectrum
  • Scientific Sessions Abstracts
  • Standards of Medical Care in Diabetes
  • BMJ Open - Diabetes Research & Care
  • Professional Books
  • Diabetes Forecast

 

  • DiabetesJournals.org
  • Diabetes Core Update
  • ADA's DiabetesPro
  • ADA Member Directory
  • Diabetes.org

© 2021 by the American Diabetes Association. Diabetes Care Print ISSN: 0149-5992, Online ISSN: 1935-5548.