Validation of Prediction of Diabetes by the Archimedes Model and Comparison With Other Predicting Models

The SAHS is a prospective cohort study consisting of 3,682 individuals (62% Mexican American and 38% non-Hispanic white) followed for 7–8 years (4). The SAHS predicting model is a multiple logistic regression model with incident diabetes as the dependent variable and a panel of baseline characteristics that are ordinarily available in a routine clinical setting as independent variables (2). The ARIC predicting model is a similarly constructed logistic regression model (3).

The Archimedes model is built from underlying anatomy and physiology and uses scores of ordinary and differential equations to represent metabolic pathways, occurrence and progression of diseases, signs and symptoms, treatments, and outcomes. A practical, free, readily available tool derived from the Archimedes model is the American Diabetes Association's Diabetes PHD (Personal Health Decisions; available at http://diabetes.org/diabetesPHD). Diabetes PHD can simultaneously predict the risk of diabetes and numerous other outcomes, including the effects of a wide variety of treatments in many different populations (e.g., those with diabetes). It was used here to provide external validation of its prediction of the incidence of diabetes.

Among the 3,228 individuals in the SAHS who were nondiabetic at baseline, 295 developed diabetes over the 7–8 years of follow-up. All the required elements for the Archimedes risk estimation were available in the subjects selected for the present analyses. The present analyses were restricted to the recent cohort 2 of SAHS, which included 1,734 nondiabetic individuals, 195 of whom were diabetic at follow-up. Within the SAHS database, we selected 100 individuals at random, 50 of whom were diabetic at follow-up and 50 who remained free of diabetes at follow-up. This sample size would provide 80% power to detect an aROC significantly (P < 0.05) greater than 0.70 (the low end of acceptable discrimination [5]) if the true aROC was >0.80 and 90% power if the true aROC was 0.83 (benchmark values near that of other established models) (2,3).

The risk of developing diabetes for each individual was determined according to the years of follow-up for that individual (rounded to the nearest year), which ranged from 6–9 with a mean of 7.5. Data from each individual were entered into Diabetes PHD and the results obtained from the graphical output displayed on the computer screen. A second person confirmed the accuracy of the input and, in a random sample of 20 forms, also confirmed the output from Diabetes PHD.

We also estimated the risk of diabetes for the same 100 individuals using both the SAHS diabetes predicting model and the ARIC predicting model. The aROC's and CIs for all three models were computed and compared (6). Finally, we computed the Spearman correlation coefficients between the risk estimates obtained from each pair of predicting models.

The aROC for Diabetes PHD was 0.818 (95% CI 0.739–0.899) and was not statistically different than the aROC of the SAHS model (0.869 [95% CI 0.801- 0.936]) or the ARIC model (0.870 [0.802–0.937]) (Fig. 1). The risk estimates from the SAHS model and ARIC model were highly correlated (r = 0.962), and both correlated well with Diabetes PHD (r = 0.834 and 0.842, respectively).

With an aROC of 0.818, it is evident that the accuracy of Diabetes PHD (i.e., Archimedes) to predict an individual's risk of diabetes is excellent—almost as high as models specifically designed and used only for that purpose. The SAHS model may have had an unfair advantage over Archimedes because it was designed and optimized using the SAHS database and could be overfitted to the subset of SAHS cases selected for this analysis. It was for that reason that we used the ARIC predicting model: the latter was developed in an entirely independent dataset and performed as well as the SAHS model.

Both the SAHS and ARIC models were built from person-specific data and optimized specifically for predicting incident diabetes. In contrast, Archimedes was designed to be used for a very wide range of purposes, calculates many different outcomes, was not built from person-specific data, and was not calibrated to determine the incidence of diabetes. Also, several of the variables Archimedes uses that may have enhanced its predictive capability were not included in this analysis.

This report extends the validation of Archimedes and demonstrates its excellent ability to discriminate between individuals who will or will not develop diabetes. Its utility is comparable with models developed solely for that purpose. Because Diabetes PHD, derived from Archimedes, is freely available on the internet and calculates many additional outcomes, it is a powerful tool that can be reliably used for comprehensive risk assessment and decision making. Diabetes PHD is now widely accessed (∼80,000 users per year) for use in comprehensive risk assessment of cardiometabolic disease over a 30-year period and that helps diabetic patients better appreciate the likely benefits of risk factor reduction. Although the tool currently uses complex distributive computing, which limits its speed and capacity, a much more rapid version will soon become available with unlimited capacity. This will allow for widespread promotion.

This study was supported by grants from the National Heart, Lung, and Blood Institute (R01 HL24799 and R01 HL36820).