Validation of Risk Equations for Complications of Type 2 Diabetes (RECODe) Using Individual Participant Data From Diverse Longitudinal Cohorts in the U.S.
Abstract
OBJECTIVE We sought to validate Risk Equations for Complications of Type 2 Diabetes (RECODe) among diverse populations.
RESEARCH DESIGN AND METHODS We compared risk predictions from RECODe equations and from two alternative risk models (UK Prospective Diabetes Study Outcomes Model 2 [UKPDS OM2] and American College of Cardiology/American Heart Association Pooled Cohort Equations) to observed outcomes in two studies: the Multi-Ethnic Study of Atherosclerosis (MESA, n = 1,555 adults with type 2 diabetes, median follow-up 9.1 years) and the Jackson Heart Study (JHS, n = 1,746 adults with type 2 diabetes, median follow-up 8.0 years). Outcomes included nephropathy by multiple measures (microalbuminuria, macroalbuminuria, renal failure, end-stage renal disease, and reduction in glomerular filtration rate), moderate to severe diabetic retinopathy by Airlie House classification, fatal or nonfatal myocardial infarction, fatal or nonfatal stroke, congestive heart failure, and all-cause mortality.
RESULTS RECODe equations for microvascular and cardiovascular outcomes had C-statistics for discrimination ranging from 0.71 to 0.85 in MESA and 0.64 to 0.91 in JHS for alternative outcomes. Calibration slopes in MESA ranged from 0.62 for a composite nephropathy outcome, 0.83–1.04 for individual nephropathy outcomes, 1.07 for retinopathy, 1.00–1.05 for cardiovascular outcomes, and 1.03 for all-cause mortality. Slopes in JHS ranged from 0.47 for retinopathy, 0.97–1.16 for nephropathy, 0.72–1.05 for cardiovascular outcomes, and 1.01 for all-cause mortality. The alternative models had C-statistics 0.50–0.72 and calibration slopes 0.07–0.60.
CONCLUSIONS RECODe equations improved risk estimation for diverse patients with type 2 diabetes, as compared with two commonly used alternatives.
Introduction
Risk equations for complications of type 2 diabetes can help guide clinical decisions and population health management, such as to identify patients for intensive outreach (1,2), particularly as hemoglobin A1c alone is a limited predictor of outcomes (3). Risk equations can facilitate decision making (1,4) and assist comparative effectiveness research on clinical guidelines (5–7).
Previous risk equations systematically misestimate both microvascular and cardiovascular complications among modern populations (8–10). We recently derived new Risk Equations for Complications of Type 2 Diabetes (RECODe) (11) (Supplementary Tables 1 and 2). RECODe equations were derived from the Action to Control Cardiovascular Risk in Diabetes (ACCORD) study (12–14) and validated with the Diabetes Prevention Program Outcomes Study (DPPOS) (15) and Look AHEAD (Action for Health in Diabetes) (16). We observed that predictions from the RECODe equations improved discrimination and calibration as compared with the UK Prospective Diabetes Study Outcomes Model 2 (UKPDS OM2; for microvascular and cardiovascular outcomes) (17) and the American College of Cardiology (ACC)/American Heart Association (AHA) Pooled Cohort Equations (PCEs; for atherosclerotic cardiovascular disease [ASCVD] events) (18).
Because the development and validation of the RECODe equations were performed using data from randomized clinical trials, the equations may have limited generalizability. Here, we sought to evaluate the validity of the RECODe equations against two longitudinal cohort studies: the Multi-Ethnic Study of Atherosclerosis (MESA) and the Jackson Heart Study (JHS). MESA and JHS were chosen because they included rigorous adjudication of both microvascular and cardiovascular outcomes; included modern populations among whom older equations are thought to be miscalibrated (8–10); included minorities; and have made de-identified individual participant data available for research use, which improves research transparency and reproducibility.
Research Design and Methods
Sources of Data for Validation of Equations
We used data from two longitudinal, observational cohort studies: MESA (2000–2012, n = 1,555 people with type 2 diabetes) and JHS (2000–2012, n = 1,746 people with type 2 diabetes). See the Supplementary Data for the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) checklist (19).
Participants
MESA participants were selected from six general population geographic areas across the U.S. MESA participants experienced baseline examinations from July 2000 to July 2002, three subsequent examinations each 17–20 months apart, and a fifth follow-up examination between April 2010 and January 2012, with follow-up phone calls between examinations. Eligibility criteria included age 45–84 years and the absence of the following: prior physician diagnosis of angina, myocardial infarction (MI), stroke, or congestive heart failure (CHF); atrial fibrillation at enrollment; prior cardiovascular procedure; active cancer; pregnancy; serious medical condition preventing long-term participation; weight >136 kg (300 lb); cognitive incapacity; living in a nursing home or planning to leave the community within 5 years; inability to communicate in English, Spanish, Cantonese, or Mandarin; and chest computed tomography scan in the year prior to enrollment (20).
JHS participants were recruited via community-based sampling from the urban and rural African American population of three counties in Mississippi, U.S., and included a subset of participants previously in the Atherosclerosis Risk in Communities (ARIC) study. JHS participants experienced baseline examinations from 2000 to 2004, a second examination from 2005 to 2008, and a third from 2009 to 2012, with follow-up phone calls between examinations. Eligibility criteria included the following: age 35–84 years, except in a nested family cohort subset, for which eligibility included people 21–34 years old (21).
Outcomes
Separate RECODe equations were developed for each of several microvascular and cardiovascular outcomes, with unique equations for each alternative definition of each outcome. Participants with each of the outcomes at baseline were excluded, to focus on incident outcomes.
Microvascular outcomes included nephropathy and retinopathy; neuropathy was not assessed in MESA or JHS. Separate RECODe equations were evaluated for each of several outcome definitions of nephropathy: 1) development of microalbuminuria (albumin-to-creatinine ratio ≥30 mg albumin per gram creatinine in urine obtained in exams 1, 2, 3, and 5 in MESA and exams 1–3 in JHS); 2) development of macroalbuminuria (albumin-to-creatinine ratio ≥300 mg albumin per gram creatinine in urine); 3) renal failure or end-stage renal disease (ESRD; dialysis) or serum creatinine >291.7 μmol/L (3.3 mg/dL, obtained in exams 1, 3, 4, and 5 in MESA and 1–3 in JHS); 4) doubling of serum creatinine or >20 mL/min decrease in estimated glomerular filtration rate (eGFR) based on the Modification of Diet in Renal Disease (MDRD) study equation (22); 5) any of the outcomes 1, 3, or 4; and 6) an additional composite outcome for any of 1, 2, or 3. The RECODe equation for diabetic retinopathy was evaluated against the diagnosis of new moderate to severe diabetic retinopathy in either eye, adjudicated by ophthalmologists from bilateral retinal photographs using the Airlie House classification system modified for the Early Treatment of Diabetic Retinopathy Study (ETDRS) (23). The photographs were obtained through ancillary studies on a subset of participants in both MESA (a subset of n = 949 subjects with retinal photographs with exam 2 and n = 905 with exam 5) and JHS (a subset of n = 886 subjects with diabetes or impaired fasting glucose photographed with exam 3) (24,25). Further details about the outcome examinations, definitions, and timing are provided in Table 2. Our analysis was limited to incident microvascular events, meaning that participants who lacked evidence of the outcome at the first examination after diagnosis of diabetes and had the outcome at subsequent examinations were treated as incident. For the retinopathy outcome in JHS, people without a prior history of self-reported vision loss or evidence of prior photocoagulation or vitrectomy at retinal exam were included as incident cases if they had retinopathy on exam 3.
Separate RECODe equations were also evaluated for each of several cardiovascular outcome definitions: 1) composite ASCVD (defined as first fatal or nonfatal MI or stroke); 2) fatal or nonfatal MI; 3) fatal or nonfatal stroke; 4) CHF; or 5) death from any cardiovascular cause, each assessed during every MESA and JHS follow-up period, with exact date of events obtained from health care records. As with the microvascular outcomes, our analyses were limited to incident cardiovascular events after diabetes diagnosis.
We additionally assessed the RECODe equation for all-cause mortality (including both cardiovascular and noncardiovascular mortality).
Predictors
Predictors included age (in years), sex, race/ethnicity (black race and/or Hispanic ethnicity), current medication use (each of the following: statin, oral diabetes medications including metformin, and anticoagulants other than aspirin), current tobacco smoking, systolic blood pressure, and biomarkers (total cholesterol, HDL cholesterol, urine creatinine, urine microalbumin-to-creatinine ratio, and hemoglobin A1c) (see Table 1). The first available value for each predictor after diagnosis of diabetes was entered into the RECODe equations. Time-varying covariates were not included because the RECODe equations were intended for use in clinical settings to assist initial treatment decisions.
Characteristics of the MESA (2000–2012, n = 1,555 people with type 2 diabetes) and JHS (2000–2012, n = 1,746 people with type 2 diabetes) study participants included for validation of RECODe equations
Sample Size
The final sample size of participants (n = 1,555 from MESA and n = 1,746 from JHS) included those with type 2 diabetes. Type 2 diabetes diagnosis was based on prior physician diagnosis, use of insulin or oral hypoglycemia medications without a history of type 1 diabetes, or 2010 American Diabetes Association diagnostic criteria (fasting plasma glucose ≥7.0 mmol/L [126 mg/dL], 2-h plasma glucose ≥11.1 mmol/L [200 mg/dL] during oral glucose tolerance test, hemoglobin A1c ≥48 mmol/mol [6.5%], or random plasma glucose ≥11.1 mmol/L [200 mg/dL] with accompanying hyperglycemia symptoms) (26). Further characteristics of people with type 2 diabetes in each sample are provided in Table 2.
Characteristics of people with type 2 diabetes in each cohort, along with incident microvascular and cardiovascular outcome definitions and timing
Missing Data
In the MESA sample, 11% of participants (n = 172) with type 2 diabetes had one or more predictor variables missing. In the JHS sample, 7% of participants (n = 128) with type 2 diabetes had one or more predictor variables missing. Missing values for predictor variables in both samples were imputed with fivefold multiple imputation by chained equations using the predictive mean matching method, followed by bootstrap resampling across imputations for subsequent computation of confidence intervals around the statistical metrics described below (27).
Statistical Analysis Methods
Performance of the RECODe equations was compared with that of the UKPDS OM2 (17) and the ACC/AHA PCEs (18) by comparing model calibration, discrimination, and reclassification. We do not compare between cohorts (i.e., with the UKPDS cohort) but between equations (i.e., with the UKPDS risk equations). The RECODe equations are Cox proportional hazards models, as are the AHA/ACC PCEs, whereas the UKPDS equations take various functional forms specific to each outcome (Gompertz, Weibull, logistic, or exponential). Model calibration was assessed by the slope and intercept of the line between predicted and observed Kaplan-Meier event rates for each outcome over 10 years by deciles of risk (Greenwood-Nam-D’Agostino [GND] test) (28). The calibration slope is ideally 1 and intercept ideally 0, reflecting a perfect correspondence between predicted and observed event rates of each outcome. The GND test nonparametrically evaluates the distance between predicted and observed Kaplan-Meier outcome rates, so that higher P values, indicating greater concordance (less difference) between predicted and observed outcome rates, are desirable. Model discrimination was assessed by the C-statistic (area under the receiver operating characteristic curve) (29). Following the TRIPOD guidelines (19), time to event for outcomes was taken from the time of study enrollment or the time of diabetes diagnosis (in the event of new diabetes diagnosed during the study), whichever came later, to mimic application of the equations in clinical settings in which patients may be previously diagnosed before application of the equations, or newly diagnosed at the time of application of the equations. Reclassification referred to the ability of the older models versus the newer models to distinguish high-risk from low-risk patients (10-year risk ≥10% vs. <10% for nephropathy, retinopathy, ASCVD, and CHF; and ≥5% vs. <5% for MI and stroke), the cut points often suggested for consideration of more intensive blood pressure, lipid, or glycemic therapies (5,30). The net reclassification index (NRI) was calculated, which is the number of people who had an event who are correctly reclassified as high risk by a new model minus those incorrectly reclassified to low risk, plus the number who did not have an event who are correctly reclassified as low risk by a new model minus those incorrectly reclassified to high risk (i.e., a positive NRI indicates improvement of the new model versus the older model) (30).
Analyses were performed in R and were approved by the Stanford University Institutional Review Board (e-Protocol ID 39274).
Results
Participants
The eligible participant sample from MESA (Table 1) averaged 63.0 years old, was 49.6% female, and was followed for a median of 9.1 years. Of eligible participants, 396 (25%) developed microalbuminuria, macroalbuminuria, or renal failure/ESRD; 34 (2%) developed retinopathy; 92 (6%) had an MI (fatal or nonfatal); 89 (6%) had a stroke (fatal or nonfatal); 117 (8%) developed CHF; 88 (6%) died of cardiovascular disease (CVD); and 323 (21%) died of all causes during the study.
The eligible participant sample from JHS (Table 1) averaged 57.5 years old, was 64.7% female, and was followed for a median of 8.0 years. Of the eligible participants, 280 (16%) developed microalbuminuria, macroalbuminuria, or renal failure/ESRD; 94 (5%) developed retinopathy; 151 (5%) had an MI (fatal or nonfatal); 142 (8%) had a stroke (fatal or nonfatal); 161 (9%) developed CHF; 64 (4%) died of CVD; and 795 (46%) died of all causes during the study. Further details about the disaggregated incident outcomes, timing of incident outcomes, and outcome adjudication in both MESA and JHS are provided in Table 2.
The eligible participant samples from MESA and JHS were of similar age and sex distribution but had greater race/ethnic diversity compared with the ACCORD participant sample (from whom the RECODe equations were derived) and the DPPOS and Look AHEAD samples (among whom the RECODe equations were initially validated) (Supplementary Table 3). The eligible participant sample from MESA and JHS also had fewer participants with a history of pharmaceutical therapy for CVD risk factors (blood pressure and/or statin therapy) as compared with the ACCORD, DPPOS, or Look AHEAD samples (31) (Supplementary Table 3).
Model Performance
In external validation against MESA data (Fig. 1 and Table 3), RECODe equations predicted retinopathy with a calibration slope of 1.07 between expected and observed event rates, a calibration intercept of −0.007, calibration GND test P value of 0.05, and a C-statistic of 0.76. RECODe equations for nephropathy performed best on the calibration and discrimination metrics for the outcomes of microalbuminuria, macroalbuminuria, or the composite outcome of microalbuminuria, macroalbuminuria, or renal failure/ESRD; for these outcomes, calibration slopes were between 0.99 and 1.04, intercepts between 0.004 and 0.032, GND P values 0.08–0.12, and C-statistics 0.76–0.85 (Fig. 1 and Table 3). For the composite outcome of either doubling of serum creatinine or >20 mL/min decrease in eGFR, the RECODe equations had a calibration slope of 0.83, intercept of 0.080, GND P value of 0.05, and C-statistic of 0.77. The RECODe equation performed more poorly for the composite outcome of macroalbuminuria, renal failure, doubling of creatinine, or >20 mL/min decrease in eGFR, which had a calibration slope of 0.62, intercept of 0.157, GND P value of <0.001, and C-statistic of 0.71. The MESA sample did not have an adequate number of patients with ESRD events to conduct calibration testing on ESRD as an isolated outcome (n = 13). RECODe equations predicted cardiovascular outcomes with calibration slopes of 1.00–1.05, calibration intercepts of −0.005 to 0.024, GND P values >0.05 for all outcomes except stroke, and C-statistics varying from 0.73 (for MI) to 0.81 (for CVD death) (Fig. 1 and Table 3). The RECODe equation for all-cause mortality applied to MESA had a calibration slope of 1.03, intercept of −0.009, GND P value of 0.08, and C-statistic of 0.81.
Calibration plots. Plots display Kaplan-Meier event rates over 10 years predicted by the RECODe equations vs. observed rates among MESA (2000–2012, n = 1,555 people with type 2 diabetes) and JHS (2000–2012, n = 1,746 people with type 2 diabetes). Also displayed, where available for each outcome, are predictions from UKPDS OM2. Each set of equations was applied to each data set to provide the comparisons shown here. Points are displayed for deciles of predicted and observed Kaplan-Meier event rates, with lower numbers of centiles than deciles used if less than five events are observed per group, to prevent unstable inferences per current guidelines. A: Nephropathy. B: Retinopathy. C: ASCVD. D: MI. E: Stroke. F: CHF. G: Mortality. 2xScr, doubling of serum creatinine; macroalb, macroalbuminuria; microalb, microalbuminuria.
Validation statistics for microvascular and macrovascular RECODe equations
In external validation against JHS data (Fig. 1 and Table 3), the RECODe equation for retinopathy had a calibration slope of 0.47, intercept of 0.119, GND P value of 0.14, and C-statistic of 0.64 (Fig. 1 and Table 3). The RECODe equations for nephropathy performed best for the outcomes of macroalbuminuria and the composite outcome of microalbuminuria, macroalbuminuria, or renal failure/ESRD, with calibration slopes of 0.98–1.16, intercepts −0.029 to 0.008, GND P values 0.05–0.09, and C-statistics 0.71–0.77 (Fig. 1 and Table 3). For the other nephropathy outcomes, calibration slopes varied from 0.97 (for microalbuminuria) to 1.16 (for the composite of microalbuminuria, macroalbuminuria, or renal falure/ESRD), intercepts from −0.029 to 0.106, GND P values <0.001, and C-statistics from 0.62 (for the composite outcome of macroalbuminuria, renal failure/ESRD, doubling of creatinine, or >20 mL/min decrease in eGFR) to 0.91 for renal failure/ESRD (Fig. 1 and Table 3). The JHS sample did not have adequate ESRD events to conduct calibration testing on ESRD as an isolated outcome (n = 27). RECODe equations predicted cardiovascular outcomes with calibration slopes of 0.94–1.10; calibration intercepts of −0.023 to 0.091; GND P values >0.05 for MI, CHF, and CVD death; and C-statistics varying from 0.72 (for stroke) to 0.87 (for CVD death) (Fig. 1 and Table 3). The RECODe equation for all-cause mortality applied to JHS had a calibration slope of 1.01, intercept of 0.007, GND P value of 0.95, and C-statistic of 0.78 (Fig. 1 and Table 3).
Model performance did not vary substantially when evaluated among subgroups of only participants who were diagnosed with diabetes prior to or at the baseline study visit, or only participants who were diagnosed during the course of the study (Supplementary Fig. 1).
Comparison With Alternative Risk Equations
Compared with the RECODe equations, the UKPDS OM2 equations and the ACC/AHA PCEs had lower C-statistics and calibration slopes in both the MESA and JHS samples (Table 3). For the outcome of retinopathy, the UKPDS OM2 predicted outcomes less well than random chance, with calibration slopes of 0.17 and 0.60 in MESA and JHS, respectively, calibration intercepts of 0.042 and 0.180, GND P values 0.17 and <0.001, and C-statistics of 0.50 and 0.52 (Table 3). For the outcome of ESRD, the UKPDS OM2 had a C-statistic of 0.72 in MESA and 0.56 in JHS; calibration could not be tested due to insufficient events. The UKPDS OM2 lacked equations for the remaining nephropathy outcomes. For cardiovascular outcomes, the UKPDS OM2 equations had calibration slopes ranging from 0.07 to 0.27, intercepts from 0.003 to 0.163, GND P values all <0.001, and C-statistics ranging from 0.54 to 0.61 across the two data sets. For the cardiovascular outcome of ASCVD, the ACC/AHA PCEs had calibration slopes of 0.19 and 0.42, intercepts of 0.019 and 0.0002, GND P values <0.001, and C-statistics of 0.60 and 0.63 in MESA and JHS, respectively (Table 3).
The reclassification of high- versus low-risk people by the RECODe equations as compared with the older sets of equations is shown in Supplementary Table 4 for each outcome. RECODe equations were more likely than the UKPDS OM2 to correctly distinguish high-risk from low-risk patients for each of the microvascular and cardiovascular outcomes (10-year risk ≥10% vs. <10% for nephropathy [NRI = 0.01 in MESA and 0.72 in JHS], retinopathy [NRI = 0.02 in MESA and 0.08 in JHS], ASCVD [NRI = 1.87 in MESA and 9.00 in JHS], and CHF [0.17 in MESA and 0.19 in JHS]; and ≥5% vs. <5% for MI [NRI = 8.09 in MESA and 26.00 in JHS] and stroke [0.51 in MESA and 6.14 in JHS]). RECODe equations were also more likely than the ACC/AHA PCEs to correctly distinguish high-risk from low-risk patients for ASCVD (10-year risk ≥10% vs. <10% for ASCVD; NRI = 0.09 in MESA and 0.19 in JHS).
Conclusions
In this work, we validated the RECODe equations for predicting microvascular and macrovascular outcomes in type 2 diabetes among sociodemographically diverse, community-based participants in two different cohort studies. We have previously published a validation of the RECODe equation in two randomized trials, the DPPOS and Look AHEAD studies, comprising mainly non-Hispanic white subjects with type 2 diabetes (11). The RECODe equations improved upon prior risk equations for the prediction of key outcomes of interest among people with type 2 diabetes. The RECODe equations outperformed the UKPDS OM2 and ACC/AHA PCE risk equations based on metrics for discrimination and calibration, whereas the latter two risk equation sets exhibited substantial overprediction of both macrovascular and microvascular events in the studied cohorts. The RECODe equations nevertheless did not perform ideally; for the outcome of retinopathy in particular, the equations still overpredicted risk. The RECODe equations also performed well for most nephropathy outcomes, but not in MESA for the composite ACCORD outcome of macroalbuminuria or renal failure/ESRD or doubling of serum creatinine or >20 mL/min decrease in eGFR. The RECODe equations were validated here against data available to other researchers, were derived through open-source statistical code, and have an online risk calculator (32). Unlike prior studies of the UKPDS OM2, our external validation used individual patient data rather than only aggregate study outcomes (8).
The RECODe equations greatly mitigated problems with older equations. The UKPDS equations typically overestimated risk by four- to sevenfold in this assessment and reflected the health and racial diversity of the U.K., not the U.S. The ACC/AHA equations did not include microvascular complications, omitted important covariates not widely available in their development data sets (e.g., statin treatment), and substantially overestimated risk, consistent with prior reports (9,33,34). This may be due to the younger age of diagnosis and earlier diagnosis (i.e., diagnosis at a lower baseline hemoglobin A1c) among modern U.S. populations than the original UKPDS cohort. In particular, we found that this study verifies that the RECODe equations are more accurate for culturally diverse, American patients with diabetes, including those both previously diagnosed and newly diagnosed. Risk equations should be valid for clinical assessments immediately upon diagnosis and when physicians care for a patient who was previously diagnosed but with new therapeutic considerations; the latter patients will have different input parameters and risk estimates from the equations.
The equations have direct clinical use, as the American Diabetes Association and other groups already recommend that clinicians consider overall risk of diabetes complications and “shared decision making” when deciding upon therapy. Shared decision making necessarily involves discussing the benefits and risks of a therapy with the patient (4,35). As some populations in the U.S. appear to be at particularly high risk of overtreatment, leading to serious adverse events (36,37), having more accurate risk estimates could help avert drug-related morbidity and mortality among populations for whom older equations overestimated risk; by contrast, more aggressive but potentially higher-risk treatments for blood pressure, lipids, and glucose can be concentrated among those patients for whom the risk of complications and potential benefits of more aggressive therapy might be more favorable (38). The equations can also inform comparative effectiveness and cost-effectiveness research for population health interventions for which accurate risk models for modern populations are necessary.
The RECODe equations had good discrimination and were well calibrated as compared with alternatives, indicating that both higher- and lower-risk subjects would have more accurate risk estimates when using RECODe.
Limitations
The available studies lacked neuropathy as an outcome variable. Relatedly, due to differences in study end point definitions, we could only validate a subset of possible microvascular end point definitions and notably did not have a sufficient sample to validate against the important outcome of ESRD. Retinopathy, in particular, occurred in a limited set of patients, and in the JHS, only one retinal examination was available, such that baseline prior retinopathy definition was imprecise due to self-report and in turn incident retinopathy was imprecise as well. Further study should evaluate the utility of the equations for retinopathy assessment among blacks. Finally, longer periods of follow-up may enable the extension of equations to the full life-course, enabling linkage to long-term life expectancy outcomes for which we lack sufficient statistical power at present.
A key set of limitations to this validation concerns the populations against which we validated. Because the RECODe equations were validated previously in predominantly white populations (n = 11,036 white, 2,854 black, and 1,523 Hispanic) (11), we validated them here in predominantly minority populations (1,746 black in JHS; 369 white, 548 black, 187 Asian, and 451 Hispanic in MESA). Hence, our findings do not generalize to that of the broad U.S. population, although there are disproportionately black adults represented in the U.S. population of patients with type 2 diabetes. All risk equation evaluations should consider the social and historical conditions against which the validation populations are assessed; in our case, the JHS black population in particular is from Mississippi and may face high competing risks of other disease, and which differs markedly from the social and historical context of the UKPDS cohort. Hence, we used calibration and discrimination tests that account for censorship due to other conditions (28,39) and validated both CVD and all-cause mortality equations. Nevertheless, clinical practitioners should use risk equations that validated in populations most similar to their practice population. We also highlight that readers should not wrongly interpret that risk is lower for all populations when using RECODe than older risk equations. We found, instead, that although older equations may produce inflated risk estimates on average, many high-risk people (18–100% of those with an event in JHS and 3–97% in MESA) were wrongly predicted as low risk by the older equations and were caught by the new equations. Hence, our goal is accurate risk estimation, not underestimation. We also suggest that in further work, the equations should be recalibrated if necessary to local populations that are being treated, once electronic health record registries have sufficiently accurate adjudication to perform more localized inferences.
Article Information
Acknowledgments. The authors thank the other investigators, the staff, and the participants of the MESA study for their valuable contributions. A full list of participating MESA investigators and institutions can be found at http://www.mesa-nhlbi.org. The authors also thank the staff and participants of the JHS.
Funding. The research reported in this publication was supported by the National Heart, Lung, and Blood Institute (NHLBI) (K08HL121056 to S.B. and R01HL117285 to A.G.B.), the National Institute on Minority Health and Health Disparities (NIMHD) (DP2MD010478 to S.B. and U54MD010724 to S.B.), the National Institute of Diabetes and Digestive and Kidney Diseases (P30DK092926 to R.A.H. and J.B.S. and K23DK109200 to S.A.B.), and the U.S. Department of Veterans Affairs (IIR11-088 to J.B.S. and R.A.H. and CDA13-021 to J.B.S. and R.A.H.). MESA is supported by the NHLBI (contracts HHSN268201500003I, N01-HC-95159, N01-HC-95160, N01-HC-95161, N01-HC-95162, N01-HC-95163, N01-HC-95164, N01-HC-95165, N01-HC-95166, N01-HC-95167, N01-HC-95168, and N01-HC-95169) and the National Center for Advancing Translational Sciences (grants UL1-TR-000040, UL1-TR-001079, and UL1-TR-001420). JHS is supported and conducted in collaboration with Jackson State University (HHSN268201300049C and HHSN268201300050C), Tougaloo College (HHSN268201300048C), and the University of Mississippi Medical Center (HHSN268201300046C and HHSN268201300047C) and is supported by contracts from the NHLBI and the NIMHD.
The views expressed in the manuscript are those of the authors and do not necessarily represent the views of the National Heart, Lung, and Blood Institute, the National Institutes of Health, or the U.S. Department of Health and Human Services.
Duality of Interest. No potential conflicts of interest relevant to this article were reported.
Author Contributions. S.B. conceived of the study, had access to raw data and analytic code, interpreted results of the analysis, wrote the first manuscript draft, and edited the manuscript for important intellectual content. J.B.S., S.A.B., and R.A.H. interpreted results of the analysis and edited the manuscript for important intellectual content. A.G.B., A.C., and S.M. had access to raw data and analytic code, interpreted results of the analysis, and edited the manuscript for important intellectual content. J.S.Y. conceived of the study, interpreted results of the analysis, and edited the manuscript for important intellectual content. S.B. is the guarantor of this work and, as such, had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Footnotes
This article contains Supplementary Data online at http://care.diabetesjournals.org/lookup/suppl/doi:10.2337/dc17-2002/-/DC1.
- Received September 27, 2017.
- Accepted November 22, 2017.
- © 2017 by the American Diabetes Association.
Readers may use this article as long as the work is properly cited, the use is educational and not for profit, and the work is not altered. More information is available at http://www.diabetesjournals.org/content/license.
References
In this Issue
