Deficiencies of Cardiovascular Risk Prediction Models for Type 1 Diabetes

These analyses used data from the Epidemiology of Diabetes Complications (EDC) study. The EDC study includes subjects with childhood-onset type 1 diabetes diagnosed between 1950 and 1980 before the age of 17 years. All subjects were seen within 1 year of diagnosis at Children’s Hospital of Pittsburgh. Although this population is clinic based, it has been shown to be epidemiologically representative of all type 1 diabetic cases in Allegheny County, Pennsylvania (17). A total of 658 subjects participated in baseline exams between 1986 and 1988. These analyses consist of those subjects with complete 10-year follow-up data and include 108 incident coronary artery disease (CAD) events and 429 subjects who did not experience an event. Fifty-two subjects with prevalent events were not included in these analyses.

CHD outcomes were defined as fatal CHD or nonfatal myocardial infarction confirmed by medical records or Q-waves according to Minnesota codes 1.1 or 1.2; revascularization procedures including coronary artery bypass graft, angioplasty, coronary endarterectomy, thrombolysis, and stenosis >50% without revascularization, also confirmed by medical records; ischemic electrocardiogram (ECG) by Minnesota codes 1.3, 4.1–4.3, 5.1–5.3, and 7.1; and angina confirmed by an EDC study physician. The Minnesota Code is a classification system for the ECG that utilizes a defined set of measurement rules to assign specific numerical codes according to severity of ECG findings. The Minnesota Code incorporates ECG classification criteria that are validated and provide an objective ECG classification system free of bias (18).

When CHD risk was estimated using the EDC data, three sets of outcomes were defined: 1) fatal or nonfatal myocardial infarction, CHD death, or Q-waves, which was the same outcome predicted by the UKPDS Risk Engine and Framingham equations; 2) the first set of outcomes plus revascularization; and 3) the first two sets of outcomes plus ischemic ECG or angina. The equations used in these analyses are presented in Table 1.

Blood pressure was measured with a random-zero sphygmomanometer, according to the Hypertension Detection and Follow-up Program protocol, after a 5-min rest (19). Hypertension was defined as ≥140/90 mmHg or use of antihypertensive medication. Stable HbA1 (upper limit of normal 7.3%) was measured by ion exchange chromatography (Isolab, Akron, OH) and subsequently by automated high-performance liquid chromatography (Diamat; BioRad, Hercules, CA). Laboratory values with the two methods are almost identical (r = 0.95). HDL cholesterol was determined by a precipitation technique (heparin and manganese chloride) with a modification (20) of the Lipid Research Clinics method (21). Cholesterol and triglycerides were measured enzymatically (22,23). Demographic and lifestyle characteristics were obtained by questionnaire.

All data were analyzed using SAS version 8.2 (Cary, NC). Descriptive statistics, including means and frequencies, were conducted using baseline data to examine the distribution and counts of the data. Univariate analyses of baseline characteristics included the Student’s t test for continuous variables and the Pearson’s χ² test for categorical variables. Triglycerides were natural log transformed before analyses due to their nonnormal distribution. Differences were considered significant at the P < 0.05 level.

EDC data were used to calculate the 10-year probability of CHD according to the Framingham Risk equations and the UKPDS Risk Engine (Table 1) for each of the three sets of CHD outcomes previously described. The Framingham model included a term for diabetes (yes/no), while the UKPDS model included a term for HbA1. Variables in the existing equations were used in these analyses. Ten-year expected probabilities were calculated and divided into deciles. Actual events in the population were then tabulated in each risk decile. Model calibration was tested using the Hosmer and Lemeshow goodness-of-fit χ² statistic (24). This tested if the predicted probability of an event calculated from existing equations differed from what was observed in the EDC cohort across deciles of risk. We tested model discrimination using the c-statistic from a logistic regression model. Presence or absence of CHD was the dependent variable, and predicted probability calculated from the model equations was the explanatory variable. Discrimination values range from 0.5 (nondiscrimination) to 1.0 (perfect discrimination) (25).

The University of Pittsburgh Institutional Review Board approved the study protocol. Investigators obtained informed consent from all participants before procedures on the day of their clinic visit.

The population included in these analyses is described in Table 2 stratified by the development of subsequent CHD events. There were 108 incident cases and 429 subjects who did not experience an event, resulting in a 10-year incidence of CHD of 18%. Incident cases were significantly older, had longer diabetes duration, and had worse CHD risk profiles (e.g., blood pressures, lipid profiles, and smoking) compared with those who remained free of CHD throughout the follow-up period. There was not a significant difference in HbA1 levels or sex. There were 36 hard CHD events (fatal/nonfatal myocardial infarction, fatal CHD, and Q-waves).

Observed and expected probabilities (generated from the UKPDS Risk Engine) of a hard CHD event were compared and are presented in Fig. 1. There was moderate discrimination (c-statistic of 0.76) and poor calibration (χ² = 324.1; P < 0.0001). The most notable differences were in the highest-risk deciles. Adding revascularization to the model slightly improved discrimination (c-statistic 0.77) but did not improve calibration. The largest discrepancies between observed and expected events were in the highest-risk categories in all models. Finally, a third model that expanded the CHD outcomes to include all of the previously modeled outcomes plus ischemic ECG and angina demonstrated poor discrimination (0.67) and a significant lack of calibration (P < 0.0001).

The observed and predicted probabilities for the Framingham risk equation are also shown in Fig. 1 for both men and women. For fatal and nonfatal myocardial infarction, there was a significant lack of calibration (men χ² = 310.3, women χ² = 6,873.9; P < 0.0001), with moderate discrimination for men (0.77), but was better for women (0.87). For men, in all models, the Framingham equations also underestimated probabilities in the lower two deciles. The same pattern was observed for women in the model for all CHD outcomes. All models again underestimated the probabilities in the higher deciles of risk.

While tools are available to predict the risk of CHD events in the general population and type 2 diabetes (12–16), they have not been validated or tested in patients with type 1 diabetes. Current models used in clinical practice may therefore provide inaccurate estimates of risk to both providers and type 1 diabetic patients. The most notable models for predicting cardiovascular events in the general population are those derived from the Framingham data (13) and for type 2 diabetes, the UKPDS Risk Engine (12). We used data from an epidemiologically representative type 1 diabetic cohort to test these equations. Results demonstrated that both the Framingham equations and the UKPDS Risk Engine poorly predict the risk of a CHD event in those with type 1 diabetes as observed, and expected probabilities differed significantly for both hard and total CHD outcomes. Consistently, poorest prediction was observed in those at highest risk.

These analyses have important implications for people with type 1 diabetes and their health care providers. Because those with type 1 diabetes tend to be younger than those with type 2 diabetes or the general population with CHD, neither patients nor physicians may view CHD as a major complication for type 1 patients. Further, using models currently available to predict risk of an event in these patients may give both providers and patients a false interpretation of risk, which was grossly underestimated as shown in our data. This hypothesis is supported by evidence of poor risk factor control in the type 1 diabetic population (9–11), which is far worse than in those with type 2 diabetes or in the general population (26), particularly in younger type 1 diabetic subjects (27).

Models derived from the Framingham Heart Study consider the combination and level of individual risk factors modeled through regression equations, which estimate a person’s risk for future events during some fixed period of time (13,28). Additionally, scores developed from these equations allow users to assign points to risk factors and by totaling the score, obtain an estimation of CHD risk over a specified time period (29). The purpose of developing these functions and risk scores is to estimate the “global risk” or probability of a CHD event. Once this risk is estimated, targeted interventions may be more appropriately implemented to reduce CHD risk (30). Such an approach has become the standard to determine, for example, whether drugs should be used in borderline hypercholesterolemia (e.g., National Cholesterol Education Program Adult Treatment Panel III-2% annual risk in the presence of two or more risk factors for LDL cholesterol level for drug therapy to 130 mg/dl) (31). The validity of the Framingham model has been tested (32,33) in a variety of ethnic groups (34) and in those with other chronic diseases (35–37); however, specific application to cardiovascular risk prediction in those with type 1 diabetes is lacking. For example, the total incidence of CHD in this population, including angina and revascularization, was 18% over 10 years of follow-up, while for hard events, it was 6.8%. The Adult Treatment Panel III assumes a 20% 10-year risk; however, this is in a much older population and is also inflated for type 2 diabetic patients aged <55 years (as age is the major determinant of risk). The mean age of this population is ∼30 years; thus, the CHD incidence is quite significant for this age-group and >10-fold higher than in the general population.

Extrapolating models to type 1 diabetic patients has several limitations. The models currently used in practice address cardiovascular risk in the age ranges where CHD is most likely to occur in the general population. For example, the UKPDS Risk Engine has age centered on the median of 55 years. This is a significant limitation if applied to type 1 diabetes. For example, the mean age of CHD onset in the EDC study cohort was 39 years (interquartile range 34–44), far younger than the ages of events in the UKPDS. Further, the mean age of events in the EDC cohort is much lower than in the general population, which for men is 65.8 years and 70.4 years for women (38).

The Framingham models are applied in people between 30 and 74 years old, where men and women under the age of 40 years are considered low risk (negative scores in the risk prediction charts) (39), which is a significant limitation when attempting to translate these models to type 1 diabetes. The original Framingham cohort was at least age 50 at study entry; thus, estimates in those who are younger may be less accurate. Other available models have only investigated the use of risk prediction equation in older adults. The Prospective Cardiovascular Munster models consider a narrow age range of 45–65 years and only in men, while the cardiovasuclar event reduction tool model is based on those aged 45–64. As data from the Joslin Clinic (2) and Pittsburgh (3) both demonstrate that hard CHD events (myocardial infarction or CHD death) can occur in the early 30s in people with type 1 diabetes, data used to calibrate risk prediction models should include these younger ages if they are to be generalized to this population. In the EDC study population, 27 CAD related events were in subjects under the age of 30, therefore the age component of the existing models is not comparable or applicable in younger populations. These differences in mean age at event demonstrate the considerable lack of generalizability of these equations to type 1 diabetic populations.

Another limitation of existing CHD risk prediction models is that none incorporate the effect of renal disease. This is a significant limitation, as renal disease is an important predictor of CHD in type 1 diabetes (4,40). Further, the Framingham risk equations (13) and UKPDS Risk Engine (12) do not consider risk factor treatment in their basic equations. The CERT (cardiovascular event reduction tool), developed from the West of Scotland Coronary Prevention Study, considers the impact of lipid treatment using pravastatin on cardiovascular risk (37), while the PRECARD program assesses the relative impact of modifiable risk factors (e.g., lipids and blood pressure) based on clinical trial data (16). However, models that do not consider treatment effect assume that the risk of an event is the same for certain levels of risk factors (e.g., blood pressure of 130/80 mmHg) whether naturally occurring or due to treatment to that level. This is a limitation that ignores duration of exposure to uncontrolled risk factors.

When diabetes has been considered in risk prediction models, there has been no specific reference to type of diabetes. The presence of diabetes is entered as a dichotomous variable, most commonly, without a measure of glycemic control, although the UKPDS Risk Engine does consider HbA1 level (see Table 1). Further, the Framingham equations were based on a cohort of 5,209 subjects where only 6% of men and 8% of women had diabetes, the majority of which had type 2 diabetes (41). While type 2 diabetes is clearly more prevalent in the general population compared with type 1 diabetes, the underlying disease process is different and should be specifically considered in these models. Though evidence that insulin resistance is a key component for cardiovascular risk in both types of diabetes exists (4, 42), the interplay of age and comorbid conditions is likely to be very different. This may lend further credence to questionable risk prediction by these models in type 1 diabetes. Again, because of a younger age of onset of hyperglycemia, hyperlipidemia, and hypertension, those with type 1 diabetes are likely to be exposed to these risk factors for a longer duration than those with type 2 diabetes, despite the existence of disturbed risk factors in the pre-diabetic state (43).

While this study was conducted in an epidemiologically representative population, there are certain limitations to the data. The EDC study includes subjects with long-duration type 1 diabetes and may therefore be subject to survivor bias. Those at highest risk for CHD among longer-duration subsets may have already died. It is unlikely that these data suffer from bias in ascertainment of cases as CHD events are quantified using a standard methodology throughout the study. Lipids and blood pressure, however, were measured at one point in time and are thus not diagnostic. This may however be more reflective of the practical clinical application of these models. Another limitation of this study is that we did not explore variables that may account for the underestimation of the Framingham and UKPDS equations. This work is currently underway, as is the exploration of new prediction models specific to type 1 diabetes.

In conclusion, people with type 1 diabetes suffer a disproportionate risk for CAD compared with those without diabetes. Given the limitations of the existing prediction models, including the disregard of younger age and comorbid conditions, there is a clear need to develop CAD risk prediction models for type 1 diabetes that incorporate these factors. The translation of more appropriate models for these patients into clinical practice could lead to decreased morbidity and mortality in this population. Future research should focus on the development of models specific to this high-risk group for use in clinical practice.

This study was funded by National Institutes of Health Grant DK34818 and the American Diabetes Association Junior Faculty Award.

This study was previously presented as a poster at the 64th Scientific Sessions of the American Diabetes Association, Orlando, Florida, on 4–8 June 2004.