Outcome Measures for Artificial Pancreas Clinical Trials: A Consensus Report

Objective and Rationale

In this report, members of the JDRF Artificial Pancreas Project Consortium in collaboration with the wider AP community 1) advocate for the use of continuous glucose monitoring (CGM) glucose metrics as glycemic outcome measures in AP trials, in addition to HbA_1c, and 2) identify a short set of basic, easily interpreted outcome measures to be reported in AP studies whenever feasible. Currently, the U.S. Food and Drug Administration accepts the use of various CGM glucose metrics in AP trials (9), but investigators do not always use a consistent set of measures that enables comparison. Thus, one rationale for the current report is to enable basic comparison between different AP research studies and with other clinical studies on glycemic control in type 1 diabetes. We acknowledge there are methodological limitations with between-study comparison that require careful consideration of study design differences. However, the standardization of these simple metrics provides a starting point for regulators, payers, health care providers, and patients to interpret AP and other study data with interventions on glycemic control. This will be especially important as AP systems become part of the daily lives of people with type 1 diabetes. Standardization of these measures does not preclude the addition of other metrics specific for a particular AP approach or used by particular research groups. In this report, we specifically advocate for the use of a basic set of CGM glucose metrics in AP studies. We suggest that their use in general type 1 diabetes studies is broadly applicable and highly relevant given the increasing adoption of CGM in research and clinical care (7).

Improvements in and adoption of AP-related technology, particularly in the reliability and accuracy of CGM systems, have focused attention on determining the best metrics for assessing outcomes in studies with people with type 1 diabetes (10–16). Currently available CGM systems with glucose readings up to every 5 min, or 288 times daily, provide considerably more data than do the American Diabetes Association recommendation of checking blood glucose 6–10 times daily (17) or the 7-point blood glucose measurements performed quarterly for research purposes in the DCCT. Although 7-point blood glucose profiles do provide insights into glycemic excursions that are not apparent with HbA_1c, the profiles are very dependent on patient motivation and the chosen day of performance and provide only limited information about glucose control compared with glucose values provided by CGM systems. From a patient perspective, a CGM glycemic profile is more meaningful in that it shows highs, lows, trends, and variability as well as the effect of behaviors on glucose levels (18–21). This contrasts with HbA_1c as a metric of integrated glycemic exposure over time. In particular, HbA_1c does not provide information on frequency and extent of hypo- or hyperglycemia, which is a crucial aspect to evaluating glucose control in people with type 1 diabetes.

Basic Outcome Measures

The recommended basic set of outcome measures presented in Table 1 includes CGM glucose metrics to define time spent in desired ranges as well as time in hypo- and hyperglycemia, measures of CGM glucose variability, safety measures such as SH and diabetic ketoacidosis, and technical metrics to evaluate AP system performance. It is intended that these measures be applicable across a wide range of AP study designs, including both short-term pilot studies and longer-term in-home or pivotal studies. Many of the glucose cut points and ranges are based on convention in AP research but were chosen to allow for comparison between studies.

HbA_1c remains the best currently available measure to assess long-term glycemic control and should be assessed in any AP study of 3 months or longer. However, it is clear that HbA_1c only captures average glycemia and does not provide information on the frequency or severity of hypoglycemic events. Although HbA_1c is currently the most accepted metric for risk stratification of long-term complications of diabetes, the proposed metrics more comprehensively describe glycemia.

Additional Recommendations and Limitations

Graphical presentation of outcome data is also important, although standardization is less straightforward than with tabular data. A common figure for visualization of pooled-subject AP performance is the modal day glycemic control plot (or analogous insulin delivery plot) with median line and interquartile range bands. Inclusion of a cumulative histogram of CGM data would support the extraction of arbitrary glycemic ranges for comparison purposes (22). Numerous other graphical representations of data have been developed, and the choice of figures should be individualized for the data and the target audience (23).

The number of symptomatic hypoglycemia events per week may also be valuable as a meaningful clinical index of diabetes burden to the patient (24–27). Indeed, time spent below targeted glucose range according to CGM data may not fully capture the patient’s experience with debilitating glucose-related events, which might better illustrate diabetes burden. Because reliable capture of symptomatic hypoglycemia events requiring treatment may be challenging in longer-term AP studies, biochemical hypoglycemia event rate as measured by CGM could be reported as a proxy. For example, the rate of CGM excursions below 70 or 55 mg/dL (3.9 or 3.0 mg/dL) for at least 10 or 30 min or longer time periods could be reported, as could other metrics including area under the curve (28–32). Many other novel measures of AP performance and algorithms have been developed (33–36), and this is an area of active research.

CGM and pump make and model and the kind of device running the control algorithm (e.g., laptop, smartphone) should be specified, including any relevant CGM signal conditioning algorithm details. We note that bias can occur when the same CGM that informs the AP controller is also used to assess glycemic outcomes (37), but there often is no practical alternative to this approach. Any special system- or protocol-related design elements should be disclosed if they are intended to improve safety or impact glycemic control or if they place additional burden on the user. The amount and timing of contact between study staff and participants in both the AP and comparator arms of the studies should be reported for in-home studies. In addition, CGM calibration logistics should be disclosed, along with a description of how conventional capillary blood glucose measurements are performed. Determination of median (or mean) absolute relative difference (MARD) for CGM versus capillary blood glucose is often used to characterize CGM accuracy in AP studies, though blood glucose sampling bias may limit the generalizability of these results (38).

Depending on the study design, the outcomes described could be reported for the entire cohort of a study or the study participants could be stratified into relevant subgroups with outcomes reported separately. For example, improved HbA_1c without increased risk for hypoglycemia could be reported for those who were poorly controlled at the baseline (e.g., baseline HbA_1c >8% [>64 mmol/mol]), whereas reduced incidence of hypoglycemia without deterioration in HbA_1c could be reported for those with well-controlled average glycemia at the baseline. The analysis of the primary and other important outcomes should be performed on an intention-to-treat basis.

Future areas of need for AP technology include expanded standardized metrics to evaluate the technical performance of AP systems (22) and to assess patient/caregiver usability, including psychosocial metrics such as quality of life and other measures of reduction of burden, which need to be developed, including stress, anxiety, depression, and use during exercise (39). In addition, the development of standard measures to assess patient preference may be used to support regulatory approval and to serve to inform health care providers and patients of the potential impact of the use of AP systems (40). Compelling health economic measures comparing AP system costs with the potential short- and long-term economic benefits are required to establish the financial viability of these systems and to drive acceptance by health care providers and people with diabetes (8).

Multiple large and longer in-home clinical trials will soon begin with different AP systems supported by the National Institute of Diabetes and Digestive and Kidney Diseases, JDRF, the Helmsley Charitable Trust, and other funders, as well as those being supported by the industry. Some of these have been designed as pivotal trials to provide data about relevant end points to be presented to U.S. Food and Drug Administration and other regulatory authorities for approval of AP products for clinical use and to support reimbursement. This report emphasizes the need for a set of basic, uniform, standardized, and comparable outcome measures with different AP systems. As AP technologies become available for clinical use, common data reports that compare systems will be desired for health care providers, payers, and people with type 1 diabetes and their families.

In summary, members of the JDRF Artificial Pancreas Project Consortium and the larger AP community advocate for the adoption of a set of basic outcome metrics that will allow for comparison between studies with different AP systems. This, in turn, will facilitate the interpretation of the information from trials and contribute to the ultimate goal of widespread adoption of AP technology to improve the life of people with type 1 diabetes.