Economic Evaluation of Quality Improvement Interventions Designed to Improve Glycemic Control in Diabetes: A Systematic Review and Weighted Regression Analysis

Improving the value of health care has become a priority, and optimizing glycemic control among patients with diabetes may represent a promising opportunity. In the short term, poor control can affect clinic visits, emergency department visits, and hospitalizations related to hyper- and hypoglycemia. Over the long term, poor control leads to disabling and costly complications including cardiovascular disease, blindness, kidney disease, and neuropathy (1–3). Yet suboptimal control remains common, with HbA_1c levels exceeding individualized targets in 16%–72% of patients, particularly younger adults with complications (4). Thus, quality improvement (QI) interventions related to glycemic control may increase value by improving health outcomes and reducing costs (5). If net costs decline substantially, there may be a “business case” for improving quality.

QI interventions represent systematic and continuous efforts to achieve measurable improvements in the structure, processes, or outcomes of care, particularly the health of targeted patient populations, by means of an organizational or structural change (6–9). QI interventions frequently emphasize teamwork and can involve the combined efforts of health care organizations, clinicians, and patients and their families (8). Interventions related to glycemic control can involve three basic types of QI strategies: changing care systems, optimizing practitioner behavior, and supporting behavior change by patients (10). Systems of care can be restructured by instituting disease/case management, creating multidisciplinary teams, establishing electronic patient registries, and relaying information to clinicians, among others. Desired actions by practitioners can be fostered through audit and feedback, education, reminders, and financial incentives. Behavior change by patients can be supported through tailored care, education, self-management training, reminders, and financial incentives (11).

Although QI interventions can be effective at lowering HbA_1c (11), their economic value does not appear to have been evaluated systematically (5). It remains unclear whether QI interventions tend to produce net savings or losses in the short and long term and how costs compare with health gains. We sought to systematically review economic evaluations of QI interventions designed to improve glycemic control among adults with diabetes. Accordingly, we examined changes in HbA_1c and incremental net costs in the short term (within up to 3 years) and incremental cost-effectiveness in the long term (over 20 years or more). To estimate mean changes in HbA_1c and net costs in the short term, we performed weighted regression analyses that combined study results statistically.

We report this review in accordance with Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (12), including posting the study protocol on the Prospero registry (CRD42015014950) (13). A technical expert panel offered guidance during key stages of the project.

A reference librarian developed search terms related to diabetes and expanded on published terms related to economic evaluation (Supplementary Data 1) (14). Databases included MEDLINE, EconLit, and the Centre for Reviews and Dissemination. To identify grey literature, we searched New York Academy of Medicine's Grey Literature Report and WorldCat and invited expert panelists to suggest studies. Searches were restricted to English-language documents from January 2004 to August 2016, as clinical practices and cost structures have evolved over time. We hand-searched citations from previous systematic reviews (15–17) and other sources (11,18).

English-language studies were eligible if they represented original investigations, addressed QI interventions designed to improve glycemic control among adult outpatients with diabetes (type 1, type 2, or unspecified), measured or modeled the cost of the QI intervention, and compared alternatives (e.g., QI intervention vs. status quo). Studies needed to report both program costs (costs of implementation) and costs related to health care utilization for diabetes. We excluded studies from low- to middle-income countries due to differences in care practices and cost structures (19). We included all time horizons, clinical study designs, economic evaluation approaches, and analytical perspectives. Interventions that sought to influence only patient behavior without involving changes to systems of care (e.g., diet or exercise programs) or that tested new equipment or medications were not considered QI. Two trained reviewers determined eligibility by independently examining titles, abstracts, and full-text publications; discrepancies were resolved by consensus or, if needed, discussion with the research team.

We obtained any additional publications related to the eligible analyses (such as study designs or clinical results published separately from economic analyses), and then pairs of experienced investigators with training in quality of care, population health, and economic evaluation extracted data from the articles. Discrepancies were resolved by consensus or, if needed, through discussion with the research team.

Reviewers applied 16 categories of strategies for improving glycemic control, including 7 system-oriented strategies, 4 practitioner-oriented strategies, and 5 patient-oriented strategies (Supplementary Data 2) (11,20). System-oriented strategies included disease management, changes to the health care team, use of electronic registries, facilitated relay of information, continuous QI, enhancing efficiency, and standardizing care. Practitioner-oriented strategies included audit and feedback, provider education, provider decision support, and incentives for practitioners. Patient-oriented strategies included tailoring care for subgroups of patients, patient education, patient self-management, patient reminders, and incentives for patients.

Contextual variables included country, academic status (major, minor, nonteaching), setting (e.g., clinic, pharmacy, etc.), and location (urban, suburban/small city, rural). Clinical study designs included randomized controlled trial (RCT), controlled before-after analysis, uncontrolled before-after analysis, interrupted time series and repeated-measures studies, and modeling studies (21). Following a best evidence approach, we emphasize findings based on RCTs but include studies with nonrandomized designs because many QI interventions use such designs (22).

To assess risk of bias, we used the Cochrane Risk of Bias Tool for randomized trials and the Newcastle-Ottawa Scale for nonrandomized studies (23,24). To assess whether authors reported key information about QI interventions, we applied items (3–5,10,11,14,15,17) from the Quality Intervention Minimum Quality Criteria Set (QI-MQCS) (25). Funding sources included government, nonprofit, commercial, and none.

The primary clinical outcome was the change in HbA_1c. When studies used controlled designs, the change in HbA_1c represented differences between the control and intervention groups in changes from baseline to follow-up (i.e., difference in differences). When studies used uncontrolled designs, the change in HbA_1c represented changes from baseline to follow-up for the intervention group. Studies generally reported follow-up HbA_1c tests at 1 to 3 years, irrespective of the time horizon for the economic analysis. For each study, we extracted numbers of individuals in intervention and any comparison groups, duration of the intervention, baseline HbA_1c, change in HbA_1c with the intervention, and timing of follow-up HbA_1c tests. For long-term analyses, we also extracted quality-adjusted life years (QALYs) and years of life gained.

Reviewers extracted the evaluation approach (cost analyses including cost, cost consequences, and business case analyses and cost-effectiveness analyses including cost utility, cost-benefit, and related analyses), perspective (clinic/provider, health system, payer, society), discount rate (when applicable), and year and currency of costs.

Reviewers classified studies by the time horizon of the economic analysis, including short term (up to 3 years), intermediate term, and long term (20 or more years). For short-term analyses, the primary economic outcome was the incremental net cost per patient per year, calculated as the sum of program and health care utilization–related costs. For long-term analyses, the primary economic outcome was the incremental cost-effectiveness ratio (ICER), calculated as the incremental net cost divided by the incremental QALYs per patient over the study time horizon. Few studies used intermediate time horizons.

We used the Quality-Cost Framework, which defines structure-, process-, and outcome-related costs, building on the Donabedian model of quality (26). Structure-related costs were the fixed costs associated with start-up and maintenance, such as labor costs associated creating new protocols and training providers. Process-related costs were variable, recurring costs resulting from the care of individual patients, such as physician visits and medications. Outcome-related costs included health care related to diabetes-related hospitalization, cardiovascular disease, blindness, nephropathy, etc. For this analysis, studies reported structure-related costs as program costs and health care utilization–related costs that included process- and outcome-related costs. When studies reported results for more than one discount rate, we extracted results based on a 3% rate (27).

We applied currency conversion and inflation factors to standardize costs per patient to 2015 U.S. dollars and the health system perspective. For short-term analyses, we graphed the relationship between the change in HbA_1c and standardized incremental net costs. For long-term analyses, we graphed the relationship between QALYs and standardized incremental net costs.

To assess whether economic evaluations met basic standards, reviewers applied a modified version of the Quality of Health Economics Studies Checklist (mQHES), as reported previously (28–30).

We conducted weighted regression analyses to identify factors potentially associated with changes in HbA_1c. We identified these factors a priori. Analyses were prespecified, unadjusted, and stratified by study design (RCT, controlled before-after analysis, and uncontrolled before-after analysis or other design). Independent variables examined for each study design included the baseline HbA_1c (continuous); study timing (year for cost data, continuous); numbers of system-oriented, practitioner-oriented, and patient-oriented strategies, respectively (continuous); and each of the 15 different individual QI strategies (dichotomous).

Weighted regression is similar to meta-regression in that studies are the unit of analysis in the model. The difference is the way studies are weighted. In the former, studies are weighted by the number of participants. In the latter, studies are usually weighted by the inverse of the study variance. We performed weighted regression because very few primary studies in our data set reported variance for cost estimates. The two approaches yield equivalent results, and weighted regression performs better when there may be concerns about publication bias or small sample sizes (31). When studies reported data separately for multiple subpopulations, we treated each subpopulation as a separate observation. Studies did not report enough data on variance to formally assess publication bias.

For analyses with short-term horizons, we used weighted least-squares regression to calculate mean incremental net costs along with 95% CIs, stratifying by study design. We could not examine predictors of net costs due to insufficient numbers of studies.

We identified 3,646 records, selecting 222 for full-text review; 45 articles met all eligibility criteria. Three articles reported results separately for different subpopulations, stratifying by baseline HbA_1c (32), sex (33), and payer (34). One article reported results for two different interventions (35), bringing the total number of unique studies to 46. Searches of grey literature did not identify eligible articles. Supplementary Data 3 includes the PRISMA flow diagram, and Supplementary Data 4 includes tables listing extracted data.

Interventions involved a median of 4.5 different QI strategies. Thirty-one studies included disease/case management, 35 studies involved patient education, 29 studies promoted patient self-management, and 22 studies involved changes to the clinical team. Other strategies were used less frequently (Table 1 and Supplementary Data 4).

Thirty-one studies focused on type 2 diabetes, 4 included type 1 and 2 diabetes, and 11 did not specify type. Three studies examined populations with diabetes or cardiovascular disease risk factors. The median baseline HbA_1c was 8.1% (65 mmol/mol) among 42 studies (including subpopulations); baseline HbA_1c was missing for 4 studies. Twenty-five studies were based in the U.S., and 33 were implemented in clinics. Nineteen studies were based on RCTs, including 9 cluster RCTs. Eleven studies were based on controlled before-after designs. Thirteen studies were based on uncontrolled before-after designs and 2 on serial cross-sectional analyses. One was a modeling analysis based on published literature (excluded from regression analyses).

Overall, the RCTs were at moderate risk of bias because the authors did not document random sequence generation and could not conceal allocation. Observational studies were also at moderate risk of bias because of concerns about the representativeness of study cohorts, selection of comparison groups, and adequacy of follow-up. Supplementary Data 4 includes items in the QI-MQCS and funding sources.

There were 17 cost analyses and 29 cost-effectiveness analyses. Twenty studies took the health system perspective and 1 study reported results for both integrated health care system and societal perspectives. Eighteen studies considered the health care payer perspective, 7 other studies took the societal perspective, and 1 took the perspective of a hospital/clinic.

Twenty-three unique studies used short-term horizons, 4 studies used intermediate-term horizons, and 19 studies used long-term horizons. In the 23 studies with short-term economic evaluations, QI interventions were implemented and clinical outcomes and costs were examined over similar time frames, including 18 studies lasting up to 1 year, and 5 studies lasting 1.5–3 years.

The 19 studies with long-term economic evaluations involved modeling long-term effectiveness and costs based on shorter-term data. In 12 studies, authors assumed that both the intervention and any associated decline in HbA_1c were sustained over the full time horizon of the economic analysis; 7 of these studies measured HbA_1c at 1 year, 1 study at 1.5 years, 1 study at 3 years, 1 study at 4 years, and 1 study at 5 years; 1 study reported a short-term change in HbA_1c without specifying the timing of measurements. In 7 studies, the authors did not assume that the intervention or change in HbA_1c was sustained long term; these studies measured HbA_1c at 1 to 4 years. This assumption did not appear to affect results.

Overall, economic evaluation methods met basic standards, with a median mQHES score of 105 across the 46 unique studies.

On the basis of weighted regression analyses, the QI interventions were associated with significant improvements in HbA_1c across all three types of study designs. Findings were generally more favorable in studies based on nonrandomized designs, although differences did not reach statistical significance (P = 0.87). Among the 19 RCTs that reported changes in HbA_1c for intervention and control groups, the weighted mean improvement in HbA_1c was 0.26% (95% CI 0.17–0.35), or 3 mmol/mol (2–4), based on the difference in differences. Among 9 controlled before-after studies that reported changes in HbA_1c for intervention and control groups, the weighted mean improvement was 0.62% (0.37–0.88), or 7 mmol/mol (4–10), based on the difference in differences. Among 15 studies that used uncontrolled before-after or other designs, the weighted mean improvement in HbA_1c from baseline to follow-up was 0.41% (0.08–0.73), or 4 mmol/mol (1–8).

In unadjusted weighted regression analyses limited to RCTs, baseline HbA_1c was the only significant predictor of the change in HbA_1c (P = 0.010). With an increase in baseline HbA_1c from 7.5–8.5% (58–69 mmol/mol), for example, the improvement in HbA_1c relative to the control group increased from 0.22% (95% CI 0.14–0.29), or 2 mmol/mol (2–3), to 0.40% (0.29–0.52), or 4 mmol/mol (3–6). Study timing; numbers of system-, practitioner-, and patient-oriented strategies; and the specific QI strategies used were not significant predictors in unadjusted analyses (P < 0.05) (Supplementary Data 5). Excluding an RCT that differed from the others in terms of target population, intervention, and results did not alter findings (36). Results were generally similar for studies that used nonrandomized designs, except that among uncontrolled before-after analyses, larger declines in HbA_1c were observed among earlier studies (P < 0.001) and among interventions that used fewer system-oriented QI strategies (P < 0.001), more practitioner-oriented strategies (P < 0.001), and more patient-oriented strategies (P < 0.001). We did not include supplementary data for these data due to the lower quality of the studies (data are available from authors upon request).

Fifteen long-term studies reported years of life saved, which ranged from 0.0245 to 1.100 years (Supplementary Data 4).

Figure 1 shows standardized program, health care utilization–related, and net costs per patient per year across 23 short-term analyses, where negative costs reflect savings (see Supplementary Data 6 for calculations). Across these studies, the median cost of implementing a QI intervention was $525 per patient per year, which was offset by a median change in health care expenditures of −$302 per patient per year.

Including both program costs and changes in health care expenditures, the mean incremental net cost per patient per year was not significantly different from zero, based on weighted regression analyses. This was true across all three study designs. The net cost was $116 (95% CI −612 to 843) among eight RCTs, −$831 (−1,527 to −134) among seven studies using controlled before-after designs, and −$401 (−1,255 to 453) in eight studies with uncontrolled before-after or other designs. The weighted mean net costs per patient per year was significantly higher for RCTs than for controlled before-after studies (P = 0.02).

Figure 2 shows the net cost per patient per year in relation to the change in HbA_1c, where each data point represents a unique study that reported both measures. We were unable to formally test whether larger improvements in HbA_1c were associated with greater net savings due to the small number of studies with each type of design.

Figure 3 shows the cost-effectiveness plane with willingness-to-pay thresholds of $50,000 and $100,000 per QALY. Lower costs and better health are toward the lower right, and each data point reflects a unique study or subpopulation, with RCTs represented by circles. All analyses yielded ICERs below $115,000 per QALY over 20 or more years. The ICER was $100,000–$115,000 per QALY in three RCTs (36,38,39), $50,000–$99,999 per QALY in one RCT (36), $0–$49,999 per QALY in four RCTs (including one with two study subpopulations) (32,39–41), and dominant (more effective and less costly than the status quo) in one RCT (42).

Results were somewhat more favorable in analyses based on nonrandomized designs. In seven studies based on controlled before-after designs, the ICER was $104,132 per QALY in one subpopulation (34), $50,000–$99,999 per QALY in one subpopulation (35), and $0–$49,999 per QALY in five studies or study subpopulations (33,34,43). In seven studies based on uncontrolled before-after and other designs, the ICER was $50,000–$99,999 per QALY in one study (44), $0–$49,999 per QALY in four studies (45–48), and dominant with more than $5,000 in net savings in two studies due to avoiding complications including renal disease (49,50).

This systematic review examined economic evaluations of 46 multifaceted QI interventions designed to improve glycemic control among adults with diabetes, including 19 RCTs that included over 33,000 patients. There are three key findings. First, the studied QI interventions were effective, leading to average declines in HbA_1c of 0.26%, or 3 mmol/mol, based on RCT data. Second, the cost of implementing QI interventions was generally offset by reductions in health care expenditures in the short term, such that net costs to the health system were not significantly different from zero. Third, over 20 years or longer, costs rose along with survival, but the ICER was under $115,000 per QALY in all studies and populations. Declines in HbA_1c, short-term costs, and long-term cost-effectiveness were more favorable in studies based on nonrandomized designs.

The interventions that we examined emphasized QI strategies that have been recommended by the American Diabetes Association and found to be effective in prior systematic reviews, including patient self-management support, changes to the health care team, disease management, patient education, use of electronic registries, and clinical decision support (10,11). In a prior meta-analysis of 120 RCTs on QI strategies for glycemic control, HbA_1c declined by an average of 0.37% (95% CI 0.28–0.45), or 4 mmol/mol (3–5), overall, including declines of 0.57% (0.31–0.83), or 6 mmol/mol (3–9), for patient self-management support; 0.57% (0.42–0.71), or 6 mmol/mol (5–8), for team changes; 0.50% (0.36–0.65), or 5 mmol/mol (4–7), for disease/case management; and 0.48% (0.34–0.61), or 5 mmol/mol (4–7), for patient education (11). In a systematic review of QI interventions that included components of the Chronic Care Model (CCM), the mean decline in HbA_1c was 0.5% (0.3–0.6), or 5 mmol/mol (3–7), across 48 primary studies (10,51). The CCM includes patient self-management support, delivery system redesign through team changes and clinical information systems such as patient registries, and clinical decision support (52).

Our work adds to this literature by demonstrating that QI interventions designed to improve glycemic control increased value from the health system perspective. However, these increases in value were entirely attributable to improvements in health outcomes as costs did not fall. In the short term, improving HbA_1c at no net cost is clearly a good value. In the long term, the interpretation of ICERs requires consideration of a society’s willingness to pay for improvements in health. In the U.K., the National Institute for Health and Care Excellence currently considers health interventions that cost under £20,000–£30,000 ($23,815–$35,723) per QALY to be cost-effective. In the U.S., the Office of Management and Budget recommends that analyses supporting government regulations use a value of a statistical life of $9.6 million, which equates to a value of per discounted QALY of over $300,000 (53). Interventions that cost under $50,000 per QALY have been considered cost-effective since the 1970s. Accounting for inflation in prices, this equates to about $300,000 per QALY today. Some authors have suggested that, based on temporal trends in health and health care spending, society appears willing to pay at least $200,000 per QALY (54).

Yet, the fact that we found net cost savings to be unlikely in the short or long term is noteworthy. Glucose control in diabetes would seem to be the archetypal situation in which improving quality might lead to financial savings because underuse of evidence-based care may increase visits related to hyper- and hypoglycemia in the short term and contribute to costly complications in the long term. Prior studies indicate that improvements in HbA_1c can be associated with declines in health care utilization and expenditures in the short and long term—but these studies overlook costs associated with implementing interventions to change clinical practice (15,55–59). Furthermore, total lifetime health care expenditures rise with increases in survival. Our findings imply that investing in efforts to improve glycemic control are not likely to yield direct financial benefits to health systems and physician practices, which incur implementation costs and lose revenue when utilization declines. Accordingly, public reporting and value-based payment programs, such as the U.S. National Committee for Quality Assurance’s Healthcare Effectiveness Data and Information Set (HEDIS) program and Centers for Medicare & Medicaid Services Quality Payment Program (60–62) or the U.K. National Health Service’s Quality and Outcomes Framework, are designed to create external incentives for investing in QI (63–65).

This analysis has several limitations. Although we focused on HbA_1c, some eligible studies estimated the combined clinical benefits and costs of controlling HbA_1c and managing other cardiovascular disease risk factors (46). For example, intensive blood pressure control can be cost saving in patients with diabetes, excluding the cost of any QI interventions that might be implemented to attain such control. For the weighted regression analyses, stratifying by clinical study design reduced statistical power and, thus, the ability to detect factors associated with effectiveness and net costs, increasing the possibility of type II error (false negatives). A larger number of high-quality RCTs would be needed to conclude which types of QI strategies work best or are most cost efficient in which settings. We emphasize RCTs because the nonrandomized studies had more favorable findings and higher risks of bias. Measurement error may have occurred when assigning categories of QI studies to individual articles because this depended on clear and complete reporting by the original authors. QI interventions are context dependent, but we examined studies in diverse populations and settings in developed nations; lower cost interventions are likely to be emphasized in low- and middle-income countries. Studies were generally at moderate risk of bias related to study design. We were unable to formally assess publication bias or heterogeneity in costs because data on variance were limited to absent. Authors may not perform economic analyses until clinical effectiveness has been demonstrated; however, we found that changes in HbA_1c were somewhat smaller than in prior systematic reviews (11).

In conclusion, diverse multifaceted QI interventions designed to improve glycemic control improve health outcomes and appear to be a fair-to-good value relative to usual care, depending on society’s willingness to pay for improvements in health. Given that the QI interventions do not yield net cost savings to the health system, a business case based solely on reducing costs appears unlikely.

Acknowledgments. The authors would like to acknowledge the contributions of the following individuals. Members of the technical expert panel: Craig Blackmore, MD (Virginia Mason Hospital), Jo Carol Hiatt, MD (Kaiser Permanente Southern California), Catherine MacLean, MD (WellPoint [prior affiliation]), Emma Hoo (Pacific Business Group Health), Paul McGann, MD (Centers for Medicare & Medicaid Services), Lucy Johns, MPH (Health Care Planning and Policy), Julie Cerese, RN, MSN (University HealthSystem Consortium), David Meltzer, MD, PhD (University of Chicago). Research assistants and fellows who contributed to the work: Philip Ros, MPH (University of California, Los Angeles), Courtney Coles, MPH (University of California, Los Angeles), Paul Di Capua, MD (VA Greater Los Angeles Healthcare System). Individuals who assisted with data collection and analysis: Daniel Waxman, MD, PhD (RAND Corp.), Martha Timmer, MA (RAND Corp.). Aneesa Motala (RAND Corp.) and Margaret Kelley, MS (Cedars-Sinai Medical Center) provided logistical and managerial support throughout the project, and Patricia Smith (RAND Corp.) also assisted with manuscript preparation.

Funding. The Agency for Healthcare Research and Quality (R01 HS22644-01) funded the work but did not participate in study design, data acquisition, analysis, or interpretation of results. The views reported are those of the authors.

Duality of Interest. No potential conflicts of interest relevant to this article were reported.

Author Contributions. T.K.N. and P.S. contributed to study concept and design. T.K.N., E.K., L.J.A., J.G., S.C.M., B.J.D., K.S., A.A., M.B., and R.S. contributed to the acquisition, analysis, or interpretation of data. T.K.N. and R.S. drafted the manuscript. T.K.N., E.K., L.J.A., J.G., S.C.M., B.J.D., K.S., A.A., M.B., and P.S. contributed to the critical revision of the manuscript for important intellectual content. T.K.N., E.K., L.J.A., S.C.M., and M.B. contributed to statistical analysis. T.K.N. obtained funding. T.K.N., L.J.A., B.J.D., and R.S. provided administrative, technical, or material support. T.K.N., S.C.M., and P.S. supervised the study.