## Abstract

Objective

To determine the effectiveness of a provider-based education and implementation intervention for improving diabetes control.

Design

Cluster-randomized trial with baseline and follow-up cross sections of diabetes patients in each participating physician's practice.

Setting

Eleven US Southeastern states, 2006–08.

Participants

Two hundred and five rural primary care physicians.

Intervention

Multi-component interactive intervention including Web-based continuing medical education, performance feedback and quality improvement tools.

Primary Outcome Measures

‘Acceptable control’ [hemoglobin A1c ≤9%, blood pressure (BP) <140/90 mmHg, low-density lipoprotein cholesterol (LDL) <130 mg/dl] and ‘optimal control’ (A1c <7%, BP <130/80 mmHg, LDL <100 mg/dl).

Results

Of 364 physicians attempting to register, 205 were randomized to the intervention (n= 102) or control arms (n= 103). Baseline and follow-up data were provided by 95 physicians (2127 patients). The proportion of patients with A1c ≤9% was similar at baseline and follow-up in both the control [adjusted odds ratio (AOR): 0.94; 95% confidence interval (CI): 0.61, 1.47] and intervention arms [AOR: 1.16 (95% CI: 0.80, 1.69)]; BP <140/90 mmHg and LDL <130 mg/dl were also similar at both measurement points (P= 0.66, P= 0.46; respectively). We observed no significant effect on diabetes control attributable to the intervention for any of the primary outcome measures. Intervention physicians engaged with the Website over a median of 64.7 weeks [interquartile range (IQR): 45.4–81.8) for a median total of 37 min (IQR: 16–66).

Conclusions

A wide-reach, low-intensity, Web-based interactive multi-component intervention did not improve control of glucose, BP or lipids for patients with diabetes of physicians practicing in the rural Southeastern US.

## Introduction

An estimated 23.6 million adults have diabetes in the USA [1]. Although quality of care process measures such as assessing hemoglobin A1c, blood pressure (BP) and low-density lipoprotein cholesterol (LDL) are performed at high rates, the outcome measures of their appropriate control lag behind. For example, in one US study, >40% adults with diabetes had hemoglobin A1c ≥7%, 70% had BP ≥130/80 mmHg and 55% had total cholesterol ≥200 mg/dl [2]. In the USA, diabetes is more prevalent in the South, but individuals with diabetes in Southern states have worse control [3, 4], especially rural residents [5, 6]. Hence, public health interventions aimed at improving diabetes control in patients living in Southern rural areas are warranted.

Because distance barriers are considerable in rural settings, it is reasonable to explore wide-reach, low-intensity interventions [7–9] to improving diabetes control. One approach that has shown some promise [10, 11] is Web-based continuing medical education (CME) programs. In fact, professional educational programs designed to improve the quality of health care and patient clinical outcomes are of increasing interest [12–15], but evidence on their effectiveness in improving patient outcomes is scant.

Therefore, our objective was to test a multi-component intervention including Web-based CME, performance feedback and quality improvement tools [16] targeted at primary care physicians designed to improve diabetes care—hemoglobin A1c, BP and LDL control—in the rural Southeastern USA.

## Methods

### Study design and setting

The Rural Diabetes Online Care (R-DOC) study was a cluster-randomized trial (Clinical Trials.gov identifier: NCT00403091). The protocol was approved by the local institutional review boards. Participants were family, general and internal medicine physicians located in rural areas of 11 Southeastern USA (Alabama, Arkansas, Florida, Georgia, Kentucky, Mississippi, Missouri, North Carolina, South Carolina, Tennessee and West Virginia). Details of the recruitment strategy are available elsewhere [17]; rural areas were identified by population size <25 000. Physicians practicing in rural areas were invited via letters/e-mails to log on to the Website.

Physicians were enrolled sequentially between 28 September 2006 and 13 September 2008 by visiting the study Website. During the first visit to the Website, physicians provided informed consent online and were randomized to the intervention or control arms using block randomization (the block size of four was concealed to the investigators and statistician). Only one physician per practice was enrolled. Participants were offered up to 1 h of American Medical Association Physicians Recognition Award (AMA PRA) Category 1 CME credit™ at no charge upon completion of each section of the Website (maximum 12 h).

### Intervention arm description

Website development included a formative evaluation [18] and focused on helping physicians to achieve A1c, BP and LDL control in their diabetic patients. The Website included case-based learning, personalized audit and feedback, and tools designed to facilitate the provision of high-quality care. Specifically, the intervention site contained: (i) challenging cases; (ii) individualized diabetes performance feedback reports based on the physician's own practice; (iii) practice timesavers; (iv) practical goals and guidelines, including guidance for quality improvement and systems redesign [16]; (v) patient resources; (vi) and an area to track and view CME credit. Additional details of the Website components are available elsewhere [19].

Intervention arm physicians received e-mail reminders every 1–3 weeks about Website updates. E-mails were tailored for participants who had not completed specific sections of the Website. In keeping with the wide-reach, low-intensity CME framework, we designed the study to be purely Web-based with only electronic communication and without mail, phone calls or other retention activities that would limit generalizability and prevent scaling up.

Individualized performance feedback reports were based on a review of records of 15 of each physician's patients with diabetes (baseline data, see below). These reports were sequentially added 12–18 months after launching the Website. We provided three feedback reports, graphically comparing each intervention arm participant's personal performance compared with the performance for the top 10% of other intervention arm participants [20]. The feedback reports consisted of: (i) % patients with A1c <7% in the past year, % with systolic BP <130 mmHg at the most recent visit and % with LDL <100 mg/dl in the past year; (ii) counseling on diet or exercise at any of the three most recent visits in the past year; and, (iii) medication added or increased for patients with A1c >8%, BP >140/90 mmHg if on less than four medications and LDL >130 mg/dl.

### Control arm description

The control Website contained: (i) links to diabetes practice guidelines and patient education materials; (ii) a list of educational conferences on general medical topics (updated monthly); (iii) an area to track and view their CME credit; and (iv) a link to an external medical blog. Physicians in the control group did not receive performance feedback reports or electronic communications.

### Data sources

All participating physicians provided copies of records of 15 (intervention arm physicians) or 10 (control arm physicians) consecutively seen patients with diabetes at baseline and again at follow-up (representing two cross-sectional views of each physician's practice). The number of records was selected to balance rigor and cost (see also statistical section below). Patient inclusion criteria were having at least two office visits during the past year and no dialysis, dementia, organ transplantation, HIV/AIDS, terminal illness or malignancy (except skin and prostate cancer). Data abstraction was performed by trained personnel on blinded records sent to the study center (or abstracted on site). Data included patient demographics, insurance status, medical conditions, diabetes complications, medications, BP readings and laboratory values. Thus, data were provided on patients fulfilling the above inclusion criteria; we included patients' data for visits prior to randomization (baseline assessment) and prior to study closure (follow-up assessment). Quality control conducted on 5% of records revealed agreement of 90–95% between abstractors. Each physician received a $200 incentive for providing records for baseline data (staff received$50 for photocopying charts) and \$500 for providing records for follow-up data.

### Outcomes

The primary outcomes were measures of ‘acceptable’ and ‘optimal’ diabetes control [21, 22]. Acceptable control was the proportion of patients with A1c ≤9%, BP <140/90 mmHg and LDL <130 mg/dl. These thresholds were selected to be consistent with thresholds of poor control as defined by the National Committee for Quality Assurance; however, missing values were not counted in the denominator. Optimal control was A1c <7%, BP <130/80 mmHg and LDL <100 mg/dl, reflecting guideline recommendations [21, 22].

Secondary outcomes were the rate of assessment of A1c and LDL over the previous year and 2 years, respectively, and BP at the last visit; and A1c, systolic BP and LDL group mean levels.

### Statistical approach

For analytical purposes and based on clinically important differences, the study required 100 physicians per trial arm to detect a minimum of 0.4% difference in A1c, 6 mmHg in BP and 6 mg/dl in LDL for the main outcomes (power 80%, alpha = 0.05, up to 20% dropout) (10 patients per physician at baseline and at follow-up). However, intervention arm physicians were asked to provide 15 patient records at baseline because of the need to construct audit and feedback reports for A1c, BP and lipids, reflecting the fact that not all diabetes patients also have hypertension and hyperlipidemia [23, 24].

All analyses followed the intention-to-treat principle. Patient characteristics were compared using bivariate analysis at baseline and follow-up with the Chi-square test or the t-test as appropriate.

The relationship between the outcome variables and intervention effect was examined with generalized linear mixed models (GLMM), accounting for clustering of patients within physicians. Odds ratios for follow-up vs. baseline within the two groups were calculated, adjusting for covariates that differed between baseline and follow-up patient populations (i.e. age, race and clinical diagnosis of hypertension or depression). GLMM was implemented by PROC GLIMMIX in SAS. Thus, the group coefficient represented differences between the intervention and control groups at baseline; the time coefficient represented changes in the control group over time, thus capturing temporal trends. Finally, the group-by-time interaction coefficient represented the difference-of-differences for over-time change in the intervention versus control group, and was a direct comparison of over-time change for the intervention versus the control group. A positive odds ratio meant that the odds of a patient being in control increased more over time for the intervention versus control group. All analyses were performed in SAS version 9.2 (Cary, NC).

We also conducted per-protocol analyses by examining the effect of web engagement (above vs. below median number of weeks of exposure to the Website) on optimal A1c control by using generalized linear latent and mixed models (GLLAMM) in STATA 11.1 (College Station, TX).

## Results

### Recruitment scheme, patient characteristics and web utilization

Figure 1 displays the recruitment scheme of the study. We obtained baseline and follow-up data on 95 physicians and 1182 of their diabetes patients at baseline and 945 diabetes patients at follow-up. Ninety (94.7%) physicians had access to the Internet in the office. The characteristics of the baseline and follow-up patients collectively are shown in Table 1. At baseline, intervention group physicians provided records for fewer African American patients and more patients with hypertension and depression compared with control group physicians. The median duration of control group physicians' visits to the Website was 5 min [interquartile range (IQR): 3–18] over a median of 0.14 weeks (IQR: 0.14–7.15); intervention group physicians' median duration of visits to the Website was 37 min (IQR: 16–66) over a median of 64.7 weeks (IQR: 45.4–81.8)].

Table 1

Characteristics of diabetes patients of 95 physicians randomized to intervention (n= 48) or control (n= 47) who completed follow-up

Patient characteristic Baseline

Follow-up

Intervention (n= 715) Control (n= 467) P value Intervention (n= 479) Control (n= 466) P value
Age, years 58.7 (13.6) 60.6 (13.8) 0.02 61.3 (13.4) 60.5 (12.7) 0.38
Gender, female 360 (51%) 230 (49%) 0.58 260 (55%) 252 (54%) 0.98
Race, African American 97 (14%) 99 (21%) 0.001 102 (21%) 143 (31%) 0.001
Obesitya 274 (38%) 193 (41%) 0.30 97 (20%) 75 (16%) 0.10
Smoker, current 85 (12%) 55 (12%) 0.92 63 (14%) 53 (13%) 0.72
No self-testing 168 (34%) 114 (36%) 0.43 189 (41%) 183 (45%) 0.32
Non-adherence to appointments 75 (11%) 54 (12%) 0.62 10 (2%) 23 (6%) 0.01
Insurance, Medicaid 65 (9%) 51 (11%) 0.30 66 (14%) 58 (12%) 0.54
Comorbidities
Diabetes complicationsb 147 (21%) 110 (24%) 0.22 156 (33%) 134 (29%) 0.20
Insulin use 103 (14%) 63 (14%) 0.90 84 (18%) 68 (15%) 0.22
Hypertension 473 (66%) 304 (65%) 0.71 327 (68%) 283 (61%) 0.02
Hyperlipidemia 110 (15%) 85 (18%) 0.20 11 (2.3%) 17 (3.7%) 0.22
Peripheral vascular disease 47 (7%) 31 (7%) 0.97 41 (9%) 35 (8%) 0.55
Coronary artery disease 135 (19%) 83 (18) 0.63 111 (23%) 95 (20%) 0.30
Vascular interventionc 60 (8%) 47 (910%) 0.33 36 (8%) 40 (9%) 0.55
Depression 105 (15%) 61 (13%) 0.43 84 (18%) 60 (913%) 0.05
Patient characteristic Baseline

Follow-up

Intervention (n= 715) Control (n= 467) P value Intervention (n= 479) Control (n= 466) P value
Age, years 58.7 (13.6) 60.6 (13.8) 0.02 61.3 (13.4) 60.5 (12.7) 0.38
Gender, female 360 (51%) 230 (49%) 0.58 260 (55%) 252 (54%) 0.98
Race, African American 97 (14%) 99 (21%) 0.001 102 (21%) 143 (31%) 0.001
Obesitya 274 (38%) 193 (41%) 0.30 97 (20%) 75 (16%) 0.10
Smoker, current 85 (12%) 55 (12%) 0.92 63 (14%) 53 (13%) 0.72
No self-testing 168 (34%) 114 (36%) 0.43 189 (41%) 183 (45%) 0.32
Non-adherence to appointments 75 (11%) 54 (12%) 0.62 10 (2%) 23 (6%) 0.01
Insurance, Medicaid 65 (9%) 51 (11%) 0.30 66 (14%) 58 (12%) 0.54
Comorbidities
Diabetes complicationsb 147 (21%) 110 (24%) 0.22 156 (33%) 134 (29%) 0.20
Insulin use 103 (14%) 63 (14%) 0.90 84 (18%) 68 (15%) 0.22
Hypertension 473 (66%) 304 (65%) 0.71 327 (68%) 283 (61%) 0.02
Hyperlipidemia 110 (15%) 85 (18%) 0.20 11 (2.3%) 17 (3.7%) 0.22
Peripheral vascular disease 47 (7%) 31 (7%) 0.97 41 (9%) 35 (8%) 0.55
Coronary artery disease 135 (19%) 83 (18) 0.63 111 (23%) 95 (20%) 0.30
Vascular interventionc 60 (8%) 47 (910%) 0.33 36 (8%) 40 (9%) 0.55
Depression 105 (15%) 61 (13%) 0.43 84 (18%) 60 (913%) 0.05

Values are mean (SD) or n (%).

aObesity: body mass index ≥30 or clinical diagnosis. bDiabetes complications: retinopathy, neuropathy or nephropathy. cVascular intervention: coronary artery bypass grafting, stent, percutaneous coronary angioplasty.

Figure 1

Consolidated standards of reporting trials (CONSORT) diagram.

Figure 1

Consolidated standards of reporting trials (CONSORT) diagram.

### Main outcomes—A1c, BP, LDL control

Acceptable A1c control (<9%) was similar at baseline and follow-up in both trial arms [control, adjusted odds ratio (AOR): 0.94; 95% confidence interval (CI): 0.61, 1.47; intervention, AOR: 1.16 (95% CI: 0.80, 1.69)], Figure 2 (top panel). Analogously, acceptable BP (<140/90 mmHg) and LDL control (<130 mg/dl) were also similar at baseline and follow-up in both trial arms [BP control arm, AOR: 1.19 (95% CI: 0.89, 1.58), P= 0.24; BP intervention arm, AOR: 1.06 (95% CI: 0.82, 1.38), P= 0.66] [LDL control arm, AOR: 1.08 (95% CI: 0.69, 1.69), P= 0.74; LDL intervention arm, AOR: 1.16 (95% CI: 0.78, 1.73), P= 0.46].

Figure 2

Main outcomes (top, middle panels). Proportions of patients with acceptable and optimal hemoglobin A1c, BP (BP, mmHg) and cholesterol (LDL, mg/dl) control for intervention (n= 48) or control (n= 47) physicians, at baseline (pre, n= 1182 patients) and follow-up (post, n= 945 patients). Secondary outcomes (bottom panel). Proportion assessed of hemoglobin A1c (past year), BP (last visit) and LDL (past 2 years). *P < 0.001, **P= 0.05.

Figure 2

Main outcomes (top, middle panels). Proportions of patients with acceptable and optimal hemoglobin A1c, BP (BP, mmHg) and cholesterol (LDL, mg/dl) control for intervention (n= 48) or control (n= 47) physicians, at baseline (pre, n= 1182 patients) and follow-up (post, n= 945 patients). Secondary outcomes (bottom panel). Proportion assessed of hemoglobin A1c (past year), BP (last visit) and LDL (past 2 years). *P < 0.001, **P= 0.05.

Optimal A1c control (<7%) was also similar at baseline and follow-up in both trial arms [control, AOR: 0.82 (95% CI: 0.60, 1.13); intervention, AOR: 1.02 (95% CI: 0.77, 1.34)], Figure 2 (middle panel). Optimal BP (<130/80 mmHg) was also similar at baseline and follow-up in both arms [BP control, AOR: 1.20 (95% CI: 0.88, 1.63); BP intervention, AOR: 1.15 (95% CI: 0.88, 1.50)]. For optimal LDL control (<100 mg/dl), the control arm was modestly improved at follow-up (AOR: 1.35 (95% CI: 0.96, 1.90)] but the intervention arm at baseline and follow-up was similar [AOR: 0.95 (95% CI: 0.71, 1.28)].

We observed no significant effect attributable to the intervention as reflected in the group-by-time interaction term for either acceptable (A1c: P= 0.45, BP: P= 0.46 and LDL: P= 0.68) or for optimal control measures (A1c: P= 0.28, BP: P= 0.74 and LDL: P= 0.19). Because of the large variability in time spent on the Website, we also examined whether time engaged in the Website was associated with improved patient outcomes. Although not statistically significant, as web engagement increased in the intervention group, the proportion of patients with optimal A1c control increased (above vs. below median number of weeks, AOR: 1.32 (95% CI: 0.90, 1.94), P= 0.16).

### Secondary outcomes—process measures

We observed notable increases in A1c assessment for both trial arms at follow-up relative to baseline (control, P < 0.001; intervention, P < 0.001), Figure 2 (bottom panel). BP assessment was high at both baseline and follow-up in both trial arms (control, P= 0.48; intervention, P= 0.05). Similar to A1c, LDL assessment increased for both trial arms at follow-up relative to baseline (control, P < 0.001; intervention, P < 0.001).

### Secondary outcomes—group mean A1c, BP and LDL

The group mean A1c did not differ between baseline and follow-up for both trial arms (control, P= 0.19; intervention, P= 0.70), Figure 3. Similarly, mean systolic BP was similar at baseline and follow-up across trial arms (control, P= 0.40; intervention, P= 0.68). Finally, LDL also did not differ significantly at baseline and follow-up across trial arms, although there were non-significant trends toward improvement in both arms (control, P= 0.31; intervention, P= 0.20).

Figure 3

Secondary outcomes. Hemoglobin A1c, systolic and diastolic BP (mmHg) and cholesterol (LDL, mg/dl) for patients in the control and intervention groups at baseline (pre) and follow-up (post) (values are means and 95% confidence intervals).

Figure 3

Secondary outcomes. Hemoglobin A1c, systolic and diastolic BP (mmHg) and cholesterol (LDL, mg/dl) for patients in the control and intervention groups at baseline (pre) and follow-up (post) (values are means and 95% confidence intervals).

## Discussion

In this cluster-randomized trial, a wide-reach, low-intensity Web-based multi-component intervention for primary care physicians in the rural Southern USA had no significant effect on their diabetes patients' glycemic, BP or lipid control. These findings have important implications for the recent enthusiasm in the CME community for education/quality improvement programs that aim to improve not only learning outcomes, but also higher level patient outcomes. While the type of intervention tested here could potentially be widely disseminated, the results suggest that in order to improve patient outcomes, perhaps a higher intensity approach, and concomitant cost, may be needed.

One plausible explanation for the lack of improvement in patient outcomes was the relatively low Web engagement by physician participants. This observation has important implications for Web-based CME programs, which by one estimate made up 7–9% of CME in 2008, but may make up as much as 50% by 2018 [25]. As we previously reported, intervention physicians whose patients were in the worst quartile of A1c control—in greatest need for improvement—spent substantially less time on the Website than physicians whose patients were in the best quartile of A1c control (33 vs. 80 min, respectively, P= 0.003) [19]. The low engagement occurred in spite of highly tailored retention activities and personalized feedback reports reflecting the physician's own practice. We purposely designed this intervention to be relatively low in intensity but wider in reach [9], maximizing generalizability and dissemination possibilities. However, our findings suggest that to impact patient outcomes, more intensive approaches may be required to better engage physicians, especially those physicians whose quality measures are suboptimal.

A second contributor to our findings could be the higher-than anticipated attrition rate (50 vs. 20% projected). High attrition is not unusual in Web-based interventions [26]. The marked loss to follow-up undoubtedly was related to time constraints, since the Health Insurance Protection and Portability Act required us to request practice staff to blind records sent to us. Notably, practice closures were also higher than we anticipated, with a total of 20 practices closing over the 18 months between baseline and follow-up, reflecting economic conditions in the South. Attrition has important implications for performance improvement CME, which is even more demanding of physician time than this study. We conducted the record review and practice staff-copied records, but in performance improvement CME, physician themselves are often asked to conduct the review. While this could increase engagement, it could also result in substantial barriers to participation and retention, as we experienced in our study.

Our findings are consistent with some but not all physician Web-based education studies. For example, process but not outcome measures improved in a group-randomized controlled trial of a Web-based provider decision support tool among 26 urban academic medical center practitioners and 598 of their patients [27]. Intervention providers could access patient data, treatment recommendations and Web-based resources. Annual foot examinations and A1c and LDL testing increased in the intervention group. However, measures of A1c, BP or LDL control did not improve in either study arm. In our study, physicians in both trial arms demonstrated both statistically significant and clinically important improvements in the assessment of hemoglobin A1c and LDL cholesterol. These findings suggest that low-intensity, wide-reach CME programs may be more effective at improving processes but not outcomes of care; however, the lack of significant differences between the intervention and control groups suggests a Hawthorne's effect (behavior change simply due to the study itself, rather than the specific intervention).

More recently, in a pre–post study of 18 rural physicians, a Web-based educational intervention was associated with improved patient outcomes [28] in three cross-sectional samples of patients. The Web-based intervention was composed of 13 educational chapters, practice tools, patient education resources and self-assessment. Glucose control (A1c <7%) increased by 5% at 24 months; BP control (≤140/90 mmHg) increased by 11%; and LDL control (<100 mg/dl) increased by 5%. In contrast to our CME program, this quality improvement intervention was far more intensive, and the lack of a control group makes the reported results difficult to interpret.

Our study has several important implications. First, the quest for effective wide-reach and low-intensity interventions, applicable to rural settings, is important. The needs to update practicing physicians on the rapidly expanding body of knowledge as well as to optimize the quality of care they provide are very real, and finding effective strategies to accomplish continuing professional development for physicians practicing in remote locations is an important objective. Web-based CME activities are widely accessible and are therefore attractive options. However, our study suggests that more work is needed to identify strategies that can achieve better physician engagement [15]. We hoped to have physicians apply quality improvement principles in their practices; however, they were particularly disengaged in this area. If better engagement can be achieved, tests of whether Web-based programs can actually improve patient outcomes are needed. Notable is the paucity of rigorous evaluation of CME programs per se. Only studies with control groups and rigorous design can provide information on the effectiveness of such programs, and our study is an example of how such programs can be tested.

Our study had several notable limitations. The non-random sampling of physicians may have introduced selection bias. Second, physicians were asked to provide records of consecutively seen patients, and while we were unable to monitor compliance with this request, the wide range of A1c, BP and LDL values suggests that not only well-controlled patients were selected, but poorly-controlled patients as well. The fiscal realities of our project prevented a more comprehensive assessment of each physician's practice, and it is possible that 10 or 15 records did not represent the physician's diabetes patients in general. The high attrition may have introduced biases. Although it was designed with wide reach and scaling up in mind, our budget for Website development and implementation likely exceeded that available in most academic CME divisions, raising concerns about sponsorship of such programs.

In conclusion, a rigorous test of a state-of-the-art multi-component Web-based intervention designed to improve the diabetes care of rural primary care physicians showed no improvement in their patients' glycemic, BP and lipid control. Despite low engagement in the Web-based program, process of care measures did improve in both arms of the study. Our study raises important issues for wide-reach, low-intensity CME programs that seek to improve patient outcomes.

## Funding

Awards by the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) 5R18DK065001 to Dr J.J.A. Drs A.H.S and C.A.E. were supported by the Veterans Affairs National Quality Scholars Program.

## Acknowledgements

Sharina Person, PhD, for prior statistical analysis assistance.

## References

1
Centers for Disease Control and Prevention
National Diabetes Fact Sheet: General Information and National Estimates on Diabetes in the United States, 2007
,
2008
Atlanta, GA
U.S. Department of Health and Human Services, Centers for Disease Control and Prevention
2
Mann
DM
Woodward
M
Ye
F
, et al.  .
Trends in medication use among US adults with diabetes mellitus: glycemic control at the expense of controlling cardiovascular risk factors
Arch Intern Med
,
2009
, vol.
169
(pg.
1718
-
20
)
3
Jencks
SF
Cuerdon
T
Burwen
DR
, et al.  .
Quality of medical care delivered to Medicare beneficiaries: a profile at state and national levels
JAMA
,
2000
, vol.
284
(pg.
1670
-
6
)
4
Weingarten
JP
Jr
Brittman
S
Hu
W
, et al.  .
The state of diabetes care provided to Medicare beneficiaries living in rural America
J Rural Health
,
2006
, vol.
22
(pg.
351
-
8
)
5
Andrus
MR
Kelley
KW
Murphey
LM
, et al.  .
A comparison of diabetes care in rural and urban medical clinics in Alabama
J Community Health
,
2004
, vol.
29
(pg.
29
-
44
)
6
Krishna
S
Gillespie
KN
McBride
TM
Diabetes burden and access to preventive care in the rural United States
J Rural Health
,
2010
, vol.
26
(pg.
3
-
11
)
7
Glasgow
RE
Emmons
KM
How can we increase translation of research into practice? Types of evidence needed
Annu Rev Public Health
,
2007
, vol.
28
(pg.
413
-
33
)
8
Glasgow
RE
Klesges
LM
Dzewaltowski
DA
, et al.  .
Evaluating the impact of health promotion programs: using the RE-AIM framework to form summary measures for decision making involving complex issues
Health Educ Res
,
2006
, vol.
21
(pg.
688
-
94
)
9
Glasgow
RE
Vogt
TM
Boles
SM
Evaluating the public health impact of health promotion interventions: the RE-AIM framework
Am J Public Health
,
1999
, vol.
89
(pg.
1322
-
7
)
10
Fordis
M
King
JE
Ballantyne
CM
, et al.  .
Comparison of the instructional efficacy of internet-based CME with live interactive CME workshops: a randomized controlled trial
JAMA
,
2005
, vol.
294
(pg.
1043
-
51
)
11
Allison
JJ
Kiefe
CI
Wall
T
, et al.  .
Multicomponent Internet continuing medical education to promote chlamydia screening
Am J Prev Med
,
2005
, vol.
28
(pg.
285
-
90
)
12
Moore
DE
Jr
Green
JS
Gallis
HA
Achieving desired results and improved outcomes: integrating planning and assessment throughout learning activities
J Contin Educ Health Prof
,
2009
, vol.
29
(pg.
1
-
15
)
13
Moore
DE
Jr
Pennington
FC
Practice-based learning and improvement
J Contin Educ Health Prof
,
2003
, vol.
23

Suppl. 1
(pg.
S73
-
80
)
14
Davis
N
Davis
D
Bloch
R
Continuing medical education: AMEE Education Guide No 35
Med Teach
,
2008
, vol.
30
(pg.
652
-
66
)
15
Duffy
FD
Lynn
LA
Didura
H
, et al.  .
Self-assessment of practice performance: development of the ABIM Practice Improvement Module (PIM)
J Contin Educ Health Prof
,
2008
, vol.
28
(pg.
38
-
46
)
16
Salanitro
AS
CA
Allison
JJ
Glasser
S
Implementation research: beyond the traditional randomized controlled trial
Essentials of Clinical Research
,
2008
New York: Springer Science
(pg.
217
-
44
)
17
Foster
PP
Williams
JH
CA
, et al.  .
Recruitment of rural physicians in a diabetes internet intervention study: overcoming challenges and barriers
J Natl Med Assoc
,
2010
, vol.
102
(pg.
101
-
7
)
18
Safford
MM
Shewchuk
R
Qu
H
, et al.  .
Reasons for not intensifying medications: differentiating ‘clinical inertia’ from appropriate care
J Gen Intern Med
,
2007
, vol.
22
(pg.
1648
-
55
)
19
Crenshaw
K
Salanitro
AH
Safford
MM
, et al.  .
Are there differences in engagement in a web-based intervention among physicians of patients with diabetes: the Rural Diabetes Online Care (RDOC) Study
,
2010
, vol.
85
(pg.
1511
-
7
)
20
Weissman
NW
Allison
JJ
Kiefe
CI
, et al.  .
Achievable benchmarks of care: the ABCs of benchmarking
J Eval Clin Pract
,
1999
, vol.
5
(pg.
269
-
81
)
21
American Diabetes Association
Standards of medical care in diabetes—2006
Diabetes Care
,
2006
, vol.
29

Suppl. 1
(pg.
S4
-
42
)
22
National Committee for Quality Assurance
Comprehensive diabetes care
2008

23
Allison
JJ
Calhoun
JW
Wall
TC
, et al.  .
Optimal reporting of health care process measures: inferential statistics as help or hindrance?
Manag Care Q
,
2000
, vol.
8
(pg.
1
-
10
)
24
Holmboe
ES
Meehan
TP
Lynn
L
, et al.  .
Promoting physicians' self-assessment and quality improvement: the ABIM diabetes practice improvement module
J Contin Educ Health Prof
,
2006
, vol.
26
(pg.
109
-
19
)
25
Harris
JM
Jr
Sklar
BM
Amend
RW
, et al.  .
The growth, characteristics, and future of online CME
J Contin Educ Health Prof
,
2010
, vol.
30
(pg.
3
-
10
)
26
Bennett
GG
Glasgow
RE
The delivery of public health interventions via the Internet: actualizing their potential
Annu Rev Public Health
,
2009
, vol.
30
(pg.
273
-
92
)
27
Meigs
JB
Cagliero
E
Dubey
A
, et al.  .
A controlled trial of web-based diabetes disease management: the MGH diabetes primary care improvement project
Diabetes Care
,
2003
, vol.
26
(pg.
750
-
7
)
28
Gannon
M
Qaseem
A
Snow
V
Community-based primary care: improving and assessing diabetes management
Am J Med Qual
,
2010
, vol.
25
(pg.
6
-
12
)