Abstract

Background and aims. The aim of the present study was to evaluate whether three reminder letters mailed to GPs after dissemination of a Dementia Guideline increased the GPs’ use of the corresponding e-learning programme (ELP).

Methods. Single-blinded randomized trial among all GPs in Copenhagen Municipality from 1 November 2006 to 1 May 2007.

Results. A total of 15 of 320 GPs (4.7%) had a web-based logon during the study period. The intervention group had a significantly increased frequency of web-based logons (P = 0.0192) equivalent to a hazard ratio of 8.0 (95% CI: 1.03–66.1; P = 0.047). NNT was calculated to 22.2. We could not detect any significant differences in any of the secondary outcomes.

Conclusions. Three reminder letters added to a nation-wide dissemination increased the probability for a GP logon in the ELP by a Factor 8. However, in total, only a small proportion used the ELP. Thus, further research is needed in order to consider future implementation strategies for Internet-based Continuous Medical Education activities among not primed GPs.

Introduction

Internet-based education offers an innovative possibility for professional education and training of physicians. It refers to the use of Internet technologies to deliver a broad array of solutions that enhance knowledge and performance for the user.1 The literature in medical education indicates that Internet-based programmes are as effective in increasing participant knowledge as traditional formats.2–4 Studies in both medical and non-medical literature have consistently demonstrated a high learner satisfaction with e-learning and the satisfaction rates increase with e-learning compared to traditional learning, along with perceived ease of use and access, navigation, interactivity and user-friendly interface design.5

Clinical guidelines, in the form of systematically developed statements to assist physician–patient decisions on appropriate health care in specific circumstances, are produced in many countries.6 However, the dissemination of guidelines alone rarely brings about improvements in clinical practice7 and even an implementation of guidelines may not change clinical practice.8,9 Multiple strategies for implementing guidelines appear to be more effective than single ones.7,10. However, well-designed empirical research looking into various implementation strategies is still needed in this area.11

Dementia is a common condition in the elderly, with an estimated prevalence of 7.5% in those >65 years of age.12 However, previous surveys have indicated that a substantial fraction of patients with cognitive symptoms are not diagnosed by GPs,13 that diagnostic evaluation in general practice is difficult9,14 and that several barriers for diagnostic evaluation in patients and their caregivers as well as among GPs exists.13 Previously, we have reported that only few GPs used a corresponding e-learning programme (ELP) launched at the same time as the dissemination of a Dementia Guideline.15 Due to the assumption that the use of web-based Continuous Medical Education (CME) will increase in the future and the lack of knowledge regarding implementation of both guidelines and web-based CME, we found it of interest to study whether three postal reminder letters mailed to GPs after dissemination of a guideline on dementia increased the GPs’ use of the corresponding ELP.

Material and methods

The study was conducted among all 339 GPs working in the municipality of Copenhagen from 1 November 2006 until 1 May 2007.

Participants

The GPs were identified by Quality Development Centre for General Practice in Copenhagen Municipality (KvEAP). GPs moving or retiring during the study period were excluded. Characteristics of the GPs are presented in Table 1.

TABLE 1

Baseline characteristics of GPs

 Intervention (n = 160) Controls (n = 160) 
GP characteristics 
    No (%) male 87 (54.4) 83 (51.9) 
    Age (95% CI) 52.9 (51.7–54.1) 54.5 (53.3–55.6) 
    No (%) from single-handed practice 110 (68.8) 109 (68.1) 
    Years in practice (95% CI) 12.5 (11.2–13.9) 14.4 (13.0–15.8) 
    Average patients per GP (95% CI) 1825 (1627–2021) 1645 (1535–1756) 
Primary outcomea 
    Numbers of GPs logging on the ELP 4 (2.5%) 2 (1.1%) 
 Intervention (n = 160) Controls (n = 160) 
GP characteristics 
    No (%) male 87 (54.4) 83 (51.9) 
    Age (95% CI) 52.9 (51.7–54.1) 54.5 (53.3–55.6) 
    No (%) from single-handed practice 110 (68.8) 109 (68.1) 
    Years in practice (95% CI) 12.5 (11.2–13.9) 14.4 (13.0–15.8) 
    Average patients per GP (95% CI) 1825 (1627–2021) 1645 (1535–1756) 
Primary outcomea 
    Numbers of GPs logging on the ELP 4 (2.5%) 2 (1.1%) 
a

Period 1 November 2006 to 8 January 2007.

The guideline and the ELP

The Dementia Guideline was developed by the Danish College of General Practitioners (DSAM) and mailed to all Danish GPs in October 2006 (the guideline is available at the following Internet address: http://www.dsam.dk/files/9/demens_2006.pdf). The ELP was launched 31 October 2006 and followed the recommendation of the Guideline and consisted of five sections: suspicion of dementia, identification, diagnostics, evaluation and follow-up. The ELP was based on interactive parts using slides with audio, video-cases and self-study parts. The estimated time to complete the ELP was 90 minutes and the producers did not incorporate an evaluation of performance. All GPs had free access to the ELP from the homepages of DSAM as well as Danish Medical Association (DMA) by a unique username and password. The Danish Ministry of Health financially supported the Guideline and the ELP. The GPs were not offered any incentives for using the ELP.

General nation-wide implementation

The dissemination strategy was chosen by DSAM and the ELP producer jointly. At the launching, the ELP was promoted at the homepages of the DMA and DSAM, the ELP was mentioned in the weekly journal of DMA (Ugeskrift for Læger) as well as the journal of DSAM (Practicus). Furthermore, DSAM has sent one e-mail to all members with a stated e-mail address.

Specific implementation (intervention)

GPs were randomized to receive either three reminder letters mailed from KvEAP or nothing in the period from December 2006 to February 2007.

The first reminder

The first reminder letter was mailed 9 January 2007 and introduced the ELP and explained in detail how to find the ELP at either DSAMs or DMA homepage. If any problems occurred, the GPs had the possibility to contact two computer specialists employed by the municipality that could help the GPs.

The second reminder

The second reminder letter was mailed 24 January 2007 and was divided into two sections. The first section was a ‘test your knowledge’ section consisting of four questions that could be solved by the ELP:

  1. How common is Frontotemporal dementia?

  2. How common is Lewy Body Dementia?

  3. Which type of dementia is characterized by hallucination early in the disease?

  4. Are there any risk factors for development of Frontotemporal dementia?

The second section reminded the GPs how to log logon to the ELP.

The third reminder

The third reminder letter was mailed 13 February 2007 and was divided into two sections as well. The first section was a test your knowledge section consisting of four questions that could be solved by the ELP:

  1. What is the most important differentia diagnosis to Alzheimer's disease?

  2. What type of job had case 1 (Anne Hansen) in the ELP?

  3. What was the Mini Mental State Examination of Anne Hansen?

  4. How can you distinguish the course of Lewy Body dementia and Parkinson Dementia?

The second section reminded the GPs how to find logon to the ELP.

Data collection

Data were collected by means of databases from KvEAP, log files, as well as from databases at the DMA. The GPs identified by KvEAP were handed over to the DMA. Outcome measures were provided by the registration department of DMA in form of files with plain text (ASCII-files). For each randomization group, we received two files, one from the first period and one for the second period. The randomization code and identification variable were delivered to DMA after the follow-up period and the log files from the ELP were merged with the randomization code and identification variable. Then the identification variable was deleted from the file, and an electronic file was then returned for outcome analysis. Thus, data on the individual GP use of the ELP was kept anonymous for the authors.

KvEAP data

From KvEAP we obtained the following data at the start of the study: gender, age, number of years in practice, the number of GPs working together and the number of patients enlisted at the clinic.

Log files

The software provider (Cursum) retrieved the log files and handed them over to DMA. Each click by a mouse is filed in a log file, which makes it possible to examine the use and time spent on the ELP. When no mouse click was detected for 20 minutes, the programme was terminated and no time was calculated. Log files were obtained as plain text.

Data from the DMA

Informations regarding sex, age, year of graduation from university, working address, type of practice and membership of the DSAM and log files were retrieved from the DMA and merged with the log files in May 2007. Subsequently, a constructed identification code was obtained and the original identification was deleted. These files were handed over to FBW and VS for analysis.

Sample size

It was estimated that the intervention group would increase the use of the ELP by 15% in absolute figures. We assumed that ∼20% would logon the web page. Given a power of 80%, and a significance level at 5%, it was estimated that ∼300 GPs should participate in the study.

Randomization

The randomization was carried out independently by an employee at the research unit of general practice. The practices were stratified before randomization by single-handed practices and practices with more than one GP working full time by means of PROC PLAN (SAS version 9). The first half of the randomly generated list was allocated to intervention and the rest to control.

Outcome measures

The primary outcome measure was the proportion of GPs with at least one log-on procedure at the ELP. Secondary outcome measures were the proportion of GPs with more than one log-on procedure and the number of slides viewed at the ELP. The baseline period was defined as from the launch of the ELP up to the dissemination of the first reminder letter.

Statistics

We used chi-square and t-tests to compare differences at baseline. The distributions of the (first) logins over time are illustrated with Kaplan–Meier estimate of the cumulative time distribution for each of the trial arms. Hazard ratio (HR) estimates for the intervention effect are obtained from a Cox regression analysis departing from the 9th of January 2007 (the start of the intervention). For those GPs that logged on in the period from the start of the programme to the 1st of May 2007, the difference in number of slides visited between the trial arms is tested with Kruskal–Wallis non-parametric tests. A significance level of 5% is considered significant. All statistical analyses were performed using SAS, version 9.1. NNT was calculated as the inverse of the difference in log on proportion between intervention and control groups.

Ethics

The Scientific Ethical Committee for Copenhagen and Frederiksberg Municipalities evaluated the project. The Danish Data Protection Agency and the DSAM Study Committee approved the project. The project was registered in ClinicalTrials.gov Identifier: NCT00394017.

Results

A total of 339 physicians in 280 different practices were working as GPs in the municipality of Copenhagen. During the study period, a total of 19 GPs retired leaving 320 GPs for analysis, see Figure 1. No statically significant differences between intervention and control GPs were detected at the start of the study (Table 1).

FIGURE 1

Trial flow of participants

FIGURE 1

Trial flow of participants

A total of 15 of 320 (4.7%) GPs had at least one log-on session at the e-learning site. Stratified by intervention/control group, a total of 12 (7.5%) came from the intervention group and 3 (1.9%) came from the control group. The group of GPs receiving reminder letters had, after intervention, a significant 8-fold increased probability for a log-on procedure at the e-learning site compared with controls after the dissemination of the reminder letters (HR = 8.0, 95% CI: 1.03–66.1; P = 0.047). NNT was calculated to 22.2. We did not find any statistical differences between intervention and controls regarding secondary outcomes, see Table 2 and Figure 2.

TABLE 2

Outcomes at the end of the study based on figures from the dissemination of the first reminder letter (9 January 2007 to 1 May 2007)

 Intervention (n = 160) Control (n = 160) P-value 
Primary outcome 
    Logins, n (%) 8/156a (5.1%) 1/158b (0.6%) 0.0192 
Secondary outcomes 
    Median relog frequency – 
    Median slides viewed (min–max) 3 (0–7) 2 (2–2) 0.8445 
 Intervention (n = 160) Control (n = 160) P-value 
Primary outcome 
    Logins, n (%) 8/156a (5.1%) 1/158b (0.6%) 0.0192 
Secondary outcomes 
    Median relog frequency – 
    Median slides viewed (min–max) 3 (0–7) 2 (2–2) 0.8445 
a

Four GPs were censored from this analysis because they were had logged on the ELP during baseline.

b

Tow GPs were censored from this analysis because they were had logged on the ELP during baseline.

FIGURE 2

The proportion of GPs logging on the e-learning site during the study period stratified by intervention and control (N = 320)

FIGURE 2

The proportion of GPs logging on the e-learning site during the study period stratified by intervention and control (N = 320)

Discussion

In comparison with the control group GPs, intervention group GPs had an increased probability for a logon at web page by a Factor 8. Thus, in this study, postal reminders did seem to improve the uptake of the ELP among GPs.

A number of factors must be considered. Due to the low number of doctors who used the ELP, our study may be underpowered. In our power calculation, we assumed that 32 of the GPs would use the ELP in the control group and 56 in the intervention group. However, in total, only 15 GPs did logon the programme, some before the start of the intervention. The low log-on frequency may explain that we did not identify statistically significant effect on the individual secondary outcomes. NNT for a logon at the ELP based on three reminder letters were 22.2. This may seem high and underscore the low effect of postal reminder letters.

Second, e-mails have successfully been used in several studies to implement health behaviours changes at risk populations16,17 and web-based CME among health care providers.18 In general, these studies reported that e-mail reminders did increase both the use of ELP as well as reinforced the utilization of the programme among the target population.18 We considered using e-mails as our intervention as well; however, because of Danish privacy legislation, we were not able to use this method.

In accordance with other studies, we used log-files from the ELP software provider as an outcome measure.19 This methodology ensures that our data are complete and correct and that the proportion of participants therefore can be considered accurate. Thus, we believe that this result could be generalized in a Danish setting. However, this outcome measure does not evaluate the actual learning progress of the individual physician using the ELP. Likewise, GPs with a different learning profile may have gained knowledge from other sources and the reminder letters may have prompted other forms of CME activities, which our outcome measure did not detect.

Third, single interventions only rarely improve physicians’ performance, whereas combined strategies seem more effective.7,10 In our study, we added three reminder letter mailed directly to each GPs to the general nation-wide implementation. Thus, the intervention may have been considered a multifaceted intervention, even though we only studied reminder letters in this study.

Conclusions

In conclusion, three reminder letters added to a nation-wide dissemination increased the probability for a GP logon the ELP. However, only a small proportion of the GPs used the ELP. Further research is needed in order to consider future implementation strategies for Internet-based CME activities among not primed GPs. This research should also incorporate qualitative aspects.

Declaration

Funding: The Danish Ministry of Health.

Ethical approval: The Scientific Ethical Committee for Copenhagen and Frederiksberg Municipalities evaluated the project.

Conflict of interest: none.

Authors’ contribution: FBW participated in the design of the study and acquisition of the data, analysed data, drafting and revising of the manuscript. VS analysed data and participated together with BN, APS and FB in the interpretation of data and revising the manuscript. All authors read and approved the final manuscript.

References

1
Rosenberg
MJ
E-Learning: Strategies for Delivering Knowledge in the Digital Age
 , 
2001
New York
McGraw-Hill
2
Cobb
SC
Internet continuing education for health care professionals: an integrative review
J Contin Educ Health Prof
 , 
2004
, vol. 
24
 (pg. 
171
-
80
)
3
Ruiz
JG
Mintzer
MJ
Leipzig
RM
The impact of E-learning in medical education
Acad Med
 , 
2006
, vol. 
81
 (pg. 
207
-
12
)
4
Wutoh
R
Boren
SA
Balas
EA
eLearning: a review of Internet-based continuing medical education
J Contin Educ Health Prof
 , 
2004
, vol. 
24
 (pg. 
20
-
30
)
5
Gibbons
A
Fairweather
P
Tobias
S
Fletcher
J
Computer-based instruction
Training & Retraining: A Handbook for Business, Industry, Government and the Military
 , 
2002
New York
Macmillian Reference
(pg. 
410
-
42
)
6
Grimshaw
J
Freemantle
N
Wallace
S
, et al.  . 
Developing and implementing clinical practice guidelines
Qual Health Care
 , 
1995
, vol. 
4
 (pg. 
55
-
64
)
7
Wensing
M
van der
WT
Grol
R
Implementing guidelines and innovations in general practice: which interventions are effective?
Br J Gen Pract
 , 
1998
, vol. 
48
 (pg. 
991
-
7
)
8
Bro
F
Waldorff
FB
Guidelines—let's take a break and then move forward together!
Scand J Prim Health Care
 , 
2004
, vol. 
22
 (pg. 
2
-
5
)
9
Waldorff
FB
Almind
G
Makela
M
Moller
S
Waldemar
G
Implementation of a clinical dementia guideline. A controlled study on the effect of a multifaceted strategy
Scand J Prim Health Care
 , 
2003
, vol. 
21
 (pg. 
142
-
7
)
10
Oxman
AD
Thomson
MA
Davis
DA
Haynes
RB
No magic bullets: a systematic review of 102 trials of interventions to improve professional practice
CMAJ
 , 
1995
, vol. 
153
 (pg. 
1423
-
31
)
11
Fretheim
A
Schunemann
HJ
Oxman
AD
Improving the use of research evidence in guideline development: 15. Disseminating and implementing guidelines
Health Res Policy Syst
 , 
2006
, vol. 
4
 pg. 
27
 
12
Lobo
A
Launer
LJ
Fratiglioni
L
, et al.  . 
Prevalence of dementia and major subtypes in Europe: a collaborative study of population-based cohorts. Neurologic Diseases in the Elderly Research Group
Neurology
 , 
2000
, vol. 
54
 (pg. 
S4
-
9
)
13
Waldorff
FB
Rishoj
S
Waldemar
G
Identification and diagnostic evaluation of possible dementia in general practice. A prospective study
Scand J Prim Health Care
 , 
2005
, vol. 
23
 (pg. 
221
-
6
)
14
Renshaw
J
Scurfield
P
Cloke
L
Orrell
M
General practitioners’ views on the early diagnosis of dementia
Br J Gen Pract
 , 
2001
, vol. 
51
 (pg. 
37
-
8
)
15
Waldorff
FB
Steenstrup
AP
Nielsen
B
Rubak
J
Bro
F
Diffusion of an e-learning programme among Danish General Practitioners: a nation-wide prospective survey
BMC Fam Pract
 , 
2008
, vol. 
9
 pg. 
24
 
16
Woodall
WG
Buller
DB
Saba
L
, et al.  . 
Effect of emailed messages on return use of a nutrition education website and subsequent changes in dietary behavior
J Med Internet Res
 , 
2007
, vol. 
9
 pg. 
e27
 
17
Franklin
PD
Rosenbaum
PF
Carey
MP
Roizen
MF
Using sequential e-mail messages to promote health behaviors: evidence of feasibility and reach in a worksite sample
J Med Internet Res
 , 
2006
, vol. 
8
 pg. 
e3
 
18
Abdolrasulnia
M
Collins
BC
Casebeer
L
, et al.  . 
Using email reminders to engage physicians in an Internet-based CME intervention
BMC Med Educ
 , 
2004
, vol. 
4
 pg. 
17
 
19
Filippucci
E
Meenagh
G
Ciapetti
A
, et al.  . 
E-learning in ultrasonography: a web-based approach
Ann Rheum Dis
 , 
2007
, vol. 
66
 (pg. 
962
-
5
)

Author notes

Waldorff FB, Siersma V, Nielsen B, Steenstrup AP and Bro F. The effect of reminder letters on the uptake of an e-learning programme on dementia: a randomized trial in general practice. Family Practice 2009; 26: 466–471.