ABSTRACT

Objective. The prevalence of chronic nonmalignant pain (CNMP), the lack of confidence and reward among trainees and providers caring for patients with CNMP, and the lack of a comprehensive curriculum in pain management prompted the creation of the Virginia Commonwealth University (VCU) Chronic Nonmalignant Pain Management curriculum, an innovative e-learning resource. This article describes the development of the curriculum and presents initial evaluation data.

Design. The curriculum is organized into six modules that cover 20 specific Accreditation Council of Graduate Medical Education competency-based objectives. Broad content and effective instructional design elements promote its utility among a range of learner levels in a variety of medical disciplines.

Results. Twenty-four physician reviewers and over 430 trainees (medical students and graduate medical residents) have evaluated the curriculum. Of the respondents to course evaluation questions, 85.7% (366/427) stated that they would access the practice resources again, 86.3% (366/424) agreed that the treatment of CNMP was more important to them after completing the curriculum, 73.9% (312/422) stated that they would make changes in their behavior or practice, and 92.3% (386/418) stated that they would recommend the curriculum to their colleagues. Qualitative data are uniformly positive. Results of pretest and posttest scores and item analyses have been used to make content changes.

Conclusions. The VCU Chronic Nonmalignant Pain Management curriculum is an e-learning resource that has the potential to fill a significant training void. Design and content changes have been made as a result of initial evaluation data. Data from ongoing evauation will allow curricular refinement.

Introduction

National Perspective

Due to well-described management challenges, residents and practicing physicians indicate a lack of confidence and reward in managing patients with chronic nonmalignant pain (CNMP) [1–3]. Inadequate training has been cited as one reason for providers feeling ill-prepared to manage this problem [4].

Both the Joint Commission and the U.S. Congress, which declared 2001–2011 as the “Decade of Pain Control and Research,” stress the importance of effective pain management [5,6]. There has also been a call for establishing widespread training requirements in pain management [7–10]. Importantly, providers and trainees themselves have indicated a need for relevant training [10–12].

Curricular Efforts by Others

Data on several pain curricula have been published. Each of these curricula has limitations: some target only a specific audience or specialty; some address a selected aspect of chronic pain management, such as prescribing opioid analgesics; and some require significant delivery resources, such as standardized patients [13–19].

E-learning programs are growing in number, content, and educational efficacy [20–22]. However, there are no published descriptions of e-learning programs in chronic pain management. Several pain organizations offer online programs funded by the pharmaceutical industry, but evaluation and efficacy data are lacking. No extant online programs feature delineation of objectives by standard Accreditation Council of Graduate Medical Education (ACGME) competencies or offer tracking of user completion, both particularly relevant for clerkship and program directors.

Institutional Perspective

Consistent with national training deficiencies, our institution, Virginia Commonwealth University (VCU) Medical Center, lacked a comprehensive approach to education about chronic pain management at both the undergraduate and graduate training levels. Lack of knowledge contributed to practice deficiencies, lack of preparation undermined trainee confidence, and lack of awareness and access resulted in underutilization of practice resources. Curricular aims were to address these three broad areas: knowledge, confidence, and access to resources.

In recommended format [23], this article describes the process of curriculum development, the content and elements of instructional design, initial evaluation data, and revisions based on these evaluation data.

Development Process

Needs Assessment

We conducted a needs assessment at VCU Medical Center, which included an internal medicine residents' questionnaire on CNMP, as well as a retrospective chart review in the internal medicine resident continuity clinic [24]. The results of the questionnaire revealed a self-perceived lack of preparation in managing chronic pain, a lack of personal reward when treating patients with chronic pain, and a significant negative impact on residents' view of primary care as a career. The retrospective chart review identified significant practice deficiencies in risk assessment and prescription drug misuse monitoring in patients treated with opioids for CNMP.

Decisions about Scope, Selection of Topics, and Design

The topic of chronic pain management spans disciplines. Several studies have documented improved patient outcomes with a multidisciplinary approach to treatment [25,26]. The content writers for our curriculum included general internists, a palliative care specialist, an addiction specialist with additional training in psychiatry, a pharmacist, and a toxicologist. While not primary content writers, experts from other disciplines including psychiatry, rheumatology, and neurology were consulted as curriculum objectives and content were developed. The curriculum was written over a span of 4 months in 2005 and included the following topics in systematic progression: assessing chronic pain, treating chronic pain with a multidisciplinary approach, reviewing the management of specific pain syndromes, confronting challenges including prescription drug misuse, and reviewing laws and regulations that govern controlled substance prescribing.

Several key references in chronic pain management were germane to the choice of module headings and development of measurable objectives including published resources for developing pain curricula in medical schools, comprehensive pain management text books, and federal documents from the Drug Enforcement Administration and the Federation of State Medical Boards [27–32]. Over 6 months, the objectives, questions, and content were transformed into an e-learning resource through a partnership with VCU School of Medicine's Office of Faculty and Instructional Development.

We chose e-learning for several reasons. First, trainees have indicated satisfaction with and preference for e-learning [33]. Additionally, the curriculum is accessible at any time, regardless of clinical schedules or duty hours, and from anywhere, considering that computers are now ubiquitous. Ease of accessibility in a busy practice setting is especially beneficial with the inclusion of practice resources including pharmacology tables, summary tables, tools, references, and Web links allowing rapid access to information. Online delivery also encourages widespread use within the academic medical center and among affiliated community-based providers. Finally, online delivery allows a potential training resource to be shared with other institutions [34].

Design decisions focused on optimizing the curriculum for use among a range of learner levels in a variety of medical disciplines. The process followed “Ten Steps to Effective Web-Based Learning” and included “Characteristics of Effectively Designed Webpages,” as described by Cook and Dupras [35].

Curriculum Description

Aims

The VCU Chronic Nonmalignant Pain Management curriculum was created to fill a void in training with aims to improve knowledge, improve confidence by better preparing students and residents to manage a challenging problem, and to provide access to practice resources.

Curricular Topics

The curriculum is composed of six modules: Overview and Assessment of Chronic Nonmalignant Pain; Treatment of Chronic Nonmalignant Pain; Common Pain Syndromes: Fibromyalgia; Common Pain Syndromes: Neuropathic Pain; Identifying and Meeting Challenges; and Legal and Regulatory Aspects of Prescribing Controlled Substances. Three to four objectives are housed within each module. Each of the 20 objectives is tied to one of the six standard ACGME competencies, based on consensus from several expert clinician educators. These objectives and associated competencies can be viewed in Table 1.

Table 1

Module headings and specific objectives

 Associated ACGME Competency 
Module 1: Overview and assessment of chronic nonmalignant pain 
Recognize the responsibility of the health care provider in caring for patients with chronic nonmalignant pain Professionalism 
Identify elements of the chronic pain assessment that are essential for developing a patient-centered treatment regimen Patient care 
Describe methods of assessing chronic pain in the elderly, specifically in those with cognitive and/or functional impairment Interpersonal and communication skills 
Describe features and examples of non-nociceptive or maladaptive pain Medical knowledge 
Module 2: Treatment of chronic nonmalignant pain 
Apply a multidisciplinary approach to the management of patients with chronic nonmalignant pain Patient care 
Identify effective nonpharmacologic therapies for chronic nonmalignant pain Medical knowledge 
Describe and apply principles of non-opioid pharmacologic therapy for the treatment of chronic nonmalignant pain Medical knowledge 
Describe and apply principles of opioid use for the treatment of chronic nonmalignant pain Medical knowledge 
Module 3: Common pain syndromes: fibromyalgia 
Recognize the clinical syndrome of fibromyalgia Patient care 
Apply a stepwise approach when managing patients with fibromyalgia Patient care 
Choose specific pharmacologic treatment options for patients with fibromyalgia Medical knowledge 
Module 4: Common pain syndromes: neuropathic pain 
Recognize the major diagnostic features of neuropathic pain as well as common neuropathic pain syndromes Patient care 
Advanced content: review concepts in neuropathic pain pathophysiology Medical knowledge 
Name the evidence-based pharmacologic treatment options for neuropathic pain Medical knowledge 
Recall the management of specific neuropathic pain syndromes including: diabetic polyneuropathy; trigeminal neuralgia; complex regional pain syndrome Medical knowledge 
Module 5: Identifying and meeting challenges 
With an understanding of chronic pain terms, recognize drug-seeking behaviors consistent with either: pseudoaddiction or addiction Patient care 
Recall specific office practice tools that help avert patient misbehaviors and achieve consistency in treatment of chronic nonmalignant pain Systems-based practice 
Identify the use for and limitations of urine drug screening Systems-based practice 
Module 6: Legal and regulatory aspects of prescribing controlled substances 
Distinguish the differences between classifications of medications in the U.S. Controlled Substance scheduling system Medical knowledge 
Define the role of law enforcement authorities regarding the prescribing of controlled substances Systems-based practice 
Recognize and apply the requirements for proper and safe controlled substance prescribing Professionalism 
 Associated ACGME Competency 
Module 1: Overview and assessment of chronic nonmalignant pain 
Recognize the responsibility of the health care provider in caring for patients with chronic nonmalignant pain Professionalism 
Identify elements of the chronic pain assessment that are essential for developing a patient-centered treatment regimen Patient care 
Describe methods of assessing chronic pain in the elderly, specifically in those with cognitive and/or functional impairment Interpersonal and communication skills 
Describe features and examples of non-nociceptive or maladaptive pain Medical knowledge 
Module 2: Treatment of chronic nonmalignant pain 
Apply a multidisciplinary approach to the management of patients with chronic nonmalignant pain Patient care 
Identify effective nonpharmacologic therapies for chronic nonmalignant pain Medical knowledge 
Describe and apply principles of non-opioid pharmacologic therapy for the treatment of chronic nonmalignant pain Medical knowledge 
Describe and apply principles of opioid use for the treatment of chronic nonmalignant pain Medical knowledge 
Module 3: Common pain syndromes: fibromyalgia 
Recognize the clinical syndrome of fibromyalgia Patient care 
Apply a stepwise approach when managing patients with fibromyalgia Patient care 
Choose specific pharmacologic treatment options for patients with fibromyalgia Medical knowledge 
Module 4: Common pain syndromes: neuropathic pain 
Recognize the major diagnostic features of neuropathic pain as well as common neuropathic pain syndromes Patient care 
Advanced content: review concepts in neuropathic pain pathophysiology Medical knowledge 
Name the evidence-based pharmacologic treatment options for neuropathic pain Medical knowledge 
Recall the management of specific neuropathic pain syndromes including: diabetic polyneuropathy; trigeminal neuralgia; complex regional pain syndrome Medical knowledge 
Module 5: Identifying and meeting challenges 
With an understanding of chronic pain terms, recognize drug-seeking behaviors consistent with either: pseudoaddiction or addiction Patient care 
Recall specific office practice tools that help avert patient misbehaviors and achieve consistency in treatment of chronic nonmalignant pain Systems-based practice 
Identify the use for and limitations of urine drug screening Systems-based practice 
Module 6: Legal and regulatory aspects of prescribing controlled substances 
Distinguish the differences between classifications of medications in the U.S. Controlled Substance scheduling system Medical knowledge 
Define the role of law enforcement authorities regarding the prescribing of controlled substances Systems-based practice 
Recognize and apply the requirements for proper and safe controlled substance prescribing Professionalism 

ACGME = Accreditation Council of Graduate Medical Education.

Learning Strategies (See Figures 1–3)

Figure 1

(1) Module titles are clickable navigation tools. (2) Module pretest is completed before proceeding to module content. (2.1) Case-based pretest questions encourage application and problem solving. (2.2, 2.3) Pretest question feedback emphasizes self-assessment. (3) Learners enter a 20-question posttest that applies content from module objectives.

Figure 1

(1) Module titles are clickable navigation tools. (2) Module pretest is completed before proceeding to module content. (2.1) Case-based pretest questions encourage application and problem solving. (2.2, 2.3) Pretest question feedback emphasizes self-assessment. (3) Learners enter a 20-question posttest that applies content from module objectives.

Figure 2

(1, 1.1) Learning objectives are used as navigation tools and are paired with (2, 2.1) Accreditation Council of Graduate Medical Education (ACGME) competencies. (3) Pharm tables resource tab provides access to original tables for use in practice. (4) Tools resource tab provides access to practice tools, as illustrated by this original patient education template.

Figure 2

(1, 1.1) Learning objectives are used as navigation tools and are paired with (2, 2.1) Accreditation Council of Graduate Medical Education (ACGME) competencies. (3) Pharm tables resource tab provides access to original tables for use in practice. (4) Tools resource tab provides access to practice tools, as illustrated by this original patient education template.

Figure 3

(1) Case-based self-assessment questions are interwoven into content. (1.1.) Feedback button provides the learner with immediate feedback. (2–6) Resource tabs allow immediate accessibility of practice resources from anywhere in the curriculum. (5,7,8) Reference links and citations provide evidence-based support for validity of content. (6) Key point summaries are presented at the end of each module and are also accessible from this tab.

Figure 3

(1) Case-based self-assessment questions are interwoven into content. (1.1.) Feedback button provides the learner with immediate feedback. (2–6) Resource tabs allow immediate accessibility of practice resources from anywhere in the curriculum. (5,7,8) Reference links and citations provide evidence-based support for validity of content. (6) Key point summaries are presented at the end of each module and are also accessible from this tab.

Effective Website design enhances principles of active learning. For our curriculum, learners progress through the modules systematically, starting with a case-based module pretest with answer feedback. Learning objectives, paired with standard ACGME competencies, are used as navigation tools, promoting clear delineation of content purpose and emphasis of key points. Case-based self-assessment questions are woven into the text to enhance application of didactic content. Immediate feedback appears in pop-up windows to reinforce learner comprehension [33]. Tabbed resource pages collect and organize practice resources including pharmacology tables, summary tables, and tools, encouraging utilization beyond curriculum completion. Extensive references and Web links to more in-depth information support the authority of the program content and model effective evidence-based practice. These also help to keep text streamlined, letting learners decide when and if to pursue further information. To support varying learner levels from undergraduate students to practicing providers, the curriculum incorporates a tiered approach with optional advanced content. Additionally, the printable practice resources serve as didactic content while also supporting a provider's need for quick clinical references and documentation tools. Key point summaries, available after module completion, may also be viewed under tab headings. Finally, unique posttest questions reflect content from each of the 20 objectives, thus encouraging review and durable synthesis of information. A printable certificate reflects the number of correct posttest responses within each ACGME competency.

Results

Implementation

Because our needs assessment revealed that internal medicine residents were not exposed to coursework in CNMP during their medical school experience, we decided to integrate the curriculum into the ambulatory rotation of the third-year VCU School of Medicine Internal Medicine clerkship. As part of the evaluation phase, the curriculum was also piloted in 13 different graduate medical education (GME) programs in several institutions from December 2006 to June 2007. As of June 30, 2007, 439 trainees (161 medical students and 278 graduate medial residents) had completed the curriculum.

Curriculum Evaluation Strategies

During the 2006–2007 academic year, evaluation of the VCU Chronic Nonmalignant Pain Management curriculum focused on the examination of 1) ease of use of the e-learning design; 2) completeness and accuracy of online course content; 3) initial curriculum impact through qualitative and quantitative assessments; and 4) impact based on quantitative analysis of pretest and posttest scores and item analyses. The underlying assumption was that the initial evaluation data would allow focused curricular revision to positively impact learners' clinical approach to patients with CNMP through enhancement of knowledge, confidence, and utilization of practice resources.

Evaluation data were collected from a reviewer response form that prompted feedback and suggestions on the curriculum's content and design elements, a voluntary Web-based course evaluation form available after the completion of the posttest, pretest and posttest scores, and item analyses of individual pretest and posttest question responses.

Twenty-four knowledgeable physicians from 18 academic institutions were recruited as reviewers through listservs and personal contacts in pain management and medical education. Twelve physician reviewers completed the reviewer response form and 14 completed the course evaluation form. Additionally, 161 third-year medical students and 278 medical residents from 13 different GME specialities and 3 institutions completed the online curriculum (see Table 2).

Table 2

Graduate medical resident completion

Graduate Medical Program Number Percentage (%) 
Internal medicine 116  41.7 
Family medicine  57  20.5 
Psychiatry  28  10.1 
Orthopedics  24   8.6 
Physical medicine and rehabilitation  17   6.1 
Medicine–pediatrics  13   4.7 
Neurology   9   3.2 
Anesthesia   7   2.5 
Geriatrics   2   0.7 
Rheumatology   2   0.7 
Hematology/oncology   1   0.4 
Infectious disease   1   0.4 
Neurosurgery   1   0.4 
Total 278 100.0 
Graduate Medical Program Number Percentage (%) 
Internal medicine 116  41.7 
Family medicine  57  20.5 
Psychiatry  28  10.1 
Orthopedics  24   8.6 
Physical medicine and rehabilitation  17   6.1 
Medicine–pediatrics  13   4.7 
Neurology   9   3.2 
Anesthesia   7   2.5 
Geriatrics   2   0.7 
Rheumatology   2   0.7 
Hematology/oncology   1   0.4 
Infectious disease   1   0.4 
Neurosurgery   1   0.4 
Total 278 100.0 

To evaluate ease of use and completeness and accuracy of course content, physician reviewers and trainees (medical students and graduate medical residents) responded to the following two questions using a 5-point Likert scale: “Rate the ease of use of this curriculum” (1 = very difficult to 5 = very easy) and for each of the six modules “Rate the extent to which the objectives of the curriculum were met” (1 = strongly disagree to 5 = strongly agree). A repeated-measures mixed model assuming an unstructured correlation structure was used to model the mean score across the seven measures separately for each group as the response to the measures given by each person was highly correlated (r ≥ 0.4076). Comments from the reviewer response form were collected and are summarized under the Summarized Evaluation Data section below.

Additionally, comments on ease and use and completeness and accuracy of course content from the reviewer response form were collected and summarized (see “Summarized Evaluation Data”). Physician reviewer, student, and resident responses to four yes/no questions on the course evaluation form were used to provide quantitative data on initial curriculum impact. Fisher exact tests were used for each question to determine if the proportion of “yes” responses was different across the three groups.

  1. “Having completed the curriculum, will you access any of the resources?”

  2. “Is the assessment and treatment of patients with CNMP more important to you as a result of using this curriculum?”

  3. “As a result of the curriculum, will you make any changes in your behavior or practice?”

  4. “Would you recommmend this curriculum to your colleagues?”

We then analyzed student and resident pretest and posttest scores. For the student and resident responses, a repeated-measures analysis assuming an unstructured correlation structure (r = 0.13) was used to determine the following:

  1. Are the pretest to posttest score changes significant for students and residents?

  2. Are the pretest scores different between the two groups?

  3. Are the posttest scores different between the two groups?

  4. Are the pretest to posttest score changes different between the two groups?

Because the physician reviewers, most of whom were likely to be practicing pain management at a high level, were recruited primarily to provide feedback about the curriculum and to assess the accuracy of course content, pretest and posttest score analyses were performed only as pilot testing on this group. Thus, a paired t-test was used to test the changes in the pretest to posttest scores for the physician reviewers.

Item analyses of individual pretest and posttest question responses were analyzed. Difficulty and discrimination were assessed based on all the responses for each pretest and posttest item. Difficulty is defined as the percentage of correct responders (irrespective of total score). Discrimination (D) is defined as the difference in the proportion of correct responders in the upper group (upper 27% based on total score) and the proportion of correct responders in the lower group (lower 27% based on total score) [36]. For our initial item analyses, we sought a difficulty of between 45% and 75% and a discrimination of 0.3 or greater. We categorized questions as “best,” “acceptable,” “review for editing,” or “edit or eliminate” based on the description in Table 3.

Table 3

Categorization of questions by difficulty and discrimination

Category Difficulty* Discrimination* Pretest Questions (20) Posttest Questions (20) All Questions (40) 
Best 45–75% ≥0.3 5 (25%) 6 (30%) 11 (27.5%) 
Acceptable 45–75%  0.15–0.29 2 (10%) 1 (5%) 11 (27.5%) 
76–91% ≥0.3 2 (10%)  
25–44% ≥0.3 2 (10%) 4 (20%) 
Review for editing 45–75% <0.15   8 (20%) 
76–91% <0.29 2 (10%) 6 (30%) 
25–44% <0.29   
Edit or eliminate >24 or <91% Any 7 (35%) 3 (15%) 10 (25%) 
Category Difficulty* Discrimination* Pretest Questions (20) Posttest Questions (20) All Questions (40) 
Best 45–75% ≥0.3 5 (25%) 6 (30%) 11 (27.5%) 
Acceptable 45–75%  0.15–0.29 2 (10%) 1 (5%) 11 (27.5%) 
76–91% ≥0.3 2 (10%)  
25–44% ≥0.3 2 (10%) 4 (20%) 
Review for editing 45–75% <0.15   8 (20%) 
76–91% <0.29 2 (10%) 6 (30%) 
25–44% <0.29   
Edit or eliminate >24 or <91% Any 7 (35%) 3 (15%) 10 (25%) 
*

Difficulty is defined as the percentage of correct responders, and discrimination is defined as the difference in the proportions of correct responders in the upper and lower 27th percentiles.

Each pretest and posttest question was a unique question.

To fall into any category, a question had to meet criteria of both difficulty and discrimination.

Summarized Evaluation Data

Responses from physician reviewers were overwhelmingly positive on open-ended prompts regarding curriculum design and structure. Physician reviewers found the program easy to use––mean (X) = 4.64, standard deviation (SD) = 0.63 on a 5-point Likert scale: 1 = very difficult to 5 = very easy. Student and resident mean scores were lower and similar to each other (X = 3.60, SD = 0.91; X = 3.64, SD = 0.91). Based on the results from the mixed model, the trends in mean rating across the six measures were not significantly different between the groups (P value = 0.1412). However, the physician reviewers rated the accomplishment of each of the curricular objectives much higher than did either the students or the residents (P value = 0.0001).

From the reviewer response form, physician reviewers found the six module topics to be appropriate for training. Five reviewers suggested the addition of a specific musculoskeletal module. All reviewers agreed that the online material adequately covered the stated objectives. Ninety-two percent (11/12) felt the module objectives matched the designated ACGME competency well. One respondent commented that several objectives may correlate with more than one ACGME competency. All respondents affirmed that the curriculum is evidence-based and reported consulting listed references.

The distribution by group of the “yes” responses to four yes/no “course evaluation questions” is summarized in Table 4. Students, residents, and physician reviewers gave similar percentages of “yes” responses to questions 1, 3, and 4 (P values > 0.08). There was a statistically significant difference among the groups (P value = 0.0027) in the percentage of “yes” responses to question 2: “Is the assessment and treatment of patients with CNMP more important to you as a result of using this curriculum?” The percentage reporting “yes” was greatest among students (92.4%), followed by residents (83.2%), and then physician reviewers (61.5%).

Table 4

Responses to course evaluation questions

Question Students (N = 161)* Residents (N = 278)* Reviewers (N = 14)* Fisher P Value 

 
Percent “Yes” (Ratio) 
1. Having completed the curriculum, will you access any of the resources? 89.1% (131/147) 83.2% (222/267) 100.0% (13/13) 0.0873 
2. Is the assessment and treatment of patients with CNMP more important to you as a result of using this curriculum? 92.4% (133/144) 84.3% (225/267)  61.5% (8/13) 0.0027 
3. As a result of the curriculum, will you make any changes in your behavior or practice? 74.7% (106/142) 73.8% (197/267)  69.2% (9/13) 0.8677 
4. Would you recommend this curriculum to your colleagues? 95.7% (134/140) 90.2% (239/265) 100.0% (13/13) 0.1020 
Question Students (N = 161)* Residents (N = 278)* Reviewers (N = 14)* Fisher P Value 

 
Percent “Yes” (Ratio) 
1. Having completed the curriculum, will you access any of the resources? 89.1% (131/147) 83.2% (222/267) 100.0% (13/13) 0.0873 
2. Is the assessment and treatment of patients with CNMP more important to you as a result of using this curriculum? 92.4% (133/144) 84.3% (225/267)  61.5% (8/13) 0.0027 
3. As a result of the curriculum, will you make any changes in your behavior or practice? 74.7% (106/142) 73.8% (197/267)  69.2% (9/13) 0.8677 
4. Would you recommend this curriculum to your colleagues? 95.7% (134/140) 90.2% (239/265) 100.0% (13/13) 0.1020 
*

Sample sizes for specific questions may vary due to missing data.

Indicates statistical significance at α = 0.05.

CNMP = chronic nonmalignant pain.

From the reviewer response form, physician reviewers found the practice resources valuable. Table 4 reveals that the majority of students, residents, and physician reviewers would access the practice resources again. Physician reviewers in particular commented positively on the pharmacology and summary tables as well as tools (pain assessment tools, urine drug screening tables, sample policy documents and consent forms). The patient education tool featuring a multidimensional approach to therapy was also well received. Students, residents, and physician reviewers agreed that as a result of the curriculum, they would make changes in their clinical practice. Selected comments are presented in Table 5.

Table 5

Selected student, resident, and reviewer comments

Student Comments 

 
I will be less hasty in making assumptions about patients and their relationship to their pain. I will be more open to treating and understanding their pain as it relates to them, rather than judging them on their past medical history. 
I will be more vigilant about pain management than I likely would have without the program. 
I will feel more comfortable treating patients with chronic pain and identifying those patients who misuse scheduled medications. I also feel less bias. 
I will follow the guidelines outlined in this curriculum for diagnosing and treating chronic pain. 
It was very helpful in understanding chronic pain. It makes working with patients less frustrating and makes me more understanding of what they are dealing with. I am not sure where I would have gotten this content in my regular curriculum and it is really important. 
All of the links and tables were excellent. The pretest—the whole thing. Excellent. 

 
Resident Comments 

 
Now I am more aware of CNMP; I can treat my patients better without hesitating to give them proper medication. 
I will take chronic pain more seriously and make every attempt to ease patient discomfort. 
I will treat fibromyalgia more as a “real” condition and use a multimodal approach. 
I feel more prepared to address this pathology. Helpful to have new ideas and other avenues with which to treat chronic pain. 
Thank you so much for your involvement in the modules. They are extremely helpful and the embedded documents/tables I am finding to be very valuable. 

 
Reviewer Comments 

 
Thanks very much for sharing your work with us. The Website you've developed is impressive for its comprehensive coverage of chronic pain issues and for the resources you've gathered together in a user-friendly site. 
I just finished reviewing the course and thought it was great. I liked the content, the format, the fact that you could link directly to so many things, the actual items you need to conduct a proper pain management visit were there. Also, the pretest and posttest were excellent. I would certainly endorse this for our residency program. 
I reviewed the curriculum, think it is a great educational resource and look forward to getting my residency program involved. I also think the curriculum does a good job incorporating aspects of the chronic care model. 
We deal with chronic nonmalignant pain all the time and are challenged. I commend you for taking on such a challenge; our residents and fellows will benefit. 
Student Comments 

 
I will be less hasty in making assumptions about patients and their relationship to their pain. I will be more open to treating and understanding their pain as it relates to them, rather than judging them on their past medical history. 
I will be more vigilant about pain management than I likely would have without the program. 
I will feel more comfortable treating patients with chronic pain and identifying those patients who misuse scheduled medications. I also feel less bias. 
I will follow the guidelines outlined in this curriculum for diagnosing and treating chronic pain. 
It was very helpful in understanding chronic pain. It makes working with patients less frustrating and makes me more understanding of what they are dealing with. I am not sure where I would have gotten this content in my regular curriculum and it is really important. 
All of the links and tables were excellent. The pretest—the whole thing. Excellent. 

 
Resident Comments 

 
Now I am more aware of CNMP; I can treat my patients better without hesitating to give them proper medication. 
I will take chronic pain more seriously and make every attempt to ease patient discomfort. 
I will treat fibromyalgia more as a “real” condition and use a multimodal approach. 
I feel more prepared to address this pathology. Helpful to have new ideas and other avenues with which to treat chronic pain. 
Thank you so much for your involvement in the modules. They are extremely helpful and the embedded documents/tables I am finding to be very valuable. 

 
Reviewer Comments 

 
Thanks very much for sharing your work with us. The Website you've developed is impressive for its comprehensive coverage of chronic pain issues and for the resources you've gathered together in a user-friendly site. 
I just finished reviewing the course and thought it was great. I liked the content, the format, the fact that you could link directly to so many things, the actual items you need to conduct a proper pain management visit were there. Also, the pretest and posttest were excellent. I would certainly endorse this for our residency program. 
I reviewed the curriculum, think it is a great educational resource and look forward to getting my residency program involved. I also think the curriculum does a good job incorporating aspects of the chronic care model. 
We deal with chronic nonmalignant pain all the time and are challenged. I commend you for taking on such a challenge; our residents and fellows will benefit. 

CNMP = chronic nonmalignant pain.

Student comments suggest increased awareness of all aspects of chronic pain management and self-reported improved attitudes including being less judgmental, less skeptical, and more empathetic toward patients with chronic pain syndromes. Resident comments suggest improved knowledge of chronic pain assessment and treatment, enhanced appreciation for the complexity of chronic pain syndromes, willingness to expand treatment plans beyond a single therapeutic approach, and increased awareness of monitoring and documentation tools for patients on controlled substance medications. Examples of stated changes from physician reviewers included increased use of non-opioid pain medications for pain syndromes and increased utilization of both medication agreements and urine drug screening.

From the reviewer evaluation form, physician reviewers identified multiple user groups at all levels of training and practice as appropriate target audiences. They also identified multiple specialties and/or disciplines, including non-physician providers who may benefit from its use. All respondents agreed that if they were residency program directors, they would utilize the program for their trainees.

For the reviewers, the mean of the pretest scores was 67.14 (N = 14, SD = 9.35, range 55–85) and 70.00 (N = 13, SD = 11.37, range 50–85) for the mean posttest score, resulting in a nominal but insignificant increase of 2.69 (N = 13, 95% confidence interval = −6.08, SD = 11.47, P value = 0.5165).

On the average, students (N = 161) scored 62.3% (SD = 11.32, range = 10–85) on the pretest and 64.0% (SD = 12.83, range = 25–100) on the posttest. Residents (N = 278) scored 63.8% (SD = 11.73, range 15–95) on the pretest and 61.7% (SD = 12.74, range 25–100) on the posttest. Based on the results from the mixed model, the mean pretest score was not significantly different between the students and residents (P value = 0.1926) nor was the mean posttest score (P value = 0.0724).

While there was a nominal improvement of 1.71 (standard error [SE] = 1.27) pretest to posttest for the students, this increase was not statistically significant (P = 0.1783). The residents, however, showed a significant decrease in scores of 2.07 (SE = 0.97; P = 0.0325).

The difficulty and discrimination for each pretest and posttest item was assessed based on all question responses. These measures are summarized in Table 3: categorization of questions by difficulty and discrimination. Of the 20 pretest questions, 11 questions (55%) were classified as “best” or “acceptable,” 2 questions (10%) were classified as “review for editing,” and 7 questions (35%) were classified as “edit or eliminate.” Of the 20 posttest questions, 11 questions (55%) were classified as “best” or “acceptable,” 6 questions (30%) were classified as “review for editing,” and 3 questions (15%) were classified as “edit or eliminate.”

Discussion

The development of the VCU Chronic Nonmalignant Pain Management curriculum closely followed published recommendations for developing e-learning curricula including the incorporation of changes based on evaluation data [35]. Initial evaluation data have resulted in revisions to support our curricular aims of knowledge acquisition, improved confidence in management, and utilization of practice resources.

The ease of use and course content were highly rated. Of note is the significantly greater physician reviewer rating for each of the curriculum modules. Possible explanations for this difference include the fact that the statistical power was less for this group due to comparatively low numbers, recognition of the importance of the modules based on physician reviewer experience, self-selection of reviewers through willingness to participate in the review, and the desire to provide positive feedback for a new training resource with the potential to fill a training void.

Quantitative data on curriculum impact was uniformly positive and significant. Interestingly, physician reviewers were less likely than students or residents to indicate an increase in importance in the topic of CNMP as a result of completing the curriculum. The most likely explanation for this is that these reviewers were already clinically experienced in pain management and well aware of its importance.

Initial evaluation data did not corroborate an increase in knowledge through pretest to posttest scores, as qualitative information suggested. There are several important reasons why posttest scores did not increase. First, the question writers were not question-writing experts. Item analyses have been helpful in identifying questions needing revision. Second, we consciously chose to create different questions for the pretest and posttest to illustrate numerous case-based scenarios and thus, in theory, maximize application of content. Third, the structure of the curriculum is unique; while the pretests are imbedded in each module, the posttest is completed in entirety after all six modules are finished. While data are not available on time delays, by report, many learners completed the curriculum over a course of several weeks, even as long as 2 months, which may have negatively impacted the posttest scores. Finally, the pretests and posttest are not “high-stakes” tests, meaning that the questions were written to illustrate important practice points rather than to reflect knowledge gains. The writers felt strongly that these principles would challenge the learner and emphasize application and problem solving (representative of higher-level learning) over the memorization of content [37].

Revisions Based on Evaluation Data

Changes in the curriculum have already been made as a result of the initial evaluation phase. These include technical revisions, design changes that incorporated suggestions for improving ease of use, content updates, and question revisions. From the item analyses, approximately 55% of all questions were “best” or “acceptable.” While the original questions met several criteria for effective multiple-choice questions and Web cases, they also lacked some important elements [37,38]. These were identified and questions have since been appropriately modified. Item content was modified or changed for several questions based on physician reviewer comments to remove inaccuracies and/or test its associated objective; question stems were modified to more closely conform to recommendations (complete statement, shorter length, avoidance of absolute or imprecise terms) and question options were modified to approximate the same length, remove implausible distracters, and to follow a logical or consistent order (alphabetical, for example). Because completing the 20-question posttest at the end of the curriculum may have been cumbersome for some learners (thus affecting posttest scores), we have now separated the curriculum posttest (20 items) into discrete module posttests (three to four items).

The second version of the curriculum (demonstration version accessible: http://www.paineducation.vcu.edu) is housed within an original VCU-designed learning management system, allowing program administrator access, the ability to choose specific modules based on the needs of various learner groups (students, postgraduate trainees, practicing physicians, nurses, and/or other ancillary health providers), and the ability to generate reports based on user group. Additionally, a new state-specific module supported by the Virginia Department of Health Professions has been developed (accessible: http://www.dhp.virginia.gov; click “Prescription Monitoring Program”).

Other changes may include the addition of several objectives and/or modules, notably the diagnosis and management of psychiatric comorbidities in chronic pain syndromes, an inpatient pain management module, and a pediatric pain module. Technology suggestions have included the incorporation of a search feature and the potential to download resources to a personal digital assistant.

Further evaluation is in process with a pre- and postcurriculum survey of attitudes in third-year medical students and a pre- and postcurriculum survey of attitudes and self-assessment of clinical skills in graduate medical residents. To assess the effect of the curriculum on knowledge and practice, potential evaluation strategies for the second version include delayed (3–6 months) postcurriculum knowledge assessments, postcurriculum retrospective chart review for documentation of pain assessment and pain management strategies as taught in the curriculum, and qualtitative and quantitative assessment of practice resource usage (tables, tools, references).

Summary

Developing e-learning resources should be considered for medical education topics that span disciplines, apply to various learner levels, and can be enriched by the availability of practice resources. Though this curriculum was created without any grant or pharmaceutical funding, an enormous amount of time was involved in writing content, creating resources, designing technical features, analyzing evaluation data, and integrating improvements. Thus, we hope to share this resource with other institutions to ameliorate the burden of recreating such a comprehensive curriculum.

Results of the initial evaluation of the VCU Chronic Nonmalignant Pain Management curriculum are positive and support curricular aims of improving knowledge, confidence, and access to practice resources. Additional outcome data are being collected and will help refine the curriculum by learner level and discipline.

Acknowledgments

The authors wish to acknowledge the following individuals involved in the writing, design, and evaluation of the VCU Chronic Nonmalignant Pain Management curriculum: Laura A. Morgan, PharmD; Mike F. Weaver, MD; Carl E. Wolf, PhD; Chris L. Stephens, MS; Sonya Ranson, PhD; and Semhar B. Ogbagaber, MA. The authors also wish to acknowledge the physician reviewers for the VCU Chronic Nonmalignant Pain Management curriculum.

References

1
Weinstein
SM
Laux
LF
Thornby
JI
et al
Medical students' attitudes toward pain and the use of opioid analgesics: Implications for changing medical school curriculum
.
South Med J
 
2000
;
93
(
5
):
472
8
.
2
Weinstein
SM
Laux
LF
Thornby
JI
et al
Physicians' attitudes toward pain and the use of opioid analgesics: Results of a survey from the Texas cancer pain initiative
.
South Med J
 
2000
;
93
(
5
):
479
87
.
3
O'Rorke
JE
Chen
I
Genao
I
Panda
M
Cykert
S
.
Physicians' comfort in caring for patients with chronic nonmalignant pain
.
Am J Med Sci
 
2007
;
333
(
2
):
93
100
.
4
Upshur
CC
Luckmann
RS
Savageau
JA
.
Primary care provider concerns about management of chronic pain in community clinic populations
.
J Gen Intern Med
 
2006
;
21
(
6
):
652
5
.
5
Gallagher
RM
.
Physician variability in pain management: Are the JCAHO standards enough
?
Pain Med
 
2003
;
4
(
1
):
1
3
.
6
Lippe
PM
.
The decade of pain control and research
.
Pain Med
 
2000
;
1
(
4
):
286
.
7
Katz
NP
Adams
EH
Benneyan
JC
et al
Foundations of opioid risk management
.
Clin J Pain
 
2007
;
23
(
2
):
103
18
.
8
Chang
HM
.
Educating medical students in pain medicine and palliative care
.
Pain Med
 
2002
;
3
(
3
):
194
5
.
9
Gallagher
RM
.
Pain education and training: Progress or paralysis?
Pain Med
 
2002
;
3
(
3
):
196
7
.
10
Stannard
C
Johnson
M
.
Chronic pain management—Can we do better? An interview-based survey in primary care
.
Curr Med Res Opin
 
2003
;
19
(
8
):
703
6
.
11
Chen
JT
Fagan
MJ
Diaz
JA
Reinert
SE
.
Is treating chronic pain torture? Internal medicine residents' experience with patients with chronic nonmalignant pain
.
Teach Learn Med
 
2007
;
19
(
2
):
101
5
.
12
Nelson
R
.
Physicians need resources on chronic pain tx
.
Internal Medicine News
  1 October
2006
;
39
(
19
):
18
.
13
Wilson
JF
Brockopp
GW
Kryst
S
Steger
H
Witt
WO
.
Medical students' attitudes toward pain before and after a brief course on pain
.
Pain
 
1992
;
50
(
3
):
251
6
.
14
Ury
WA
Arnold
RM
Tulsky
JA
.
Palliative care curriculum development: A model for a content and process-based approach
.
J Palliat Med
 
2002
;
5
(
4
):
539
48
.
15
Watt-Watson
J
Hunter
J
Pennefather
P
et al
An integrated undergraduate pain curriculum, based on IASP curricula, for six health science faculties
.
Pain
 
2004
;
110
(
1–2
):
140
8
.
16
Maclaren
JE
Cohen
LL
.
Teaching behavioral pain management to healthcare professionals: A systematic review of research in training programs
.
J Pain
 
2005
;
6
(
8
):
481
92
.
17
Leila
NM
Pirkko
H
Eeva
P
Eija
K
Reino
P
.
Training medical students to manage a chronic pain patient: Both knowledge and communication skills are needed
.
Eur J Pain
 
2006
;
10
(
2
):
167
70
.
18
Sullivan
MD
Leigh
J
Gaster
B
.
Brief report: Training internists in shared decision making about chronic opioid treatment for noncancer pain
.
J Gen Intern Med
 
2006
;
21
(
4
):
360
2
.
19
Chen
I
Goodman
B
3rd
Galicia-Castillo
M
et al
The EVMS pain education initiative: A multifaceted approach to resident education
.
J Pain
 
2007
;
8
(
2
):
152
60
.
20
Chumley-Jones
HS
Dobbie
A
Alford
CL
.
Web-based learning: Sound educational method or hype? A review of the evaluation literature
.
Acad Med
 
2002
;
77
(
Suppl 10
):
S86
93
.
21
Ruiz
JG
Mintzer
MJ
Leipzig
RM
.
The impact of e-learning in medical education
.
Acad Med
 
2006
;
81
(
3
):
207
12
.
22
Kerfoot
BP
Baker
H
Jackson
TL
et al
A multi-institutional randomized controlled trial of adjuvant web-based teaching to medical students
.
Acad Med
 
2006
;
81
(
3
):
224
30
.
23
Reznich
CB
Anderson
WA
.
A suggested outline for writing curriculum development journal articles: The IDCRD format
.
Teach Learn Med
 
2001
;
13
(
1
):
4
8
.
24
Yanni
LM
Weaver
MF
Johnson
BA
Morgan
LA
Harrington
SE
Ketchum
JM
.
Management of chronic nonmalignant pain: A needs assessment in an Internal Medicine Resident Continuity Clinic
.
J Opioid Manage
 
2008
(in press).
25
Chelminski
PR
Ives
TJ
Felix
KM
et al
A primary care, multi-disciplinary disease management program for opioid-treated patients with chronic non-cancer pain and a high burden of psychiatric comorbidity
.
BMC Health Serv Res
 
2005
;
5
(
1
):
3
.
26
Flor
H
Fydrich
T
Turk
DC
.
Efficacy of multidisciplinary pain treatment centers: A meta-analytic review
.
Pain
 
1992
;
49
(
2
):
221
30
.
27
Pilowsky
I
.
An outline curriculum on pain for medical schools
.
Pain
 
1988
;
33
(
1
):
1
2
.
28
Turner
GH
Weiner
DK
.
Essential components of a medical student curriculum on chronic pain management in older adults: Results of a modified delphi process
.
Pain Med
 
2002
;
3
(
3
):
240
52
.
29
Boswell
MV
Cole
BE
.
Weiner's Pain Management: A Practical Guide for Clinicians
 .
7th
edition.
Philadelphia, PA
:
CRC Press
;
2005
.
30
McCarberg
B
Passik
SD
.
Expert Guide to Pain Management
 .
Philadelphia, PA
:
American College of Physicians
;
2005
.
31
Drug Enforcement Administration, U.S. Department of Justice Last Acts Partnership, Pain and Policies Study Group, University of Wisconsin Medical School
.
Prescription pain medications: Frequently asked questions and answers for health care professionals, and law enforcement personnel
.
J Pain Palliat Care Pharmacother
 
2005
;
19
(
1
):
71
104
.
32
Federation of State Medical Boards of the United States, Inc
.
Model policy for the use of controlled substances for the treatment of pain
.
J Pain Palliat Care Pharmacother
 
2005
;
19
(
2
):
73
8
.
33
Cook
DA
Thompson
WG
Thomas
KG
Thomas
MR
Pankratz
VS
.
Impact of self-assessment questions and learning styles in web-based learning: A randomized, controlled, crossover trial
.
Acad Med
 
2006
;
81
(
3
):
231
8
.
34
Sisson
SD
Rastegar
DA
Rice
TN
Hughes
MT
.
Multicenter implementation of a shared graduate medical education resource
.
Arch Intern Med
 
2007
;
167
(
22
):
2476
80
.
35
Cook
DA
Dupras
DM
.
A practical guide to developing effective web-based learning
.
J Gen Intern Med
 
2004
;
19
(
6
):
698
707
.
36
Wiersma
W
Jurs
SG
.
Educational Measurement and Testing
 .
2nd
edition.
Boston, MA
:
Allyn and Bacon
;
1990
.
37
Collins
J
.
Education techniques for lifelong learning: Writing multiple-choice questions for continuing medical education activities and self-assessment modules
.
Radiographics
 
2006
;
26
(
2
):
543
51
.
38
Kim
S
Phillips
WR
Huntington
J
et al
Medical case teaching on the web
.
Teach Learn Med
 
2007
;
19
(
2
):
106
14
.