Abstract

Objective As part of the Heath Information Technology for Economic and Clinical Health (HITECH) Act, the Office of the National Coordinator for Health Information Technology (ONC) implemented its Workforce Development Program, which included initiatives to train health information technology (HIT) professionals in 12 workforce roles, half of them in community colleges. To achieve this, the ONC tasked five universities with established informatics programs with creating curricular materials that could be used by community colleges. The five universities created 20 components that were made available for downloading from the National Training and Dissemination Center (NTDC) website. This paper describes an evaluation of the curricular materials by its intended audience of educators.

Methods We measured the quantity of downloads from the NTDC site and administered a survey about the curricular materials to its registered users to determine use patterns and user characteristics. The survey was evaluated using mixed methods. Registered users downloaded nearly half a million units or components from the NTDC website. We surveyed these 9835 registered users.

Results 1269 individuals completed all or part of the survey, of whom 339 identified themselves as educators (26.7% of all respondents). This paper addresses the survey responses of educators.

Discussion Successful aspects of the curriculum included its breadth, convenience, hands-on and course planning capabilities. Several areas were identified for potential improvement.

Conclusions The ONC HIT curriculum met its goals for community college programs and will likely continue to be a valuable resource for the larger informatics community in the future.

Background and significance

The Heath Information Technology for Economic and Clinical Health (HITECH) Act addressed barriers to health information technology (HIT) and its potential to improve the quality and safety of healthcare while reducing its costs.1 While most of HITECH was focused on addressing the technical and financial impediments to electronic health record (EHR) adoption, one section of the legislation earmarked funding to deal with another barrier, the need for a well-trained professional workforce to implement the known science and best practices of EHR adoption.

This known workforce need was codified in Section 3016 of HITECH, which the Office of the National Coordinator for Health Information Technology (ONC), provided with US$118 million of funds, implemented as its Workforce Development Program to significantly expand the cadre of professionals needed to develop, implement, and evaluate EHRs being adopted under HITECH. Drawing on available data2 and internal estimates of workforce requirements for increasing HIT adoption, ONC estimated that a workforce of approximately 51 000 HIT professionals would be required to help eligible hospitals and professionals achieve meaningful use of the EHR.

In addition, based on expert opinion, ONC determined that HIT professionals in 12 workforce roles would be required, and grouped them into three categories. The first category would be mobile personnel who move from site to site to assist in implementing EHR systems. They would be followed by more permanent staff who would maintain and support the implemented EHR systems. A third category would consist of clinical and public health informatics experts who would manage, evaluate, educate, and carry out further research and development of these systems. Personnel for half of these HIT workforce roles would be trained in 6-month certificate programs in community colleges, while the other half were to be trained for 1–2 years in university-based programs (see box 1 for details of categories and roles).

Box 1
Categories and roles of health information technology professionals comprising the informatics workforce
  • Category 1: Mobile adoption support roles

  • Implementation support specialist*

  • Practice workflow and information management redesign specialist*

  • Clinician consultant*

  • Implementation manager*

  • Category 2: Permanent staff of healthcare delivery and public health sites

  • Technical/software support staff*

  • Trainer*

  • Clinician/public health leader†

  • Health information management and exchange specialist†

  • Health information privacy and security specialist†

  • Category 3: Healthcare and public health informaticians

  • Research and development scientist†

  • Programmer and software engineer†

  • Health IT sub-specialist†

*Personnel to be trained in community colleges.

†Personnel to be trained in university-based settings.

When launched in 2009, the ONC Workforce Development Program consisted of four specific initiatives to train staff for the various workforce roles:

  1. Community College Consortia (CCC)3—82 Community colleges, grouped into five regional consortia, funded to offer 6-month certificate programs for the first six workforce roles.

  2. Curriculum Development Centers (CDCs)4—Because community colleges did not have suitable curricula, five universities with established informatics programs (Columbia University, Duke University, John Hopkins University, Oregon Health & Science University (OHSU), and the University of Alabama at Birmingham) received awards to develop curricular components that were then to be developed into courses by the community colleges. In addition, OHSU was additionally designated the National Training and Dissemination Center (NTDC), tasked with developing the web site for dissemination of the materials and providing training for community colleges.

  3. Competency examinations5—A single entity was funded to develop, validate, and deploy examinations to test the competencies gained by graduates of the community college programs.

  4. University-based training (UBT)6—Additional training funds were awarded to nine universities for longer-term UBT in the last six workforce roles.

Drawing from its experts, ONC stipulated that the curriculum would consist of 20 ‘components,’ each of which would be roughly equivalent in content and depth to a college course. In addition, a matrix was established and vetted by ONC, external experts, and the leads from the CDCs that stipulated which components should be required or optional for each of the six community college workforce roles7 (box 2).

Box 2
Titles of the 20 components
  1. Introduction to Health Care and Public Health in the U.S.

  2. The Culture of Health Care

  3. Terminology in Health Care and Public Health Settings

  4. Introduction to Information and Computer Science

  5. History of Health Information Technology in the U.S.

  6. Health Management Information Systems

  7. Working with Health IT Systems*

  8. Installation and Maintenance of Health IT Systems*

  9. Networking and Health Information Exchange

  10. Fundamentals of Health Workflow Process Analysis & Redesign

  11. Configuring EHRs*

  12. Quality Improvement

  13. Public Health IT

  14. Special Topics Course on Vendor-Specific Systems

  15. Usability and Human Factors

  16. Professionalism/Customer Service in the Health Environment

  17. Working in Teams

  18. Planning, Management and Leadership for Health IT

  19. Introduction to Project Management

  20. Training and Instructional Design

*Hands-on laboratory courses.

As the curriculum was developed, each of the 20 components was divided into 8–12 units, which consisted of learning objectives, lectures (in PowerPoint (Microsoft, Redmond, WA) as well as Flash-format voice-over-slides (Adobe, San Jose, CA), student exercises, and self-assessment activities. An instructor manual was also included, which provided a detailed description of the material for each component and suggested reading material. Three of the components included ‘lab’ units, providing hands-on laboratory exercises using and configuring an example EHR (the VistA system from the Department of Veterans Affairs). In addition, design of the curricular materials was based on Section 504 guidelines to ensure maximum accessibility by people with disabilities.

All materials developed as part of this work were made available from the NTDC Web site at http://www.onc-ntdc.info/.

The requirements for the CDCs mandated that curricular materials be produced and then undergo two revisions over the 2 years of funding. After the first revision (V.2.0), the curriculum was made available to the general public, distributed under a Creative Commons Attribution-NonCommercial-ShareAlike License, and available for use by any educational program without restrictions. The third and final version of the curriculum was released in Spring 2012.

Curricular materials were developed for an audience of instructors and not HIT students, and intended as a resource for faculty to develop coursework in HIT and clinical informatics. While anyone could download and use the materials, they were intended to be used in a learning environment led by an educator, and not as a standalone self-study course in HIT. The entire V.3 materials were 11.2 gigabytes in size and contained in 18 072 files. As shown in table 1, the 20 components contained 9974 PowerPoint slides and audio lasting over 136 h. Table 1 also provides general data regarding various attributes of the downloadable material.

Table 1

The average file numbers and sizes for the different file types and their attributes from V.3 of the ONC HIT curriculum

File type/attribute Total for 20 components Average per component SD per component Range (minimum–maximum) per component 
Word files 1403 70.1 15.5 (40–93) 
Word file size (MB) 96.7 4.83 3.78 (1.55–17.8) 
PPT files 454 22.7 8.45 (8–39) 
PPT slides 9974 498.7 201.7 (204–862) 
PPT file size (MB) 612.3 30.6 19.3 (9.42–86.5) 
MP3 files 455 22.8 8.38 (8–39) 
MP3 file size (MB) 4620.7 231 79.2 (60.5–362.1) 
MP3 total length (h:min) 136:04 6:48 2:39 (2:06–12:57) 
File type/attribute Total for 20 components Average per component SD per component Range (minimum–maximum) per component 
Word files 1403 70.1 15.5 (40–93) 
Word file size (MB) 96.7 4.83 3.78 (1.55–17.8) 
PPT files 454 22.7 8.45 (8–39) 
PPT slides 9974 498.7 201.7 (204–862) 
PPT file size (MB) 612.3 30.6 19.3 (9.42–86.5) 
MP3 files 455 22.8 8.38 (8–39) 
MP3 file size (MB) 4620.7 231 79.2 (60.5–362.1) 
MP3 total length (h:min) 136:04 6:48 2:39 (2:06–12:57) 

In total, the 20 components contain 454 PowerPoint files and 9974 PowerPoint slides, with an average of 498.7 slides per component, with a SD of 201.7 and ranging from 204 to 862

HIT, health information technology; ONC, Office of the National Coordinator for Health Information Technology.

To enable easier access and use of the materials from the NTDC website, a search engine was created for the text-based files, in which 38 181 unique words were indexed. A manually constructed topical index was also made available. Once opened to the public, access to the materials on the NTDC site rapidly escalated. At the time that the user survey featured in this paper was conducted, a total of 9835 individuals from around the globe had created logins to download from the NTDC site.

The CDCs encountered a number of challenges in developing the curricular materials, which were compounded by tight timelines and regulatory oversight. A prominent challenge was the additional work required because of our commitment to accommodate students with visual and hearing disabilities and adhere to Section 504 guidelines.8 For the hard of hearing, we included transcripts of all audio materials. Visual materials (eg, slides) were designed to be accessed by electronic screen readers which read aloud a description of the images. To ensure this was possible, every photograph, chart, text box, and smart art image in the materials was ‘tagged’ with a full description of the image so a person with a visual disability could essentially have the same understanding of the image as someone without vision impairment. All images were selected from those that were in the public domain, in order to comply with intellectual property and copyright concerns.

Another challenge was for the experienced faculty developers to produce materials for other educators to teach, rather than to deploy the courses themselves. This was made more difficult by the relatively new notion of the workforce roles. While all of the CDC leaders were experienced educators in biomedical informatics, they were less certain of the backgrounds and workplace competencies of the students who would be enrolling in the community college programs. To overcome the unique challenges associated with this project, university-based informatics educators worked in collaboration with community college educators to iteratively develop curricular material. A final challenge was accommodating the short time frame for developing the materials, that is, an initial version and two revisions due over 2 years.

As part of our ongoing evaluation of the use and usefulness of the CDC developed materials, and in our quest to determine options for sustainability, we conducted a survey of NTDC users. Aside from a greater understanding of who was downloading materials, we were interested in how the materials were being used, where they were being used, and in what fashion they were being deployed. For example, because the materials were developed as components for use by other educators to convert into official courses, we were interested in determining the extent of modification, the degree of use, and user perceptions (students and faculty).

Objective

The goal of the conducted evaluation study is to assess the use and usefulness of the curricular materials, with a focus on their use by the intended audience, namely educators. We describe both the quantity of downloads as well an analysis of the responses of our target audience of educators. This is followed by a discussion of the lessons learned from our experience and how this work can be best leveraged now and in the future.

Methods

Users obtained curricular material from the NTDC website. At the time of their initial visit, users were required to create an account and register with their email addresses prior to downloading content. Community college curriculum educators specifically self-identified themselves during the registration process. The NTDC also recorded details of the number of downloaded files, by user and by curriculum component. Download totals, account data, self-reported information about user workforce roles (administrator, IT support, instructor, learning management software (LMS) coordinator, or other) as well as the type of institution they were affiliated with (community college, graduate and undergraduate university, supplemental education organization or other educational institution) were collected, tabulated, and shared with the ONC.

A survey was developed for administration to all registered users of the NTDC site, including 556 CCC faculty and 9279 other users. The elements and branching logic of the survey are provided in the online supplementary appendix. Survey questions were iteratively reviewed by the authors asynchronously and edited to improve the question set and the initial question set was assessed by team members who had not previously been involved in the process of initial question creation. After approval by the OHSU Institutional Review Board, the survey was administered in SurveyMonkey (Palo Alto, California) and required respondents to indicate whether they were an educator, a student, or neither. The survey was open online for a period of 1 month, and reminder emails to users who had not yet participated were sent during week 2 and 3. We focused our results analysis on those who self-identified as educators (n=339), since they were the primary audience for the curriculum.

In the survey, specific aspects of the curricular materials were evaluated by asking respondents Likert-style as well as open-ended questions. As noted earlier, the survey elements can be viewed in the online supplementary appendix, and we only report the results obtained from the filtered group of 339 self-identified educators in this paper. Descriptive statistics were computed to determine respondent characteristics, to assess the educational level of the intended student audience, to determine the nature of their course offerings, to gauge the degree and nature of modification and customization of NTDC-provided content, and to obtain their level of satisfaction with curricular materials.

Using grounded theory and an iterative approach, a subset of the authors (led by VM) qualitatively reviewed open-ended survey results for repetitions and utilized a constant comparison method9 to search for similarities and contrasts by making systematic comparisons across responses. We then sorted identified individual expressions into themes by conducting a card sort, arranging them into piles at different levels of data abstraction,10 and considering each card in relation to other cards. All steps were carefully documented, and collected data were stored in an easily retrievable format online for analysis. Researchers who analyzed data engaged in self-reflection to promote the validity of the study,11 and once themes were identified, data were again triangulated by comparing data to ensure accuracy.

Results

Data downloads

The NTDC website allowed users to download curricular material for each of the 20 components that were developed by the CDCs. Users could choose to download an entire component en masse or to download individual units within a single component. Additional documentation such as instructor manuals, blueprints, installation guidelines for laboratory components, and change logs between versions were also available for download separately. A total of 493 237 files have been downloaded from the NTDC website to date: 26 458 by users from CCC and 466 779 files by those who identified themselves as public users (general members of the public who accessed the NTDC site with the intention of reviewing or downloading curricular material).

Educator survey

Of the 9835 individuals who were polled, 1269 (12.9%) completed all or part of the survey. We identified 556 individuals as community college faculty users, of whom 99 (17.8%) responded. Among public user respondents, 240 of the 1170 (20.5%) identified themselves as educators. Thus, a total of 339 respondents identified themselves as educators.

Many of the educators who responded to our survey were experienced teachers. A total of 215 respondents (63.4%) had been teaching for more than 5 years, with 65 (19.2%) reporting 2–5 years of teaching experience and only 47 (13.9%) having less than 2 years of teaching experience.

These educators became aware of the existence of materials generated by the ONC curriculum grant in a variety of ways. Of the 307 respondents who answered this question, 91 (29.6%) knew of the material developed by ONC CDCs since they were members of the CCC that used the content. A total of 104 educators (33.9%) reported they had first heard about the course materials by reading about them online, while 87 (28.3%) heard about the course materials from colleagues. A smaller percentage of educators (seven respondents, 2.3%) read about the materials in a print magazine or journal, and 45 (14.7%) heard about the course materials from other sources, including professional associations such as the American Medical Informatics Association (AMIA) or the Health Information and Management Systems Society (HIMSS). Some learned about the ONC components from faculty involved in curriculum creation, or even randomly while on the internet; as one educator frankly noted: ‘I stumbled into it.’

Over half of the 332 educators who reported they had downloaded course materials (180 respondents, 54.2%) planned to use the materials to teach a course. Respondents indicated that they would be using curricular material to meet a plethora of educational aspirations, including delivering courses for college-level as well as professional undergraduate and post-graduate programs, for on the job and hospital-based training programs, for HIT certificate programs, and even to educate students in vocational schools and high schools, and government personnel working in the HIT community as trainers (figure 1).

Figure 1

Intended audience for Office of the National Coordinator for Health Information Technology curricular materials (176 respondents answered the question “At what educational level do you plan to teach your course?”).

Figure 1

Intended audience for Office of the National Coordinator for Health Information Technology curricular materials (176 respondents answered the question “At what educational level do you plan to teach your course?”).

In our survey, we asked educators how they planned to deliver their course materials. Of the 174 educators who responded to this question, only a minority (32 respondents, 18.4%) expressed a desire to offer a purely in-class course. Seventy-five educators (43.1%) reported that they deployed hybrid courses using a combination of in-class and online methods, while 67 (38.5%) reported that they planned to deliver course material entirely online. Most (144 of the 178 respondents to this question) used a learning management system (LMS) to manage online course material and student experiences.

We asked educators if they modified the materials they downloaded. Of the 111 respondents to this question, only a very small percentage of educators (five, 4.5%) reported that they offered course materials to students without making any changes at all. Educators reported adding and deleting component materials, reorganizing lecture content, and adding additional visual elements to them. They reported that they frequently combined materials from different lectures into one lecture, and created or modified additional learning content. Table 2 details some of the customizations implemented by educators, and the frequency of these modifications. In addition to reorganizing materials, educators tailored content and course objectives to best suit their own programs and the environment of the learner, and often fine-tuned the level of relevancy and detail of material to their audience.

Table 2

Educator responses regarding customization of downloaded content

Modification Number and percentage of respondents who modified downloaded curricular course material (n=111, response percentage in parentheses) (%) 
Did not make any changes 5 (4.5%) 
Added materials 63 (56.8%) 
Deleted materials 61 (55.0%) 
Added visuals or other graphic elements 36 (32.4%) 
Combined materials from different lectures into a single lecture 64 (57.7%) 
Reorganized material within a lecture 46 (41.4%) 
Changed the template for the lecture 23 (20.7%) 
Changed or modified the quiz questions 55 (49.5%) 
Modification Number and percentage of respondents who modified downloaded curricular course material (n=111, response percentage in parentheses) (%) 
Did not make any changes 5 (4.5%) 
Added materials 63 (56.8%) 
Deleted materials 61 (55.0%) 
Added visuals or other graphic elements 36 (32.4%) 
Combined materials from different lectures into a single lecture 64 (57.7%) 
Reorganized material within a lecture 46 (41.4%) 
Changed the template for the lecture 23 (20.7%) 
Changed or modified the quiz questions 55 (49.5%) 

111 Respondents answered the question. Respondents could select more than one choice.

When asked about the level of usefulness of downloaded materials, of the 297 educators who answered the question, 152 (51.2%) found the materials ‘extremely useful,’ 124 (41.7%) found them ‘somewhat useful,’ and only 21 (7.1%) found them ‘not very useful’ or ‘not useful at all.’ Figure 2 details the elements of curricular components that were perceived to be valuable.

Figure 2

Elements of curricular components that were felt to be useful by educators. Respondents could select more than one option. The number of respondents is given in parentheses; 286 respondents answered the question.

Figure 2

Elements of curricular components that were felt to be useful by educators. Respondents could select more than one option. The number of respondents is given in parentheses; 286 respondents answered the question.

Successful aspects of curricular material

Breadth

One primary theme that emerged from data analysis was that educators felt that the broad scope of topics in curricular material added to its utility. One educator noted that the ‘historical perspective on medical and IT developments, an[d] understanding of HIT from the perspective of government and academia’ was particularly effective.

Convenience

As discussed earlier, the components were offered in different formats such as PowerPoint and Flash, written transcripts of lectures, MP3 files for audio only, and the occasional video excerpts or demonstrations. This diversity of content delivery was another theme identified during the card sort that educators perceived to be helpful and convenient. ‘Students enjoyed having options with the learning materials,’ one educator noted. The audio-only versions of lectures were felt to be particularly advantageous to students. ‘Many students love the audio recordings, and download them to iPods to study’ one educator responded, while another felt that ‘audio lectures were convenient for students, particularly when driving,’ a reflection of the fast pace of today's students, the need for alternative content delivery modalities, and their ability to literally learn on the go.

Laboratory components

Analysis of narrative comments revealed many educators' comments on the utility of adding practical, hands-on laboratory components to the curriculum. While these laboratories used the Veterans Administration's VistA CPRS (Computerized Patient Record System) to fulfill educational goals, this content was not meant to train students specifically on VistA, but rather to generically demonstrate the use and configuration of EHRs. ‘Labs using VistA were very useful,’ one educator noted, and others echoed this sentiment. One educator commented that the laboratories might have been improved if the designers had chosen to ‘use more th[a]n one vendor.’

Assisting course planning

Some educators felt that the availability of ready-to-use, complete units was reassuring, especially since the comprehensive content and materials available to them could help fill in gaps in their own expertise. One educator noted that the curricular material ‘…provided, in some cases, very useful starting points for areas I have less experti[s]e in.’ Another found ‘…the audio transcripts and the PowerPoints very useful for developing study guides and practice test questions.’ However, it should be noted that only 61 educators (of the 176 who answered this question) planned to use downloaded materials in an ‘as is' fashion. A majority (116 educators, 65.3%) planned to modify materials to best suit their courses.

Improving the curriculum

Obsolescence and updates

The first iteration of the complete curricular materials was completed by October 2010. Two revised and improved versions were released in June 2011 and May 2012 via the NTDC website. However, given the rapidly changing world of health IT, materials such as these run the risk of quickly becoming obsolete. Despite the efforts of curriculum developers to provide up-to-date material at each iteration, the rate of change in the field was at a pace hard to match with static materials. This was particularly true of links to websites on the internet: ‘links are quite often old or ‘broken,’’ one educator noted, and another suggested that ‘…the handouts need to be updated from time to time.’ One educator suggested bringing the content ‘up to date with today's technology.’ The need to regularly update the course curriculum was emphasized, and 89% of educators (276 of the 310 educators who responded to this question) indicated that they would be interested in downloading future versions of components with updated content if and when they became available.

Certification value

The process of developing the certification exam was occurring concurrently with the development of the curriculum components, making coordination difficult. This factor impacted user opinion and emerged in the survey results. One educator felt that ‘repetition between components' was an issue which could affect students' preparation for the HIT competency evaluation examination. Analysis of narrative comments revealed that educators did feel that the modifications made to curricular materials in subsequent versions (which were released as versions 2 and 3) were useful: ‘[t]his reorganization was done with a combination of V2+V3 material and resulted in an above average gradepoint and HITPro pass rate for students using the reorganized material,’ one respondent noted.

Packaging and finding content

One theme that emerged from the card sort analysis was that some educators felt that the packaging of content could be improved. ‘The material is heavy and slow to download and difficult to browse to find pieces of interest,’ one educator noted. Finding relevant topics within components was also perceived to be cumbersome at times; one educator wished there was a ‘better search mechanism on [the] website…’ and went on to note that ‘… information in more than one component may fit a particular lecture in my course and [I] would like to pull the information by subject matter that slightly differs from the way it is organized by NTDC.’

Lectures and quizzes

The CDCs opted to use professional voice talent to record lectures, primarily in order to standardize quality and enhance clarity and consistency, thus offering a uniform experience to students. Paradoxically, this decision was perceived by some educators to detract from the learning experience. One educator described it as: ‘I understand that accents and differences in technological capabilities made it difficult to have the original grantees do the recordings, but the professional voices that were used were SO dry…’ Often times, a highlight of lectures delivered by the same educators who wrote the content is that the enthusiasm and subject matter expertise of the person delivering the lecture carry through. One unintended consequence of using professional narration was that this delivery method resulted in considerable loss of instructor inflections, accents, and enthusiasm, which might have detracted from the overall reception of the lectures by students. Another area of comment related to the self-testing tools that were packaged with each learning unit. ‘I wish the quizzes were more consistent,’ one educator noted; another felt that the quizzes could be improved since ‘some were too easy, some did not always follow the objectives.’ Another educator felt quiz questions were at some times ambiguous for students who were more often expecting only one answer as correct. This reflects some of the intrinsic difficulties associated with ‘leveling’ or writing self-assessment questions for students with varying backgrounds and levels of familiarity with health IT related topics.

There was also a perception that lectures were not visually stimulating, and, in the words of one educator, ‘make no visual impact.’ Given the copyright concerns, Section 504 compliance issues, and time constraints in developing the materials, the use of images, color, and animation was stunted. In addition, component developers were operating under the belief that CCC educators and others would simply use the content and overlay backgrounds on existing slides, or choose to print slides in lower-cost black and white format. This illustrates that some of the faculty who downloaded the components may have been using them without any modifications, retaining the look and feel of rather sterile, but safe, curriculum components.

Broadening curricular perspectives

More than one educator felt that a broader perspective would be beneficial. ‘Students provided feedback that some lectures were very physician-centric and could be more inclusive of additional provider types, including nurses and public health professionals,’ one educator said, and another noted that ‘… inclusion of materials from the industry's perspective (end users, vendors, journalists)’ would improve the curriculum. One educator noted that the ‘material needs to be less ‘academic’ and theoretical and more practical and based on industry information from HIMSS [Healthcare Information and Management Systems Society], AHIMA [the American Health Information Management Association], etc.’

Making learning fun

Educators noted that course material related to HIT, by its very nature, could be quite technical, and at times uninteresting. ‘The material as presented is very dry—I'm not sure how it could be made more interesting,’ one respondent noted; another candidly pointed out that ‘some of the lectures were quite dry and quite frankly, a little boring.’ This could reflect the fact that while online lectures offer greater accessibility to a larger audience in comparison to an in-person experience, educators are unable to respond to students' unique needs for clarification, customizing for the audience or allaying boredom by changing the tempo and interactivity level of the lecture on the fly. Another educator provided a possible solution, suggesting that curriculum developers ‘incorporate more ‘fun’ learning materials such as ONC's CyberSecure Games12 to create interest and motivation for learning.’

Box 3 lists the major themes that emerged during the analysis of this survey.

Box 3
Summary of the challenges associated with developing a large-volume health information technology curriculum with a wide range of topics for a disparate audience
  • Challenges associated with determining the level of complexity of courses

    • Challenges posed by students with varying degrees of familiarity with content

    • Challenges posed by appropriately calibrating the degree of difficulty of courses

  • Challenges associated with hands-on access to electronic health records (EHRs)

    • Challenges involved in enhancing the curriculum by allowing student access to an EHR

    • Challenges due to lack of involvement by commercial proprietary EHR vendors in the education program

  • Challenges associated with technical issues

    • Challenges due to duplication of materials and redundancy of concepts across components

    • Challenges in reducing the variability of technical aspects across lectures and ancillary materials

    • Challenges in maintaining the interest levels of students for health information technology-related topics

Discussion

The purpose of this evaluation study was to determine use patterns, usefulness, and areas for improvement of the CDC curricular materials, with a specific focus on respondents who self-identified as educators. While the response rate was lower than we would have wished, valuable insights emerged—some were expected and others unanticipated. Of particular interest were the attitudes of respondents to the balance between Section 504 compliance and a perception of lifeless materials, methods for maintaining currency, the value of the materials as a support mechanism for inexperienced faculty, the pros and cons of file size and accessibility, and the inherent difficulties of leveling materials for widely divergent students. By unearthing these perceptions, we see opportunity for continued development and enhancement, particularly as one considers the 89% of respondents who expressed a desire for updated versions of the materials. Three additional themes are discussed below that have bearing on the interpretations of the findings, and plans for continued growth of this valuable and federally funded resource.

Determining the level of course complexity

One characteristic worthy of deeper discussion is in the development of health IT courses for students who exhibit variable degrees of familiarity with both the healthcare and/or the IT environment. This often leads to issues associated with calibrating the level of the course, and instructors may find that the level of the curriculum may not perfectly match the learning needs of all students. Because this effort was undertaken as part of the workforce development aspect of the American Recovery and Reinvestment Act of 2009 (ARRA), students with vastly different backgrounds were interested in taking the courses. This phenomenon was reflected in educator evaluations; while one educator noted, ‘the coverage seem[s] to be too superficial and provides too little introductory context for my students,’ another felt that ‘the curriculum was advanced, 90% of the students needed basic course development.’ And while some educators wished for more ancillary material, case examples, and comprehensive references, another wanted to ‘decrease the numerous study citations that impede learning the actual material, and limit examples.’ Concerns with ensuring the ideal level of course difficulty is not uncommon in informatics courses, where the issue might be mitigated by requiring prerequisite courses that address basic concepts, or by incorporating additional adaptive content geared specifically towards advanced learners. However, in the case of the ONC curricular material, the differences in educator perceptions of course content and learner proficiency require a much greater effort than building a one-size-fits-all curriculum. It requires skilled implementation of the curricular components, customized to student characteristics. Similar to the tenets of implementation science, multi-methods and adaptive approaches are required to extract the most value from the NTDC resource.

Integration with an EHR

Educators believed that the HIT training curriculum was enhanced by allowing student access to an EHR, which would enable them to learn about EHR installation by working with the VA's VistA CPRS. One educator noted that ‘integration with VISTA (sic) is especially helpful.’ Emphasizing a laboratory component allows students to gain hands-on exposure to the EHR environment, which is an important skill in the health IT workforce. Although it was not possible to include experience with proprietary and commercially marketed systems as part of this federally funded program, the experience with VistA was perceived to be of high value. Future efforts may benefit by including other systems which could demonstrate aspects of informatics and HIT that the VistA system does not easily support.

Technical issues and solutions

Duplication of some material was inevitable given the breadth of topics covered in the 20 components that were produced by the five CDCs. One educator noted that ‘the material was repetitive in some of the components,’ a sentiment that was shared by others. While we developed a search engine that educators could access from the NTDC website that allowed educators to efficiently find topics across components, many educators were not aware that it was available and did not use it. While eliminating redundancy would certainly have reduced the size of individual components, including various perspectives on the same topics permits the user more choice in what materials to use, allowing them to ‘mix and match’ content to best suit their audiences.

Despite attempts to standardize the technical aspects of lectures and ancillary materials across components, some variability was noted by educators. Close attention to quality and content consistency, with an emphasis on aspects such as audio and video clarity, and copy editing audio transcripts can improve the technical quality of similar future offerings.

Educators felt that some health IT topics were ‘dry,’ resulting in decreased student interest. Health IT topics often tend to be technical, which exacerbates disinterest. While the intrinsic nature of the content to some degree determines the interest level that students are able to generate, topics can be made more attractive by incorporating innovative educational tools, increasing interactivity, and emphasizing the relevancy of health IT use, perhaps by providing, as one educator noted, ‘real examples from the events in the world (past, historic, current)’ or ‘more scenario based application questions,’ as suggested by another educator.

Limitations

The survey was administered to all registered users of the NTDC website, who self-identified themselves as educators or other users. Despite the initial email invitation and two subsequent reminders, the response rate was only 12.9% (17.8% for educators). While some studies have suggested that response rates for email and web-based surveys may not be as robust as terrestrial mail surveys,13,14 online surveys are fast supplanting more traditional questionnaires. The typical response rates for mail-based surveys is in the region of 55%,15 and some authors have claimed even higher effective response rates.16 However, response rates for email surveys can be highly variable. Respondents may change email addresses and internet service providers between the time they register on the website and receive the online survey, respondents may have multiple email addresses that they do not check with equal frequencies, and some users may enter invalid email addresses in order to gain entry to the website (the NTDC registration protocol did not verify the authenticity of emails entered by registrants). Additionally, Krosnick13 also points out that response rates for surveys in general in the USA have been falling since the 1950s.

A limitation of our survey was the relative lack of validation built into the methodology. We also acknowledge the potential influence of response bias in our results—respondents critical of the curricular materials or the program may be more likely to respond than those whose experience with the content was favorable, or participants from funded CCC sites be subject to a social desirability bias. We attempted to minimize this by using neutral phrasing, avoiding leading questions, and allowing for a robust free text element within the survey to elicit as many qualitative comments as possible. Of course, a non-response bias is a distinct possibility here as well. However, we believe that the non-response rate may be more related to the non-invested nature of the largest majority of users as discussed earlier. Regardless, the response rate remains low, and is a key limitation of this study.

Conclusions

The ONC HIT curriculum has been a valuable resource not only for its initial intended audience of community college faculty but also for other educators worldwide. Despite a number of challenges for the curriculum developers, the materials were generally well received and adapted for a variety of types of education. This project also, however, illustrates the complexity associated with developing HIT curricula intended for students with diverse needs and learning goals, and the difficulty involved in creating highly technical content that is amenable for use by educators with diverse agendas, experience levels, and teaching styles. Embarking on such an initiative must be approached from an implementation science perspective, abandoning the notion that one size can fit all, understanding that a mix of models and approaches is necessary, and that what may work in one college or class will not necessarily transfer successfully to another. In short, this necessitates that these materials are used and adapted by skilled educators, and implemented in ways that are expedient, but congruent with academic workflow. These materials were never intended to be used directly ‘out of the box.’

Now that the HITECH funding for this project is ending, further maintenance and updating of the project is undetermined. Sustainability is a key factor in curricula involving HIT, since the topology of the field changes so rapidly. Without a long-term plan to curate, update, and maintain curricular material that has been developed by the CDCs, there is a risk of rapid obsolescence, with the consequence that subsequent cohorts of students will learn from outdated course materials or that the materials will fade away. Thus, there is some urgency to ensure continued support for the curriculum developed during this project. Regardless, even if there is no further central development of these materials, their development and dissemination will have greatly advanced the baseline of educational materials available for scaling-up the HIT workforce. The work of the CDCs has already disseminated widely, generating the development and distribution of materials and workforce required to support our rapidly transforming digital environment.

Acknowledgements

We first and foremost acknowledge Charles Friedman, PhD, who in the role of the Office of the National Coordinator for Health Information Technology (ONC) Chief Science Officer conceptualized and implemented the ONC Workforce Development Program, including the Curriculum Development Centers (CDCs) initiative. We also acknowledge Michelle Murray, who served as Project Officer through development of the first three versions of the curriculum, and Chitra Mohla and Cinyon Reid, who served as project officers later in the project. In addition, we acknowledge the key staff of the CDCs from Oregon Health & Science University (Chris Weldon, Nathan Skidmore), Columbia University (Syncia Sabain), Duke University (Serae Thomas, Vivian West), Johns Hopkins University (Tricia Francis, Michael Vaughn), and the University of Alabama at Birmingham (Amanda Dorsey, Meg Bruck, Kay Clements, and Beth Elias). All of the CDCs also express great gratitude to the many subject matter experts and instructional designers who contributed to the curriculum while working with their centers.

Contributors

VM designed the data collection tools, monitored data collection, analyzed data, and drafted and revised the paper. He is guarantor. PA and ESB assisted with data analysis and revised the draft paper. SA and CD assisted with designing data collection tools, monitored data collection, and analyzed data. WEH and RK revised the draft paper. WH assisted with designing data collection tools, analyzed data, and revised the draft paper.

Funding

This work was funded by the Office of the National Coordinator for Health Information Technology (ONC) via cooperative agreements 1U24OC000015 (OHSU), 1U24OC000023 (UAB), 1U24OC000013 (JHU), 1U24OC000003 (Columbia), and 1U24OC000024 (Duke).

Competing interests

None.

Provenance and peer review

Not commissioned; externally peer reviewed.

References

1
Blumenthal
D
.
Launching HITECH
.
N Engl J Med
 
2010
;
362
:
382
5
.
2
Hersh
W
Wright
A
.
What workforce is needed to implement the health information technology agenda? An analysis from the HIMSS Analytics™ Database
.
AMIA Annual Symposium Proceedings
 ;
2008
;
Washington, DC
:
American Medical Informatics Association
.
3
HealthIT.gov.
Workforce Development Program: Community College Consortia
 . http://www.healthit.gov/policy-researchers-implementers/community-college-consortia (accessed 24 Jun 2013).
4
HealthIT Workforce Development Program: Curriculum Development Centers
. http://www.healthit.gov/policy-researchers-implementers/curriculum-development-centers (accessed 24 Jun 2013).
5
HealthIT Workforce Development Program: Competency Examination Program
. http://www.healthit.gov/policy-researchers-implementers/competency-examination-program (accessed 24 Jun 2013).
6
HealthIT Workforce Development Program: Program of Assistance for University-Based Training
. http://www.healthit.gov/policy-researchers-implementers/program-assistance-university-based-training (accessed 24 Jun 2013).
7
HealthIT.gov
.
Matrix of HIT curriculum components by role
 . http://www.healthit.gov/sites/default/files/matrix_of_curriculum_components_by_6_month_role_as_of_06_30_10.pdf (accessed 24 Jun 2013).
8
U.S. Department of Health & Human Services Office for Civil Rights.
Discrimination on the basis of disability
 . http://www.hhs.gov/ocr/civilrights/understanding/disability/index.html (accessed 24 Jun 2013).
9
Glaser
BG
Straus
A
.
The discovery of grounded theory: strategies for qualitative research
 .
New York
:
Aldine
,
1967
:
101
16
.
10
Ryan
GW
Bernard
HR
.
Techniques to identify themes
.
Field Methods
 
2003
;
15
:
86
109
.
11
Patton
MQ
.
Qualitative evaluation and research methods
 . Newbury Park, CA:
Sage Publications
,
1990
.
12
HealthIT.gov
.
Privacy & Security Training Games
 . http://www.healthit.gov/providers-professionals/privacy-security-training-games (accessed 24 Jun 2013).
13
Krosnick
J
.
Survey research
.
Annu Rev Psychol
 
1999
;
50
:
537
67
.
14
Weible
R
Wallace
J
.
The impact of the internet on data collection
.
Market Research
 
1998
;
10
:
19
23
.
15
Baruch
Y
.
Response rates in academic studies—a comparative analysis
.
Human Relations
 
1999
;
52
:
421
34
.
16
Dillman
D
.
Mail and internet surveys: the total design method
 .
2nd ed.
New York
:
Wiley
,
2000
.

Comments

0 Comments