Abstract

Today’s molecular life scientists are well educated in the emerging experimental tools of their trade, but when it comes to training on the myriad of resources and tools for dealing with biological data, a less ideal situation emerges. Often bioinformatics users receive no formal training on how to make the most of the bioinformatics resources and tools available in the public domain. The European Bioinformatics Institute, which is part of the European Molecular Biology Laboratory (EMBL-EBI), holds the world’s most comprehensive collection of molecular data, and training the research community to exploit this information is embedded in the EBI’s mission. We have evaluated eLearning, in parallel with face-to-face courses, as a means of training users of our data resources and tools. We anticipate that eLearning will become an increasingly important vehicle for delivering training to our growing user base, so we have undertaken an extensive review of Learning Content Management Systems (LCMSs). Here, we describe the process that we used, which considered the requirements of trainees, trainers and systems administrators, as well as taking into account our organizational values and needs. This review describes the literature survey, user discussions and scripted platform testing that we performed to narrow down our choice of platform from 36 to a single platform. We hope that it will serve as guidance for others who are seeking to incorporate eLearning into their bioinformatics training programmes.

INTRODUCTION

The huge amount of data generated by new technologies such as next-generation sequencing requires that today’s life scientists not only keep up with their field of study, but also with the rapidly evolving tools and databases used to collect, store and analyse biological data. While the classical academic career path for molecular life scientists provides a thorough education in bench-based research, it tends not to prioritize formal training on the databases and tools that allow the efficient extraction and exploitation of the data. The rapid advancement of eLearning technologies to enhance and complement traditional classroom teaching has enjoyed wide uptake in many fields, including bioinformatics. Bioinformatics is a fast-paced discipline, requiring continuous training of its users in emerging areas and concepts [1]. In view of this, eLearning could provide a firm basis for the continuing professional development of molecular life scientists [2].

EMBL-EBI is one of the few places in the world that has the resources and expertise to maintain and provide public access to a comprehensive collection of large biological data sets, ranging from DNA and protein sequences through to models of complex pathways [3]. Hand-in-hand with our mission to make biological information freely available in ways that promote scientific progress is a need to train our users to exploit the data to their full potential. A core part of EMBL-EBI’s mission is therefore to provide user-orientated bioinformatics training to researchers at all levels, from PhD students to independent investigators. Our user base spans wet-lab experimentalists to computational scientists, basic biologists to clinical researchers and early-stage researchers to experienced investigators, including those who need to train others. The EMBL-EBI offers several opportunities aimed at fulfilling this:

  • Support and training to researchers who use EMBL-EBI’s resources via courses and workshops held at EMBL-EBI;

  • Road shows to train researchers on site at their own institutions;

  • Support for other bioinformatics trainers, including community building and resource sharing as discussed in Schneider et al. (this issue);

  • Online training for the potential and current users of EMBL-EBI resources, including other trainers.

There is general consensus that, if used correctly, eLearning can be a powerful educational tool. Indeed, Zhang and colleagues [4] describe eLearning as an ‘indispensable part of academic and professional education’ and they believe that eLearning is ‘a promising alternative to traditional classroom learning, which is especially beneficial to remote and lifelong learning and training’. Kruse [5] also explains the value of eLearning: ‘it is unarguable that eLearning is rapidly growing as a form of training delivery and most are finding that the clear benefits to eLearning will guarantee it a role in their overall learning strategy’. eLearning employs interactive systems and communication technologies to improve learning. It makes possible for a truly ambitious education system for a future learning society [2].

We anticipate that online training will become an increasingly important training vehicle, not only enabling the EMBL-EBI to train more users than we can physically accommodate in our face-to-face training courses, but also as an adjunct to face-to-face training and as a means of sharing training materials among a community of trainers. Having previously run a small pilot project to develop a prototype eLearning portal, the EMBL-EBI is currently in the process of redeveloping its eLearning resources. The first part of this process has been an extensive review of currently available Learning Content Management Systems (LCMSs)—see Box 1.

Box 1. What is a learning content management system?

Content management systems (CMSs) are now widely used to manage large volumes of dynamic web-based material. A learning content management system is a specialized type of CMS fine tuned to different types of users who need to interact with online learning resources. When designing and providing a programme of eLearning courses it is common to use an LCMS as the ‘container’. According to Colace De Santo [17], an LCMS ‘includes all the functions enabling creation, description, importation or exportation of contents as well as their reuse and sharing.’ Cohen and Nycz [6] explains that an LCMS ‘makes it possible for technologists, content specialists, instructional specialists and students to work seamlessly to create and refine eLearning’. eLearning is very much a user motivated method for learning and the quality of the learning environment is very important. Bouchard [16] explains that it must be possible to clearly define learning objectives, decide on the content structure, personalize learning and change the pace of the learning activities for learning to be successful, thus the platform needs to support these.

In this review we describe the process used to select an appropriate LCMS for EMBL-EBI’s online training programme. We generated a list of requirements for our LCMS by reviewing the current literature, soliciting users’ views and reviewing organizational and technical requirements. The most suitable platform was chosen through completion of scripted platform testing and face-to-face discussion groups from the perspective of the trainers, trainees and systems administrators. This review seeks to aid those who find themselves in a similar situation. We believe that every organization’s requirements are unique and that there are several excellent LCMSs available. Our focus here is to describe, and make available to all, the procedure we used to match the attributes of the LCMS to the requirements of our users and trainers.

WHAT DO WE MEAN BY eLEARNING?

There are several definitions for the term eLearning and the perception of what it means is still evolving. The delivery of knowledge, and thus learning, can occur in a variety of ways. These include: the traditional face-to-face method used by most universities and educational institutes; the delivery of courses using only electronic (typically internet-based) media or a combination of both (blended learning). There are lots of different implementations of eLearning and numerous types of media used. Cohen and Nycz [6] describe eLearning in the broader context of using technology to meet society’s needs for learning. Expanding on this they add: ‘eLearning is the current technological solution to the problem of finding the best match between the needs of a given set of learners to learn a given content, using a given set of learning tools’. Koohang and Harman [7] define eLearning as ‘the delivery of education (all activities relevant to instructing, teaching and learning) through various electronic media. The electronic medium could be the Internet, intranets, extranets, satellite TV, video/audio tape, and/or CD ROM.’ At the EMBL-EBI, we define eLearning quite broadly, as the delivery of education through electronic media. The most important concept in this definition is that of ‘education’ as opposed to ‘information’. Learning is an active process that requires, in order to be successful, the use of a range of different techniques to keep the learner engaged (see below). This may involve practicing and testing one’s knowledge, as well as absorbing new information through reading, listening or viewing. We aim to build a system that promotes active learning, not just an encyclopaedia of learning materials.

USEFUL DEFINITIONS AND CONCEPTS

eLearning can be subdivided into trainer-led courses, which are typically delivered to a specific, pre-advertised schedule, and trainee-led (self-directed) courses, which may also be self-paced. eLearning can be delivered on a one-way (asynchronous) or two-way (synchronous) basis. Asynchronous technologies deliver content one way at one point in time and can include: email, video, blogs and audio aids. Synchronous technologies deliver two or more technologies at the same time and can include teleconferencing, chat rooms and web conferencing [8]. It may be safe to say that synchronous technologies are typically used for trainer-led eLearning, and that asynchronous technologies may be typically used for trainee-led eLearning. Asynchronous interaction is also beneficial when students are geographically dispersed and it is difficult to assemble them at the same time [9].

Defining our users from all sides

Before we began working on our selection criteria, we felt that it was important to define our main groups of users. Whilst trainees—the end users of our online courses—are clearly an important group of stakeholders, others need to be taken into account. Trainers—the people who will create and modify our online course content—are an equally important user group. Systems administrators—the people who will install, customize and update the LCMS and any associated modules or plugins—are also vital to the success of any eLearning portal. We therefore sought to structure our selection process around the needs of these three ‘stakeholder’ groups.

Defining priorities for selecting an LCMS

There are many advantages and disadvantages of eLearning and these can be viewed from different perspectives, for example, from the learner’s perspective or from that of the organization seeking to implement the eLearning project. Kruse [5] suggests that the pros and cons of eLearning vary depending on programme goals, target audience and organizational infrastructure and culture. As a starting point to our selection process, we wanted to define which of these pros and cons were important to us, so that we could build a prioritized list of requirements for our LCMS. To do this, we conducted a literature-based review. The literature review included searching Google Scholar [10] and SCOPUS [11] with key terms for hits in the article title, abstract and keywords: ‘eLearning’; ‘eLearning benefits’; ‘eLearning advantages’ and ‘eLearning disadvantages’. Using the term ‘e learning’ as opposed to ‘eLearning’ yielded more results but also includes many unwanted ones on learning alone. The highly cited top hits published within the last 10 years were then reviewed for relevant information. Key articles used in this literature review include Kruse [5] and Zhang [4]. From the literature review we categorized the main advantages and disadvantages of eLearning, from the trainees’ perspective (Table 1), from a systems administration perspective (Table 2) and from the trainers’ perspective (Table 3). We then prioritized which advantages were important to meet our requirements, and which disadvantages it was important to address, using a star-rating system, where two stars means ‘must have’ (for advantages) or ‘must address’ (for disadvantages) and one star means ‘good to have’ or ‘good to address’. These tables do not seek to provide an exhaustive list of all of the advantages and disadvantages of eLearning but aim to provide an overview of the important requirements to consider when approaching eLearning as a training method. It is important to understand that the advantages and disadvantages from each perspective assume that all of the requirements from the other perspectives have been met: this avoids redundancy in the tables. For example, the features from a trainee’s perspective assume that technical requirements, such as browser compatibility, have been met.

Table 1:

Summary of the advantages and disadvantages of eLearning from the trainees’ perspective

Advantages of eLearning from the trainees’ perspective Disadvantages of eLearning from the trainees’ perspective 
Available** Complete training whenever Knowledge Requires knowledge of computers 
Self-pacing** Learn at different speeds Lack of skills No social/cultural interaction 
Portable** Complete training wherever Incorrect** Content may be out-of-date/wrong 
Accessible** Intuitive and easy to use Discipline* Need motivation to work 
Interactive** Engages and motivates users Support** May be no/little support available 
Long-lasting* Available as a refresher course Information* Too much/little information on web 
Individual* Target individual learning styles Technology** Issues with software/web sites 
Skills Increases computer skills Styles Constrained to chosen web style 
Modularity** Complete the relevant parts Culture No access to computers (languages) 
Feedback Quick and interactive feedback   
Costs Only internet/electricity required   
Private Overcomes issues of shyness   
Boundaries Less physical/social boundaries   
Consistency* Similar styled courses   
Subjects Learn outside specialism   
Reviews* Learners can rate courses   
Advantages of eLearning from the trainees’ perspective Disadvantages of eLearning from the trainees’ perspective 
Available** Complete training whenever Knowledge Requires knowledge of computers 
Self-pacing** Learn at different speeds Lack of skills No social/cultural interaction 
Portable** Complete training wherever Incorrect** Content may be out-of-date/wrong 
Accessible** Intuitive and easy to use Discipline* Need motivation to work 
Interactive** Engages and motivates users Support** May be no/little support available 
Long-lasting* Available as a refresher course Information* Too much/little information on web 
Individual* Target individual learning styles Technology** Issues with software/web sites 
Skills Increases computer skills Styles Constrained to chosen web style 
Modularity** Complete the relevant parts Culture No access to computers (languages) 
Feedback Quick and interactive feedback   
Costs Only internet/electricity required   
Private Overcomes issues of shyness   
Boundaries Less physical/social boundaries   
Consistency* Similar styled courses   
Subjects Learn outside specialism   
Reviews* Learners can rate courses   

We define a trainee as an end user of our online courses. We prioritized which advantages were important to meet our requirements, and which disadvantages it was important to address, using a star-rating system, where two stars means ‘must have’ (for advantages) or ‘must address’ (for disadvantages) and one star means ‘good to have’ or ‘good to address’.

Table 2:

Summary of the benefits and difficulties of eLearning from a systems administration perspective

Benefits of eLearning from the technical perspective Difficulties of eLearning from the technical perspective 
Knowledge Acquisition/storage of data Portability Does not rival that of books 
Standardization** Standardization of website Technology Compatibility and bandwidth issues 
Single site Single site back-ups/updates Upgrades* Regular upgrades may be needed 
Repository** One place for all materials Security** Data may not be secure 
Data* Monitor website usage Systems Systems can crash 
Storage Larger storage capacity Redundancy Content may be duplicated 
Tracking* Able to track course usage Maintaining* Requires maintenance and support 
Benefits of eLearning from the technical perspective Difficulties of eLearning from the technical perspective 
Knowledge Acquisition/storage of data Portability Does not rival that of books 
Standardization** Standardization of website Technology Compatibility and bandwidth issues 
Single site Single site back-ups/updates Upgrades* Regular upgrades may be needed 
Repository** One place for all materials Security** Data may not be secure 
Data* Monitor website usage Systems Systems can crash 
Storage Larger storage capacity Redundancy Content may be duplicated 
Tracking* Able to track course usage Maintaining* Requires maintenance and support 

We define a system administrator as a user who will install, customize and update the LCMS and any associated modules or plugins. We prioritized which advantages were important to meet our requirements, and which disadvantages it was important to address, using a star-rating system, where two stars means ‘must have’ (for advantages) or ‘must address’ (for disadvantages) and one star means ‘good to have’ or ‘good to address’.

Table 3:

Summary of the advantages and disadvantages of eLearning from the trainers’ perspective

Advantages of eLearning from the trainers’ perspective Disadvantages of eLearning from the trainers’ perspective 
Low costs* Long-term costs reduced High costs* Large development costs 
Distribution* Faster material distribution Content* Difficult to aim it at the correct level 
Audience* Maximizes participants Time* Initial set up requires a lot of time 
Evaluation* Evaluate learners’ progress Training* Training on the software is needed 
Quality Reviewing content increases quality Instruction* Inferior to face-to-face teaching? 
Community Knowledge can be shared Support* Leaner support may be required 
Reusable* Content is easily reused Updates* Course content needs to be updated 
Promotion* Promotion of other courses   
Advantages of eLearning from the trainers’ perspective Disadvantages of eLearning from the trainers’ perspective 
Low costs* Long-term costs reduced High costs* Large development costs 
Distribution* Faster material distribution Content* Difficult to aim it at the correct level 
Audience* Maximizes participants Time* Initial set up requires a lot of time 
Evaluation* Evaluate learners’ progress Training* Training on the software is needed 
Quality Reviewing content increases quality Instruction* Inferior to face-to-face teaching? 
Community Knowledge can be shared Support* Leaner support may be required 
Reusable* Content is easily reused Updates* Course content needs to be updated 
Promotion* Promotion of other courses   

We define a trainer as someone who will create and modify our online course content. We prioritized which advantages were important to meet our requirements, and which disadvantages it was important to address, using a star-rating system, where two stars means ‘must have’ (for advantages) or ‘must address’ (for disadvantages) and one star means ‘good to have’ or ‘good to address’.

Methods for selecting an LCMS

Evaluating an eLearning platform is not a trivial task; there are many aspects to consider and multiple methods, models and frameworks to evaluate them. Most published literature evaluates platforms based on identified required features but this alone is not always adequate. To elicit the best method for evaluating LCMSs another literature review was conducted, this literature review also revealed potential LCMSs to evaluate. This literature review included searching Google Scholar [10] and SCOPUS [11] with the keywords: ‘eLearning platform’; ‘learning content management systems’; ‘evaluating LCMS’; ‘evaluating eLearning platform’; ‘selecting eLearning platform’ and ‘selecting LCMS’. Again, the highly cited top hits published within the past 10 years were viewed for relevant information. Key articles used in this literature review are discussed below.

The success of an online course is closely linked to the ability of the chosen LCMS to fulfil the requirements of its stakeholders, so what makes a good LCMS? Box 2 lists some successful examples of eLearning programmes in bioinformatics: University of Manchester online Masters level courses [12], Bioinformatics Information Resource and eLearning Center (BIREC) [13], the Education Department of the Bielefeld Bioinformatics Server (BiBiServ) [14] and BioManager [15]. Bouchard [16] suggests one of the most important factors in the success of any learning environment is the ability of the environment to support learner self-direction. Bouchard explains that it must be possible to define learning goals, decide on the learning sequence, personalize learning and change the pace of the learning activities for learning to be successful, thus the LCMS needs to support these. Cohen and Nycz [6] suggest that the teaching of adults (andragogy) should be approached in a different way to the teaching of children (pedagogy) and they advise that problem-based learning (PBL) should be supported by the platform. PBL aims to provide the information and tools needed to solve presented problems in the proper order, with the appropriate level of depth and presented in the correct format. The idea that a successful learner is an active learner (as opposed to a passive learner) is embedded in the constructivist theory of learning. In a constructivist environment, knowledge construction takes place in individual contexts and through social disclosure, collaboration and experience [6]; learning becomes an active process of constructing knowledge in such a way that the learner builds on prior knowledge and experience to draw meaning and construct new knowledge [7]. EMBL-EBI focus on training rather than education and so the teaching of adults is highly important when designing a learning environment. Thus, for the online learning environment to be successful, the leaner should be active, the andragogical approach should be implemented as PBL and the LCMS should support this [17]. Colace and De Santo [17] also explain that a more advanced LCMS should be able to store interactions between the user and each learning object, aiming at gathering detailed information about their utilization and efficacy.

Box 2. Successful examples of eLearning programmes in bioinformatics

eLearning has become an accepted part of education in many fields and is, for example, widely used in medical education [1]. Perhaps due to the difficulty in providing online training in such a fast-paced environment, eLearning is only just emerging as an educational tool for bioinformatics. Examples of successful eLearning programmes for bioinformatics include the following:

  • The University of Manchester runs online Masters level courses [12] as well as biological data analysis for computer statistics modules and computational modelling courses specifically designed for biologists;

  • Bioinformatics Information Resource and eLearning Center (BIREC) [13] was created to provide free bioinformatics information resources and an online self-learning platform. Their goal is to provide maximum amount of information in one place and present the information focussed on the audience;

  • In the Education Department of the Bielefeld Bioinformatics Server (BiBiServ) [14], two online courses are offered, one introductory and one on the advanced level: (i) Sequence Analysis with Distributed Resources (SADR) and (ii) Algebraic Dynamic Programming in Bioinformatics (ADP);

  • BioManager [15] is a web-based bioinformatics application which integrates a variety of common bioinformatics tools, for teaching, BioManager has been utilized as a training environment.

Colace and De Santo [17] also suggest that the features of the system as well as how they are integrated to facilitate learning and training should be examined. They suggest that to evaluate the platforms properly, both andragogical and technological aspects must be carefully investigated, thus, the features used to evaluate the platforms are usually separated into areas [18].

Graf [19] tested nine LCMSs in detail. The author used an example course, which simulated a real life teaching situation, to test each system. Graf split the main evaluation categories into subcategories, which were weighted and evaluated based on the experience from the usage of each LCMS when conducting the example course. Graf used a qualitative weight and sum approach which used six levels of importance for the weights, from ‘essential’ to ‘not valuable’. This approach provided a good starting point for developing our own rating system. We based our ranking system, on which features and requirements were important for us to assess in a LCMS, on this idea. However, we used two weights (‘must have’ and ‘nice to have’) instead of six, mainly to keep it simple but also because this was an approach to decide which the important features and requirements to test were, rather than the actual testing of the LCMS. It was useful to implement Graf’s suggestion of splitting categories into sub-categories in our testing, as this allowed us to incorporate more details in the investigation. We also put into effect the idea of using an example course when testing the LCMSs.

Kurilovas and Dagiene [20] also use a rating system (but with only five weights) and propose using subcategories of features to evaluate an LCMS properly, as does Graf. They do, however, concentrate only the technical evaluation of the LCMSs. The tool is based on analysis of the level of support for the features and the level of modification needed to reach the desired level of support. They give each feature a rating between 0 and 4 (a rating of 0 implies that the feature has failed or it does not exist and a rating of 4 means that the feature has excellent support and meets the criteria out of the box with minimal effort). The technical criteria used by this tool provided a basis for our organization to generate a list of features to incorporate when evaluating the LCMSs (see below). This rating tool was specifically designed for the technical evaluation of LCMSs and our investigation extended beyond this to include the assessment of the entire platform, communication tools for example; thus, this tool was not directly relevant for us to use, but provided guidance for the technical evaluation of the LCMSs.

Graf [18] and Kurilovas and Dagiene [20] both provide a good starting point when generating a list of ranked requirements and features with which to evaluate LCMSs. However, as mentioned previously, the list of features and requirements (and indeed the importance of the requirements and features) is specific to each organization, so after reviewing the sub-categories and categories suggested in the papers, we developed our own weighting system and our own list of features and requirements.

There is a substantial amount of heterogeneous but valid information on several previous attempts to compare different LCMSs. We established that our first step was to define a set of features and requirements with which we would evaluate the LCMSs. The results from our second literature are summarized in Tables 4 and 5. Two categories of features have been taken into consideration: (i) communication and andragogy tools (Table 4) and (ii) technical requirements and features (Table 5).

Table 4:

This table displays features (from the andragogical and communication perspective) that EMBL-EBI considered in the evaluation of LCMSs

Category Features 
Communication tools  
 File exchange Easy file management** 
 Media and collaboration tools Polls** 
Webcasts** 
RSS Feeds** 
Photos** 
Text chat (real time help)** 
Forums* 
External email* 
Video conferencing* 
Wiki functionality* 
Podcasts* 
Animations* 
Administration tools  
 Monitoring progress User progress tracking** 
Users can view own grades** 
Course progress tracking** 
Communication tools  
 Monitoring participation view stats on:
  • – The users on the course**

  • – How the course is being used**

 
Personal user profile* 
Username and password assignment* 
Productivity tools  
 Navigation Multimedia indexing and searching (of e.g. lecture videos)** 
 Users can view course map** 
Course delivery tools  
 Course templates Support to trainers for development of modules and courses. Available course templates** 
 Pedagogy Quizzes** 
Assessment tools** 
Glossary and references** 
Learning objects creation and repository** 
Continuous feedback (may be automated)** 
Gradebook and associated tools* 
 Study tools Resume reading content at previous point** 
On-line note taking or add notes to course content* 
 Course management Updates to users for news about the course** 
Publication of calendar for upcoming events** 
 Course evaluation Course evaluation feature** 
Category Features 
Communication tools  
 File exchange Easy file management** 
 Media and collaboration tools Polls** 
Webcasts** 
RSS Feeds** 
Photos** 
Text chat (real time help)** 
Forums* 
External email* 
Video conferencing* 
Wiki functionality* 
Podcasts* 
Animations* 
Administration tools  
 Monitoring progress User progress tracking** 
Users can view own grades** 
Course progress tracking** 
Communication tools  
 Monitoring participation view stats on:
  • – The users on the course**

  • – How the course is being used**

 
Personal user profile* 
Username and password assignment* 
Productivity tools  
 Navigation Multimedia indexing and searching (of e.g. lecture videos)** 
 Users can view course map** 
Course delivery tools  
 Course templates Support to trainers for development of modules and courses. Available course templates** 
 Pedagogy Quizzes** 
Assessment tools** 
Glossary and references** 
Learning objects creation and repository** 
Continuous feedback (may be automated)** 
Gradebook and associated tools* 
 Study tools Resume reading content at previous point** 
On-line note taking or add notes to course content* 
 Course management Updates to users for news about the course** 
Publication of calendar for upcoming events** 
 Course evaluation Course evaluation feature** 

We used a star-rating system to define high-priority features: two stars mean ‘must have’ and one star means ‘good to have’.

Table 5:

Features (from the systems administration perspective) that EMBL-EBI considered in the evaluation of LCMSs

Category Features 
Adaptability  
 Graphical flexibility Can the organization customize the look of the layout?** 
 Location flexibility Can content be delivered in multiple languages?* 
 Personalization Can the user customize his/her own view of the platform?* 
Security  
 Permissions Can it assign users permissions to edit by a specific criteria, thus controlling who can visit/edit the site?** 
 Security Is it secure?** 
Are the vulnerabilities resolved quickly?** 
Costs  
 Costs Is it open source? Is it free?** 
Support  
 Community Is it easy to get answers to questions about the system?** 
Is there enough documentation?** 
Is the product available in your area?** 
Are the support teams are local?** 
System  
 Programming Does it use appropriate programming languages?** 
 Web Based Is it 100% web-based** 
Compatibility and portability  
 Browser compatibility Does the platform refer to the SCORM standards?** 
What browsers are compatible? (versions)** 
 Portability Can the software be installed on multiple OS? (versions)** 
Is it easy to import content?* 
 VLE Standards Is the platform compatible with Mobile devices? (e.g. PDAs, laptops, iPods, cell phones)* 
Installation, maintenance and scalability 
 Installation Is it easy to find and install add-on modules?** 
Is it easy to set up the pages for a simple site?** 
Is it easy to build a more advanced site?** 
 Accessibility Is the navigation scheme simple?** 
Does it have scalable fonts and graphics?** 
 Maintenance and updates Is it easy to update the site—add pages, images, links?** 
Is it easy to update the system?** 
Is there a system to monitor technical and content changes and give any errors?** 
Can you create content online?* 
Is it easy to download the code and files and install it?* 
Category Features 
Adaptability  
 Graphical flexibility Can the organization customize the look of the layout?** 
 Location flexibility Can content be delivered in multiple languages?* 
 Personalization Can the user customize his/her own view of the platform?* 
Security  
 Permissions Can it assign users permissions to edit by a specific criteria, thus controlling who can visit/edit the site?** 
 Security Is it secure?** 
Are the vulnerabilities resolved quickly?** 
Costs  
 Costs Is it open source? Is it free?** 
Support  
 Community Is it easy to get answers to questions about the system?** 
Is there enough documentation?** 
Is the product available in your area?** 
Are the support teams are local?** 
System  
 Programming Does it use appropriate programming languages?** 
 Web Based Is it 100% web-based** 
Compatibility and portability  
 Browser compatibility Does the platform refer to the SCORM standards?** 
What browsers are compatible? (versions)** 
 Portability Can the software be installed on multiple OS? (versions)** 
Is it easy to import content?* 
 VLE Standards Is the platform compatible with Mobile devices? (e.g. PDAs, laptops, iPods, cell phones)* 
Installation, maintenance and scalability 
 Installation Is it easy to find and install add-on modules?** 
Is it easy to set up the pages for a simple site?** 
Is it easy to build a more advanced site?** 
 Accessibility Is the navigation scheme simple?** 
Does it have scalable fonts and graphics?** 
 Maintenance and updates Is it easy to update the site—add pages, images, links?** 
Is it easy to update the system?** 
Is there a system to monitor technical and content changes and give any errors?** 
Can you create content online?* 
Is it easy to download the code and files and install it?* 

We used a star-rating system to define high-priority features: two stars mean ‘must have’ and one star means ‘good to have’.

In our search for a suitable LCMS for the EMBL-EBI we took into account the following requirements:

  • easy to use from all perspectives (systems administration, trainers, trainees and the organization);

  • secure storage of the data, allowing only those users with the correct privileges to change the information (different levels of administration rights);

  • easy to maintain and update;

  • allows our organization to provide a consistent look and feel to the courses that is customisable to EMBL-EBI’s design guidelines;

  • allows EMBL-EBI to build courses that cater for different learning styles and incorporate different types of materials;

  • space for users to add their comments and feedback;

  • compliance with eLearning standards;

  • repository for materials from face-to-face training courses;

Organizational requirements that we took into account include the following:

  • Free/open source—In accord with EMBL-EBI’s philosophy of supporting and contributing to open-source software projects, we are strongly in favour of using an open-source LCMS so long as it meets our requirements. Cohen and Nycz [6] add that open source platforms offer better reusability, reduced costs, and rapid fixes to bugs/problems.

  • Comparison with tried and tested systems—Our eLearning pilot project was built using Moodle, so it was important to us that we compared other platforms to Moodle, hence, it had a place as one the final three LCMS lined up for testing. Moodle was previously chosen as the LCMS for the pilot eLearning portal as, at the time of choice, it was a free and popular LCMS which was considered to require minimal installation support and maintenance. The EMBL-EBI has also selected Drupal as a Content Management System (CMS) for its website, so there would be some obvious benefits to using the same system as an LCMS. Of course both of these platforms are successful and popular LCMSs [20, 21].

  • Compatibility with EMBL-EBI software—The EMBL-EBI uses Linux as the main operating system for its servers, PhP to create dynamic web content and MySQL to store data. Compatibility with this software was therefore a prerequisite for our LCMS.

The selection process

To select an LCMS most suited to EMBL-EBI’s requirements from the vast list of the current popular eLearning platforms, we broke down the selection process into these phases, summarized in Figure 1:

  • Phase 1: We generated a list of eLearning platforms by reviewing the literature (the same literature review process was used for finding LCMSs as was used to discover the most suitable methods for evaluating LCMSs—see above) to discern which platforms were worth investigating. This generated a list of 36 platforms (Table 6).

  • Phase 2: In accordance with EMBL-EBI’s philosophy of supporting open-source systems whenever possible, we selected only open-source LCMSs from our phase 1 list, thereby reducing the number of LCMSs to investigate to 13 (Table 6);

  • Phase 3: We (the EBI’s eLearning Content Developer and the Training Programme Project Leader) then performed a short investigation into each remaining LCMS. This involved: checking the platform’s website, including any online demonstration material; researching the literature produced by the platform author and by other authors; investigating the level of support provided for each platform and briefly evaluating the platform against the criteria summarized in Tables 4 and 5. This reduced our list to only three platforms—Drupal, ATutor and Moodle—to consider for further testing.

  • Phase 4: We (the EBI’s training team, members of the ES team and the Outreach and Training group leader) used the prioritized lists in Tables 1–5 to define exactly what our evaluators would test during the scripted platform testing.

  • Phase 5: Scripted testing of each of the three LCMSs was carried out by three sets of evaluators: (i) trainers (12 people), who were asked to created a course using each LCMS; (ii) trainees (nine people), who were asked to complete a course on each LCMS) and (iii) systems administrators (four people). The systems administrators were further divided into two groups: one group was asked to install the LCMS and any modules/plugins required, and will ultimately be responsible for building and maintaining the EMBL-EBI’s LCMS; the second group was asked to investigate the site and its content from an administrative perspective; for example, to find out how many users were enrolled on a course. This group will be responsible for administration of the online training programme and therefore will have access rights greater than those of trainers. All three LCMSs were installed on the EMBL-EBI web server and an identical sample course (a method supported by Graf [19]) was set up on each platform. We chose to use a topic unrelated to bioinformatics, because we felt that this would best enable our platform testers to focus on the usability of the LCMS as opposed to the quality of the content. The evaluators were assigned a role (e.g. trainer) and asked to follow a guideline script, complete a questionnaire and partake in discussions about the platforms (Supplementary Data). The feedback provided through this process provided us with guidelines for determining which platform best meets the needs of trainers, trainees and systems administrators. Full guidelines used for the course creation as well as the forms used for the evaluation of the LCMSs are available as Supplementary Data.

  • Phase 6: After the physical testing of the platform the feedback was collated and analysed and a platform was chosen. The evaluation of the results is presented in the following section.

Figure 1:

Flow diagram of the different phases of the project and the stakeholders at each stage. The inner square boxes describe which stakeholders had input in that phase. OTT is the EMBL-EBI’s Outreach and Training Team, which is responsible for coordinating our communications and training programmes; ES is the External Services Team, which is responsible for maintaining and developing the EMBL-EBI website.

Figure 1:

Flow diagram of the different phases of the project and the stakeholders at each stage. The inner square boxes describe which stakeholders had input in that phase. OTT is the EMBL-EBI’s Outreach and Training Team, which is responsible for coordinating our communications and training programmes; ES is the External Services Team, which is responsible for maintaining and developing the EMBL-EBI website.

Table 6:

List of popular LCMSs was generated by reviewing past and present literature to discern which platforms to include on our ‘long list’

Platform Free open source? 
A Tutor Yes 
Adobe Connect Pro No 
Apriori No 
BELTS No 
Blackboard No 
Bodington Yes 
Claroline Yes 
Cognitivity No 
Docebo Yes 
Dokeos Yes 
Drupal Yes 
Ecollege No 
EduZope No 
Efront Yes 
ILIAS Yes 
iLinc No 
Interact No 
Intralearn No 
Joomla (LMS) No 
LAMS No 
LON-CAPA Yes 
.LRN Yes 
Meridian KSI No 
Moodle Yes 
OpenLMS Yes 
Plateau No 
Saba Suite No 
Sakai (CMS) Yes 
SharepointLMS No 
Spiral universe No 
Sumtotal systems No 
SwitchPort No 
Telania No 
Thinking cap No 
TopClass (WBT) No 
Platform Free open source? 
A Tutor Yes 
Adobe Connect Pro No 
Apriori No 
BELTS No 
Blackboard No 
Bodington Yes 
Claroline Yes 
Cognitivity No 
Docebo Yes 
Dokeos Yes 
Drupal Yes 
Ecollege No 
EduZope No 
Efront Yes 
ILIAS Yes 
iLinc No 
Interact No 
Intralearn No 
Joomla (LMS) No 
LAMS No 
LON-CAPA Yes 
.LRN Yes 
Meridian KSI No 
Moodle Yes 
OpenLMS Yes 
Plateau No 
Saba Suite No 
Sakai (CMS) Yes 
SharepointLMS No 
Spiral universe No 
Sumtotal systems No 
SwitchPort No 
Telania No 
Thinking cap No 
TopClass (WBT) No 

The table also describes which platforms are freely available and which are open source (the source code is published and made available to the public, so it can be copied, modified and redistributed by anyone for free).

EVALUATING THE RESULTS

To choose the most suitable LCMS for EMBL-EBI out of the three potential platforms (Drupal, Moodle and ATutor), the user feedback from the scripted testing day (Phase 5) was collated and analysed:

  • The type of user supplying the feedback (systems admin, trainer and trainee) was taken into consideration at all points in the feedback evaluation process;

  • The platforms were ranked according to which LCMS the user had chosen as their favourite and their least favourite platform;

  • Qualitative information, provided as comments in the evaluation forms (for example, on what the user liked the most and least about the platform) were consolidated to create a list of the main advantages and disadvantages of each platform;

  • Individual feature evaluations from all perspectives were examined for each platform, for example, how easy it was to create (trainer) and complete a quiz (trainee);

  • The users’ feedback after completing testing of all three platforms was discussed;

  • The feedback and options were reviewed by the authors of this paper and a platform was chosen;

  • A list of requirements, based on feedback from all three groups, was compiled and used as the basis of a prototype for the EMBL-EBI’s eLearning platform, which is now in development.

RELATED WORK AND FUTURE DIRECTIONS

We have now begun the task of translating our list of requirements into a portal; of course, the portal itself is merely an empty vessel, which will then need to be filled with courses. In parallel with work on identifying a suitable LCMS, we have gathered comments from potential users of our eLearning platform, including molecular life scientists attending our hands-on training courses, delegates at our roadshows, and masters’ students at our open days. Their feedback, and our own previous experience of creating course materials, is being used to create a prototype course, which will be tested by trainees and refined in light of their feedback. As we learn from this process, we will apply our learning to the development of further courses, initially focusing on Europe’s core data resources and on our most frequently used bioinformatics tools.

An important aim of this project, in line with EMBL-EBI’s philosophy of making the results of its work openly accessible, is to enable others to reuse and build upon our training materials. We have therefore begun to engage with a broader community of trainers from beyond the EMBL-EBI to gather their requirements in the hope that we may be able to support some of their training needs. This work is discussed in more detail in the accompanying paper by Schneider et al. We warmly encourage readers of this paper to contact us with comments and suggestions, by e-mailing us at elearning@ebi.ac.uk. Should you wish to learn about future eLearning developments at the EMBL-EBI, we encourage you to join our mailing list: http://listserver.ebi.ac.uk/mailman/listinfo/elearning.

CONCLUSION

Choosing an appropriate LCMS is a complex task that requires you to:

  1. define and involve the different types of users (administrators, trainers and end users);

  2. prioritize your requirements;

  3. find an unbiased way of scoring these requirements against each LCMS being tested and

  4. recognize which issues raised during the testing are crucial to a go/no go decision.

SUPPLEMENTARY DATA

Supplementary data are available online at http://bib.oxfordjournals.org/.

Key Points

  • Users of bioinformatics data resources and tools require training on how to exploit these services to their full potential; providing such training is a core part of the EMBL-EBI’s mission.

  • eLearning has the potential to provide a valuable complement to face-to-face training; it is likely to become increasingly important as increasing numbers of bioscientists, from a diversifying range of fields, begin to exploit bioinformatics data resources and tools.

  • Choosing an appropriate learning content management system (LCMS) is an important first step towards developing an online training programme. This review summarizes the process that the EMBL-EBI went through to make this choice.

  • The selection of an LCMS is specific to each organization and this review makes no claims for the LCMSs that it shortlisted; we focus instead on the process that we went through, in the hope that our experience may be valuable to other bioinformatics trainers who need to make their own choices.

References

1
Kowalewska
A
The possibilities of using e-learning in students' training in the field of health education
Rocz Panstw Zakl Hig
 , 
2006
, vol. 
57
 (pg. 
29
-
33
)
2
Saravanana
V
Shanmughavel
P
E-Learning as a new tool in bioinformatics teaching
Bioinformation
 , 
2007
, vol. 
2
 (pg. 
83
-
5
)
3
Brooksbank
C
Cameron
G
Thornton
J
The European Bioinformatics Institute's data resources
Nucleic Acids Res
 , 
2010
, vol. 
38
 (pg. 
D17
-
25
)
4
Zhang
D
Zhao
L
Nunamaker
L
E-learning replace classroom learning?
Commun ACM
 , 
2004
, vol. 
47
 (pg. 
74
-
9
)
5
Kruse
K
The benefits and drawbacks of e-Learning
(20 July 2009, date last accessed) 
6
Cohen
E
Nycz
M
Learning objects and e-Learning: an informing science perspective
Interdisciplinary J Knowledge Learn Objects
 , 
2006
, vol. 
2
 (pg. 
23
-
34
)
7
Koohang
A
Harman
K
Open source: a metaphor for e-Learning
Inform Sci J
 , 
2005
, vol. 
8
 (pg. 
75
-
86
)
8
Mircea
G
The future of e-Learning
Designing Tomorrow's Education
 , 
2006
(23 June 2010, date last accessed) 
Available at SSRN: http://ssrn.com/abstract=904201
9
Haverila
M
Barkhi
R
The influence of experience, ability and Interest on e-learning effectiveness
Eur J Open Distance E-Learning
 , 
2009
 
http://www.eurodl.org/?keyword=collaborative%20learning&article=359 (23 June 2010, date last accessed)
10
Google
Google scholar
(22 September 2009, date last accessed) 
11
Scopus
Elsevier B
(21 September 2009, date last accessed) 
12
Manchester
U
Bioinformatics online modular degree programme
(16 March 2010, date last accessed) 
13
BIREC
Bioinformatics information resource and eLearning center
(16 March 2010, date last accessed) 
14
Sczyrba
A
Konermann
S
Giegerich
R
Two interactive bioinformatics courses at the Bielefeld University Bioinformatics Server
Briefings in Bioinformatics
 , 
2008
, vol. 
9
 (pg. 
243
-
9
)
15
Cattley
S
Arthur
JW
BioManager: the use of bioinformatics web application as a teaching tool in undergraduate bioinformatics training
Briefings in Bioinformatics
 , 
2007
, vol. 
8
 (pg. 
457
-
65
)
16
Bouchard
P
Some factors to consider when designing semi-autonomous learning environments
Electronic J e-Learn
 , 
2009
, vol. 
7
 (pg. 
93
-
100
)
17
Colace
F
De Santo
M
Evaluating on-line learning platforms: a case study
Proceedings of the 36th Hawaii International Conference on System Sciences 2003
 , 
2003
Big Island, HI, USA
IEEE Computer Society
18
Graf
S
List
B
An evaluation of open source e-learning platforms stressing adaptation issues
Proceedings of the Fifth IEEE International Conference on Advanced Learning Technologies
 , 
2005
Washington, DC, USA
IEEE Computer Society
(pg. 
163
-
5
)
19
Graf
S
Kinshuk
Liu
T
Supporting teachers in identifying students' learning styles in learning management systems: an automatic student modelling approach
Educational Technology & Society
 , 
2009
, vol. 
4
 (pg. 
3
-
14
)
20
Kurilovas
E
Dagiene
V
Learning objects and virtual learning environments technical evaluation criteria
Electronic Journal of e-Learning
 , 
2009
, vol. 
7
 (pg. 
127
-
36
)
21
Mooney
S
Baenziger
P
Extensible open source content management systems and frameworks: a solution for many needs of a bioinformatics group
Briefings in Bioinformatics
 , 
2008
, vol. 
9
 (pg. 
69
-
74
)

Supplementary data