Citizen science enlists the public in collecting large quantities of data across an array of habitats and locations over long spans of time. Citizen science projects have been remarkably successful in advancing scientific knowledge, and contributions from citizen scientists now provide a vast quantity of data about species occurrence and distribution around the world. Most citizen science projects also strive to help participants learn about the organisms they are observing and to experience the process by which scientific investigations are conducted. Developing and implementing public data-collection projects that yield both scientific and educational outcomes requires significant effort. This article describes the model for building and operating citizen science projects that has evolved at the Cornell Lab of Ornithology over the past two decades. We hope that our model will inform the fields of biodiversity monitoring, biological research, and science education while providing a window into the culture of citizen science.
Studying large-scale patterns in nature requires a vast amount of data to be collected across an array of locations and habitats over spans of years or even decades. One way to obtain such data is through citizen science, a research technique that enlists the public in gathering scientific information (Bhattacharjee 2005). Large-scale projects can engage participants in continental or even global data-gathering networks. Pooled data can be analyzed to illuminate population trends, range changes, and shifts in phenologies. Results can be published in the scientific literature and used to inform population management decisions.
The Cornell Lab of Ornithology (CLO), home base for the authors of this article, has welcomed public participation in its research for decades. Today, CLO operates numerous citizen science projects of various sizes, each designed to answer scientific questions while helping the public learn about birds and the process of science (www.cornellcitizenscience.org; table 1). In the past two decades, CLO's projects have engaged thousands of individuals in collecting and submitting data on bird observations, reading about project findings, visualizing data through Web-based graphs and maps, and even analyzing data themselves. Collectively, the projects gather tens of millions of observations each year.
Citizen science projects have been remarkably successful in advancing scientific knowledge. Recent publications using data collected by CLO projects have examined how bird populations change in distribution over time and space (Wells et al. 1998, Hochachka et al. 1999, Cooper et al. 2007, Bonter and Harvey 2008, Bonter et al. 2009); how breeding success is affected by environmental change (Rosenberg et al. 1999a, Hames et al. 2002a); how emerging infectious diseases spread through wild animal populations (Hochachka and Dhondt 2000, Hartup et al. 2001, Altizer et al. 2004, Hochachka et al. 2004, Dhondt et al. 2005); how acid rain affects bird populations (Hames et al. 2002b); how seasonal clutch-size variation is affected by latitude (Cooper et al. 2005a, 2005b, 2006); and how databases can be mined and models constructed to discover patterns and processes in ecological systems (Caruana et al. 2006, Hochachka et al. 2007, Fink and Hochachka 2009, Kelling et al. 2009).
The CLO citizen science projects also strive to help participants learn about birds and experience the process by which scientific investigations are conducted. Evaluations have shown that in addition to learning science facts (Brossard et al. 2005, Trumbull et al. 2005), some participants have used appropriate scientific processes and principles when making decisions about experimental design (Trumbull et al. 2000). Individuals also have increased the numbers of days that they watched birds and recorded information about them after participating in a project (Thompson and Bonney 2007). Still other participants have demonstrated that they can use online data tools to answer a variety of questions about bird distribution and abundance, such as when certain species were present in their area, where rare birds were seen, and how populations change over time (Bonney 2007).
Developing and implementing public participation projects that yield both scientific and educational outcomes requires careful planning. This article describes the model for building and operating citizen science projects that has evolved at CLO over the past two decades. We hope that our model will inform the fields of biodiversity monitoring, biological research, and science education while providing a window into the culture of citizen science.
Citizen science project design
Public participation in scientific research is not new. Lighthouse keepers began collecting data about bird strikes as long ago as 1880; the National Weather Service Cooperative Observer Program began in 1890; and the National Audubon Society started its annual Christmas Bird Count in 1900 (Droege 2007). Throughout the 20th century, thousands of public volunteers participated in projects to monitor water quality, document the distribution of breeding birds, and scour the night skies for new stars and galaxies. The current concept of citizen science, however, with its integration of explicit and tested protocols for collecting data, vetting of data by professional biologists, and inclusion of specific and measurable goals for public education, has evolved primarily over the past two decades (Bonney 2007, Cohn 2008).
Citizen science projects at the CLO are driven by a research question or monitoring agenda that fits within the organization's science or conservation mission. Projects range from focused studies, such as the House Finch (Carpodacus mexicanus) Disease Survey, which engages a few hundred participants in watching feeders for signs of avian conjunctivitis, to monitoring projects like eBird, which collects more than 1 million bird observations each month from a legion of birdwatchers around the world (table 1). In all CLO projects, participants are asked to follow specific protocols, collect data about birds and their environments, and submit the data to the CLO's database. The protocols are provided in field-tested instruction booklets or on Web pages and are enhanced by educational supports such as posters, identification guides, and CDs or online multimedia presentations. Each project is maintained by at least one full-time staff person whose tasks include responding to participants' questions.
Once data are entered into the database, anyone with Internet access can explore the information using a variety of data-visualization techniques. Explorations can be species based (Where do northern cardinals [Cardinalis cardinalis] occur?), place based (What species will I see if I visit a certain national wildlife refuge?), temporal (Have house finch populations declined in recent years?), or a combination of these (What are the average clutch sizes for tree swallows [Tachycineta bicolor] in California and New York?). The data-visualization tools are used by thousands of project participants each month to see how their contributions relate to those of others. The tools are also used by scientists, land managers, and conservationists who search for patterns of species occurrence and changes in abundance over time. Results of analyses are presented on the CLO Web site (http://birds.cornell.edu); in the CLO's newsletter, BirdScope; and in a variety of scientific publications, including numerous peer-reviewed journals.
All of the data contributed to CLO citizen science databases are provided by the public and are available at no charge to anyone, amateur or professional, for any noncommercial use. Maintenance and security are provided by database managers housed within the CLO's information science department. Raw data are available either from individual project Web sites or through the Avian Knowledge Network (www.avianknowledge.net).
Citizen science program model
The CLO's model for developing and implementing a citizen science project has been worked out over time by a group of individuals with expertise in education, population biology, conservation biology, information science, computational statistics, and program evaluation. We have found that projects whose developers follow this model can simultaneously fulfill their goals of recruitment, research, conservation, and education (box 1).
Choose a scientific question.
Form a scientist/educator/technologist/evaluator team.
Develop, test, and refine protocols, data forms, and educational support materials.
Accept, edit, and display data.
Analyze and interpret data.
Choose a scientific question. Citizen science is particularly helpful to investigators who are interested in answering questions that have a large spatial or temporal scope. For example, two questions that have been addressed by CLO projects are, “What are the patterns of irruption in winter finch populations?” (Project FeederWatch) and “How do clutch sizes of eastern bluebirds (Sialia sialis) vary with latitude?” (NestWatch). When choosing questions, project developers must consider that most participants will be amateur observers. Thus, questions for which data collection relies on basic skills, such as counting a few species of birds at feeders or determining the number of eggs in a nest, are more appropriate than questions that require higher levels of skill or knowledge, such as determining the level of courtship intensity of a breeding pair of birds. Projects demanding high skill levels from participants can be successfully developed, but they require significant participant training and support materials such as training videos.
Monitoring studies designed to detect patterns of species occurrence over time or space are especially well suited for citizen science. Broadscale surveys gather tremendous quantities of data that can be used to explore trends in species occurrence across broad geographic landscapes (e.g., Robbins et al. 1989, Hochachka et al. 2007), water quality across watersheds (e.g., EPA 2006), or trends in population interactions over time (e.g., Cooper et al. 2007). But citizen science can involve complex designs and even experiments, which provide excellent teaching opportunities. For example, the CLO's Seed Preference Test involved thousands of participants who examined food preferences of birds by providing different types of seeds in a continent-wide experiment (Trumbull et al. 2000). And participants in the Birds in Forested Landscapes project are required to select survey sites, describe site habitats in detail, and use playbacks of recorded songs and calls to locate and map breeding birds.
Because complicated projects tend to attract fewer participants, project designers who wish to reach large audiences need to keep projects simple. However, even simple projects can address complicated questions by recruiting a subset of participants into more complex tasks. For example, when clutch-size data from NestWatch yielded discoveries about geographic trends in incubation period and hatching failure that required study outside the scope of the original protocol, project staff launched a new study for which participants installed data loggers to record time and temperature inside bluebird nests. Similarly, when the Birds in Forested Landscapes project revealed a relationship between declining forest birds and acid rain (Hames et al. 2002b), researchers devised a supplemental protocol to measure the availability of calcium-rich prey in leaf litter at hundreds of study sites.
Form a team of scientists, educators, technologists, and evaluators. A successful citizen science project requires a development team comprising multiple disciplines. A researcher is required to ensure the project's scientific integrity, to develop protocols that will lead to the collection of quality data, and to analyze and publish the data after they are collected. An educator is required to explain the project's importance and significance to participants, to pilot- and field-test protocols with potential participants, to develop clear and comprehensive project support materials, and to ensure appropriate participant feedback. A computational statistician or information scientist is needed to develop both the database infrastructure and the technology required to receive, archive, analyze, visualize, and disseminate project data and results. An evaluator is needed first to ensure that the project begins with measurable objectives, and second to gather data to assess project success based on those objectives, both during and after project implementation.
Small groups or organizations that do not have internal access to all disciplines can partner with other organizations or adapt national citizen science projects for use at local or regional scales. Projects and collaborators can be located at CLO's citizen science toolkit Web site, www.citizenscience.org.
Develop, test, and refine protocols, data forms, and educational support materials. Data quality is a critical issue for any citizen science project. Ensuring that the public can collect and submit accurate data depends on three things: providing clear data collection protocols, providing simple and logical data forms, and providing support for participants to understand how to follow the protocols and submit their information. Even with these safeguards in place, we have discovered that certain concepts require special attention. These involve issues of bias—a tendency to overreport certain species and to underreport others—and a general reluctance of observers to enter data when they see only common birds or no birds at all.
Protocols. Citizen science data are gathered through protocols that specify when, where, and how data should be collected. Protocols must define a formal design or action plan for data collection that will allow observations made by multiple participants in many locations to be combined for analysis (University of Washington Health Services 2000). Protocols used for citizen science should be easy to perform, explainable in a clear and straightforward manner, and engaging for volunteer participants.
Pilot-testing protocols with naive audiences is crucial and is most valuable when directed at a wide swath of potential participants. For example, CLO project designers have tested draft protocols with local bird clubs, school groups, and youth leaders by accompanying participants in the field and observing them as they collect and submit data. CLO staff have also tested protocols at distant locations by collecting feedback online, usually from CLO members or from individuals who have participated in previous projects. When protocols prove to be confusing or overly complicated, they can be simplified, clarified, or otherwise modified until the participants can follow them with ease. For example, when developing the House Finch Disease Survey, CLO staff realized that participants were more likely to report the presence rather than the absence of the malady. To overcome this problem, they produced new educational materials explaining that reporting “negative” data (no diseased birds seen) is just as important as reporting “positive” data (eye disease present).
Data forms. Designing data forms that are easy to understand and fill in is best done in conjunction with protocol design. Quality data forms mirror project protocols and help to prepare data for analysis. For instance, eBird data forms ask participants to note if they are reporting all of the species that they observed in a given time at a given place. This information allows analysts to determine if an unrecorded species was not detected in an area or simply wasn't reported, critical knowledge for scientists who are analyzing species' presence or absence. Online data forms also can ensure that all essential information is provided by preventing participants from proceeding with data entry until all of the required fields are filled.
Errors resulting from misidentified species can be a major issue for citizen science because many cryptic, unusual, or similar species can be confused (Kelling 2008). Online data forms can help with this problem by filtering anomalous records before they enter the database. Records that do not fit within the limits of the filter can be flagged for further review. For example, in eBird and Project FeederWatch, if an observer enters data that are outside the filter limits—such as a species outside its range, or an unusually large number of individuals in a given area—a friendly message asks the user to double-check the entry. If the observer is confident about the observation, the “error” message can be overridden, but the record remains flagged until it is reviewed by a regional editor. Flagged records are not used in data visualizations nor made available for analysis until confirmed. Experts in species distribution and abundance continually adjust the filters to reflect the current understanding of species occurrence. Thus online data entry and subsequent vetting allow errors to be caught before they enter the database, while enabling project participants to visualize project data as quickly as they are submitted (Sullivan et al. 2009).
Educational materials. A variety of materials can be offered to support participant understanding and satisfactory completion of project protocols. Support materials include identification guides, posters, manuals, videos, podcasts, newsletters, and FAQs (frequently asked questions) that discuss the challenges in making observations or in filling out data forms. For example, CLO's NestWatch provides an interactive quiz based on the Nest Monitoring Code of Conduct. Users who answer all questions correctly can download a personalized nest-monitor certificate. The desire to be certified encourages volunteers to read and understand the code of conduct while allowing project administrators to track the progress of individual users each time they take the quiz.
Online forums offer additional learning opportunities. For example, NestWatch operates several forums that allow participants to ask and answer questions related to breeding biology, data protocols, and online data entry and retrieval. Because these technologies are relatively new, their impact on learning outcomes has yet to be fully determined.
Recruit participants. Recruiting participants can be very simple or extremely challenging, depending on a project's goals and audience. If a project has been developed for the general public, participants can be recruited by a variety of techniques, such as press releases, listservs, direct mailings, advertisements, public service announcements, magazine and newspaper articles, brochures, fliers, and presentations, including posters and workshops at conferences of potential participants or their leaders.
If a project has been developed for specific audiences, such as youth groups, then recruitment materials should be targeted to those audiences. However, recruiting defined audiences can be challenging without partnering. For example, youth groups such as scouts or Boys and Girls Clubs typically have unique objectives, agendas, projects, and methods of presenting materials. Simply offering project support materials, such as leaders' guides, to individual groups or teachers rarely leads to project adoption.
However, deliberate partnering during the course of project development can yield projects that do meld with existing programs. For example, the CLO has adapted eBird for use in middle schools by developing a standards- and inquiry-based curriculum, BirdSleuth, which was developed over three years with extensive input from more than 100 middle-school teachers across North America. Because these teachers helped to develop, pilot, and field-test the curriculum, it covers subject matter (e.g., diversity, adaptation, and graphing skills) that teachers can easily integrate into their lessons.
Train participants. Providing participants with the support they require to digest project materials and gain confidence in their data-collection skills is critical. Early CLO projects employed a printed research kit consisting of project instructions, background reading, and support materials such as bird identification posters, CDs of bird sounds, and instructions on building birdhouses and feeders. Such support information is still provided to project participants, but is usually delivered over the Internet through downloads and project videos. Individual participants must take responsibility for reading and studying project materials and for calling or e-mailing for help if they are confused.
Projects that are carried out by groups provide further opportunities for training, because project staff can provide guidance and information to group leaders. Regional projects can hold training workshops, and large-scale projects can hold regional workshops in partnership with project collaborators. For example, in CLO's Celebrate Urban Birds! and NestWatch projects, training workshops have been held at partner sites such as science museums and youth centers.
Accept, edit, and display data. Whether a project employs paper or electronic data forms, all of the information must be accepted, edited, and made available for analysis, not only by professional scientists but also by the public. Indeed, allowing and encouraging participants to manipulate and study project data is one of the most educational features of citizen science.
Current CLO projects allow participants to view a diverse set of graphs, maps, histograms, and other visualizations that immediately show how their data are being used. The projects also supply personal data-management tools such as those that create sophisticated birding life lists or compare information on breeding success in nest boxes from one year to the next. Such tools have been popular with project participants and have increased project participation. In April 2006, the eBird Web site was upgraded with new features that allow participants to track their own observations and to explore how their reports compare with others. Immediately after these features were implemented, the number of individuals submitting data nearly tripled (Sullivan et al. 2009).
Analyze and interpret data. Citizen science projects tend to produce coarse data sets that can present significant challenges for analysis and interpretation. Fortunately, the large size of most citizen science data sets creates a favorable signal-to-noise ratio, yielding strong patterns that are easy to interpret. In addition, researchers working with large data sets can develop criteria for identifying data that contain systematic errors, such as species misidentifications or misinterpretations of protocols, and can omit such data from analysis without compromising the goals of the project. For example, techniques have recently been developed to estimate detectability in observational data and to incorporate differences in detectability (e.g., between different observers) into data analysis (Fink and Hochachka 2009). Also, if analysis techniques and constraints are determined during protocol development, potential biases or errors can be minimized as data are being collected.
Because of difficulties inherent in estimating and controlling for detectability, citizen science data are often more suitable for computing indices of relative abundance than estimates of absolute abundance. Also, because observation points do not always constitute a random or stratified sample, making inferences beyond the actual data points may be difficult. However, presenting valid assumptions about the presence of systematic errors or sampling biases can facilitate geographical comparisons. For example, because estimating an actual number of nest attempts requires observations of banded individuals and a way to estimate detectability, Cooper and colleagues (2005a, 2005b) computed the relative number of nest attempts by eastern bluebirds for comparisons across latitudes. And Fink and Hochachka (2009) developed new analytical techniques that allow more accurate representations of species occurrence and relative abundance, as well as regional and temporal comparisons. An example of the way in which data from eBird can be used to describe seasonal patterns of relative abundance for the eastern phoebe (Sayornis phoebe), a common Neotropical migrant, is provided in figure 1.
Often citizen science data will show general phenomena or patterns that must be examined further with smaller, more focused studies. Combining multiple data sets can illustrate fine-resolution, small-scale results in the context of large-scale patterns. In addition, large-scale data sets obtained through citizen science can be leveraged with sensor- or professional-based, large-scale data sets that can be interpolated. Hames and colleagues (2002b) combined data collected by Birds in Forested Landscapes participants on the presence and absence of breeding wood thrush (Hylocichla mustelina) in forest fragments with data interpolated from US Geological Survey and NOAA (National Oceanic and Atmospheric Administration) to investigate the synergistic effects of acid rain and habitat fragmentation. Also, inferences made from large-scale coarse patterns detected with citizen science data are stronger when complemented with small-scale, fine-grained studies. For example, patterns of the spread of eye disease across house finch populations spawned an intense investigation of disease transmission using captive experiments, local field studies of banded individuals, and modeling specifically aimed at understanding the patterns observed across North America (Dhondt et al. 1998, 2006, Hochachka and Dhondt 2000).
Disseminate results. Results from CLO citizen science projects have appeared in a range of scientific journals including Ibis, Ecology, Conservation Biology, Journal of Avian Biology, Journal of Animal Ecology, and Proceedings of the National Academy of Sciences. In addition, many projects publish technical reports to disseminate results to target audiences. For example, Project Tanager and the Birds in Forested Landscapes project resulted in a series of forest management guidelines intended for public and private landowners (Rosenberg et al. 1999b, 2003). More recently, results of citizen science monitoring projects have been used to develop online decision-support tools for policymakers and land managers through regional data nodes of the Avian Knowledge Network (www.avianknowledge.net).
Results are also published for the public through project Web sites and through the CLO's quarterly newsletter, BirdScope. In addition, results frequently are reported in popular literature such as newspapers, magazines, and newsletters published by a variety of organizations ranging from bird clubs to state and national biodiversity conservation organizations. Such publications are important not only for general interest but also for showing the public how fellow citizens are contributing to science and, we hope, for motivating new individuals to participate themselves.
Measure impacts. A final step in the citizen science model involves measuring project outputs and outcomes to ensure that both scientific and educational objectives have been met. If they have, publications can elaborate these successes for others to use as models. If they have not, evaluations can illuminate how to improve the project or how to design better projects in the future.
Outputs and outcomes can be gauged in many ways. Some measures reflect greater knowledge in scientific fields, some reflect improved scientific literacy among the public, and some reflect both.
Measures of scientific contribution. Measuring contributions to science is reasonably straightforward. Possible measures include the following:
Numbers of papers published in peer-reviewed journals
Numbers of citations of results
Numbers of researchers publishing citizen science research papers
Numbers and sizes of grants received for citizen science research
Size and quality of citizen science databases
Numbers of graduate theses completed using citizen science data
Frequency of media exposure of results
Scientific literacy outcomes. Measuring improvement in public scientific literacy is more challenging. Among possible measures are the following:
Duration of involvement by project participants
Numbers of participant visits to project Web sites
Improved participant understanding of science content
Enhanced participant understanding of science process
Better participant attitudes toward science
Improved participant skills for conducting science
Increased participant interest in science as a career
Measurement methods can include pre- and postproject surveys for project participants, examinations of e-mail and listserv messages from project participants, surveys of self-reported knowledge gains among participants, focus groups, and in-depth interviews. Such techniques require an understanding of research methodology in the social sciences (Bonney et al. 2009).
Costs. An effective citizen science program requires staff dedicated to direct and manage project development; participant support; and data collection, analysis, and curation. Such a program can be costly; CLO's current citizen science budget exceeds $1 million each year. Since 1992, the projects have been supported largely by grants, including several from the National Science Foundation (NSF), primarily from its education and informatics programs. Except for FeederWatch and BirdSleuth, the CLO does not currently charge anyone to participate in its projects or to use its data; thus, sustaining long-term projects is an ongoing challenge. However, considering the quantity of high-quality data that citizen science projects are able to collect once the infrastructure for a project is created, the citizen science model is cost-effective over the long term.
Also, new projects can build on previously developed efforts; several organizations are now adapting eBird technology to collect new types of data. Moreover, many open-source technologies can be modified for specific projects. Google Maps, for example, can be customized and integrated into Web sites at little expense.
We believe that the full potential for citizen science is just beginning to be understood. As one step in moving the field forward, the CLO hosted an invitational conference on citizen science project development and implementation from 20 to 23 June 2007, in Ithaca, New York. The conference, sponsored by the NSF, brought together 54 citizen science practitioners and evaluators from across North America who discussed various models of public participation in research and showcased projects in a variety of disciplines. Discussions revealed that most citizen science projects developed so far have been in disciplines that historically have embraced volunteer involvement: ornithology, paleontology, astronomy, and atmospheric sciences. But many other fields, such as botany and herpetology, are beginning to develop successful projects as well. The full conference proceedings, and additional information about designing, implementing, and evaluating citizen science projects, can be found at www.citizenscience.org.
Participants agreed that the field of citizen science is ripe for development. However, as citizen science efforts grow in scope and level of public involvement, the need for innovative tools in database management, scientific analysis, and educational research will be greater. For example, networking technologies and distributed database solutions will be imperative, as will computationally efficient geospatial analysis and imaging techniques. Innovative and rigorous statistical analysis methods will be required to handle the massive amounts of monitoring data that will be collected across vast geographic scales. Systems for ensuring high-quality data through interactive technological and educational techniques will have to be developed. Research on the best ways for people to learn through the citizen science process, and on how that process may differ among different cultures and languages, also will be needed. To fulfill these requirements, expertise from a diversity of science, education, engineering, and other fields must be harnessed in a collaborative, integrated research effort.
Many individuals beyond the authors of this article have contributed extensively to the development of the citizen science program at the Cornell Lab of Ornithology and the techniques described in this article, including Paul Allen, David Bonter, Greg Butcher, Andre Dhondt, Erica Dunn, Jennifer Fee, Daniel Fink, John Fitzpatrick, Tom Fredericks, Jeff Gerbracht, Stefan Hames, Wesley Hochachka, Marshall Iliff, Tim Lenz, Tim Levatich, Karen Purcell, Ron Rohrbaugh, Roger Slothower, Brian Sullivan, Diane Tessaglia-Hymes, and Christopher Wood. Development of the CLO citizen science program has been supported by the National Science Foundation (NSF) under grants ESI-9155700, ESI-9550541, ESI-9627280, ESI-0087760, ESI-0125633, ESI-0242666, ESI-0296149, ITR-0427914, ESI-0540185, DBI-0542868, ESI-0610363, IIS-0612031, and DUE-0734857. Any opinions, findings, and conclusions or recommendations expressed in this manuscript are those of the authors and do not necessarily reflect the views of the NSF. Additional funding has come from the Robert F. and Marilyn H. Schumann Foundation, the Park Foundation, the Leon Levy Foundation, and the Wolf Creek Foundation.