https://orcid.org/0000-0002-2947-6665
Abstract
While museum voucher specimens continue to be the standard for species identifications, biodiversity data are increasingly represented by photographic records from camera traps and amateur naturalists. Some species are easily recognized in these pictures, others are impossible to distinguish. Here we quantify the extent to which 335 terrestrial nonvolant North American mammals can be identified in typical photographs, with and without considering species range maps. We evaluated all pairwise comparisons of species and judged, based on professional opinion, whether they are visually distinguishable in typical pictures from camera traps or the iNaturalist crowdsourced platform on a 4-point scale: (1) always, (2) usually, (3) rarely, or (4) never. Most (96.5%) of the 55,944 pairwise comparisons were ranked as always or usually distinguishable in a photograph, leaving exactly 2,000 pairs of species that can rarely or never be distinguished from typical pictures, primarily within clades such as shrews and small-bodied rodents. Accounting for a species geographic range eliminates many problematic comparisons, such that the average number of difficult or impossible-to-distinguish species pairs from any location was 7.3 when considering all species, or 0.37 when considering only those typically surveyed with camera traps. The greatest diversity of difficult-to-distinguish species was in Arizona and New Mexico, with 57 difficult pairs of species, suggesting the problem scales with overall species diversity. Our results show which species are most readily differentiated by photographic data and which taxa should be identified only to higher taxonomic levels (e.g., genus). Our results are relevant to ecologists, as well as those using artificial intelligence to identify species in photographs, but also serve as a reminder that continued study of mammals through museum vouchers is critical since it is the only way to accurately identify many smaller species, provides a wealth of data unattainable from photographs, and constrains photographic records via accurate range maps. Ongoing specimen voucher collection, in addition to photographs, will become even more important as species ranges change, and photographic evidence alone will not be sufficient to document these dynamics for many species.
The rise of born-digital biodiversity data has led to more and more photographs serving as visual vouchers for species occurrences (Kays et al. 2020). This is especially true on crowdsourced platforms, such as iNaturalist (inaturalist.org), where georeferenced photographs are posted and collectively identified by the user community. If a photograph in iNaturalist is identified to species (or occasionally genus or family) by the community (>2 people), and is georeferenced, time-stamped, and not captive/cultivated, the record is judged as “research grade” and aggregated in larger data portals, including the Global Biodiversity Information Facility (GBIF). The scale of current observations is impressive, with over 3 million photo-vouchered Animalia records from iNaturalist archived at GBIF in 2019, compared to ~209,000 new museum specimens (GBIF.org 2020). Camera traps provide another growing collection of photo-vouchered biodiversity records, primarily for birds and mammals. Recent studies have shown the utility of classifying millions of photographs from thousands of locations across states or countries (Cove et al. 2021; Lasky et al. 2021).
Although there has been extensive discussion—and controversy—about whether photographs should be used as type specimens when describing a new species (Krell and Marshall 2017), there has been little discussion about when these are appropriate for other applications. In particular, some species are easy to identify through photographs, whereas others are impossible because they require examination of detailed morphological characters difficult or impossible to see in photos, color differences missed by nocturnal camera trap pictures, or even genetic evidence (Kays and Wilson 2009). Smaller species—which comprise the majority of mammalian diversity—are also more difficult to photograph, less likely to trigger camera traps, and less likely to have obvious external features useful for identification. Indeed, Potter et al. (2018) found that smaller mammal species were more difficult to distinguish than larger mammals in camera trap pictures from Australia. Yet the importance of accurately identifying mammals from photographs has not been systematically assessed.
The monumental task of identifying species in photographs globally is increasingly being performed by artificial intelligence (AI). AI uses human-identified photographs to train algorithms to make subsequent identifications automatically (Norouzzadeh et al. 2021). The accuracy of these tools has improved with larger training data sets and increasingly sophisticated algorithms (Ahmed et al. 2020), including the use of spatial priors to account for geographic range (Mac Aodha et al. 2019). However, these efforts usually start from the assumption that all species can be distinguished and accurately identified in a photograph. Although we typically do not know if algorithms are using the same physical criteria to recognize species as humans, they are working off the same visual image, and thus would encounter the same challenges humans face when trying to distinguish visually similar species.
Here we quantify how often this assumption is violated by evaluating which species of 335 North American terrestrial mammals could theoretically be identified from typical camera trap or citizen science photographs. We use a 4-point scale based on the consensus expert opinion of the authors that simultaneously considers the characteristics used to distinguish between each species pair and whether those characteristics would be captured in a typical photograph (1) always, (2) usually, (3) rarely, or (4) never (Fig. 1). To evaluate the importance of geography in aiding species identification, we compare geographically naive scores of identification difficulties with maps that plot the overlapping geographic ranges for species pairs judged to be difficult or impossible to identify from photographs. Our results set a baseline for how well we should expect people and AI to identify mammal species and highlight which species and geographic areas can be most confidently studied with photographs.
Criteria used to classify each pair of species as to whether they could be distinguished in a typical camera trap or citizen science photograph always, usually, rarely, or never. Top four pictures credit to eMammal, bottom four to iNaturalist users Dario, Christoph Moning, Ken Kneidel, and John Sankey.
Materials and Methods
We used a list of North American mammals (north of Mexico), and their range maps, from International Union for Conservation of Nature (IUCN 2020) for all analyses. We focused on terrestrial, nonvolant species, and excluded domestic animals and invasive species with small ranges, as these would inflate the overall species numbers unnecessarily, but included wide-ranging, long-established invasives, that is, Rattus rattus, R. norvegicus, Mus musculus, Sus scrofa, and Myocastor coypus. We expanded the list to include every pairwise species comparison, and for each comparison we collectively scored the difficulty of distinguishing them in a photograph as follows: (1) easy with most pictures, (2) possible if the picture captures a distinguishing feature, which should happen frequently, (3) possible if the picture captures a distinguishing feature, which should happen rarely, or (4) impossible (Fig. 1). We did not systematically evaluate photographs, but assessed the overall physical similarity of species pairs using a regional field guide (Kays and Wilson 2009), and based on expert opinions for each taxonomic group from the coauthors on this paper based on their experience working with living animals, with museum specimens, and with photographs. Each author evaluated the species groups for which they had expertise, with each group having at least two experts. The penultimate draft of the matrix was discussed by the authors and any disagreements were resolved by consensus. We considered all species pairs regardless of whether they co-occur, but we take geographic range overlap into account in a secondary analysis (see below).
Our scoring of 1, 2, 3, or 4 presumes a typical picture from a camera trap or citizen scientist without special efforts to capture subtle distinguishing characteristics. This assumption makes our results relevant to the great majority of photo-vouchered data available. However, we note that specialized camera trap setups (e.g., baited, close-focus, white-flash, MCCleery et al. 2014; Herrera et al. 2021) could obtain identifiable photographs for some groups more often than we assume here. Additionally, we did not account for identifiability of some species by experts who know about, and can photograph, a distinguishing characteristic (e.g., while handling an animal). Although we did not specifically examine photographs for this assessment, these judgments of whether a species pair could always, usually, rarely, or never be distinguished were made based on the authors’ professional experience with photos from camera traps and iNaturalist photos. Our approach also inherently considers photo quality, in that our experts took into account factors such as nocturnal species typically being photographed in black and white by camera traps, and many iNaturalist contributors not typically knowing and photographing obscure morphological traits needed to identify some species. Although a more quantitative approach might be possible to evaluate the quality of available photos for particular species pairs (Thornton et al. 2019), this was not practical for the large species list and enormous photo collections we considered. Thus, we consider our approach as a first approximation and encourage future work considering photo quality and accuracy of particular taxon identifications.
Based on the final matrix, we computed a Euclidean distance matrix representing all pairwise differences in species identifiability as input to conduct a hierarchical clustering analysis, using the function hclust in R v4.0.3 (R Core Team 2020). To aid in our goal of visualizing among-cluster distances, we used the complete linkage method for dendrogram construction (the default in hclust). The resulting dendrogram was plotted as a phylo object using functions in the ape v 5.4-1 package in R (Paradis and Schliep 2019). To map species that are appropriate for sampling by camera traps, we removed genera that are typically too small to trigger a camera trap (<90 g, i.e., we removed Neotamias, Dipodomys and smaller, but kept Tamias, Glaucomys and larger) and subterranean species (i.e., gophers and moles). We used body mass data from Soria et al. (2021).
We overlapped range maps of different species groups using North American mammal species ranges downloaded from the IUCN Red List of Threatened Species (IUCN 2020). We created raster layers (50 km × 50 km pixel size) for each individual mammal species and then extracted the areas of overlap for species that were difficult to identify. We created all spatial figures in R version 4.0.3 (10 October 2020) using packages rgdal (version 1.5-23), tidyverse (version 1.3.0), sf, dplyr, raster, and ggplot2 (Wickham et al. 2019; Hijmans et al. 2013; R Core Team 2020).
Results
Most (96.5%) of the pairwise comparisons (Ntotal = 55,944) of the 335 species were considered easily or usually distinguishable from photographs, leaving 2,000 pairs of species that are rarely or never distinguishable from typical pictures (Table 1; Supplementary Data SD1). Most of these difficult comparisons involve shrews, small rodents, or rabbits (Fig. 2A), showing that smaller species and more speciose genera are more likely to be difficult to identify (Fig. 2B). Considering only species likely to be surveyed with camera traps, 98.0% of species pairs should be easily or usually distinguished by photographs (Table 1).
Scores for whether a pair of species could be distinguished by a photograph for pairs of 335 terrestrial North American mammals.
| Pairwise species comparisons (n/%) | ||
|---|---|---|
| Can be distinguished via photograph? | All species | Camera-trappable species |
| Easily | 52,899/94.6% | 9,480/96.0% |
| Usually | 1,046/1.9% | 196/2.0% |
| Rarely | 933/1.7% | 97/1.0% |
| Never | 1,066/1.9% | 97/1.0% |
| Total | 55,944 | 9,870 |
| Pairwise species comparisons (n/%) | ||
|---|---|---|
| Can be distinguished via photograph? | All species | Camera-trappable species |
| Easily | 52,899/94.6% | 9,480/96.0% |
| Usually | 1,046/1.9% | 196/2.0% |
| Rarely | 933/1.7% | 97/1.0% |
| Never | 1,066/1.9% | 97/1.0% |
| Total | 55,944 | 9,870 |
Scores for whether a pair of species could be distinguished by a photograph for pairs of 335 terrestrial North American mammals.
| Pairwise species comparisons (n/%) | ||
|---|---|---|
| Can be distinguished via photograph? | All species | Camera-trappable species |
| Easily | 52,899/94.6% | 9,480/96.0% |
| Usually | 1,046/1.9% | 196/2.0% |
| Rarely | 933/1.7% | 97/1.0% |
| Never | 1,066/1.9% | 97/1.0% |
| Total | 55,944 | 9,870 |
| Pairwise species comparisons (n/%) | ||
|---|---|---|
| Can be distinguished via photograph? | All species | Camera-trappable species |
| Easily | 52,899/94.6% | 9,480/96.0% |
| Usually | 1,046/1.9% | 196/2.0% |
| Rarely | 933/1.7% | 97/1.0% |
| Never | 1,066/1.9% | 97/1.0% |
| Total | 55,944 | 9,870 |
The extent to which genera of terrestrial North American mammals are identified plotted by taxonomy (A) and average body mass (B, exponential trend line r2 0.33, P < 0.0001). Difficulty is measured as the number of species pairs that would be impossible or difficult (class 3 or 4) to identify via photographs.
The clustering analysis provides a visualization of which species groups are most similar in appearance (Fig. 3; Supplementary Data SD2). In most of the shrews and smaller-bodied rodents, species were ranked as being usually identifiable at the generic level, but with species-level identification judged as rarely or never possible from photographs alone. Conversely, higher-level groups of small mammals could always be distinguished from each other including the vole-like animals (small ears, eyes, and short tails), pocket mice (medium ears and eyes, and long tails), deer mice and allies (large eyes and ears), kangaroo rats (large feet and tail, and bipedal posture), shrews (small size, absent or reduced ears, and pointy nose), rats (large size), moles (distinctive shape), and gophers (distinctive shape). The squirrels were separated into nine groups that were judged as always distinguishable: chipmunk-like, unstriped ground squirrels, antelope ground squirrels, spotted ground squirrels, tree squirrels, red squirrels, flying squirrels, marmots, and prairie dogs. However, within each group of squirrels, many species are very difficult to distinguish, especially the chipmunks and unstriped ground squirrels, which include many species for which skeletal or genetic characters are required. The lagomorphs were split into four groups that could be easily distinguished from one another: cottontails, Northern hares, jackrabbits, and pikas. Most (n = 10) of the species within cottontails (Sylvilagus) were judged as indistinguishable in photographs, whereas most of the jackrabbits were ranked as usually identifiable. The carnivores and ungulates included larger species that were easier to identify, including 25 unmistakable species and nine species pairs that were usually identifiable.
Hierarchical clustering diagram for terrestrial North American mammals based on how difficult it is to distinguish a pair of species based on a photograph. The color of branch tips indicates whether a taxon should be always (dark blue), usually (light blue), rarely (pink), or never (red) distinguishable in typical photographs. The large groups of red tips indicate species groups that are practically impossible to tell apart in photographs, whereas the single blue branch tips on the bottom right represent one-of-a-kind species that are unmistakable. Color of nontip branches shows to what extent higher taxonomic groups can be distinguished. Artwork from Kays and Wilson (2009). Not all groups are pictured. All species names are presented in Supplementary Data SD2.
The comparison of all possible species in North America revealed 2,000 pairs of species that would be difficult (n = 934) or impossible (n = 1,066) to identify via photograph, but our secondary mapping exercise shows that accounting for geographic range dramatically reduces this number (Fig. 4). The average number of difficult or impossible (3 or 4) pairs of species from any 50 km × 50 km grid cell was 7.3, ranging from 0 in southern Florida to 57 in the southwestern United States. When considering only larger species likely to be detected by camera traps, the average location had 0.37 difficult or impossible species pairs, ranging from 0 in the southeastern and midwestern states to 9 in portions of the western United States. Of these, the most difficult groups were Sylvilagus and Neotamias (Supplementary Data SD3).
Maps showing the total diversity of terrestrial mammals in North America (A) and the diversity of species that are likely to be detected in camera traps based on body size (B). Map (C) shows how many species overlap geographically in an area that would be difficult or impossible to identify based on a photograph for all mammals, and (D) shows this for those species typically surveyed with camera traps.
Discussion
This is a first-ever assessment of the potential to identify a group of animals from photographs, which is timely given the growing importance of born-digital biodiversity data being identified by either AI or crowdsourcing (Kays et al. 2020). This measure is also relevant for the growing effort to use AI to identify animals in photographs because assessments of algorithm accuracy should account for the fact that species-level identification is likely impossible for some groups. We found 2,000 pairs of North American mammal species that would be rarely or never distinguishable in typical photographs. However, the majority of these difficult species are not known to overlap geographically, such that the highest number of indistinguishable species pairs at any one site was constrained to southeastern Arizona (57 pairs). As expected, most of the confusing species were smaller-bodied mammals. Considering only the larger species, which are typically studied with camera traps, most sites have between 0 and 1 difficult or impossible-to-identify pairs of species. These results are important for those who use photo-based biodiversity data, because even “research-grade” observations of small species from iNaturalist might be overconfident in species identifications, and care should be taken to vet the photographs before using them in spatial modeling.
Our results confirm that photographic vouchers can be useful to identify many larger mammal species, but also emphasize the importance of traditional museum collections and continued biological surveys by experts. For instance, monitoring the distribution and abundance of large mammals, especially carnivores and ungulates, with camera trap and crowdsourced data should be possible at the species level. However, traditional collections are necessary for smaller species for accurate species identification and to map the potentially dynamic geographic ranges of such species in the Anthropocene (in addition to the wealth of additional “extended” data voucher specimens provide). AI should be broadly useful to classify photographs of most North American mammal species, although the accuracy of both human and machine identifications still needs to be guided by extensive professional training data sets. Additionally, our results do highlight some areas of the western United States where extra caution is needed even for medium- to large-bodied species, such as rabbits and hares.
Perhaps the most significant challenges with camera traps are the groups that are large enough to trigger camera traps, but difficult to identify to the species level based on photographs alone. Particularly problematic groups are the cottontails (Sylvilagus) and woodrats (Neotoma) in the southern United States and arid Mexico. The western chipmunks (Neotamias) average smaller than our weight cutoff (90 g) to be considered a target of camera traps, but are still common in some camera trap data sets (Cove et al. 2021). Most of these confusing pairs can be distinguished based on careful consideration of geographic range (Supplementary Data SD3), but many of these boundaries are shifting as a result of climate change (e.g., Moritz et al. 2008). In most of the difficult-to-identify species, individuals are recognizable at the generic level. This level of identification could still be useful for spatial modeling if the records are used as a covariate for distribution models of other species (e.g., a measure of prey or competitor abundance) or quantifying functional richness or ecosystem processes. Studies focusing on groups such as chipmunks might be able to improve identification with camera traps designed to take close-up color photographs (Gracanin et al. 2019) that can capture species-specific pelage markers or videos that include audio to capture their unique vocalizations (Gannon and Lawlor 1989; Kays and Wilson 2009). Indeed, recent work by McKibben and Frey (2021) show how one pair of similar chipmunk species, ranked 3 (rarely distinguishable in typical pictures) in our study, can reliably be distinguished in camera trap pictures with special camera protocols and training of staff on distinguishing characters from museum specimens.
The use of AI to identify species in images is a rapidly growing field and is now integrated into many platforms for different uses. For example, AI is used by Wildlife Insights (https://www.wildlifeinsights.org) to speed the processing of camera trap images (Ahumada et al. 2019) and by iNaturalist to suggest species-level and genus-level IDs (Van Horn et al. 2018; Ueda 2020). Creating these tools requires the use of annotated images to train algorithms to identify species, followed by an assessment of their accuracy by testing on a second, independent set of annotated images (He et al. 2016). Our results show the importance of not restricting these tools to species-level identifications, which are impossible for some taxa, but to report higher taxonomic-level results for the most difficult groups. We recommend that these algorithms group species that can never be identified by most photographs (class 4 in our ranking) at the generic level to avoid compounding dubious species-level identifications across other biodiversity data aggregators. Accounting for the underlying possibility of distinguishing pairs of species, identifying some groups only to the generic level, and including geographic information about species distribution (Mac Aodha et al. 2019) should result in improved algorithmic performance and realistic metrics about their accuracy.
Our analyses emphasize that photo-vouchered biodiversity records are a supplement to, but not a replacement of, traditional museum specimen collections. Physical specimens have myriad other uses unavailable from photographs, including genetic, isotopic, disease, and detailed morphological studies (Dunnum et al. 2018; Thompson et al. 2021). As expected, our results show that many smaller mammals are impossible or difficult to identify by photograph, making species-specific research on those taxa questionable from photographs alone. Our results also suggest that geographic range can, in many cases, help distinguish similar species, but this depends on range maps created from densely sampled physical vouchers that allow confident species identification and range delimitation. Detecting range shifts in response to global change for species groups that are difficult to identify thus requires continued active collection and museum vouchering, especially at long-term monitoring sites, which would pay extra dividends by also enabling updated range-assisted identification of some photo-vouchered data. Additionally, for species that can be identified by photographs, the rich data available from iNaturalist could highlight areas of species expansion that should be priorities for additional trapping surveys. Such “ground proofing” to obtain voucher specimens will verify changing distributions, and allow for the study of both ecological and evolutionary dynamics along range peripheries. Finally, species description and geographic range delimitation is still an active process for many taxonomic groups (including mammals) in many parts of the world (Burgin et al. 2018), and physical specimens will be required to voucher these discoveries and highlight distinguishing phenotypic traits that may or may not be evident in photographic data.
Our results are subjective in that they represent a consensus of the authors on the relative difficulty of distinguishing pairs of species from photographs alone, given our knowledge of mammals, their external characters, and our experiences working with pictures typical of camera traps or crowdsourcing. Although it is conceivable that a different group of experts would come to slightly different conclusions for some species pairs, we think that these results are robust in identifying the most problematic groups, the spatial distributions of these pairs, and the broad patterns across taxa. Despite these limitations, we hope others will build on this work to empirically explore the challenges of identifying particular groups of interest (e.g., Gooliaff and Hodges 2018, 2019; Thornton et al. 2019) with the community goal of increasing overall accuracy of identifications in digital biodiversity data sets at a global scale.
Supplementary Data
Supplementary data are available at Journal of Mammalogy online.
Supplementary Data SD1.—Matrix with ranks of how difficult all pairs of 335 species of terrestrial North American mammals are to distinguish from each other based on typical photographs from camera traps or citizen science projects. Scores show if a pair of species would always (score = 1), usually (2), rarely (3), or never (4) be distinguishable in photographs.
Supplementary Data SD2.—Hierarchical clustering diagram for terrestrial North American mammals based on how difficult it is to distinguish a pair of species based on a photograph with all species names provided for Fig. 3. The color of branch tips indicates whether a taxon should be always (dark blue), usually (light blue), rarely (pink), or never (red) distinguishable in typical photographs. The large groups of red tips indicate species groups that are practically impossible to tell apart in photographs, whereas the single blue branch tips on the bottom right represent one-of-a-kind species that are unmistakable. Color of nontip branches show to what extent higher taxonomic groups can be distinguished. Species names of the same genus are colored the same.
Supplementary Data SD3.—Maps showing the range overlap for species of mammals from North America within the more speciose taxa that are difficult (score 3 or 4) to identify via photographs.
Acknowledgments
Thanks to all the dedicated scientists and volunteers who have contributed specimens and photographs to public repositories that make this kind of work possible, and the results relevant.
