https://orcid.org/0000-0002-9375-843X
Abstract
Observer error is an unavoidable aspect of vegetation surveys involving human observers. We quantified four components of interobserver error associated with long-term monitoring of prairie vegetation: overlooking error, misidentification error, cautious error and estimation error. We also evaluated the association of plot size with pseudoturnover due to observer error, and how documented pseudochanges in species composition and abundance compared with recorded changes in the vegetation over a 4-year interval.
This study was conducted at Tallgrass Prairie National Preserve, Kansas. Monitoring sites contained 10 plots; each plot consisted of a series of four nested frames (0.01, 0.1, 1 and 10 m2). The herbaceous species present were recorded in each of the nested frames, and foliar cover was visually estimated within seven cover categories at the 10 m2 spatial scale only. Three hundred total plots (30 sites) were surveyed, and 28 plots selected at random were resurveyed to assess observer error. Four surveyors worked in teams of two.
At the 10 m2 spatial scale, pseudoturnover resulting from overlooking error averaged 18.6%, compared with 1.4% resulting from misidentification error and 0.6% resulting from cautious error. Pseudoturnover resulting from overlooking error increased as plot size decreased, although relocation error likely played a role. Recorded change in species composition over a 4-year interval (excluding potential misidentification error and cautious error) was 30.7%, which encompassed both pseudoturnover due to overlooking error and actual change. Given a documented overlooking error rate of 18.6%, this suggests the actual change for the 4-year period was only 12.1%. For estimation error, 26.2% of the time a different cover class was recorded. Over the 4-year interval, 46.9% of all records revealed different cover classes, suggesting that 56% of the records of change in cover between the two time periods were due to observer error.
摘要
人类观测误差是植被测量中不可避免的一个问题。我们量化了与高草草原植被长期监测相关的观测者间误差的四个组成部分：忽略误 差、误识别误差、谨慎误差和估计误差。由于观察者会产生误差，我们还评估了地块大小与伪周转率的关系，以及对比了物种组成和丰度的伪变化与四年间植被变化之间的关系。这项研究是在美国堪萨斯州的高草草原国家保护区进行的。监测点包括10个地块，每个地块由一系列的四个嵌套框架(0.01, 0.1, 1和10 m2)组成。在每个嵌套框架中记录了所有的草本物种，并且在10 m2的空间尺度下，视觉估计了7个覆盖类别内的叶面覆盖。总共调查了300个地块(30个地点)，并随机选择28个地块重新进行测量以评估观测者的误差。所有的调查由四名观测者分两组完成。研究结果表明，在10 m2空间尺度上，由忽略误差引起的伪周转率平均为18.6%，而由误识别误差和谨慎误差引起的伪周转率平均值分别为1.4%和0.6%。尽管由重新定位引起的误差可能也起一定的作用，由忽略误差导致的伪周转率随样地面积的减小而增 加。物种组成在四年期间的变化(排除潜在的误识别误差和谨慎误差)为30.7%，其中包括由忽略误差和实际变化引起的伪周转率。18.6%的忽略误差表明四年期间的实际变化只有12.1%。对于估计误差，26.2%会记录为不同的覆盖等级。在四年的时间内，46.9%的记录显示了不同的覆盖等级，这表明两个时间段间覆盖率变化的56%是由于观测者误差造成的。
INTRODUCTION
Vegetation sampling involving human observers is characterized by an unavoidable degree of observer error. The extent to which this occurs has been summarized in a review article (Morrison 2016), and further elucidated in more recent contributions (Dennett et al. 2018; Futschik et al. 2020; Groom and Whild 2017; Mason et al. 2018; Morrison 2017; Morrison et al. 2020). Three types of observer error have commonly been described: (i) overlooking error, which occurs when species actually present are not observed, (ii) misidentification error, which occurs when species are not correctly identified and (iii) estimation error, which occurs when abundances are not accurately estimated. In addition to these three, a fourth type, cautious error, may occur if different observers identify the same specimen to different taxonomic levels (i.e. species vs. genus), as this could also be interpreted as a discrepancy in comparing species lists (Groom and Whild 2017; Morrison et al. 2020).
Introduced by Lynch and Johnson (1974) in the island biogeographic literature, the term pseudoturnover refers to apparent, as opposed to actual, changes in species composition due to observer error. Following Nilsson and Nilsson’s (1982, 1983, 1985) quantification of observer error on island floras using this concept, pseudoturnover has become commonly used to document observer error in vegetation studies (Morrison 2016). Three of the four types of error listed above (all except estimation error) contribute to pseudoturnover.
Most observer errors in recording species composition that have been quantified in the literature are ‘false negative’ errors, and reflect the failure to correctly record species that are actually present. This is because true species compositions are rarely known with certainty, and studies of observer error have focused on precision among observer estimates, rather than accuracy of such estimates, which would require knowledge of true values. It is likely that ‘false positive’ errors, in which species that are not actually present are recorded, may also be common (Groom and Whild 2017), although this is rarely evaluated because of the necessity for knowledge of actual species composition.
Although some aspects of observer error have been quantified in numerous vegetation sampling studies (Morrison 2016), other aspects are still poorly known. For example, overall pseudoturnover has often been documented, yet few investigators have attempted to isolate the individual components of pseudoturnover (i.e. overlooking error, misidentification error and cautious error). Plot size may vary greatly in vegetation studies, yet few investigators have compared the observer error associated with different plot sizes. In a study of islands, Nilsson and Nilsson (1985) found a negative association between pseudoturnover and insular area. More recent studies have reported ambivalent effects of plot size (Archaux et al. 2007; Ringvall et al. 2005; Vittoz and Guisan 2007), although Morrison et al. (2020) found overlooking error to be 37% greater for smaller plots, at least for herbaceous vegetation.
For long-term monitoring studies, an important question is how the magnitude of observer error compares with the recorded degree of change. Few studies have addressed this issue. Burg et al. (2015) reported that species turnover over a century was three times greater than mean pseudoturnover between observers. Futschik et al. (2020) found that recorded change in species turnover and cover were significantly greater than pseudochanges due to observer error only for observation periods of a decade or longer.
To further investigate these issues, we quantified the observer error associated with long-term monitoring of tallgrass prairie vegetation on a preserve in Kansas. We pose the following questions: (i) How much observer error in estimating species composition and abundance characterizes sampling of this type of vegetation? (ii) What are the contributions of overlooking error, misidentification error and cautious error to overall pseudoturnover? (iii) What is the association of plot size with estimated pseudoturnover? (iv) What are the mechanisms underlying the documented error? and (v) How do documented rates of pseudoturnover compare with recorded changes in the vegetation over a 4-year interval?
MATERIALS AND METHODS
Study area
This study was conducted at Tallgrass Prairie National Preserve near Strong City, KS (712311.373 easting, 4257485.335 northing; Zone 14N, UTM NAD83), USA. This 4409 ha preserve is within the Flint Hills upland physiographic region, and is primarily covered by tallgrass prairie vegetation, dominated by big bluestem (Andropogon gerardii) and little bluestem (Schizachyrium scoparium). Relief ranges from 335 to 457 m above sea level. Formerly Spring Hill Ranch, the preserve has been continuously grazed by cattle for over 120 years. Bison were reintroduced in 2009 to a section of the preserve. Prescribed fires have occurred commonly in the spring, and since 2014 also in autumn. Fire return intervals for prairie units averaged 1–3 years. As part of a long-term monitoring program at this preserve, the vegetation is surveyed at 30 sites every 4 years. Monitoring sites were established based on a random selection of vertices of a stratified systematic grid with a random bearing (James et al. 2009).
Field surveys
Methodology
Monitoring sites were 50 m × 20 m (0.1 ha) in size with two transects bounding the site on the 50-m sides (Fig. 1). The ends of both transects were permanently marked with rebar stakes and metal tags indicating site identity and the beginning and end of each transect. A 50 m tape was stretched between the two stakes. All data were collected within 10 plots spaced evenly along the transects. Each plot consisted of a series of four nested frames differing in size by an order of magnitude (0.01, 0.1, 1 and 10 m2). Plots were centered at 10, 20, 30, 40 and 50 m along the upper transect and at 0, 10, 20, 30 and 40 m along the lower transect (Fig. 1). The largest sampling frame (10 m2) was centered directly on the tape over the relevant meter mark. The smaller frames were aligned with the perimeter of the largest frame along the tape closest to the start of the transect (Supplementary Fig. S1). A more detailed description is available in James et al. (2009).
Sampling design for each monitoring site (see text for detailed description).
The herbaceous species present were recorded in each of the nested frames, beginning with the smallest size frame and progressing to the largest. Identifications were made to species level whenever possible. Tree species were only encountered as regeneration and were not included. Foliar cover was visually estimated at the 10 m2 spatial scale only, within seven classes based on a modified Daubenmire (1959) cover class system: 1 = 0%–0.9%, 2 = 1%–5%, 3 = 6%–25%, 4 = 26%–50%, 5 = 51%–75%, 6 = 76%–95% and 7 = 96%–100%. Only species rooted in the plot were recorded and included in estimates of foliar cover. For purposes of long-term monitoring, the primary information obtained from this design includes species composition and frequency (% of plots occupied) at all four spatial scales, and foliar cover at the 10 m2 spatial scale only. Information on frequency at different spatial scales was desired because frequencies vary among spatial scales depending upon the abundance of a species, and the power to detect change is maximized at ~50% frequency for most species (Heywood and DeBacker 2007).
Observer experience and training
Four surveyors visited each site, working in teams of two. Each team surveyed one of the transects of five plots at each site. A team consisting of an observer and a recorder compiled lists of species composition and estimated cover class. The observer primarily identified plants and estimated cover, but the recorder was encouraged to contribute observations or question the data provided by the observer. The data sheet included a list of species observed in prior monitoring events, but no cover class estimates. This list aided recorders in checking for potential missed species, and as a reference to aid observers with species identification when needed. Teams were allowed to consult with each other.
Three of the surveyors were professional botanists. Two botanists (Botanists 1 and 2) had master’s degrees that emphasized botanical work and were considered to be experts with 23 and 17 years of botanical field experience, especially in prairies of the Great Plains. The third botanist (Botanist 3) was proficient with a master’s degree in natural resources with emphasis in forestry, including 9 years of botanical field experience. The fourth surveyor (Recorder 1) was inexperienced at botany and served only as a recorder. Such variability in experience was typical of teams involved with this long-term monitoring effort.
The composition of teams changed over the course of sampling different sites, but never between sampling the same plots at a site. Expert Botanists 1 and 2 were always on different teams and worked as observers; occasionally Botanist 3 provided identifications under the supervision of Botanist 1 or 2. To evaluate potential observer bias in cover estimates (see below), we compared teams consisting of the following: Team A = Botanist 2 and Recorder 1, Team B = Botanists 1 and 3; Team C = Botanists 2 and 3, Team D = Botanist 1 and Recorder 1.
Prior to sampling, all surveyors practiced estimating cover classes on three or four occasions, using paper cutouts of varying sizes, shapes and colors that were placed in the sampling frames to simulate vegetation. Surveyors also practiced estimating plant cover of lawn species during training sessions. All surveyors studied problematic plants via specimens, images and notes; and also practiced sampling of a mock site within a restored prairie.
Resurveys
Each team surveyed all five plots of one of the transects. After completion of surveys at all 10 plots at a site, 2 plots—one on each transect—were randomly selected to be resurveyed. Each team then switched transects and sampled the randomly determined plots. The transect tapes were not moved between surveys, but the plot frames were removed by the first team and repositioned by the second team. Resurveys were conducted in exactly the same manner as the initial surveys, except that teams were not allowed to consult each other.
A total of 30 sites (each with 10 plots) were surveyed as part of the long-term monitoring project. To assess observer error, we resurveyed 10% of the plots (out of 300 total). To accomplish this, half of the sites were randomly selected for quantification of observer error, and one plot from each transect at those sites was randomly selected to be resurveyed. Methodological problems resulted in 2 plots from 1 site being excluded from analyses, leaving a total of 28 resurveyed plots for comparison.
All surveys and resurveys occurred during 4–14 June 2018. To compare observer error with recorded change over a longer time interval, we used data from surveys conducted at these same sites from 11–16 June 2014. The same 28 plots were compared, using data from the initial set of surveys in 2018. The four surveyors involved in the 2014 surveys were different than those in the 2018 surveys, although the two lead botanists had similar levels of experience (15–20 years), and the same protocol was followed (James et al. 2009). Surveys have been conducted in prior years at this study area, although the methodology changed in 2014 so that prior data are not directly comparable.
Analyses
Pseudoturnover
Pseudoturnover was calculated as in Nilsson and Nilsson (1985):
where A is the number of species recorded exclusively by observer 1, B is the number of species recorded exclusively by observer 2, SA is the total number of species recorded by observer 1 and SB is the total number of species recorded by observer 2. Three types of errors could have contributed to pseudoturnover: (i) overlooking errors, in which a species was recorded by one observer but not the other, (ii) misidentification errors, in which the same specimen was apparently recorded as a different species by the two observers and (iii) cautious errors, in which one observer identified a plant to species and the other identified it only to genus.
We attempted to differentiate misidentification errors from overlooking errors by examining all survey lists and determining post hoc pairs of species with similar morphologies that were challenging to identify and may have been confused. Whenever one of the pair of species appeared in one survey and the other of the pair appeared in the other survey of the same plot, this was deemed a likely misidentification error. As in Morrison et al. (2020), we assumed that only one of the species was actually present, and in one (or potentially both) of the surveys it was misidentified. The alternative was that both species were actually present, but each was seen in only one of the surveys. We consider the former explanation to be the more parsimonious approach, as it requires only one (or potentially two) errors of misidentification, as opposed to two errors of overlooking. This may not have always been true, however, and thus this method produces a more conservative estimate of overlooking error and a more liberal estimate of misidentification error. Cautious error was assigned when a specimen was identified only to genus in one survey and an apparently matching species-level identification was present in the other survey from the same plot.
Estimation errors
To evaluate estimation errors, all cover class estimates in which the same plant apparently was recorded in both surveys were compared. This included all cases in which the same species was recorded in both surveys, cases in which misidentification error was presumed to occur and the relevant records could be associated, and cases of cautious error. Over all 28 plots there were 503 such cases. Differences in cover estimates were also evaluated for surveys conducted in 2014 compared with 2018. Again, all cover class estimates in which the same taxa apparently were recorded in both surveys were compared. Over all 28 plots there were 384 such cases.
Statistics
The following statistical analyses were performed: (i) a chi-square goodness-of-fit test to evaluate whether the distributions of cover class estimates differed among overlooked and non-overlooked plants; (ii) simple linear regressions to evaluate the relationship between species richness in a plot and overlooking rate, and between time of day (as a proxy for fatigue) and overall pseudoturnover and (iii) repeated measures analysis of variance followed by pairwise comparisons by the Bonferroni method to compare pseudoturnover estimates among spatial scales. All statistical analyses were performed with SPSS version 26 (IBM Corporation 2019).
RESULTS
Pseudoturnover
Observer error
Across all 28 plots, there was a cumulative total of 658 records at the 10 m2 spatial scale from the first set of surveys, and 662 total records from the second set of surveys. The same average number of species (22) was recorded in both the first and second surveys at the 10 m2 spatial scale (Table 1). The average numbers of species recorded from the three smaller spatial scales were also extremely similar, varying by at most 5% (i.e. 6.0 vs. 6.3 at the 0.1 m2 scale) (Table 1).
Number of species recorded, overlooked and cumulative total number of unique species per plot (mean ± SD) by spatial scale in 2018
| Plot scale | Species number recorded | Overlooked species | Cumulative unique species | ||
|---|---|---|---|---|---|
| First survey | Second survey | First survey | Second survey | ||
| 10 m2 | 22.0 ± 5.6 | 22.0 ± 4.7 | 4.1 ± 1.7 | 4.0 ± 2.2 | 26.0 ± 5.5 |
| 1 m2 | 12.6 ± 3.6 | 13.0 ± 3.9 | 3.1 ± 2.0 | 2.8 ± 1.3 | 15.7 ± 4.3 |
| 0.1 m2 | 6.0 ± 2.3 | 6.3 ± 2.4 | 1.9 ± 1.6 | 1.6 ± 1.3 | 7.9 ± 2.5 |
| 0.01 m2 | 2.6 ± 1.0 | 2.5 ± 0.8 | 1.1 ± 1.0 | 1.2 ± 0.8 | 3.6 ± 1.1 |
| Plot scale | Species number recorded | Overlooked species | Cumulative unique species | ||
|---|---|---|---|---|---|
| First survey | Second survey | First survey | Second survey | ||
| 10 m2 | 22.0 ± 5.6 | 22.0 ± 4.7 | 4.1 ± 1.7 | 4.0 ± 2.2 | 26.0 ± 5.5 |
| 1 m2 | 12.6 ± 3.6 | 13.0 ± 3.9 | 3.1 ± 2.0 | 2.8 ± 1.3 | 15.7 ± 4.3 |
| 0.1 m2 | 6.0 ± 2.3 | 6.3 ± 2.4 | 1.9 ± 1.6 | 1.6 ± 1.3 | 7.9 ± 2.5 |
| 0.01 m2 | 2.6 ± 1.0 | 2.5 ± 0.8 | 1.1 ± 1.0 | 1.2 ± 0.8 | 3.6 ± 1.1 |
Number of species recorded, overlooked and cumulative total number of unique species per plot (mean ± SD) by spatial scale in 2018
| Plot scale | Species number recorded | Overlooked species | Cumulative unique species | ||
|---|---|---|---|---|---|
| First survey | Second survey | First survey | Second survey | ||
| 10 m2 | 22.0 ± 5.6 | 22.0 ± 4.7 | 4.1 ± 1.7 | 4.0 ± 2.2 | 26.0 ± 5.5 |
| 1 m2 | 12.6 ± 3.6 | 13.0 ± 3.9 | 3.1 ± 2.0 | 2.8 ± 1.3 | 15.7 ± 4.3 |
| 0.1 m2 | 6.0 ± 2.3 | 6.3 ± 2.4 | 1.9 ± 1.6 | 1.6 ± 1.3 | 7.9 ± 2.5 |
| 0.01 m2 | 2.6 ± 1.0 | 2.5 ± 0.8 | 1.1 ± 1.0 | 1.2 ± 0.8 | 3.6 ± 1.1 |
| Plot scale | Species number recorded | Overlooked species | Cumulative unique species | ||
|---|---|---|---|---|---|
| First survey | Second survey | First survey | Second survey | ||
| 10 m2 | 22.0 ± 5.6 | 22.0 ± 4.7 | 4.1 ± 1.7 | 4.0 ± 2.2 | 26.0 ± 5.5 |
| 1 m2 | 12.6 ± 3.6 | 13.0 ± 3.9 | 3.1 ± 2.0 | 2.8 ± 1.3 | 15.7 ± 4.3 |
| 0.1 m2 | 6.0 ± 2.3 | 6.3 ± 2.4 | 1.9 ± 1.6 | 1.6 ± 1.3 | 7.9 ± 2.5 |
| 0.01 m2 | 2.6 ± 1.0 | 2.5 ± 0.8 | 1.1 ± 1.0 | 1.2 ± 0.8 | 3.6 ± 1.1 |
Despite the fact that the same or very similar numbers of species were recorded during both surveys, examination of lists revealed unique species to be present in both surveys. At the 10 m2 spatial scale, an average of ~ four unique species were found during each survey (Table 1). Pseudoturnover resulting from overlooking error averaged 18.6% at the 10 m2 spatial scale (Table 2). Comparing between species lists, unique species were found at all spatial scales (Table 1). Pseudoturnover resulting from overlooking error increased as plot size decreased, and averaged 46.9% at the smallest plot size (Table 2). There was no effect of species richness on overlooking error at any of the spatial scales (linear regressions; all P >> 0.05).
Total pseudoturnover and individual components (mean ± SD, in %) by spatial scale
| Plot scale | Total pseudoturnover | Pseudoturnover components | ||
|---|---|---|---|---|
| Overlooking | Misidentification | Cautious | ||
| 10 m2 | 20.5 ± 6.4A | 18.6 ± 5.7 | 1.4 ± 2.3 | 0.6 ± 2.0 |
| 1 m2 | 25.3 ± 10.2A,B | 23.5 ± 8.9 | 0.9 ± 2.3 | 0.9 ± 3.5 |
| 0.1 m2 | 31.7 ± 19.3B | 30.0 ± 18.0 | 1.1 ± 4.0 | 0.6 ± 3.4 |
| 0.01 m2 | 48.7 ± 31.3C | 46.9 ± 31.1 | 1.8 ± 9.5 | 0 |
| Plot scale | Total pseudoturnover | Pseudoturnover components | ||
|---|---|---|---|---|
| Overlooking | Misidentification | Cautious | ||
| 10 m2 | 20.5 ± 6.4A | 18.6 ± 5.7 | 1.4 ± 2.3 | 0.6 ± 2.0 |
| 1 m2 | 25.3 ± 10.2A,B | 23.5 ± 8.9 | 0.9 ± 2.3 | 0.9 ± 3.5 |
| 0.1 m2 | 31.7 ± 19.3B | 30.0 ± 18.0 | 1.1 ± 4.0 | 0.6 ± 3.4 |
| 0.01 m2 | 48.7 ± 31.3C | 46.9 ± 31.1 | 1.8 ± 9.5 | 0 |
Total pseudoturnover values with the same letters indicate they were not significantly different based on pairwise comparisons using the Bonferroni method following a repeated measures analysis of variance.
Total pseudoturnover and individual components (mean ± SD, in %) by spatial scale
| Plot scale | Total pseudoturnover | Pseudoturnover components | ||
|---|---|---|---|---|
| Overlooking | Misidentification | Cautious | ||
| 10 m2 | 20.5 ± 6.4A | 18.6 ± 5.7 | 1.4 ± 2.3 | 0.6 ± 2.0 |
| 1 m2 | 25.3 ± 10.2A,B | 23.5 ± 8.9 | 0.9 ± 2.3 | 0.9 ± 3.5 |
| 0.1 m2 | 31.7 ± 19.3B | 30.0 ± 18.0 | 1.1 ± 4.0 | 0.6 ± 3.4 |
| 0.01 m2 | 48.7 ± 31.3C | 46.9 ± 31.1 | 1.8 ± 9.5 | 0 |
| Plot scale | Total pseudoturnover | Pseudoturnover components | ||
|---|---|---|---|---|
| Overlooking | Misidentification | Cautious | ||
| 10 m2 | 20.5 ± 6.4A | 18.6 ± 5.7 | 1.4 ± 2.3 | 0.6 ± 2.0 |
| 1 m2 | 25.3 ± 10.2A,B | 23.5 ± 8.9 | 0.9 ± 2.3 | 0.9 ± 3.5 |
| 0.1 m2 | 31.7 ± 19.3B | 30.0 ± 18.0 | 1.1 ± 4.0 | 0.6 ± 3.4 |
| 0.01 m2 | 48.7 ± 31.3C | 46.9 ± 31.1 | 1.8 ± 9.5 | 0 |
Total pseudoturnover values with the same letters indicate they were not significantly different based on pairwise comparisons using the Bonferroni method following a repeated measures analysis of variance.
Plants that were overlooked were usually (86% of the time) estimated to be in the lowest cover class (Fig. 2). The majority of records overall were estimated to be in this class, although plants that were not overlooked (i.e. recorded in both surveys) were estimated to be in the lowest cover class only 61% of the time (Fig. 2). The distributions of overlooked and recorded plants as a function of cover class were significantly different (chi-square goodness-of-fit test, chi-square = 23.6, P < 0.001).
Frequency of plants found in both 2018 surveys compared with those found in only one survey, by cover class (1 = 0%–0.9%, 2 = 1%–5%, 3 = 6%–25%, 4 = 26%–50%, 5 = 51%–75%).
Pseudoturnover resulting from misidentification error averaged 1.4% at the 10 m2 spatial scale (Table 2). Misidentifications included five congeneric species pairs (Asclepias viridis/viridiflora, Euphorbia corollate/spathulate, Physalis pumila/virginiana, Ratibida pinnata/columnifera and Symphyotrichum ericoides/oblongifolium) and two species within different genera that were confusing (Desmodium sessilifolium/Lespedeza capitata). Misidentification error did not vary greatly with spatial scale, and ranged from 0.9 to 1.8 (Table 2).
Pseudoturnover resulting from cautious error averaged 0.6% at the 10 m2 spatial scale (Table 2). Two genera were involved: Ceanothus, which was sometimes identified as C. herbaceous, and Liatris, which was sometimes identified as L. punctata. Like misidentification error, cautious error did not vary greatly with spatial scale, ranging from 0 to 0.9 (Table 2).
Considering all three sources of error, total pseudoturnover ranged from an average of 20.5 at the 10 m2 spatial scale to an average of 48.7 at the 0.01 m2 spatial scale (Table 2). There was no significant relationship between time of day and total pseudoturnover at the 10 m2 spatial scale (linear regression; P = 0.42).
Observed changes over a 4-year interval
Total observed change in species composition between 2014 and 2018 was 39.1% ± 9.5% (at the 10 m2 spatial scale). Likely misidentification error was 2.6% ± 3.0% and cautious error was 5.8% ± 3.2%, resulting in a difference of 30.7% ± 9.8%, which encompasses both pseudoturnover due to overlooking error and actual change.
Misidentification error was slightly higher for the 2014–18 interval compared with the observer error recorded in 2018: 2.6 compared with 1.4 (both at the 10 m2 spatial scale). Some different species pairs were deemed to be involved in the 2014–18 interval. In addition to P. pumila/virginiana and S. ericoides/oblongifolium which were misidentified in the 2018 observer error surveys, Bouteloua gracilis/hirsute and Solidago missouriensis/canadensis were implicated. Three pairs of species in different genera were also considered to represent misidentification error: Achillea millefolium/Hymenopappus scabiosaeus, Psoralidium tenuiflorum/Pediomelum esculentum and Verbena simplex/Brickellia eupatorioides.
Cautious error was substantially higher for the 2014–18 interval compared with the observer error recorded in 2018: 5.8 compared with 0.6 (at the 10 m2 spatial scale). In addition to the two genera identified in the 2018 observer error surveys, three additional taxa were implicated for the 2014–18 interval: Dichanthelium which was sometimes identified as D. oligosanthes, Tragia which was sometimes identified as T. ramose and Oxalis, which was sometimes identified as O. violacea. In all instances of cautious error except one, the more cautious identification (genus level only) occurred in 2014.
Estimation error
Most species were relatively rare. Out of 503 cases in which the same plants were actually or apparently recorded in both surveys in 2018 (i.e. 1006 cover estimates), 60.8% of the estimates were class 1 (0%–0.9%), 20.9% were class 2 (1%–5%), 13.0% were class 3 (6%–25%) and only 5.3% were either class 4 (26%–50%) or 5 (51%–75%). Agreement between surveys tended to be greater for the lower cover classes. Agreement was 80.5% for a class 1 (average) estimate, 60% for a class 2 (average) estimate, 71.7% for a class 3 (average) estimate and 16.7% for a class 4 (average) estimate. (Averages were rounded down; e.g. a class 1 estimate in one survey and class 2 estimate in another were considered to be class 1 average for the above summaries.)
Out of these 503 cases in which the same plants were recorded in both surveys, in 369 of the cases (73.7%) the same cover class was recorded (Table 3). In 113 cases (22.5%) cover estimates differed by 1 class, in 17 cases (3.4%) cover estimates differed by 2 classes, and in 2 cases (0.4%) cover estimates differed by 3 classes. Overall, 12.9% of all estimates were greater in the second survey and 13.3% were greater in the first survey.
Differences in cover class estimates between surveys for observer error study (n = 503) and recorded changes from 2014 to 2018 (n = 384), for all records that could be associated in both surveys
| Difference between surveys (number of cover classes) | Percentage differences | |
|---|---|---|
| Observer error (2018) | Recorded change (2014–18) | |
| −3 | 0.20 | 0.26 |
| −2 | 0.99 | 4.94 |
| −1 | 11.73 | 20.83 |
| 0 | 73.76 | 53.12 |
| +1 | 10.74 | 16.15 |
| +2 | 2.39 | 3.91 |
| +3 | 0.20 | 0.78 |
| Difference between surveys (number of cover classes) | Percentage differences | |
|---|---|---|
| Observer error (2018) | Recorded change (2014–18) | |
| −3 | 0.20 | 0.26 |
| −2 | 0.99 | 4.94 |
| −1 | 11.73 | 20.83 |
| 0 | 73.76 | 53.12 |
| +1 | 10.74 | 16.15 |
| +2 | 2.39 | 3.91 |
| +3 | 0.20 | 0.78 |
Differences in cover class estimates between surveys for observer error study (n = 503) and recorded changes from 2014 to 2018 (n = 384), for all records that could be associated in both surveys
| Difference between surveys (number of cover classes) | Percentage differences | |
|---|---|---|
| Observer error (2018) | Recorded change (2014–18) | |
| −3 | 0.20 | 0.26 |
| −2 | 0.99 | 4.94 |
| −1 | 11.73 | 20.83 |
| 0 | 73.76 | 53.12 |
| +1 | 10.74 | 16.15 |
| +2 | 2.39 | 3.91 |
| +3 | 0.20 | 0.78 |
| Difference between surveys (number of cover classes) | Percentage differences | |
|---|---|---|
| Observer error (2018) | Recorded change (2014–18) | |
| −3 | 0.20 | 0.26 |
| −2 | 0.99 | 4.94 |
| −1 | 11.73 | 20.83 |
| 0 | 73.76 | 53.12 |
| +1 | 10.74 | 16.15 |
| +2 | 2.39 | 3.91 |
| +3 | 0.20 | 0.78 |
Considering cover estimates based on differences between Team A and B, 26.8% of Team B’s estimates differed by at least one class compared with Team A. Exactly 50% of differing estimates by Team B were greater, and 50% less than Team A. Considering cover estimates based on differences between Team C and Team D, 28.6% of Team D’s estimates differed by at least one class compared with Team C. Compared with Team C, 52.5% of differing estimates by Team D were greater and 47.5% were less.
Overall, 26.2% of all paired estimates of cover class varied from the first survey to the second in 2018 (Table 3), which obviously reflected observer error. Comparing this to the apparent changes from 2014 to 2018, overall 46.9% of all records that we could match up between the two time periods (n = 384) specified different cover classes. Thus, it can be inferred that 56% of the records of change in cover between the two time periods were due to observer error.
DISCUSSION
Comparison of pseudoturnover rates with other studies
In other studies of interobserver error, pseudoturnover has ranged from 10% to 36% (Mason et al. 2018; Morrison 2016; Morrison et al. 2020). These rates usually included both overlooking and misidentification error, as few other studies have attempted to separate different components of pseudoturnover (see Archaux et al. 2009; Gray and Azuma 2005; Groom and Whild 2017; Morrison et al. 2020; Scott and Hallam 2002 for notable exceptions). The magnitude of pseudoturnover documented for the grasslands of this study were within the range documented by previous studies for all spatial scales except the smallest.
Relatively few other studies have quantified misidentification error. Misidentification was considered to account for a 10%–20% discrepancy among observers in grasslands by Klimeš et al. (2001). Scott and Hallam (2002) reported overall misidentification rates of 5.9% at the species level from a variety of habitat types. Archaux et al. (2006) reported misidentification errors of 5.6%–10.5% in temperate lowland forests. In a comparison of vegetation layers, Archaux et al. (2009) reported misidentification rates of 5.3% at the species level for the ground vegetation layer and 2.3% for the tree layer. The misidentification rates documented in this study (0.9%–1.8%, depending upon spatial scale) are several-fold lower than that reported in other studies. The lower misidentification error may be the result of the training conducted prior to the surveys and the use of species lists based on previous surveys.
Pseudoturnover due to cautious error was also very small (0%–0.9% depending upon spatial scale). This was probably also due to some degree to training and the availability of species lists from prior surveys. Additionally, no attempts were made by any observers to identify problematic taxa such as Carex and Cyperus to species level, thus reducing the likelihood of cautious errors. In the only other study of which we are aware in which cautious error was estimated separately from overlooking and misidentification error, it ranged from 1.0% to 6.5% in wetland vegetation (Morrison et al. 2020).
Cover estimation
Error in cover estimation occurred more frequently for more abundant species (i.e. higher cover classes). Most disagreements between surveyors were only different by one class (84% of the time disagreements occurred). Overall, differences of one class composed 22.5% of all paired observations, and differences of two or more classes resulted in only 3.8% of all paired observations.
Several other studies have quantified observer error in categorical estimation of cover. All have reported higher error rates, often twice as great. The percentages differing by one category and two or more, respectively, were 39.5% and 3% in meadows, clearcuts and peat-bogs (Lepš and Hadincová 1992), 46% and 4% in grasslands (Klimeš 2003), 41% and 6% in temperate forests (Gray and Azuma 2005) and 47.5% and 11.5% in lowland temperate forest (Archaux et al. 2007). Morrison et al. (2020) reported that for wetland vegetation 39% of estimates differed by one cover class and 21% differed by two or more, although they used a scale with 10 cover classes whereas the other studies listed above all used the Braun-Blanquet 7 class cover scale.
It is not clear why estimation error was so low compared with other published studies. Estimation error was infrequent for the smallest cover class, but many species are typically rare in communities. In the wetlands study (Morrison et al. 2020), 59% of all cover estimates were within the first two categories reflecting foliar cover up to 1%, which is extremely similar to the 60.8% of average estimates of class 1 (0%–0.9%) found in this study.
Differences in cover estimation were independent of observer or team composition and appeared to have a strong random component. Other studies have investigated whether observer error is random or whether a systematic bias exists. Some studies reported little or no evidence of a systematic bias (e.g. Archaux et al. 2007; Futschik et al. 2020; Klimeš 2003; Lepš and Hadincová 1992). Other studies, however, found that some observers or teams recorded estimates that were consistently higher or lower compared with other observers or teams (e.g. Carlsson et al. 2005; Gorrod and Keith 2009; Kercher et al. 2003; Morrison and Young 2016).
Effects of spatial scale
The overlooking component of pseudoturnover increased as sample frame size decreased, which was the opposite of our expectation. We hypothesized that as sample frame size became smaller, overlooking would be less likely due to a smaller area to search and fewer species to observe. In fact, the number of species overlooked as frame size decreased did decline (from an average of 4.1 to 1.2), but the number of species present declined at a much more rapid rate (from an average of 26.0 to 3.6) (Fig. 3). Thus, the greater overlooking error at the smaller frame sizes was driven by the rapid decline in species richness as frame size decreased.
Mean total number of species, mean number of species overlooked and mean overlooking error, as a function of plot frame size for the 2018 monitoring event.
It is likely, however, that some unknown proportion of overlooking error was actually relocation error (i.e. error associated with imperfect relocation of plot boundaries). Although the three smaller frames were rigid, the largest (10 m2) was constructed of shock-corded aluminum sections that could be broken down for easier transportation. This created the potential for the frame to be distorted somewhat in shape. Two ropes, which were permanently attached to the frame and ran perpendicular to each other, intersected in the middle and were used to align the frame with the tape (Supplementary Fig. S1). By lifting the frame by the rope intersection, the perimeter could be positioned consistently in the same location on flat ground. The presence of uneven ground, dense vegetation or rocky outcrops, however, could have deformed the frame during actual sampling. This would have made little difference at the 10 m2 scale, but could have affected the placement of the smaller frames to a relatively larger degree, since they were aligned where the perimeter of the largest frame intersected the tape, rather than on the meter mark of the tape as the largest frame was. Since the purpose of the smaller frames was to generate frequency data, and plots within sites do not necessarily have to be relocated exactly (Heywood and DeBacker 2007), the potential displacement of the smaller frames was not deemed a critical issue for long-term monitoring. It could, however, have affected the observer error estimates at the smaller spatial scales.
Few other studies have evaluated the effect of plot size on pseudoturnover. Two found evidence for greater pseudoturnover in smaller areas: Nilsson and Nilsson (1985), in their classic pseudoturnover study of insular vegetation, surveyed a relatively large size range of islands (0.03–2.19 ha) and reported a negative association between pseudoturnover and island area (i.e. a greater similarity in species lists on larger islands). Morrison et al. (2020) found that, in temperate wetlands, overlooking error was greater for 100 m2 modules than for plots ranging from 400 to 1000 m2 (24.4% vs. 17.8%, respectively) for herbaceous vegetation. For woody vegetation, however, rates of overlooking error were similar (6.7 vs. 6.0, modules vs. plots respectively).
Other studies, however, have reported ambivalent results: Ringvall et al. (2005), working in boreal forests, found no effect of two different sized plots, although both were relatively small (0.01 and 0.33 m2). Vittoz and Guisan (2007), surveying alpine meadows, reported that the area of plots (0.4, 4 and 40 m2) resulted in an ‘unclear’ influence on their species lists. Archaux et al. (2007) concluded that small quadrats (2 and 4 m2) were not more reliable than larger ones (400 m2) in lowland temperate forest, based on the use of five different variables including species composition, and advocated the use of larger quadrats primarily because they contain more species.
Changes over time
The overall recorded change in species composition between 2014 and 2018 was 39.1%. We estimated that 2.6% was the result of misidentification in one year or the other, and 5.8% was the result of cautious error, resulting in a 30.7% difference, which includes both overlooking error and actual change. Given an overlooking error of 18.6% in 2018, this suggests an actual change of 12.1% for the 4-year period, or an average of 3% per year.
It is possible that a greater degree of observer error characterized the 2014 data collection compared with that in 2018. None of the surveyors in 2014 were present in 2018, although the two primary botanists in 2014 had similar levels of experience as those in 2018. Fewer species per 10 m2 plot were recorded in 2014 compared with 2018 (18.6 vs. 22, respectively), however, and surveyors were more cautious with species-level identifications in 2014 than 2018 (involving five genera compared with two, respectively). Ideally, studies of observer error should encompass all observers, although with long-term monitoring this will not often be possible.
Few other studies have attempted to compare reported change over time with observer error. Burg et al. (2015), working on high alpine summits, found that species turnover over a century was three times larger than mean pseudoturnover between observers (41.4% vs. 13.6%). Futschik et al. (2020) found that recorded change in species turnover and cover, as well as a derived metric (the thermic vegetation indicator) were significantly larger than pseudochanges due to observer error for observation periods of 10 years or longer, but over 7-year periods the recorded changes did not significantly exceed observer error noise.
In general, the longer the time period, the more community change will occur and the less important observer error will be. For smaller time scales, however, much of recorded community change could be attributable to observer error. The magnitude will likely vary with habitat types and species compositions. Ecologists must be wary of the potential of observer error in reporting monitoring results.
Dependence upon sample design
For the purposes of this observer error study, inferences are made at the level of the plot. In any given study, the effects of observer error will depend upon the sampling design. For the overall long-term monitoring project at this preserve, the unit of replication is the site, which contains 10 such plots. If a species is overlooked at one of the plots but is relatively common, it will likely be recorded at one of the others. Such overlooking would have no effect if, e.g., one is interested in species richness or composition at the site level. In the case of rare species that are present at only one or a few plots, however, the potential exists for such species to be entirely overlooked at the site level. In such a case, species richnesses would be underestimated and the identities of such rare species unrecorded. Overlooking would result in an underestimate of frequency, even for more common species, if such species are not recorded from all plots in which they exist. Thus, the overall effect of observer error in a design with multiple subunits per site may be reduced and will depend on how records are averaged or accumulated over the subunits. In designs in which a single plot, transect or relevé is the unit of replication, however, any inferences may be affected to a greater degree.
Because errors in cover estimation were essentially random (i.e. no directional bias was detected), such errors may simply cancel each other out in a design such as ours, in which cover for each species is averaged over all plots for a site-wide inference. Even in simpler designs without subunits, the random nature of errors in cover estimation may largely cancel each other out in arriving at area-wide inferences.
Sources of error
A relatively long list of potential sources of error in vegetation sampling was compiled in the review by Morrison (2016). For this particular study, we identified the following as being potentially the most problematic: (i) Because a list of species previously encountered in each plot was available, observers may have focused primarily on finding these species and potentially ignored others. (ii) Observers reported fatigue due to heat and long days (although no significant relationship between time of day and pseudoturnover was found). (iii) Some species were taxonomically problematic and difficult to identify. (iv) Over time, the cumulative knowledge of the species assemblage in this area has increased and species-level identifications for some problematic taxa have become more practical. Some confusion may have existed among surveyors as to whether species-level identifications were expected for these taxa (especially important when comparing 2014 and 2018 data when different surveyors were involved). (v) Some specimens were in poor condition or did not have the characters needed to make definitive identifications. The first two sources of error would contribute to overlooking error, whereas the third would contribute to misidentification error and the fourth to cautious error. The fifth could affect both misidentification and cautious error.
Experience levels differed somewhat among the top two botanists in both 2014 and 2018, although all four had relatively high levels of experience. The effect of experience in studies such as these is not clear; although some studies have found an effect of experience (e.g. McCune et al. 1997; Oredsson 2000; Scott and Hallam 2002; Vittoz and Guisan 2007), others have not (e.g. Burg et al. 2015; Cheal 2008; Chen et al. 2009; Moore et al. 2011; Sykes et al. 1983).
What can be done?
Numerous suggestions for reducing observer error in vegetation monitoring have been made (summarized in Morrison 2016). One of the most common is the use of multiple observers. In this study teams of two surveyed the vegetation, and although one primarily made observations, the second was encouraged to assist and contradict the observer if he disagreed. Although this does not represent a true multiple observer approach, it may have eliminated the most egregious errors. Consultations among observers in the field (when more than one exists) regarding problematic identifications will be valuable.
Training of observers is also important. Many studies (e.g. Kennedy and Addison 1987; Murphy and Lodge 2002; Stapanian et al. 1997; Symstad et al. 2008) have found that training increased the precision or accuracy of estimates (but see Archaux et al. 2009). Much misidentification error, especially of problematic taxa, could be eliminated by increased training. Much cautious error could be eliminated by determining a priori which taxa are to be identified to species and which to the level of genus or species groups. Lists of species known or expected to be present should be useful. Of course, specimens in poor condition or lacking the requisite characters will continue to be problems, and surveying new areas is likely to result in encountering unanticipated taxa.
Reducing the number of plots surveyed per day, particularly in inhospitable conditions, will likely reduce errors due to fatigue. Although fatigue has been suggested as an important contributor to overlooking error (e.g. Archaux et al. 2009), it has rarely been evaluated quantitatively. Burg et al. (2015), in evaluating pseudoturnover between observers surveying vegetation on alpine summits, found that longer ascents significantly increased the number of species overlooked. We found no significant effect of time of day, which would be expected to be positively correlated with fatigue, although there likely exists interobserver variability in susceptibility to fatigue, and our study was not designed to evaluate this rigorously.
CONCLUSIONS
Some degree of observer error will characterize all vegetation studies using human observers. When compiling species lists, overlooking species is usually the largest component of observer error, although misidentification of plants and the use of inconsistent taxonomic levels (i.e. cautious error) also contribute. In this study of grasslands, pseudoturnover due to overlooking error was 18.6%, which is in the middle of the range of values reported from other studies across different vegetation types. Pseudoturnover due to misidentification and cautious error (0.6 and 1.4, respectively) was smaller than that reported from other studies, possibly due to a higher level of training of observers. When quantifying abundance of vegetation, estimation error commonly occurs. In this study of grasslands, estimation error resulting from incorrect cover class assignment occurred in 26% of paired estimates. Taking into account observer error is particularly important when attempting to document long-term vegetation change. In this study system, at the plot level, failure to take into account observer error would result in documented change of species composition being about three times higher than the amount of actual change, and documented change of cover being about twice as high as actual change. Studies with greater observer error rates would have even more inflated estimates of vegetation change. Quantification of observer error, identification of the sources of error and reduction of such error to the degree possible, through additional training or sampling design modification, will benefit many vegetation sampling studies.
Supplementary Material
Supplementary material is available at Journal of Plant Ecology online.
Figure S1: Top: sampling frames positioned along tape. Bottom: close-up of smaller two frames.
Funding
This work was funded by the National Park Service Inventory and Monitoring Program.
Acknowledgements
Mary Short, Bradly Thorton, Kevin James, Karola Mlekush, Robin Graham, David Londe and Marcus Portofee provided valuable field assistance. We thank the staff of Tallgrass Prairie National Preserve for access and field support. Views, statements, findings, conclusions, recommendations and data in this report are those of the author(s) and do not necessarily reflect views and policies of the National Park Service, U.S. Department of the Interior. Mention of trade names or commercial products does not constitute endorsement or recommendation for use by the National Park Service.
Conflict of interest statement. None declared.
