Abstract

This study examined intra-individual variability in a large sample (n = 629) of individuals with a history of mild traumatic brain injury (mTBI) or TBI referred for neuropsychological evaluation. Variability was assessed using the overall test battery mean standard deviation (OTBM SD). We found a negative linear relation between OTBM and OTBM SD (r = −.672) in this sample with a history of neurologic pathology, indicating that the variability is inversely related to cognitive performance and contrary to what is observed in most normative data. Analyses revealed main effects for OTBM and OTBM SD across three TBI severity groups: loss of consciousness (LOC) <1 h, LOC 1 h–6 days, and LOC >6 days. These effects were found for both a valid performance group (no failed embedded validity measures; n = 504) and an invalid performance group (failed one or more embedded validity measures; n = 125). These findings support that cognitive intra-individual variability is increased uniquely by both neuropathology and suboptimal effort, there is a dose–response relationship between neuropathology and cognitive variability, and intra-individual variability may have utility as a clinical index of both.

Introduction

Performance variability has traditionally been viewed as error variance and a nuisance variable in research. However, this view is changing with performance variability now seen as an important marker of nervous system integrity and measures of cognitive variability are increasingly being used by researchers in a variety of domains, including aging (Hertzog, Dixon, & Hultsch, 1992), attention-deficit/hyperactivity disorder (Castellanos et al., 2008), and chronic fatigue syndrome (Fuentes, Hunter, Strauss, & Hultsch, 2001). Typically, researchers have focused on within-subject variance on cognitive tasks or intra-individual variability. This is conceptualized as fluctuations in the ability to maintain consistent performance independent of learning or environmental effects on performance (Nesselroade, 2001). Additionally, intra-individual variability is psychometrically distinct from the cognitive construct that is used to measure the performance (Ram, Rabbitt, Stollery, & Nesselroade, 2005) so that the absolute level of performance on a cognitive task and intra-individual variability on that task measure discrete constructs (e.g., average reaction time measures something different than variability in reaction times).

A common premise in the variability literature is that stable performance reflects a generally intact central nervous system with neuropathology resulting in less stability in the output. This is partly based on the observation that increasing performance variability is a predictor of terminal decline (MacDonald, Hultsch, & Dixon, 2008). While intra-individual variability as a marker of pathology is being explored in the aging literature (Bielak, Hultsch, Strauss, MacDonald, & Hunter, 2010a, 2010b; Duchek et al., 2009; Hultsch, MacDonald, & Dixon, 2002; MacDonald, Hultsch, & Dixon, 2003), it is not currently a major emphasis of research in traumatic brain injury (TBI). However, TBI has been consistently associated with increased intra-individual variability on cognitive tasks in the extant literature.

Stuss and colleagues (1989) published the first study examining intra-individual variability in TBI. They assessed TBI patients on both simple and complex reaction time tasks on a single day and compared them with matched controls. The TBI group's performance was found to be significantly more variable as indexed by each individual's standard deviation from their mean performance on the tests. Stuss later replicated and expanded on these findings by reporting that the degree of increased variability was not related to the severity of TBI and that the greater variability observed following TBI was generally unaffected by practice effects (Stuss, Pogue, Buckle, & Bondar, 1994). Hetherington, Stuss, and Finlayson (1996) also reported that the increases in variability in reaction time associated with TBI resolved over time and again reported that there was no dose–response relationship between TBI severity and performance variability.

There is a small literature examining intra-individual variability in response time related to mild TBI (mTBI). Bleiberg, Garmoe, Halpern, Reeves, and Nadler (1997) compared six individuals with mTBI with six matched controls on a computerized cognitive assessment battery. Controls were reported to generally improve with practice but individuals with mTBI did not show the same improvement, actually performing worse at times. Sosnoff, Broglio, Hillman, and Ferrara (2007) found that young adults who had sustained a sports-related concussion/mTBI had significantly increased intra-individual variability on a visual discrimination reaction time task compared with normal controls, but this effect disappeared after controlling for mean reaction time differences in the groups. They concluded that increased intra-individual variability in reaction time following an mTBI was related to a general increase in reaction time following mTBI and was not a unique marker for mTBI-related dysfunction. Halterman and colleagues (2006) reported similar findings. All of these studies had methodological problems such as small sample size and controlling for mean group differences which may have essentially regressed away the effect of pathology. Therefore, we believe none of the previous studies can be considered to have definitively addressed the association between mTBI and cognitive intra-individual variability.

Intra-Individual Variability

Intra-individual variability is the variability in performance in a domain of functioning of a single individual measured across multiple time points. In contrast, inter-individual variability is defined as the variability in performance in a single domain by multiple individuals at a single time point. Another way of conceptualizing the difference between these two measures of variability is to think of intra-individual variability as rows in a typical data set (within-person) and inter-individual variability as columns in a typical data set (between-persons). Both approaches assess variability in data and utilize multiple data points to calculate a standard deviation to determine whether performance was consistent or inconsistent. The difference is that the multiple data points come from the same individual in an intra-individual approach and from many different individuals in the inter-individual approach. Nesselroade (2001) has asserted that apparent between-group differences may actually be reflecting asynchronous intra-individual variability that is captured at a single point in time (i.e., cognitive differences between younger and older adults may actually represent intra-individual variability increasing with age). Stated another way, between-person variance may be driven by within-person variance, particularly if a variable used for grouping is associated with increased within-person variance.

Intra-individual variability is computed a number of ways but the general approach is to derive a single mean variability value for each individual subject on a number of performance measures across time. Thus, intra-individual variability can be thought of as the mean variability value for an individual based on multiple samples of performance consistency. This is sometimes computed as an intra-individual standard deviation, with higher standard deviations indicating greater levels of performance inconsistency. Psychometric research examining an individual's variability on cognitive assessment batteries has traditionally utilized simply the individual's range of scores (Schinka, Vanderploeg, & Curtiss, 1994; Schretlen, Munro, Anthony, & Pearlson, 2003). In this approach, the index of variability is essentially set by the individual's highest and lowest performances. These performances may not be in similar cognitive domains and no weighting is given to other test performances. Essentially, this assumes that two performances can act as proxies for variability in all other performances and violates a basic statistical assumption that outliers are typically poor measures of central tendency. It should be noted that Schretlen and colleagues (2003) attempted to control for this by calculating what they termed adjusted maximum discrepancy. As an alternative to measuring variability using discrepancy between the highest and lowest performances, Hilborn, Strauss, Hultsch, and Hunter (2009) suggested that there are two ways to view intra-individual variability. For a single individual, they termed variability on a single task inconsistency and variability across multiple tasks that tap into multiple cognitive domains dispersion. They reported that consistency and dispersion were positively related (i.e., greater variability across time is associated with greater variability across tasks) and that increased cognitive dispersion was associated with the presence of cognitive decline. It makes intrinsic sense that dispersion would be a potentially better index of variability and less prone to chance fluctuations in performance than a score discrepancy approach as multiple data points are being utilized instead of only two performances. Dispersion as an index of variability is a concept that has not yet found traction in TBI research, though it has been used to study the effects of aging and HIV infection (Morgan, Woods, Delano-Wood, Bondi, & Grant, 2011).

Variability: Pathology or Effort?

Generally, variability in performance has been attributed to the presence of central nervous system pathology in the intra-individual variability literature. This is directly at odds with an emerging belief in the clinical literature that performance variability on neuropsychological test batteries is normal and not necessarily indicative of pathology (e.g., Binder, Iverson, & Brooks, 2009; Schretlen et al., 2003). Variability in these studies and reviews was typically defined using some form of discrepancy analysis, often the discrepancy between the highest and lowest standardized scores on neuropsychological tests. Additionally, although some have argued that intra-individual variability is related to intelligence (Schinka et al., 1994), research seems to indicate that there is no clear systematic relationship between cognitive performance variability and intelligence other than that higher levels of intelligence are somewhat associated with less variability and lower levels are somewhat associated with more variability (Ram et al., 2005). We propose that increased cognitive variability has two primary sources, either central nervous system pathology or inconsistent effort on testing by the examinee. We have a broad conceptualization of pathology that encompasses all sources that could affect neurologic output from neurodegenerative processes and traumatic brain injury to minor illness and excessive daytime sleepiness. The literature looking at intra-individual variability has generally not addressed the issue of participant effort or performance validity. There have been a few previous attempts at incorporating performance consistency into the evaluation of participant effort (Demakis, 1999; Gunner, Miele, Lynch, & McCaffrey, 2012; Reitan & Wolfson, 1997; Strauss et al., 2000, 2002). Generally, these studies have not reported results with meaningful clinical application with the exception of the study by Gunner and colleagues (2012) which found that inconsistent responding across the first two trials of the Test of Memory Malingering (TOMM; Tombaugh, 1996) identified individuals who appeared to give good effort on the TOMM but performed suboptimally on the Word Memory Test (Green, 2003). We hypothesize that variability or dispersion will be positively associated with TBI severity and that variability due to effort will be greater than variability due to TBI pathology.

Method

Participants

Archival data from one of the authors' private practice (JEM) were utilized in this study. Data from 629 participants were identified who had completed a neuropsychological evaluation using the Meyers Neuropsychological Battery (MNB) and had reported a history of head trauma. Participants were excluded from the study if they were older than 80 years old or less than 16 years old. The sample was 75.8% men, 88.1% right-handed, and 89.3% Caucasian (next largest groups were 3.2% Hispanic and 2.4% African American). The mean age was 34.5 years (SD = 14.7), the mean education 12.4 years (SD = 2.3), and the mean Wechsler Adult Intelligence Scale-Third Edition (WAIS-III) Full-Scale Intelligence Quotient 89.7 (SD = 14.2). The mean MNB overall test battery mean (OTBM; seeMiller & Rohling, 2001) was 40.9 (SD = 7.6) and the mean OTBM standard deviation (OTBM SD) was 10.89 (SD = 2.74). Participants were initially grouped according to reported loss of consciousness (LOC) into the following categories: LOC up to 1 h (n = 341), LOC >1 –24 h (n = 76), LOC >1–6 days (n = 101), and LOC >6 days (n = 111). Reported LOC was derived from the individual's medical record and history. Statistical analyses revealed that the LOC >1–24 h group and the LOC >1–6 days group were not significantly different from each other on their mean OTBM leading us to combine these groups in order to avoid falsely dichotomizing these individuals on a primary outcome variable. This resulted in the following LOC groups being used in this study: Group 1, LOC up to 1 h (n = 341); Group 2, LOC >1 h–6 days (n = 177); and Group 3, LOC >6 days (n = 111). Additionally, even though LOC group 1 was set at LOC up to 1 h, the maximum LOC reported was <2 min and the mean LOC was <1 min making this essentially an mTBI group. Participants were also grouped according to whether they passed all embedded validity measures of the MNB (Rey Complex Figure Test [RCFT] memory error pattern ≤3, Reliable Digit Span ≤6, Forced Choice Test ≤10, Judgment of Line Orientation ≤12, Token Test (TT) ≤150, Dichotic Listening ≤9, Sentence Repetition ≤9, Rey Auditory Verbal Learning Test (AVLT)-Recognition ≤9, Estimated Finger Tapping ≤10; seeMeyers & Volbrecht, 2003) or failed one or more of the embedded measures into valid (n = 504) or invalid (n = 125) performance groups. The mean number of failed embedded measures for the invalid group was 1.34 (SD = 0.84). Demographics and mean values are presented for these groups in Table 1.

Table 1.

Group demographics with mean age, education, and OTBM

LOC group Valid
 
Invalid
 
n Gender Age Education OTBM n Gender Age Education OTBM 
<1 h 279 78% men 35.20 (15.12) 12.58 (2.42) 45.11 (4.95) 62 79% men 35.58 (13.10) 12.21 (1.98) 36.87 (6.34) 
1 h–6 days 56 81% men 31.49 (13.80) 12.41 (2.21) 41.33 (6.20) 20 76% men 38.17 (18.11) 12.02 (2.81) 36.02 (6.53) 
>6 days 90 63% men 36.01 (12.96) 11.89 (2.08) 35.12 (7.31) 21 62% men 28.48 (13.88) 12.71 (2.26) 26.95 (6.73) 
LOC group Valid
 
Invalid
 
n Gender Age Education OTBM n Gender Age Education OTBM 
<1 h 279 78% men 35.20 (15.12) 12.58 (2.42) 45.11 (4.95) 62 79% men 35.58 (13.10) 12.21 (1.98) 36.87 (6.34) 
1 h–6 days 56 81% men 31.49 (13.80) 12.41 (2.21) 41.33 (6.20) 20 76% men 38.17 (18.11) 12.02 (2.81) 36.02 (6.53) 
>6 days 90 63% men 36.01 (12.96) 11.89 (2.08) 35.12 (7.31) 21 62% men 28.48 (13.88) 12.71 (2.26) 26.95 (6.73) 

Notes: OTBM = overall test battery mean; LOC = loss of consciousness; standard deviations are in parentheses.

Measures

All participants completed the MNB. The battery consists of the following subtests which are listed in the order of their administration: seven selected subtests (i.e., Picture Completion, Digit Symbol, Similarities, Block Design, Arithmetic, Digit Span, Information) from the WAIS-III (Pilgrim, Meyers, Bayless, Whetstone, 1999; Ward, 1990), Forced Choice (Brandt, Rubinsky, & Larson, 1985); RCFT-Copy (Meyers & Meyers, 1995); Animal Naming (Strauss, Sherman, & Spreen, 2006); 3-min recall of RCFT (Meyers & Meyers, 1995); Controlled Oral Word Association Test (Benton & Hamsher, 1989); Dichotic Listening (Meyers, Roberts, Bayless, Volkert, & Evitts, 2002; Roberts et al., 1994); North American Adult Reading Test (Strauss et al., 2006); Sentence Repetition (Meyers, Volkert, & Diep, 2000; Strauss et al., 2006); 30-min recall of RCFT and recognition trial of RCFT (Meyers & Meyers, 1995); Rey AVLT (Strauss et al., 2006); Judgment of Line Orientation (Benton, Hamsher, Varney, & Spreen, 1983); Boston Naming Test (Strauss et al., 2006); Finger Tapping (Reitan & Wolfson, 1985); Finger Localization (Benton et al., 1983); Trail Making Test (Reitan & Wolfson, 1985); TT (Spreen & Strauss, 1998); Rey AVLT 30-min free recall and recognition (Strauss et al., 2006); and Booklet Category Test (Victoria Version; Strauss et al., 2006; Kozel & Meyers, 1998). Each subtest was administered in accordance with the authors' instructions.

Procedure

An OTBM for each participant was calculated for each participant and the individual's standard deviation around their own OTBM (the OTBM SD) was used as the measure of intra-individual variability for this study. The OTBM was calculated by converting individual test performances to a T score and averaging all T scores for each individual to obtain their OTBM. We then calculated a standard deviation for each individual around their own OTBM similar to an intra-individual dispersion approach to analyzing performance variability when multiple indexes of similar domains are given across a single evaluation day for each patient. These different performances are now directly comparable as they have been converted to the same metric individually for each patient with the average mean and standard deviation of the normative data for all measures (seeMiller & Rohling, 2001) and essentially provide multiple measures of multiple domains across a single testing session. Conceptually, it is important to realize that the standard deviation of the mean OTBM is not the same as the mean OTBM SD (the mean of individuals' standard deviations around their own OTBM) as the former is variability around the group OTBM and the latter is the mean value of our intra-individual variability measure. Additionally, in all analyses examining the relation between TBI severity and variability, we only analyzed valid data from individuals who passed all MNB embedded validity measures. For all analyses focusing on the association between performance validity and variability, we only utilized data from individuals who reported up to 1 h LOC (Group 1). As the majority of this group had LOC of a few minutes or less, we utilized this approach to minimize false-positive errors on the embedded measures possibly associated with more severe TBI.

Results

The following analyses utilized only valid data, n = 504. Length of LOC was correlated with OTBM r(502) = −.47, p < .001 and OTBM SD r(502) = .32, p < .001, indicating that greater neuropathology was associated with lower neuropsychological performance and great intra-individual variability. OTBM was negatively correlated with OTBM SD, r(502) = −.67, p < .001, indicating that lower neuropsychological performance was associated with greater intra-individual variability. Fig. 1 shows a regression scatterplot of OTBM SD predicting OTBM for valid performers only indicating a strong linear relationship. The intercept was 61.1 and the slope was −1.80. One-way analyses of variance (ANOVAs) were performed with LOC groups as the factor and OTBM and OTBM SD as dependent variables. Levene's tests of homogeneity of variance were significant for both OTBM (p < .001) and OTBM SD (p = .017). Kruskal–Wallis nonparametric tests were utilized and significant main effects were found for OTBM and OTBM SD (both p < .001). We, therefore, felt comfortable interpreting the results of the one-way ANOVAs. Significant main effects were found for OTBM, F(2, 501) = 104.1, p < .001, and OTBM SD, F(2, 501) = 55.0, p < .001. Games–Howell post hoc analyses (does not assume equal variances) found that all LOC groups were significantly different for both OTBM and OTBM SD (Tables 2 and 3). Specifically, for OTBM Group 1 > Group 2 > Group 3 and for OTBM SD Group 1 < Group 2 < Group 3. These results support our hypothesis that IIV would increase with increasing TBI pathology. To determine the clinical significance of these findings, we calculated effect sizes for the valid group based on OTBM and OTBM SD. For OTBM, Group 1 versus Group 2 yielded d = 0.70, Group 1 versus Group 3 d = 1.78, and Group 2 versus Group 3 d = 0.93. For OTBM SD, Group 1 versus Group 2 yielded d = 0.56, Group 1 versus Group 3 d = 1.27, and Group 2 versus Group 3 d = 0.64.

Table 2.

OTBM presented for each TBI severity groups by their validity status with one-way ANOVA results

LOC group Valid
 
Invalid
 
n OTBM SD F p-value n OTBM SD F p-value 
<1 h 279 45.1 5.0   62 36.9 6.3   
1 h–6 days 135 41.3 6.2   42 36.0 6.5   
>6 days 90 35.1 7.3   21 27.0 6.7   
Sample 504 42.3 6.9 104.43 <.001 125 34.9 7.4 19.35 <.001 
LOC group Valid
 
Invalid
 
n OTBM SD F p-value n OTBM SD F p-value 
<1 h 279 45.1 5.0   62 36.9 6.3   
1 h–6 days 135 41.3 6.2   42 36.0 6.5   
>6 days 90 35.1 7.3   21 27.0 6.7   
Sample 504 42.3 6.9 104.43 <.001 125 34.9 7.4 19.35 <.001 

Notes: OTBM = overall test battery mean; LOC = loss of consciousness.

OTBM presented as T score.

Table 3.

OTBM SD presented for each TBI severity groups by their validity status with one-way ANOVA results

LOC Group Valid
 
Invalid
 
n OTBM SD SD F p-value N OTBM SD SD F p-value 
<1 h 279 9.6 2.2   62 12.1 2.4   
1 h-6 days 135 10.9 2.5   42 12.2 2.7   
>6 days 90 12.5 2.5   21 15.0 2.4   
Sample 504 10.5 2.6 55.00 <.001 125 12.7 2.7 11.69 <.001 
LOC Group Valid
 
Invalid
 
n OTBM SD SD F p-value N OTBM SD SD F p-value 
<1 h 279 9.6 2.2   62 12.1 2.4   
1 h-6 days 135 10.9 2.5   42 12.2 2.7   
>6 days 90 12.5 2.5   21 15.0 2.4   
Sample 504 10.5 2.6 55.00 <.001 125 12.7 2.7 11.69 <.001 

Notes: OTBM SD = overall test battery mean standard deviation; LOC = loss of consciousness.

Fig. 1.

Scatterplot of OTBM SD predicting OTBM for valid performers.

Fig. 1.

Scatterplot of OTBM SD predicting OTBM for valid performers.

Next, we examined the invalid group data (n = 125). LOC was correlated with OTBM r(123) = −.51, p < .001 and OTBM SD r(123) = .40, p < .001, indicating that increasing neuropathology was associated with lower neuropsychological performance and increased intra-individual variability. OTBM was negatively correlated with OTBM SD, r(123) = −.67, p < .001, indicating that lower neuropsychological performance was associated with increased intra-individual variability. Fig. 2 shows a regression scatterplot of OTBM SD predicting OTBM for invalid performers only and a strong linear relationship is observed. The intercept was 57.9 and the slope was −1.82. One-way ANOVA was performed with invalid LOC groups as the factor and OTBM and OTBM SD as dependent variables. Levene's tests of homogeneity of variance were not significant for either OTBM (p = .98) or OTBM SD (p = .65). Significant main effects were found for OTBM, F(2, 122) = 19.3, p < .001, and OTBM SD, F(2, 122) = 11.7, p < .001. Fisher's least significant difference post hoc analyses found that for both OTBM and OTBM SD, LOC Groups 1 and 2 were not significantly different from each other, whereas LOC Group 3 was significantly different from all LOC groups (Tables 2 and 3). Specifically, for OTBM Group 1 = Group 2 > Group 3 and for OTBM SD Group 1 = Group 2 < Group 3. There was a linear decrease for OTBM across LOC groups, with a 22% decrease from Group 1 to Group 3 for the valid group and a 27% decrease from Group 1 to Group 3 for the invalid group. There was a linear increase for OTBM SD across LOC groups, with a 30% increase from Group 1 to Group 3 for the valid group and a 24% increase from Group 1 to Group 3 for the invalid group.

Fig. 2.

Scatterplot of OTBM SD predicting OTBM for invalid performers.

Fig. 2.

Scatterplot of OTBM SD predicting OTBM for invalid performers.

As our LOC Group 1 was the largest group for both the valid and invalid groups, LOC Group 1 is conceptually an mTBI group, and more symptom exaggeration typically occurs in mTBI than more severe TBI, we examined differences between valid and invalid performers for OTBM and OTBM SD in LOC Group 1 only. As the groups had very unequal sample sizes (valid n = 279, invalid n = 62) and there was evidence of a violation of the homogeneity of variance assumption for OTBM (Levene's test p = .004), we used nonparametric analyses. Independent-samples Mann–Whitney U-tests were significant for both OTBM and OTBM SD, p < .001 for both, indicating that in Group 1, OTBM was significantly higher for valid performers compared with invalid performers and OTBM SD was significantly lower for valid performers compared with invalid performers (see Tables 2 and 3 for means). These results supported our hypothesis that intra-individual variability (measured using the OTBM SD) associated with invalid effort would be greater than intra-individual variability associated with TBI pathology.

Discussion

Unlike many normative data sets used by clinicians where higher performing groups have higher levels of variability compared with lower performing groups (Strauss et al., 2006), we found that performance variability was inversely related to the level of performance using an OTBM approach in this TBI sample. This indicates that, generally, the lower the individual's overall cognitive ability the larger the intra-individual variability when there is a history of neurologic pathology. Our results also clearly demonstrate a linear relationship between increasing TBI severity and increasing intra-individual variability (measured using OTBM SD). Additionally, we believe that our results illustrate a relationship between variability and performance validity in some individuals with TBI. It was particularly interesting that individuals with what was conceptually an mTBI and evidence of potentially invalid performance did not differ significantly from individuals with more severe TBI; however, individuals with mTBI and valid performance were clearly discernible from more severe TBI groups. We believe our results also revealed that in mTBI, individuals with indicators of valid performance will have higher overall cognitive performance and lower intra-individual variability than individuals with evidence of invalid performance. However, we acknowledge that a limitation of our study is that we only had available for analysis embedded validity measures and not stand alone performance validity tests (PVTs). As such, it is possible that our measures of effort were contaminated by pathology. Still, this would not likely explain the significant differences in OTBM and OTBM SD that we observed between these groups. Additionally, although we would have preferred to have specified our invalid performance group at two or more embedded validity indicator failures, our data precluded this approach. In our sample, 16% of subjects failed a single embedded PVT, whereas only 4% failed two or more embedded PVTs. Specifically for individuals failing only a single embedded measure, 47% were likely mTBI (Group 1), 34% were moderate to severe TBI (Group 2), and 19% were severe TBI (Group 3). We fully acknowledge that there may be some false positives for poor effort among individuals that failed a single embedded validity indicator, particularly in Group 3. However, we believe that the majority of the subjects are likely true cases of poor effort based on looking at other factors in the data as well as the low percentage of individuals with a single failed embedded validity measure in our overall sample. Most of the individuals in this clinical sample passed all embedded validity measures. Notably, 81% of individuals with a history of LOC >6 days did not fail a single embedded measure making failures in the larger mTBI group highly suspicious. Additionally, the effect of including some individuals with valid performances in our invalid performance groups would be to mitigate our reported finding of increased variability in the invalid group compared with the valid group; it would not create the obtained result in any way. Therefore, we believe the likelihood is that there is an even larger effect of effort on performance variability than we actually report. In summary, although the current methodology may include some individuals with valid performances in our invalid performance group, this would not create our finding of increased variability in the invalid effort group and may make our current results an underestimate of the actual increase in variability attributable to invalid effort. It is also possible that our valid performance group may include some individuals who gave suboptimal effort during testing. However, we consider this to be a lesser concern and the fact that a strong dose–response relationship was observed between pathology and both neuropsychological performance and variability suggests that our findings were not driven by the contamination of the valid performance group. Further research is needed to confirm our findings.

We believe that our results demonstrate that variability in cognitive performance in TBI can be a marker of both brain dysfunction and inconsistent effort associated with performance invalidity. Intra-individual cognitive variability holds promise as a marker of poor effort in TBI and further research into this relationship is warranted. Some may argue that we are confusing effort with actual pathology in TBI. However, we believe that we minimized this concern by only analyzing the effect of effort in individual with LOC <2 min who would be considered to have sustained an mTBI. This is also readily seen in Table 3 in which the invalid performance group with the least pathology has intra-individual variability that is equivalent to the valid group with the greatest degree of pathology. Although this effect of performance invalidity increasing variability is readily apparent in our group data, it might be difficult to use an intra-individual variability cut-score as a performance validity indicator due to the degree of overlap in cognitive variability across valid and invalid LOC groups.

Psychometric Versus Pathologic Variability

Our results are contradictory to research findings examining cognitive performance variability in normal populations. The consistent finding in normal populations is that cognitive variability increases in above-average individuals and decreases in below-average individuals (Binder et al., 2009; Brooks, Strauss, Sherman, Iverson, & Slick, 2009). We call this psychometric variability to reflect what we believe is the cause of this pattern. We propose that low-performing individuals must have lower levels of variability than higher performing individuals because the variability in their performance is constrained by floor effects of cognitive tests and inconsistency does not result in better than possible performance. In contrast, ceiling effects are usually not as firm as floor effects, with the exception of motor functioning tests that are often biomechanically constrained. Therefore, higher performing individuals can perform at their typical ability level and any inconsistency is likely to fall in a downward direction but is not constrained by floor effects as occurs in low-performing individuals. We hypothesize that these psychometric constraints would essentially guarantee greater variability for the higher performing group in normal samples. This is an empirical question that requires further research.

In contrast to the effect of psychometric variability in normal individuals, we report what many would consider a counterintuitive effect in TBI where intra-individual variability increases as cognitive impairment increases. We call this pathologic variability to contrast it from psychometric variability. We believe that this variability is due to neurologic dysfunction that increases “noise” in the system resulting in inconsistent performance and believe this explains our current findings. However, extreme degrees of TBI, or any pathology, would result in almost zero variability in performances as variability is an emergent property of measurement which requires at least a minimal level of performance. We have also demonstrated that effort may also increase variability in individuals with TBI and that the increased variability attributable to effort is greater than the variability attributable to TBI pathology in the data we present here. Additionally, it is possible that the difference between our current findings and previous studies examining cognitive variability (Binder et al., 2009; Brooks et al., 2009; Schinka et al., 1994; Schretlen et al., 2003) is driven more by methodological differences in how cognitive variability is measured than psychometric factors. In the end, although the relation between cognitive ability and variability may truly depend on how you measure it, it seems logical to assume that only one methodology can be truly correct when such discrepant findings are obtained.

In summary, examining intra-individual variability in performance around an OTBM is an innovative way to assess neurologic integrity that may have particular promise in the area of TBI. Although previous research did not find a dose–response relationship between variability and TBI severity (Hetherington et al., 1996; Stuss et al., 1994), we have shown that performance variability does increase in a linear fashion with severity of TBI when you assess variability around an OTBM. We have also demonstrated that suboptimal effort may account for variability in performance and extreme variability may have utility as an effort index. We believe that cognitive dispersion measured by variability around an OTBM holds promise as an analytical methodology and should be considered for broader use by TBI researchers. Further study of cognitive variability related to TBI may improve the sensitivity of neuropsychological assessment and help explain more subtle complaints commonly reported post-injury, such as the subjective experience of not fully returning to pre-morbid cognitive baseline or inconsistency in performance.

Conflict of Interest

One of the authors (JEM) is the developer of the Meyers Neuropsychological Battery and profits from the sale of this battery.

References

Benton
A.
Hamsher
K.
Varney
N.
Spreen
O.
Contributions to neuropsychological assessment: A clinical manual
 , 
1983
New York
Oxford University Press
Benton
A. L.
Hamsher
K.
Multilingual aphasia examination
 , 
1989
2nd ed.
Iowa City, IA
AJA Associates
Bielak
A. A. M.
Hultsch
D. F.
Strauss
E.
MacDonald
S. W. S.
Hunter
M. A.
Intraindividual variability in reaction time predicts cognitive outcomes 5 years later
Neuropsychology
 , 
2010a
, vol. 
24
 
6
(pg. 
731
-
741
)
Bielak
A. A. M.
Hultsch
D. F.
Strauss
E.
MacDonald
S. W. S.
Hunter
M. A.
Intraindividual variability is related to cognitive change in older adults: Evidence for within-person coupling
Psychology and Aging
 , 
2010b
, vol. 
25
 
3
(pg. 
575
-
586
)
Binder
L. M.
Iverson
G. L.
Brooks
B. L.
To err is human: “Abnormal” neuropsychological scores and variability are common in healthy adults
Archives of Clinical Neuropsychology
 , 
2009
, vol. 
24
 
1
(pg. 
31
-
46
)
Bleiberg
J.
Garmoe
W. S.
Halpern
E. L.
Reeves
D. L.
Nadler
J. D.
Consistency within-day and across-day performance after mild brain injury
Neuropsychiatry, Neuropsychology, and Behavioral Neurology
 , 
1997
, vol. 
10
 (pg. 
247
-
253
)
Brandt
J.
Rubinsky
E.
Larson
G.
Uncovering malingered amnesia
Annals of the New York Academy of Science
 , 
1985
, vol. 
44
 (pg. 
502
-
503
)
Brooks
B. L.
Strauss
E.
Sherman
E. M. S.
Iverson
G. L.
Slick
D. J.
Developments in neuropsychological assessment: Refining psychometric and clinical interpretive methods
Canadian Psychology
 , 
2009
, vol. 
50
 (pg. 
196
-
209
)
Castellanos
F. X.
Margulies
D. S.
Kelly
C.
Uddin
L. Q.
Ghaffari
M.
Kirsch
A.
, et al.  . 
Cingulate-precuneus interactions: A new locus of dysfunction in adult attention-deficit/hyperactivity disorder
Biological Psychiatry
 , 
2008
, vol. 
63
 
3
(pg. 
332
-
337
)
Demakis
G. J.
Serial malingering on verbal and nonverbal fluency and memory measures: An analog investigation
Archives of Clinical Neuropsychology
 , 
1999
, vol. 
14
 (pg. 
401
-
410
)
Duchek
J. M.
Balota
D. A.
Tse
C. S.
Holtzman
D. M.
Fagan
A. M.
Goate
A. M.
The utility of intraindividual variability in selective attention tasks as an early marker for Alzheimer's disease
Neuropsychology
 , 
2009
, vol. 
23
 
6
(pg. 
746
-
758
)
Fuentes
K.
Hunter
M. A.
Strauss
E.
Hultsch
D. F.
Intraindividual variability in cognitive performance in persons with chronic fatigue syndrome
The Clinical Neuropsychologist
 , 
2001
, vol. 
15
 
2
(pg. 
210
-
227
)
Green
P.
Green's Word Memory Test
 , 
2003
Edmonton, CA
Green's Publishing
Gunner
J. H.
Miele
A. S.
Lynch
J. K.
McCaffrey
R. J.
The Albany consistency index for the test of memory malingering
Archives of Clinical Neuropsychology
 , 
2012
, vol. 
27
 (pg. 
1
-
9
)
Halterman
C. I.
Langan
J.
Drew
A.
Rodriguez
E.
Osternig
L. R.
Chou
L. S.
, et al.  . 
Tracking the recovery of visuospatial attention deficits in mild traumatic brain injury
Brain
 , 
2006
, vol. 
129
 (pg. 
747
-
753
)
Hertzog
C.
Dixon
R. A.
Hultsch
D. F.
Intraindividual change in text recall of the elderly
Brain and Language
 , 
1992
, vol. 
42
 (pg. 
248
-
269
)
Hetherington
C. R.
Stuss
D. T.
Finlayson
M. A. J.
Reaction time variability 5 and 10 years after traumatic brain injury
Brain Injury
 , 
1996
, vol. 
10
 (pg. 
473
-
486
)
Hilborn
J. V.
Strauss
E.
Hultsch
D. F.
Hunter
M. A.
Intraindividual variability across cognitive domains: Investigation of dispersion levels and performance profiles in older adults
Journal of Clinical and Experimental Neuropsychology
 , 
2009
, vol. 
3
 
4
(pg. 
412
-
424
)
Hultsch
D. F.
MacDonald
S. W. W.
Dixon
R. A.
Variability in reaction time performance of younger and older adults
Journal of Gerontology: Psychological Sciences
 , 
2002
, vol. 
57B
 (pg. 
P101
-
P115
)
Kozel
J. J.
Meyers
J. E.
A cross-validation study of the Victoria Revision of the Category Test
Archives of Clinical Neuropsychology
 , 
1998
, vol. 
13
 (pg. 
327
-
332
)
MacDonald
S. W. S.
Hultsch
D. F.
Dixon
R. A.
Performance variability is related to change in cognition: Evidence from the Victoria Longitudinal Study
Psychology and Aging
 , 
2003
, vol. 
18
 (pg. 
510
-
523
)
MacDonald
S. W. S.
Hultsch
D. F.
Dixon
R. A.
Predicting impending death: Inconsistency in speed is a selective and early marker
Psychology and Aging
 , 
2008
, vol. 
23
 
3
(pg. 
595
-
607
)
Meyers
J. E.
Meyers
K. R.
Rey Complex Figure Test and Recognition Trial: Professional manual
 , 
1995
Lutz, FL
Psychological Assessment Resource
Meyers
J. E.
Roberts
R. J.
Bayless
J. D.
Volkert
K.
Evitts
P. E.
Dichotic listening: Expanded norms and clinical application
Archives of Clinical Neuropsychology
 , 
2002
, vol. 
17
 (pg. 
79
-
90
)
Meyers
J. E.
Volbrecht
M. E.
A validation of multiple malingering detection methods in a large clinical sample
Archives of Clinical Neuropsychology
 , 
2003
, vol. 
18
 (pg. 
261
-
276
)
Meyers
J. E.
Volkert
K.
Diep
A.
Sentence Repetition Test: Updated norms and clinical utility
Applied Neuropsychology
 , 
2000
, vol. 
7
 (pg. 
154
-
159
)
Miller
L. S.
Rohling
M. L.
A statistical interpretive method for neuropsychological test data
Neuropsychology Review
 , 
2001
, vol. 
11
 (pg. 
143
-
169
)
Morgan
E. E.
Woods
S. P.
Delano-Wood
L.
Bondi
M. W.
Grant
I.
Intraindividual variability in HIV infection: Evidence for greater neurocognitive dispersion in older HIV seropositive adults
Neuropsychology
 , 
2011
, vol. 
25
 
5
(pg. 
645
-
654
)
Nesselroade
J. R.
Intraindividual variability in development within and between individuals
European Psychologist
 , 
2001
, vol. 
6
 
3
(pg. 
187
-
193
)
Pilgrim
B. M.
Meyers
J. E.
Bayless
J.
Whetstone
M. M.
Validity of the Ward seven-subtest WAIS-III short form in a neuropsychological population
Applied Neuropsychology
 , 
1999
, vol. 
6
 (pg. 
243
-
246
)
Ram
N.
Rabbitt
P.
Stollery
B.
Nesselroade
J. R.
Cognitive performance inconsistency: Intraindividual change and variability
Psychology and Aging
 , 
2005
, vol. 
20
 
4
(pg. 
623
-
633
)
Reitan
R.
Wolfson
D.
The Halstead-Reitan Neuropsychological Test Battery: Theory and clinical interpretation
 , 
1985
Arizona
Neuropsychology Press
Reitan
R. M.
Wolfson
D.
Consistency of neuropsychological test scores of head-injured subjects involved in litigation compared with head-injured subjects not involved in litigation: development of the Retest Consistency Index
Clinical Psychologist
 , 
1997
, vol. 
11
 (pg. 
69
-
76
)
Roberts
M.
Persinger
M.
Grote
C.
Evertowski
L.
Springer
J.
Tuten
T.
, et al.  . 
The Dichotic Word Listening Test: Preliminary observations in American and Canadian samples
Applied Neuropsychology
 , 
1994
, vol. 
1
 (pg. 
45
-
56
)
Schinka
J. A.
Vanderploeg
R. D.
Curtiss
G.
Wechsler Adult Intelligence Scale-Revised subtest scatter as a function of maximum subtest scaled score
Psychological Assessment
 , 
1994
, vol. 
6
 (pg. 
364
-
367
)
Schretlen
D. J.
Munro
C. A.
Anthony
J. C.
Pearlson
G. D.
Examining the range of normal intraindividual variability in neuropsychological test performance
Journal of the International Neuropsychological Society
 , 
2003
, vol. 
9
 (pg. 
864
-
870
)
Sosnoff
J. J.
Broglio
S. P.
Hillman
C. H.
Ferrara
M. S.
Concussion does not impact intraindividual response time variability
Neuropsychology
 , 
2007
, vol. 
21
 
6
(pg. 
796
-
802
)
Spreen
O.
Strauss
E.
A Compendium of Neuropsychological Tests: Administration, norms, and commentary
 , 
1998
2nd ed.
New York
Oxford University Press
Strauss
E.
Hultsch
D. F.
Hunter
M.
Slick
D. J.
Patry
B.
Levy-Bencheton
J.
Using intraindividual variability to detect malingering in cognitive performance
The Clinical Neuropsychologist
 , 
2000
, vol. 
14
 
4
(pg. 
420
-
432
)
Strauss
E.
Sherman
E. M. S.
Spreen
O.
A Compendium of Neuropsychological Tests: Administration, norms, and commentary
 , 
2006
3rd ed.
New York
Oxford University Press
Strauss
E.
Slick
D. J.
Levy-Bencheton
J.
Hunter
M.
MacDonald
S. W. S.
Hultsch
D. F.
Intraindividual variability as an indicator of malingering in head injury
Archives of Clinical Neuropsychology
 , 
2002
, vol. 
17
 (pg. 
423
-
444
)
Stuss
D. T.
Pogue
J.
Buckle
L.
Bondar
J.
Characterization of stability of performance in patients with traumatic brain injury: Variability and consistency on reaction time tests
Neuropsychology
 , 
1994
, vol. 
8
 (pg. 
316
-
324
)
Stuss
D. T.
Stethem
L. L.
Hugenholtz
H.
Picton
T.
Pivik
J.
Richard
M. T.
Reaction time after head injury: fatigue, divided and focused attention, and consistency of performance
Journal of Neurology, Neurosurgery, and Psychiatry
 , 
1989
, vol. 
52
 
6
(pg. 
742
-
748
)
Tombaugh
T. N.
Test of Memory Malingering (TOMM)
 , 
1996
New York
Multi Health Systems
Ward
L. C.
Prediction of verbal, performance, and full scale IQs from seven subtests of the WAIS-R
Journal of Clinical Psychology
 , 
1990
, vol. 
46
 (pg. 
436
-
440
)