Base rates of low ANAM4 TBI-MIL scores were calculated in a convenience sample of 733 healthy male active duty soldiers using available military reference values for the following cutoffs: ≤2nd percentile (2 SDs), ≤5th percentile, <10th percentile, and <16th percentile (1 SD). Rates of low scores were also calculated in 56 active duty male soldiers who sustained an mTBI an average of 23 days (SD = 36.1) prior. 22.0% of the healthy sample and 51.8% of the mTBI sample had two or more scores below 1 SD (i.e., 16th percentile). 18.8% of the healthy sample and 44.6% of the mTBI sample had one or more scores ≤5th percentile. Rates of low scores in the healthy sample were influenced by cutoffs and race/ethnicity. Importantly, some healthy soldiers obtain at least one low score on ANAM4. These base rate analyses can improve the methodology for interpreting ANAM4 performance in clinical practice and research.

Introduction

A neurocognitive evaluation, whether brief or comprehensive, routinely includes a battery of individual tests designed to assess a variety of cognitive domains (e.g., attention, speed of information processing, and memory). Multiple scores are derived from a battery of tests, and the traditional approach to interpretation is to conceptualize each score relative to the standard normal distribution. Low scores are usually conceptualized as atypical or abnormal scores. In a clinical setting, abnormally low scores are typically interpreted as evidence for cognitive impairment when test performance cannot be explained by other factors unrelated to the clinical condition (e.g., poor effort, longstanding learning disability, situational factors). However, several studies and reviews have shown that it is relatively common for healthy individuals to have one or more abnormally low scores when given a battery of neurocognitive tests (Axelrod & Wall, 2007; Binder, Iverson, & Brooks, 2009; Brooks, Iverson, Holdnack, & Feldman, 2008; Brooks, Iverson, Feldman, & Holdnack, 2009; Brooks, Iverson, Sherman, & Holdnack, 2009; Brooks, Iverson, & White, 2007; Iverson, Brooks, Langenecker, & Young, 2011; Crawford, Garthwaite, & Gault, 2007; Diaz-Asper, Schretlen, & Pearlson, 2004; Ingraham & Aiken, 1996; Palmer, Boone, Lesser, & Wohl, 1998; Patton et al., 2003; Schretlen, Testa, Winicki, Pearlson, & Gordon, 2008).

There are a number of reasons for this. First, the probability of having a low score increases as the number of tests within a battery increases as was demonstrated in studies using simulated data (Crawford et al., 2007; Ingraham & Aiken, 1996; Schretlen et al., 2008). For example, Crawford and colleagues using Monte Carlo simulation estimated that 18.5% of individuals in a population would be expected to have one abnormally low score on a hypothetical battery with four tests when none of the tests were correlated (Crawford et al., 2007). However, when the number of scores was increased to six, the percentage of the population expected to have one low score increased to 26.5. Finally, when the number of tests was increased to 10, 40.1% of individuals were expected to have one abnormally low score.

Demographic characteristics of the test takers, such as age, education, and race/ethnicity also contribute to variability in test performance that will result in some groups having a greater propensity for low scores than others. In a study of healthy adults, Schretlen and colleagues found that age, race, education, and IQ were correlated with nine different cognitive impairment index (CII) scores derived from three different cutoffs in test batteries of varied length (Schretlen et al., 2008). Additionally, in regression analyses, these demographic variables resulted in a statistically significant model for each CII score explaining from 21.1% to 53.8% of the variance across the models. In these analyses, age was generally the best predictor explaining 13.7%–30.8% of the variance followed by education or estimated IQ which explained and additional 7.7%–25.6% of the variance. Race and gender explained much less variance but were nevertheless predictive in some of the regressions. Additional evidence suggesting the impact of race on cognitive test performance was reported by Patton and colleagues who found that cognitively intact community-dwelling older adult African Americans had lower scores on RBANS than age-, education-, and gender-matched Caucasians (Patton et al., 2003).

Performance varies by intelligence level, with individuals of lower intelligence tending to have lower scores than those with higher intelligence. Diaz-Asper and colleagues studied the effects of IQ on cognitive test performance in a sample of 221 cognitively intact adults using 28 scores from 16 different neuropsychological tests (Diaz-Asper et al., 2004). They found that the group with below average IQs had significantly lower performance than those with average IQs on 25 of the 28 scores analyzed with a mean effect size (Cohen's d) of 0.73, which is a medium effect. The average IQ group had significantly lower performance than those with above average IQ on 19 of the 28 scores analyzed with mean effect size (Cohen's d) of 0.41, which is a small effect. In addition, when they examined the effects of IQ on a CII using regression analyses, they found that IQ explained from 22% to 31% of the variance (R2) depending on the type of function (linear, quadratic, and cubic). Finally, when they repeated these regression analyses using a composite performance index that was more normally distributed than the CIIs the amount of variance explained (R2) by IQ ranged from 47% to 51%.

To reduce the likelihood that low scores are misinterpreted as clinically significant when examined in isolation, some neuropsychologists have recommended that all scores from either a full test battery or a specific cognitive domain be evaluated simultaneously (Iverson, Brooks, & Holdnack, 2012). To do this, it is necessary to first determine base rates of the number of low scores in populations of interest for a given battery of neuropsychological tests. The base rates can then be used to determine how common, or uncommon, it is to have a particular number of low scores across the entire test battery under consideration.

The use of frequency data is advantageous because it allows for the application of contingency table analyses and simple but powerful statistics, such as sensitivity, specificity, odds ratios, and positive and negative predictive power to evaluate the utility of the test battery to discriminate between healthy individuals and those with cognitive impairment resulting from a given pathology (Chelune, 2010). These statistics are well suited for determining the probability of the occurrence of a discrete outcome; such as whether or not a patient's low test scores are indicative of pathology. This type of probabilistic thinking has become one of the foundations of contemporary clinical reasoning (Beck, Pyle, & Lusted, 1984; Ledley & Lusted, 1959; Lusted, 1991; Sox, 1986).

The purpose of this study is to estimate how common it for healthy service members to have one or more low scores on the Automated Neuropsychological Assessment Metrics (Version 4) Traumatic Brain Injury Military (ANAM4 TBI-MIL) (Center for the Study of Human Operator Performance, 2007) battery using various cutoffs and how the frequency of low scores can be influenced by certain demographic characteristics. Another purpose is to illustrate how base rate analyses can be used by clinicians and researchers to evaluate ANAM4 TBI-MIL's ability to identify impaired cognitive functioning following mild traumatic brain injury (mTBI). ANAM4 TBI-MIL is a battery of computerized cognitive tests currently used by the U.S. military to evaluate service members who have had an mTBI. It is part of the ANAM family of 30 tests and modules. ANAM evolved from DoD efforts begun in the 1980s to develop tests for assessing service member performance in various situations that might degrade performance, such as sustained operations, while experiencing sleep deprivation, and while taking medications, particularly drugs designed to pre-treat service members who might be exposed to chemical weapons (Reeves, Winter, Bleiberg, & Kane, 2007). The emphasis was to develop a battery of tests that were suitable for repeated measures and the generation of alternate forms in order to identify performance changes over time. The DoD has been the primary sponsor of ANAM development but in 2006 the Center for the Study of Human Operator Performance (now called the Cognitive Science Research Center) at the University of Oklahoma was licensed to begin transitioning ANAM for commercial use and is primarily responsible for the development of ANAM4.

One recently published article about the ANAM4 TBI-MIL military normative data presented base rates of low scores in healthy U.S. military service members for two cutoffs that are used to indicate “below average” (≥1.3 SD) and “clearly below average” (≥2 SDs) performance (Vincent, Roebuck-Spencer, Gilliland, & Schlegel, 2012). These were the original cutoffs provided for the commercial version of ANAM4. However, it is important to have base rates of low scores on ANAM4 TBI-MIL using additional cutoffs because various cutoffs are used by clinicians and researchers and having this information might facilitate future research comparisons. Furthermore, Vincent and colleagues did not study base rate data in depth such as examining potential variability in base rates due to demographic and education factors. They also did not present base rates of low scores for service members with mTBI and they did not illustrate how base rate analysis could be used to identify cognitive impairment with ANAM4 TBI-MIL. Therefore, this article builds upon the work on ANAM4 TBI-MIL base rates begun by Vincent and colleagues.

Methods

Materials

ANAM4 TBI-MIL is a battery of computerized cognitive tests currently used by the U.S. military to evaluate service members who have sustained an mTBI. It contains seven tests designed to assess cognitive functioning as well as a mood scale and a sleepiness scale. These are described in Table 1. For each of the seven cognitive tests, three main scores are provided: reaction time, accuracy, and throughput. Throughput is a composite of accuracy and reaction time measuring performance effectiveness or efficiency and as a matter of historical convention is the ANAM score most often used. More recently, throughput has been shown to have more attractive features for psychological and behavioral testing over reaction time and accuracy alone because it is more sensitive than either when they decrease or increase in the same direction, and it is more stable than reaction time and accuracy alone when speed-accuracy trade-offs have to be considered (Thorne, 2006). Throughput is defined as the number of correct responses on a task per unit of time available to complete the task and can be scaled to various time intervals such as seconds or minutes. Throughput is believed to best reflect the underlying processes at work in ANAM and was the score analyzed in our study (Short, Cernich, Wilken, & Kane, 2007).

Table 1.

ANAM4 TBI-MIL description

Test/module Description 
Mood Scale (MOO) Assesses the user's mood state or trait in six subcategories: Vigor, Happiness, Depression, Anger, Fatigue, and Anxiety. 
Sleepiness Scale (SLP) Identifies the user's current level of arousal. 
Simple Reaction Time (SRT) Provides an index of attention and visuo-motor response timing by having the user respond as fast as possible after a snowflake symbol that appears on the monitor. 
Code Substitution- Learning (CDS) Provides an index of complex scanning, visual tracking, and attention by requiring the user to compare a digit-symbol pair to a predefined set of digit-symbol pairs that appear on the monitor. 
Procedural Reaction Time (PRO) Measures reaction time and processing efficiency by presenting the user with a number on the monitor. The user then has to indicate whether the number is low (2 or 3) or high (4 or 5) by pressing the left or right mouse button, respectively. 
Mathematical Processing (MTH) Provides an index of basic computational skills, concentration and working memory by asking the user to solve a simple arithmetic problem and indicate whether the answer is less than or greater than five. 
Matching to Sample (MSP) Provides an index of spatial processing and visuo-spatial working memory by requiring the user to first view a pattern in a 4 × 4 sample grid and then determining which of two additional grids that appear matches the sample grid. 
Code Substitution- Delayed (CDD) Provides a measure of learning and delayed visual recognition memory by requiring the user to determine whether a digit-symbol pair matches those presented during the Code Substitution- Learning test presented earlier in the battery. 
Simple Reaction Time (Repeated) (SR2) This test is identical to the Simple Reaction Time test administered earlier in the battery and is designed to assess fatigue. 
Test/module Description 
Mood Scale (MOO) Assesses the user's mood state or trait in six subcategories: Vigor, Happiness, Depression, Anger, Fatigue, and Anxiety. 
Sleepiness Scale (SLP) Identifies the user's current level of arousal. 
Simple Reaction Time (SRT) Provides an index of attention and visuo-motor response timing by having the user respond as fast as possible after a snowflake symbol that appears on the monitor. 
Code Substitution- Learning (CDS) Provides an index of complex scanning, visual tracking, and attention by requiring the user to compare a digit-symbol pair to a predefined set of digit-symbol pairs that appear on the monitor. 
Procedural Reaction Time (PRO) Measures reaction time and processing efficiency by presenting the user with a number on the monitor. The user then has to indicate whether the number is low (2 or 3) or high (4 or 5) by pressing the left or right mouse button, respectively. 
Mathematical Processing (MTH) Provides an index of basic computational skills, concentration and working memory by asking the user to solve a simple arithmetic problem and indicate whether the answer is less than or greater than five. 
Matching to Sample (MSP) Provides an index of spatial processing and visuo-spatial working memory by requiring the user to first view a pattern in a 4 × 4 sample grid and then determining which of two additional grids that appear matches the sample grid. 
Code Substitution- Delayed (CDD) Provides a measure of learning and delayed visual recognition memory by requiring the user to determine whether a digit-symbol pair matches those presented during the Code Substitution- Learning test presented earlier in the battery. 
Simple Reaction Time (Repeated) (SR2) This test is identical to the Simple Reaction Time test administered earlier in the battery and is designed to assess fatigue. 

Invalid ANAM performance was identified by using guidelines accompanying the recently developed ANAM Effort Measure (Cognitive Science Research Center, 2011). Invalid performance is identified two ways: when percent correct on individual ANAM tests is less than 56 percent and when the value of the composite Effort Index is ≥14. The development and initial validation of the ANAM Effort Index is described in a separate publication (Roebuck-Spencer, Vincent, Gilliland, Johnson, & Cooper, 2013). In the present study, participants whose ANAM performance met either or both of these criteria were excluded from the analysis.

Participants

Data from a sample of 789 male active duty U.S. Army soldiers, 733 healthy and 56 with evidence of mTBI, were selected for this analysis from a larger convenience sample of 2,002 soldiers who participated in an IRB approved study of TBI at their home base in the United States. The larger sample consisted of 1,825 healthy male and female soldiers who may or may not have had a self-reported lifetime history of head injury or TBI and 177 soldiers who had sought treatment for a suspected TBI at either the emergency department at their base hospital or the base's TBI clinic. The head injury/TBI status of the healthy soldiers was determined by their responses to a computerized self-report questionnaire that asked if they had ever had a head injury and, if they had, enabled them to describe up to three separate injury events including month and year, cause, and whether or not the injury resulted in widely accepted signs of TBI such as being dazed or confused, inability to remember the injury event, and loss of consciousness (LOC).

Table 2 shows the number of soldiers who were excluded from the analysis and the reasons for their exclusion. Healthy soldiers who reported a lifetime history of head injury or TBI (n = 619) as well as those who did not know if they had ever had a head injury (n = 71) were excluded from the analysis. Healthy female soldiers were excluded, even if they did not have a history of head injury or TBI, because of insufficient numbers. Three healthy male soldiers who were missing demographic data were also excluded. In addition, 251 healthy soldiers were excluded from the analysis because of invalid ANAM performance, as determined by the ANAM Effort Measure, or because the validity of their overall ANAM performance could not be assessed. Data from 733 healthy soldiers were used in the analysis.

Table 2.

Number of soldiers not included by analysis group and reason for exclusion

Reasons for exclusion from analysis Healthy soldiers n (%) mTBI Sample n (%) 
Initial sample available 1,825 177 
Lifetime Head Injury History (applies to healthy only) 
 History of head injury/TBI 619 (33.9) NA 
 Unknown history of head injury/TBI 71 (3.9) NA 
Demographics 
 Female 146 (8.0) 11 (6.2) 
 Missing pay grade data 1 (0.1) 0 (0.0) 
 <12 years of education 4 (0.2) 0 (0.0) 
TBI characteristics (applies to clinical group only) 
 No TBI NA 5 (2.8) 
 Cannot determine if injury was TBI NA 4 (2.3) 
 Had TBI but cannot determine severity NA 37 (20.9) 
 TBI greater than mild severity NA 11 (6.2) 
ANAM 
 Completed <7 ANAM tests 0 (0.0) 17 (9.6) 
 Could not assess overall ANAM effort 46 (2.5) 8 (4.5) 
 Poor overall ANAM effort 53 (2.9) 14 (7.9) 
 <56% correct on one or more ANAM tests 152 (8.3) 14 (7.9) 
Total included in analysis 733 (40.2) 56 (31.6) 
Reasons for exclusion from analysis Healthy soldiers n (%) mTBI Sample n (%) 
Initial sample available 1,825 177 
Lifetime Head Injury History (applies to healthy only) 
 History of head injury/TBI 619 (33.9) NA 
 Unknown history of head injury/TBI 71 (3.9) NA 
Demographics 
 Female 146 (8.0) 11 (6.2) 
 Missing pay grade data 1 (0.1) 0 (0.0) 
 <12 years of education 4 (0.2) 0 (0.0) 
TBI characteristics (applies to clinical group only) 
 No TBI NA 5 (2.8) 
 Cannot determine if injury was TBI NA 4 (2.3) 
 Had TBI but cannot determine severity NA 37 (20.9) 
 TBI greater than mild severity NA 11 (6.2) 
ANAM 
 Completed <7 ANAM tests 0 (0.0) 17 (9.6) 
 Could not assess overall ANAM effort 46 (2.5) 8 (4.5) 
 Poor overall ANAM effort 53 (2.9) 14 (7.9) 
 <56% correct on one or more ANAM tests 152 (8.3) 14 (7.9) 
Total included in analysis 733 (40.2) 56 (31.6) 

Note: NA = not applicable.

The injury characteristics of the soldiers clinically evaluated with a suspected TBI were obtained from a computerized self-report questionnaire that asked when the injury happened, if they were dazed or confused, if they experienced any LOC, and if they could remember the injury event. If they experienced LOC, they were also asked to report the duration of LOC and if they could not remember the injury event, they were also asked to report how long it was before they started remembering new things again. Anyone who reported being dazed or confused only, LOC lasting 20 min or less, and/or having amnesia for <24 h after the injury event were considered to have had a mTBI. Eleven soldiers who reported LOC lasting >20 min or amnesia lasting >24 h were considered to have TBIs that were greater than mild and excluded from the analysis (Table 2). Clinically evaluated soldiers who did not have evidence of TBI (5), for whom TBI status could not be determined (4), or for whom TBI severity could not be determined (37) were also excluded from the analysis. Female soldiers in the clinical group (11) were excluded because of insufficient numbers. In addition, 17 soldiers from the clinical group were excluded for failing to complete all seven ANAM tests and 36 were excluded because they either had invalid ANAM performance as determined by the ANAM Effort measure, or because the validity of their ANAM performance could not be assessed. A significantly higher proportion of soldiers with mTBI were identified as having invalid scores by the ANAM effort indicators than soldiers in the healthy group (33.3% vs. 21.9%, p = .021). A total of 56 male soldiers with evidence of an mTBI were included in the analysis.

Procedures

All soldiers interested in participating were instructed to go to the Defense and Veterans Brain Injury Center test facility. Once there, the study was explained to them and they were given the opportunity to participate or decline. All soldiers agreeing to participate had to provide written informed consent. Healthy soldiers were tested in a group setting with up to 25 soldiers tested at one time and proctors available to assist them if they had questions about or problems with ANAM. Soldiers recruited because of a suspected TBI were tested individually but with a proctor present. In addition to ANAM4 TBI-MIL and the head injury/TBI questionnaires described above, all soldiers completed a computerized demographic questionnaire.

Analysis

t-Tests and Fisher's exact tests were used to compare the demographic and military characteristics of the healthy and mTBI groups. Four different clinically relevant cutoffs previously used by Iverson for classifying the cognitive functioning of athletes were used to classify ANAM4 TBI-MIL performance (Iverson, 2011). These were: first, below the 16th percentile; second, below the 10th percentile; third, at or below the 5th percentile; and fourth, at or below the 2nd percentile. Age-based reference values from ANAM4 military normative data were used for this (Cognitive Science Research Center, 2009; Vincent et al., 2012). The number of tests with throughput scores that were below the four cutoffs was calculated to establish base rates of low ANAM4 TBI-MIL scores among the healthy soldiers. We developed base rates for these soldiers overall as well as by race/ethnicity and education level. To illustrate how base rates can be used clinically, we compared the overall rates of low scores from the sample of soldiers with self-reported mTBIs to the overall rates from the healthy soldiers. We used unadjusted and adjusted risk ratios (RR) to assess the strength of association, if any, between mTBI status and abnormal ANAM4 TBI-MIL performance for each of the four cutoffs. We calculated 95% confidence intervals (CIs) for unadjusted RRs using the Taylor Series method to determine whether any RR were statistically significant (Sahai & Khurshid, 1996). Multivariable Poisson regression was used to determine if the effect of mTBI on low ANAM performance was diminished or eliminated by confounding variables, particularly race/ethnicity, education, and pay grade. Although originally developed for the analysis of count data, Poisson regression can be used to estimate prevalence ratios and RR for dichotomous-dependent variables such as those that were used in this analysis (Deddens & Peterson, 2008). The RR produced by Poisson regression are easier to interpret than odds ratios, which are produced by logistic regression. IBM SPSS Statistics, Version 19 was used for these analyses (SPSS, Inc., 2010). To do this, all independent variables were entered into the model simultaneously using the GENLIN command with a log link and robust (or Sandwich) covariance estimator. Separate univariate Poisson regression models were used to determine if time since injury influenced ANAM performance among those with mTBI. All models were checked for overdispersion, a situation in which the actual variance in the data exceeds the amount of variance assumed by the Poisson model. Overdispersion results in lower standard errors of the parameter estimates which results in CIs of the estimated RRs that are too narrow. When overdispersion was present, we adjusted the standard errors of the parameter estimates by multiplying them by the square root of the dispersion parameter as suggested by Kianifard and Gallo (1995). This enlarged the standard errors of the parameter estimates based on the amount of overdispersion, resulting in wider CIs and thereby decreasing the risk of committing Type I errors.

We used guidelines suggested by Ferguson to determine the effect size of any associations indicated by the RRs (Ferguson, 2009). If a 95% CI for an RR overlapped two or more effect size categories, such as moderate and strong, the smaller effect size was reported. When the smaller effect size of an RR for a corresponding cutoff was moderate or strong, performance at or below that cutoff was considered to be indicative of cognitive impairment. For example, if the risk of soldiers with mTBI having four or more scores below the 16th percentile is at least moderate compared with healthy soldiers, having at least that many scores at or below the 16th percentile would be considered an indication of cognitive impairment.

To further illustrate how base rate analysis can be utilized to help interpret test performance data, we compared the mean throughput scores from each ANAM4 TBI-MIL test from healthy soldiers to first, those with mTBI who were not impaired and second, those with mTBI who were impaired using t-tests. Impairment status was determined by using the results of the base rate analysis, as described above. We also compared the means from those with mTBI who were not impaired to those with mTBI who were impaired. We used Cohen's d to measure the effect sizes of the differences between the means (Cohen, 1988). Cohen suggested that the minimum value of d for a small effect is 0.2, a medium effect is 0.5, and a large effect is 0.8. A visual inspection of the throughput scores indicated that they appeared to be normally distributed for the healthy group but less normally distributed for the impaired and not impaired mTBI groups. Although t-tests are robust when some deviation from normality is present, we also compared throughput between the groups using the Mann–Whitney test.

Because all of the cognitive tests in the ANAM4 TBI-MIL battery measure reaction time and two of the tests, simple reaction time (SRT) and simple reaction time repeated (SR2) are identical, and because reaction time is a component of throughput, there is a possibility that some of the throughput scores from tests on ANAM4 TBI-MIL are highly correlated. High intercorrelation would suggest that some tests are redundant and should not be used in the simultaneous evaluation of multiple ANAM scores. This would be analogous to excluding independent variables from a regression model because of multicolinearity.

To check for the presence of multicolinearity and assess its possible impact on the simultaneous evaluation of multiple ANAM4 TBI-MIL throughput scores, we used bivariate correlation analysis to examine the intercorrelations between each ANAM4 TBI-MIL test. According to Cohen, when the value of the correlation coefficient r is at least .10, the effect is small; when r is at least .30, the effect is medium; and when r is at least .50, the effect is large (Cohen, 1988). Additionally, we used ordinary least squares (OLS) multiple regression to estimate the amount of variance, R2, in the throughput score from each ANAM4 TBI-MIL test that was explained by the other six tests combined.

Because of the large number of significance tests used in this analysis, we controlled the false discovery rate to reduce the number of type I errors. This approach to multiple testing controls the expected proportion of falsely rejected null hypotheses rather than controlling the family-wise error rate as in the Bonferroni correction (Benjamini & Hochberg, 1995). Since this approach takes into account the expected number of erroneous rejections and not only whether any error occurred, it is less stringent than the more traditional methods of controlling the family-wise error rate and is therefore more powerful, especially when the number of significance tests is large. Any p-values that did not remain significant after controlling for multiple comparisons are identified in the tables by gray shading.

Results

The injury characteristics of those with mTBI are presented in Table 3. Soldiers who sustained an mTBI were given ANAM an average of 23 days post-injury (SD = 36.1; IQR = 5.0–30.8, range = 0–245). Eleven (19.6%) reported experiencing only altered consciousness, 15 (26.8%) could not remember the injury event, 10 (17.9%) reported LOC, and 20 (35.7%) reported both LOC and not remembering the injury event. Most of those who could not remember the injury event indicated that their memory problems lasted <30 min. Many of those with LOC indicated that it lasted fewer than 5 min.

Table 3.

Injury characteristics of soldiers in the mTBI sample

 Overall (n = 56) Altered consciousness only (n = 11) Could not remember injury event (n = 15) Loss of consciousness (n = 10) Both LOC and could not remember injury event (n = 20) 
Days since injury
 
 Mean (SD23.3 (36.1) 31.8 (30.1) 16.9 (15.3) 41.5 (73.6) 14.3 (15.3) 
 Median 11.00 31.00 12.00 12.00 6.00 
 Minimum 0.00 0.00 5.00 3.00 3.00 
 25th percentile 5.00 2.00 6.00 4.75 5.00 
 75th percentile 30.75 61.00 22.00 40.50 22.75 
 Maximum 245.00 71.00 60.00 245.00 55.00 
Amnesia duration 
 1–59 s 10.7% NA 20.0% NA 15.0% 
 1–20 min 35.7% NA 60.0% NA 55.0% 
 21 to <30 min 12.5% NA 20.0% NA 20.0% 
 30–59 min 1.8% NA 0.0% NA 5.0% 
 1–24 h 1.8% NA 0.0% NA 5.0% 
 Not applicable 37.5% 100.0% 0.0% 0.0% 0.0% 
LOC duration 
 1–15 s 16.1% NA NA 60.0% 15.0% 
 16–59 s 7.1% NA NA 10.0% 15.0% 
 1 to <5 min 12.5% NA NA 20.0% 25.0% 
 5–20 min 7.1% NA NA 10.0% 15.0% 
 Do not know 10.7% NA NA 0.0% 30.0% 
 Not applicable 46.4% 100.0% 100.0% 0.0% 0.0% 
 Overall (n = 56) Altered consciousness only (n = 11) Could not remember injury event (n = 15) Loss of consciousness (n = 10) Both LOC and could not remember injury event (n = 20) 
Days since injury
 
 Mean (SD23.3 (36.1) 31.8 (30.1) 16.9 (15.3) 41.5 (73.6) 14.3 (15.3) 
 Median 11.00 31.00 12.00 12.00 6.00 
 Minimum 0.00 0.00 5.00 3.00 3.00 
 25th percentile 5.00 2.00 6.00 4.75 5.00 
 75th percentile 30.75 61.00 22.00 40.50 22.75 
 Maximum 245.00 71.00 60.00 245.00 55.00 
Amnesia duration 
 1–59 s 10.7% NA 20.0% NA 15.0% 
 1–20 min 35.7% NA 60.0% NA 55.0% 
 21 to <30 min 12.5% NA 20.0% NA 20.0% 
 30–59 min 1.8% NA 0.0% NA 5.0% 
 1–24 h 1.8% NA 0.0% NA 5.0% 
 Not applicable 37.5% 100.0% 0.0% 0.0% 0.0% 
LOC duration 
 1–15 s 16.1% NA NA 60.0% 15.0% 
 16–59 s 7.1% NA NA 10.0% 15.0% 
 1 to <5 min 12.5% NA NA 20.0% 25.0% 
 5–20 min 7.1% NA NA 10.0% 15.0% 
 Do not know 10.7% NA NA 0.0% 30.0% 
 Not applicable 46.4% 100.0% 100.0% 0.0% 0.0% 

Demographic and military characteristics of the soldiers from the healthy and mTBI groups are shown in Table 4. The healthy soldiers were mostly white with a high school education or some college and were mostly junior enlisted or non-commissioned officers. They had a mean age of 28.2 years (SD = 6.8) and had been in the Army an average of 5.4 years (SD = 5.1). The soldiers with mTBI were similar with respect to age and time spent in the Army but were more likely to be white, have some college-level education or graduated from college, and to be senior non-commissioned officers or warrant/commissioned officers.

Table 4.

Demographic and military characteristics by participant type

 Healthy soldiers (n = 733) mTBI (n = 56) p-valueb 
Age 28.2 (6.8)a 26.9 (6.5) .164 
Race 
 White 64.8% 78.6% .005 
 African American 16.4% 1.8% 
 Hispanic 13.2% 10.7% 
 Other 5.6% 8.9% 
Education 
 High school grad/GED 56.8% 37.5% .013 
 Some college 30.8% 48.2% 
 College grad 12.4% 14.3% 
Years on active duty 5.4 (5.1)a 6.0 (6.0)a .393 
Military rank 
 Junior enlisted 58.8% 55.4% .018 
 Non-commissioned officers 32.5% 23.2% 
 Senior non-commissioned officers 4.8% 10.7% 
 Warrant/commissioned officers 4.0% 10.7% 
 Healthy soldiers (n = 733) mTBI (n = 56) p-valueb 
Age 28.2 (6.8)a 26.9 (6.5) .164 
Race 
 White 64.8% 78.6% .005 
 African American 16.4% 1.8% 
 Hispanic 13.2% 10.7% 
 Other 5.6% 8.9% 
Education 
 High school grad/GED 56.8% 37.5% .013 
 Some college 30.8% 48.2% 
 College grad 12.4% 14.3% 
Years on active duty 5.4 (5.1)a 6.0 (6.0)a .393 
Military rank 
 Junior enlisted 58.8% 55.4% .018 
 Non-commissioned officers 32.5% 23.2% 
 Senior non-commissioned officers 4.8% 10.7% 
 Warrant/commissioned officers 4.0% 10.7% 

Notes: aMean (SD).

bStudent's t-test was used to compare means. Fisher's exact test was used to compare categorical variable.

The base rates of low ANAM4 throughput scores for the entire sample of healthy soldiers, and stratified by race and education, are presented in Table 5. Larger proportions of African-American and Hispanic soldiers had low scores at all four cutoffs than white soldiers, with African Americans having the largest proportions with low scores. The proportion of soldiers with low scores was not affected by education as indicated by a lack of significant differences between groups.

Table 5.

Percentages of male active duty soldiers without a history of head injury with low scores on ANAM4 TBI-MIL at selected cutoffs by race/ethnicity and education

Number of scores below cutoff Total sample (N = 733) Race/ethnicity
 
Years of education
 
White (N = 475) African American (N = 120) Hispanic (N = 97) Other (N = 41) p-valuea 12 (N = 416) 13–15 (N = 226) 16+ (N = 91) p-valuea 
<16th percentile (1 SD) 
 0 scores below cutoff 53.9 59.2 39.2 46.4 53,7  49.8 61.9 52.7  
 1 or more scores below cutoff 46.1 40.8 60.8 53.6 46.3 <.001 50.2 38.1 47.3 .012 
 2 or more scores below cutoff 22.0 16.0 39.2 30.9 19.5 <.001 23.6 19.0 22.0 .422 
 3 or more scores below cutoff 9.3 6.3 20.8 11.3 4.9 <.001 10.8 6.6 8.8 .223 
 4 or more scores below cutoff 4.0 2.3 10.0 6.2 — .001 4.3 3.1 4.4 .766 
 5 or more scores below cutoff 2.3 1.5 5.0 4.1 — .051 2.2 2.2 3.3 .728 
 6 or more scores below cutoff 0.8 0.4 1.7 2.1 — .171 0.5 1.3 1.1 .386 
 7 scores below cutoff 0.1 — — 1.0 — .188 0.2 — — 1.000 
<10th percentile 
 0 scores below cutoff 66.5 71.2 49.2 62.9 63.4  63.0 71.2 67.0  
 1 or more scores below cutoff 33.5 28.8 50.8 37.1 36.6 <.001 37.0 28.8 33.0 .108 
 2 or more scores below cutoff 11.4 7.4 24.2 17.5 9.8 <.001 13.5 9.3 8.8 .211 
 3 or more scores below cutoff 4.4 2.9 11.7 5.2 — .001 5.0 4.0 3.3 .790 
 4 or more scores below cutoff 1.9 1.1 5.0 3.1 — .026 2.2 1.3 2.2 .734 
 5 or more scores below cutoff 1.1 0.6 2.5 2.1 — .162 1.0 1.3 1.1 .882 
 6 or more scores below cutoff 0.4 — 0.8 2.1 —   .039 0.2 0.4 1.1 .358 
 7 scores below cutoff 0.1 — — 1.0 — .188 0.2 — 0.0 1.000 
≤5th percentile 
 0 scores below cutoff 81.2 84.8 66.7 82.5 78.0  80.0 84.1 79.1  
 1 or more scores below cutoff 18.8 15.2 33.3 17.5 22.0 <.001 20.0 15.9 20.9 .397 
 2 or more scores below cutoff 4.6 1.9 12.5 7.2 7.3 <.001 5.0 4.4 3.3 .851 
 3 or more scores below cutoff 1.6 0.8 4.2 3.1 — .030 1.7 1.8 1.1 1.000 
 4 or more scores below cutoff 0.7 0.4 1.7 1.0 — .287 0.5 0.9 1.1 .526 
 5 or more scores below cutoff 0.1 — — 1.0 — .188 — — 1.1 .124 
≤2nd percentile (2 SDs
 0 scores below cutoff 92.9 92.5 85.8 92.8 95.1  92.5 93.4 93.4  
 1 or more scores below cutoff 7.1 5.5 14.2 7.2 4.9 .017 7.5 6.6 6.6 .957 
 2 or more scores below cutoff 1.2 0.4 4.2 2.1 — .008 1.0 1.8 1.1 .708 
 3 or more scores below cutoff 0.3 — 0.8 1.0 — .124 — — 1.1 .328 
 4 or more scores below cutoff 0.1 — — 1.0 — .188 — — 1.1 .124 
Number of scores below cutoff Total sample (N = 733) Race/ethnicity
 
Years of education
 
White (N = 475) African American (N = 120) Hispanic (N = 97) Other (N = 41) p-valuea 12 (N = 416) 13–15 (N = 226) 16+ (N = 91) p-valuea 
<16th percentile (1 SD) 
 0 scores below cutoff 53.9 59.2 39.2 46.4 53,7  49.8 61.9 52.7  
 1 or more scores below cutoff 46.1 40.8 60.8 53.6 46.3 <.001 50.2 38.1 47.3 .012 
 2 or more scores below cutoff 22.0 16.0 39.2 30.9 19.5 <.001 23.6 19.0 22.0 .422 
 3 or more scores below cutoff 9.3 6.3 20.8 11.3 4.9 <.001 10.8 6.6 8.8 .223 
 4 or more scores below cutoff 4.0 2.3 10.0 6.2 — .001 4.3 3.1 4.4 .766 
 5 or more scores below cutoff 2.3 1.5 5.0 4.1 — .051 2.2 2.2 3.3 .728 
 6 or more scores below cutoff 0.8 0.4 1.7 2.1 — .171 0.5 1.3 1.1 .386 
 7 scores below cutoff 0.1 — — 1.0 — .188 0.2 — — 1.000 
<10th percentile 
 0 scores below cutoff 66.5 71.2 49.2 62.9 63.4  63.0 71.2 67.0  
 1 or more scores below cutoff 33.5 28.8 50.8 37.1 36.6 <.001 37.0 28.8 33.0 .108 
 2 or more scores below cutoff 11.4 7.4 24.2 17.5 9.8 <.001 13.5 9.3 8.8 .211 
 3 or more scores below cutoff 4.4 2.9 11.7 5.2 — .001 5.0 4.0 3.3 .790 
 4 or more scores below cutoff 1.9 1.1 5.0 3.1 — .026 2.2 1.3 2.2 .734 
 5 or more scores below cutoff 1.1 0.6 2.5 2.1 — .162 1.0 1.3 1.1 .882 
 6 or more scores below cutoff 0.4 — 0.8 2.1 —   .039 0.2 0.4 1.1 .358 
 7 scores below cutoff 0.1 — — 1.0 — .188 0.2 — 0.0 1.000 
≤5th percentile 
 0 scores below cutoff 81.2 84.8 66.7 82.5 78.0  80.0 84.1 79.1  
 1 or more scores below cutoff 18.8 15.2 33.3 17.5 22.0 <.001 20.0 15.9 20.9 .397 
 2 or more scores below cutoff 4.6 1.9 12.5 7.2 7.3 <.001 5.0 4.4 3.3 .851 
 3 or more scores below cutoff 1.6 0.8 4.2 3.1 — .030 1.7 1.8 1.1 1.000 
 4 or more scores below cutoff 0.7 0.4 1.7 1.0 — .287 0.5 0.9 1.1 .526 
 5 or more scores below cutoff 0.1 — — 1.0 — .188 — — 1.1 .124 
≤2nd percentile (2 SDs
 0 scores below cutoff 92.9 92.5 85.8 92.8 95.1  92.5 93.4 93.4  
 1 or more scores below cutoff 7.1 5.5 14.2 7.2 4.9 .017 7.5 6.6 6.6 .957 
 2 or more scores below cutoff 1.2 0.4 4.2 2.1 — .008 1.0 1.8 1.1 .708 
 3 or more scores below cutoff 0.3 — 0.8 1.0 — .124 — — 1.1 .328 
 4 or more scores below cutoff 0.1 — — 1.0 — .188 — — 1.1 .124 

Notes: aFisher's exact test.

Cells shaded in gray indicate that Fisher's exact test results did not remain statistically significant after controlling for multiple comparisons.

Table 6 shows the frequencies of low ANAM4 TBI-MIL scores at various cutoffs for the healthy soldiers and those with mTBIs. The proportions of soldiers with mTBIs who had low ANAM4 TBI-MIL scores were higher than the proportions of healthy soldiers with low scores at all cutoffs. For example, 51.8% of soldiers with mTBI had two or more scores below the 16th percentile compared with 22.0% of healthy soldiers (RR = 2.36, 95% CI 1.75–3.18). Moreover, at each of the four cutoffs, the association between mTBI and the risk of having some low scores was frequently moderate or strong as indicated by the effect sizes in Table 6. For example, having four, five, or six or more scores below the 16th percentile was moderately to strongly associated with mTBI. However, when the RRs indicated a moderate or strong association between mTBI and having low ANAM4 TBI-MIL scores, a relatively small minority of soldiers with mTBI had low scores. Time since injury had no effect on ANAM4 TBI-MIL performance among those with mTBI regardless of cutoff and was excluded from the multivariate analysis.

Table 6.

Risk of having low scores on ANAM4 TBI-MIL at selected cutoffs by male soldiers with mTBI compared with healthy male soldiers

Number of scores below cutoff mTBI (N = 56) Healthy (N = 733) Unadjusted risk ratio (URR) 95% CI (URR) Effect sizea Adjusted risk ratiob (ARR) 95% CI (ARR)c Effect sizea 
<16th percentile (1 SD) 
 0 scores below cutoff 26.8 53.9 0.50 0.31–0.79* NP-M 0.44 0.28–0.68* NP-M 
 1 or more scores below cutoff 73.2 46.1 1.59 1.32–1.91* NP 1.82 1.52–2.18* NP-MPE 
 2 or more scores below cutoff 51.8 22.0 2.36 1.75–3.18* NP-M 2.98 2.21–4.02* MPE-M 
 3 or more scores below cutoff 37.5 9.3 4.04 2.64–6.18* MPE-S 6.57 4.39–9.84* 
 4 or more scores below cutoff 32.1 4.0 8.12 4.72–13.99* 14.19 8.20–24.55* 
 5 or more scores below cutoff 23.2 2.3 10.01 4.98–20.09* 14.48 7.15–29.29* 
 6 or more scores below cutoff 16.1 0.8 19.63 6.96–55.35* 25.70 8.03–82.30* 
 7 scores below cutoff 5.4 0.1 39.27 3.80–404.99* MPE-S d d — 
<10th percentile 
 0 scores below cutoff 39.3 66.5 0.59 0.42–0.83* NP-MPE 0.54 0.39–0.75* NP-MPE 
 1 or more scores below cutoff 60.7 33.5 1.81 1.42–2.31* NP-MPE 2.16 1.70–2.74* NP-MPE 
 2 or more scores below cutoff 37.5 11.4 3.28 2.17–4.94* MPE-S 4.89 3.30–7.25* M-S 
 3 or more scores below cutoff 33.9 4.4 7.64 4.55–12.81* 12.28 7.31–20.63* 
 4 or more scores below cutoff 25.0 1.9 13.27 6.45–27.26* 22.51 10.48–48.36* 
 5 or more scores below cutoff 17.9 1.1 16.58 6.55–41.98* 22.32 7.21–69.18* 
 6 or more scores below cutoff 7.1 0.4 17.69 3.80–82.35* M-S d d — 
 7 scores below cutoff 1.8 0.1 13.27 0.74–237.12 NP d d — 
≤5th percentile 
 0 scores below cutoff 55.4 81.2 0.68 0.53–0.87* NP 0.64 0.51–0.81* NP 
 1 or more scores below cutoff 44.6 18.8 2.37 1.68–3.34* NP-M 3.03 2.17–4.23* NP-S 
 2 or more scores below cutoff 32.1 4.6 6.93 4.10–11.70* 11.73 6.77–20.31* 
 3 or more scores below cutoff 23.2 1.6 14.18 6.59–30.50* 24.19 9.93–58.91* 
 4 or more scores below cutoff 17.9 0.7 26.18 8.89–77.02* 43.04 8.40–220.50* 
 5 or more scores below cutoff 7.1 0.1 52.36 5.48–500.18* d d — 
 6 or more scores below cutoff 3.6 0.0 — — — — — — 
≤2nd percentile (2 SDs) 
 0 scores below cutoff 66.1 92.9 0.71 0.58–0.87* NP 0.70 0.58–0.84* NP 
 1 or more scores below cutoff 33.9 7.1 4.78 2.99–7.65* MPE-S 6.32 3.95–10.10* M-S 
 2 or more scores below cutoff 21.4 1.2 17.45 7.43–40.96* 31.88 12.22–83.17* 
 3 or more scores below cutoff 12.5 0.3 45.81 9.18–228.42* d d — 
 4 or more scores below cutoff 8.9 0.1 65.45 7.18–596.39* d d — 
 5 or more scores below cutoff 1.8 0.0 — — — — — — 
Number of scores below cutoff mTBI (N = 56) Healthy (N = 733) Unadjusted risk ratio (URR) 95% CI (URR) Effect sizea Adjusted risk ratiob (ARR) 95% CI (ARR)c Effect sizea 
<16th percentile (1 SD) 
 0 scores below cutoff 26.8 53.9 0.50 0.31–0.79* NP-M 0.44 0.28–0.68* NP-M 
 1 or more scores below cutoff 73.2 46.1 1.59 1.32–1.91* NP 1.82 1.52–2.18* NP-MPE 
 2 or more scores below cutoff 51.8 22.0 2.36 1.75–3.18* NP-M 2.98 2.21–4.02* MPE-M 
 3 or more scores below cutoff 37.5 9.3 4.04 2.64–6.18* MPE-S 6.57 4.39–9.84* 
 4 or more scores below cutoff 32.1 4.0 8.12 4.72–13.99* 14.19 8.20–24.55* 
 5 or more scores below cutoff 23.2 2.3 10.01 4.98–20.09* 14.48 7.15–29.29* 
 6 or more scores below cutoff 16.1 0.8 19.63 6.96–55.35* 25.70 8.03–82.30* 
 7 scores below cutoff 5.4 0.1 39.27 3.80–404.99* MPE-S d d — 
<10th percentile 
 0 scores below cutoff 39.3 66.5 0.59 0.42–0.83* NP-MPE 0.54 0.39–0.75* NP-MPE 
 1 or more scores below cutoff 60.7 33.5 1.81 1.42–2.31* NP-MPE 2.16 1.70–2.74* NP-MPE 
 2 or more scores below cutoff 37.5 11.4 3.28 2.17–4.94* MPE-S 4.89 3.30–7.25* M-S 
 3 or more scores below cutoff 33.9 4.4 7.64 4.55–12.81* 12.28 7.31–20.63* 
 4 or more scores below cutoff 25.0 1.9 13.27 6.45–27.26* 22.51 10.48–48.36* 
 5 or more scores below cutoff 17.9 1.1 16.58 6.55–41.98* 22.32 7.21–69.18* 
 6 or more scores below cutoff 7.1 0.4 17.69 3.80–82.35* M-S d d — 
 7 scores below cutoff 1.8 0.1 13.27 0.74–237.12 NP d d — 
≤5th percentile 
 0 scores below cutoff 55.4 81.2 0.68 0.53–0.87* NP 0.64 0.51–0.81* NP 
 1 or more scores below cutoff 44.6 18.8 2.37 1.68–3.34* NP-M 3.03 2.17–4.23* NP-S 
 2 or more scores below cutoff 32.1 4.6 6.93 4.10–11.70* 11.73 6.77–20.31* 
 3 or more scores below cutoff 23.2 1.6 14.18 6.59–30.50* 24.19 9.93–58.91* 
 4 or more scores below cutoff 17.9 0.7 26.18 8.89–77.02* 43.04 8.40–220.50* 
 5 or more scores below cutoff 7.1 0.1 52.36 5.48–500.18* d d — 
 6 or more scores below cutoff 3.6 0.0 — — — — — — 
≤2nd percentile (2 SDs) 
 0 scores below cutoff 66.1 92.9 0.71 0.58–0.87* NP 0.70 0.58–0.84* NP 
 1 or more scores below cutoff 33.9 7.1 4.78 2.99–7.65* MPE-S 6.32 3.95–10.10* M-S 
 2 or more scores below cutoff 21.4 1.2 17.45 7.43–40.96* 31.88 12.22–83.17* 
 3 or more scores below cutoff 12.5 0.3 45.81 9.18–228.42* d d — 
 4 or more scores below cutoff 8.9 0.1 65.45 7.18–596.39* d d — 
 5 or more scores below cutoff 1.8 0.0 — — — — — — 

Notes: *Statistically significant at the 0.05 level.

aNP = no practical effect (RR < 2), MPE = minimum “practical” effect (RR ≥ 2), M = moderate (RR ≥ 3), and S = strong (RR ≥ 4).

bAdjusted for effects of race/ethnicity, education, and military rank.

cWhen necessary, the standard errors of parameter estimates were multiplied by the square root of the dispersion parameter before calculating CIs to account for effects of overdispersion. This widened the CIs originally predicted by the regression model.

dModel results not reliable or could not be computed due to fit and/or calculation issues.

Table 6 also shows RR estimates and 95% CIs that were adjusted for the effects of race/ethnicity, education, and military rank. When controlling for the effects of these confounders, mTBI remained a significant predictor of having low ANAM4 TBI-MIL scores at all four cutoffs and in some cases its effect became stronger. For example, the lower boundary of the 95% CI for the unadjusted RR of having three or more scores below the 16th percentile was 2.64, which indicates a minimum practical effect (MPE) size, and was not deemed sufficient in this study for identifying cognitive impairment in those with mTBI. However, the lower boundary of the 95% CI for the adjusted RR (ARR) at the same cutoff was 4.39, indicating a strong effect. Race/ethnicity was the only other factor that was a consistently significant predictor of having low ANAM4 TBI-MIL scores in the multivariate Poisson regression models. It was associated with having low scores at all four cutoffs (data not shown). Education was only associated with having low scores below the 16th percentile. Military rank was not associated with ANAM performance at any cutoff.

Data presented in Table 7 illustrate how the results of the base rate analysis can be used to interpret the comparisons of mean throughput scores from different groups of test takers. When mean throughput scores from each of the seven ANAM4 TBI-MIL tests from healthy soldiers were compared with those from the entire mTBI group, soldiers with mTBI had significantly lower scores on all tests except for mathematical processing (MTH) and the effect sizes were small to moderate. When the soldiers in the mTBI group were classified according to cognitive impairment status (impaired or normal) using the results of the base rate analysis presented in Table 6, the results of the mean comparisons changed. Regardless of the specific cutoff used to indicate impairment, most of the mean scores from the mTBI/normal group were not significantly different from those from the healthy group.

Table 7.

Performance on ANAM4 TBI-MIL by healthy soldiers and those with mTBI grouped by impairment status

Soldier type ANAM4 throughput scores, mean (SD)
 
SRT CDS PRO MTH MSP CDD SR2 
 Healthy 733 249. 7 (34.9) 54.7 (11.36) 105.0 (14.8) 22.2 (7.2) 36.2 (11.2) 45.16 (14.8) 250.51 (34.6) 
 mTBI/all 56 212.6 (57.1) 48.7 (12.6) 93.6 (20.4) 21.7 (15.8) 30.2 (12. 6) 38.0 (15.3) 201.5 (64.5) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/all  0.65b, p< .001 0.48a, p < .001 0.56b, p < .001 0.03, p = .829** 0.48a, p < .001 0.47a, p = .001 0.76b, p < .001 
Impairment cutoff: 4 or more scores below the 16th percentile 
 mTBI/normal 38 242.8 (27.1) 53.7 (11.3) 103.4 (12.7) 23.7 (18.6) 34.9 (12.4) 44.7 (13.5) 234.1 (39.6) 
 mTBI/impaired 18 148.8 (50.8) 38.06 (7.8) 72.9 (18.0) 17.6 (5.4) 20.4 (5.1) 23.7 (6.66) 132.7 (51.4) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/normal  0.20a, p = .232 0.09, p = .608 0.10, p = .545 0.08, p = .618 0.11, p = .473 0.03, p = .850 0.41a, p = .005 
 Healthy versus mTBI/impaired  1.99c, p < .001 1.46c, p < .001 1.78c, p < .001 0.64b, p = .007 1.42c, p < .001 1.44c, p < .001 2.29c, p < .001 
 mTBI/normal versus mTBI/impaired  1.65c, p < .001 1.24c, p < .001 1.69c, p < .001 0.33a, p = .175** 1.17c, p < .001 1.55c, p < .001 1.97c, p < .001 
Impairment cutoff: 3 or more scores below the 10th percentile 
 mTBI/normal 37 242.8 (27.5) 54.2 (11.1) 103.6 (12.9) 23.9 (18.8) 35.4 (12.1) 44.9 (13.6) 238.1 (31.4) 
 mTBI/impaired 19 153.7 (53.9) 38.0 (7.6) 74.3 (18.4) 17.5 (5.2) 20.1 (5.1) 24.4 (6.9) 130.3 (51.1) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/normal  0.20a, p = .238 0.05, p.783 0.09, p = .581 0.09, p = .588 0.07, p = .675 0.02, p = .929 0.36a, p = .034 
 Healthy versus mTBI/impaired  1.78c, p < .001 1.47c, p < .001 1.67c, p < .001 0.65, p = .005 1.44c, p < .001 1.40c, p < .001 2.35c, p < .001 
 mTBI/normal versus mTBI/impaired  1.65c, p < .001 1.45c, p < .001 1.59c, p < .001 0.34a, p = .155** 1.27c, p < .001 1.51c, p < .001 2.11c, p < .001 
Impairment cutoff: 2 or more scores at or below the 5th percentile 
mTBI/normal 38 241.5 (28.2) 54.0 (11.0) 103.4 (12.8) 23. 7 (18.6) 35.0 (12.2) 44.5 (13.7) 235.7 (34.5) 
mTBI/impaired 18 151.5 (54.5) 37.4 (7.4) 73.1 (18.2) 17.7 (5.3) 20.0 (5.3) 24.2 (7.1) 129.5 (52.4) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/normal  0.23a, p = .157 0.06, p = .726 0.11, p = .518 0.08, p = .278 0.10 p = .531 0.04, p = .786 0.43a, p = .010 
 Healthy versus mTBI/impaired  1.80c, p < .001 1.52c, p < .001 1.75c, p < .001 0.63b, p = .008 1.45c, p < .001 1.41c, p < .001 2.31c, p < .001 
 mTBI/normal versus mTBI/impaired  1.65c, p < .001 1.51c, p < .001 1.66c, p < .001 0.32a, p = .186** 1.24c, p < .001 1.48c, p < .001 2.03c, p < .001 
Impairment cutoff: 2 or more scores at or below the 2nd percentile 
mTBI/normal 44 233.2 (37.6) 51.9 (11.7) 100.2 (15.1) 23.5 (17.4) 33.2 (12.3) 2.0 (14.3) 228.6 (39.2) 
mTBI/impaired 12 123.1 (53.3) 37.1 (8.8) 69.4 (19.4) 15.4 (2.5) 19.4 (5.6) 23.2 (8.0) 102.6 (34.1) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/normal  0.44a, p = .002 0.24a, p = .108 0.31a, p = .040 0.07, p = .634 0.25a, p = .083** 0.21a, p = .167 0.56b, p < .001 
 Healthy versus mTBI/impaired  2.11c, p < .001 1.55c, p < .001 1.83c, p < .001 0.94c, p < .001 1.51c, p < .001 1.48c, p < .001 4.27c, p < .001 
 mTBI/normal versus mTBI/impaired  1.80c, p < .001 1.26c, p < .001 1.59c, p < .001 0.46a, p = .118** 1.12c, p < .001 1.31c, p < .001 3.21c, p < .001 
Soldier type ANAM4 throughput scores, mean (SD)
 
SRT CDS PRO MTH MSP CDD SR2 
 Healthy 733 249. 7 (34.9) 54.7 (11.36) 105.0 (14.8) 22.2 (7.2) 36.2 (11.2) 45.16 (14.8) 250.51 (34.6) 
 mTBI/all 56 212.6 (57.1) 48.7 (12.6) 93.6 (20.4) 21.7 (15.8) 30.2 (12. 6) 38.0 (15.3) 201.5 (64.5) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/all  0.65b, p< .001 0.48a, p < .001 0.56b, p < .001 0.03, p = .829** 0.48a, p < .001 0.47a, p = .001 0.76b, p < .001 
Impairment cutoff: 4 or more scores below the 16th percentile 
 mTBI/normal 38 242.8 (27.1) 53.7 (11.3) 103.4 (12.7) 23.7 (18.6) 34.9 (12.4) 44.7 (13.5) 234.1 (39.6) 
 mTBI/impaired 18 148.8 (50.8) 38.06 (7.8) 72.9 (18.0) 17.6 (5.4) 20.4 (5.1) 23.7 (6.66) 132.7 (51.4) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/normal  0.20a, p = .232 0.09, p = .608 0.10, p = .545 0.08, p = .618 0.11, p = .473 0.03, p = .850 0.41a, p = .005 
 Healthy versus mTBI/impaired  1.99c, p < .001 1.46c, p < .001 1.78c, p < .001 0.64b, p = .007 1.42c, p < .001 1.44c, p < .001 2.29c, p < .001 
 mTBI/normal versus mTBI/impaired  1.65c, p < .001 1.24c, p < .001 1.69c, p < .001 0.33a, p = .175** 1.17c, p < .001 1.55c, p < .001 1.97c, p < .001 
Impairment cutoff: 3 or more scores below the 10th percentile 
 mTBI/normal 37 242.8 (27.5) 54.2 (11.1) 103.6 (12.9) 23.9 (18.8) 35.4 (12.1) 44.9 (13.6) 238.1 (31.4) 
 mTBI/impaired 19 153.7 (53.9) 38.0 (7.6) 74.3 (18.4) 17.5 (5.2) 20.1 (5.1) 24.4 (6.9) 130.3 (51.1) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/normal  0.20a, p = .238 0.05, p.783 0.09, p = .581 0.09, p = .588 0.07, p = .675 0.02, p = .929 0.36a, p = .034 
 Healthy versus mTBI/impaired  1.78c, p < .001 1.47c, p < .001 1.67c, p < .001 0.65, p = .005 1.44c, p < .001 1.40c, p < .001 2.35c, p < .001 
 mTBI/normal versus mTBI/impaired  1.65c, p < .001 1.45c, p < .001 1.59c, p < .001 0.34a, p = .155** 1.27c, p < .001 1.51c, p < .001 2.11c, p < .001 
Impairment cutoff: 2 or more scores at or below the 5th percentile 
mTBI/normal 38 241.5 (28.2) 54.0 (11.0) 103.4 (12.8) 23. 7 (18.6) 35.0 (12.2) 44.5 (13.7) 235.7 (34.5) 
mTBI/impaired 18 151.5 (54.5) 37.4 (7.4) 73.1 (18.2) 17.7 (5.3) 20.0 (5.3) 24.2 (7.1) 129.5 (52.4) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/normal  0.23a, p = .157 0.06, p = .726 0.11, p = .518 0.08, p = .278 0.10 p = .531 0.04, p = .786 0.43a, p = .010 
 Healthy versus mTBI/impaired  1.80c, p < .001 1.52c, p < .001 1.75c, p < .001 0.63b, p = .008 1.45c, p < .001 1.41c, p < .001 2.31c, p < .001 
 mTBI/normal versus mTBI/impaired  1.65c, p < .001 1.51c, p < .001 1.66c, p < .001 0.32a, p = .186** 1.24c, p < .001 1.48c, p < .001 2.03c, p < .001 
Impairment cutoff: 2 or more scores at or below the 2nd percentile 
mTBI/normal 44 233.2 (37.6) 51.9 (11.7) 100.2 (15.1) 23.5 (17.4) 33.2 (12.3) 2.0 (14.3) 228.6 (39.2) 
mTBI/impaired 12 123.1 (53.3) 37.1 (8.8) 69.4 (19.4) 15.4 (2.5) 19.4 (5.6) 23.2 (8.0) 102.6 (34.1) 
Effect sizes (Cohen's d with p-values from accompanying t-tests) 
 Healthy versus mTBI/normal  0.44a, p = .002 0.24a, p = .108 0.31a, p = .040 0.07, p = .634 0.25a, p = .083** 0.21a, p = .167 0.56b, p < .001 
 Healthy versus mTBI/impaired  2.11c, p < .001 1.55c, p < .001 1.83c, p < .001 0.94c, p < .001 1.51c, p < .001 1.48c, p < .001 4.27c, p < .001 
 mTBI/normal versus mTBI/impaired  1.80c, p < .001 1.26c, p < .001 1.59c, p < .001 0.46a, p = .118** 1.12c, p < .001 1.31c, p < .001 3.21c, p < .001 

Notes: Effect sizes: asmall, bmedium, and clarge. Values of Cohen's d without a symbol (<0.2) indicate no effect.

Since the distributions of throughput scores deviated from normality, Mann–Whitney tests were also performed to compare scores between groups. Except where noted by a double asterisk, Mann–Whitney tests agreed with t-tests but only the t-test p-values are shown due to lack of space.

**Mann–Whitney test yielded a statistically significant difference (p ≤ .05) between groups when t-test was not significant.

Cells shaded in gray indicate that t-test results did not remain statistically significant after controlling for multiple comparisons.

Table 7 also shows that all of the mean throughput scores from the mTBI/impaired group were significantly lower than those from the healthy group at every cutoff. In all of these cases, the effect sizes were large. In addition, most of the scores from the mTBI/impaired group were significantly lower than those from the mTBI/normal group and when they differed, the effect sizes were large. The only exceptions were mean MTH scores, which were not significantly lower.

Because the distributions of throughput scores for the mTBI group appeared to deviate from normality, we also performed Mann–Whitney tests to determine if the t-test results were valid. In all cases in which the t-tests identified a significant difference between groups, the Mann–Whitney tests also indicated significant differences. In the only instances in which the Mann–Whitney tests disagreed with the t-tests, the Mann–Whitney tests identified significant differences when the t-tests did not. These instances are identified in Table 7 by p-values marked with a double asterisk. For example, a t-test did not identify a significant difference between the MTH scores from the mTBI/impaired and mTBI/normal groups when the cutoff was 4 or more scores below the 16th percentile but the Mann–Whitney test did.

Table 8 illustrates the degree to which throughput scores are intercorrelated. In the healthy group, of the 21 bivariate correlations, 20 were small to medium and only one, SRT and SR2, was large. In the mTBI group, half of the 21 correlations were large, but 8 of these were <0.70. Furthermore, multivariate OLS regression revealed that 49%–88% of the variance in individual ANAM4 TBI-MIL throughput scores in the healthy group was unexplained by the combined performance on the other six ANAM tests. In the mTBI group, the amount of unexplained variance ranged from 26% to 53% for six of the tests and was 93% for the remaining test. Additionally, low scores occurred among all seven of the tests. For example, 43 healthy soldiers had one score at or below the 2nd percentile. Of these, 19% were from SR2 and CDD, 16% each were from CDS and MTH, 12% each were from MSP, and 9% were from PRO and SRT. In the mTBI group, 12 soldiers had scores that were greater than the 2nd percentile and at or below the 5th percentile. Of these, 33% were from MSP and 17% were from SRT, PRO, MTH, and CDD. These results suggest that the ANAM4 TBI-MIL scores are measuring different cognitive constructs and there are no redundant scores.

Table 8.

Intercorrelations between ANAM4 TBI-MIL tests from healthy soldiers and those with mTBI

 SRT CDS PRO MTH MSP CDD 
Healthy (n = 733) 
 CDS 0.24a      
 PRO 0.38b 0.45b     
 MTH 0.24a 0.24a 0.35b    
 MSP 0.24a 0.39b 0.37b 0.27a   
 CDD 0.20a 0.63c 0.31b 0.20a 0.36b  
 SR2 0.58c 0.27a 0.39b 0.19a 0.31b 0.25a 
mTBI (n = 56) 
 CDS 0.47b      
 PRO 0.65c 0.72c     
 MTH 0.13a 0.10a 0.16a    
 MSP 0.52c 0.61c 0.54c 0.18a   
 CDD 0.52c 0.80c 0.69c 0.17a 0.48b  
 SR2 0.79c 0.45b 0.65c 0.19a 0.50c 0.49b 
 SRT CDS PRO MTH MSP CDD 
Healthy (n = 733) 
 CDS 0.24a      
 PRO 0.38b 0.45b     
 MTH 0.24a 0.24a 0.35b    
 MSP 0.24a 0.39b 0.37b 0.27a   
 CDD 0.20a 0.63c 0.31b 0.20a 0.36b  
 SR2 0.58c 0.27a 0.39b 0.19a 0.31b 0.25a 
mTBI (n = 56) 
 CDS 0.47b      
 PRO 0.65c 0.72c     
 MTH 0.13a 0.10a 0.16a    
 MSP 0.52c 0.61c 0.54c 0.18a   
 CDD 0.52c 0.80c 0.69c 0.17a 0.48b  
 SR2 0.79c 0.45b 0.65c 0.19a 0.50c 0.49b 

Note: Effect sizes: asmall, bmedium, and clarge. All correlations were statistically significant at p < .001.

Discussion

This study shows that some healthy uninjured male soldiers obtain one or more low scores on ANAM4 TBI-MIL and that their performance appears to be associated with factors such as race/ethnicity. These findings are consistent with research using other neuropsychological tests that has shown that it is common for healthy people to obtain some low scores and that the number of low scores is associated with a variety of factors (Axelrod & Wall, 2007; Binder et al., 2009; Brooks, Iverson, Feldman, et al., 2009; Brooks et al., 2008; Brooks et al., 2007; Crawford et al., 2007; Diaz-Asper et al., 2004; Ingraham & Aiken, 1996; Iverson, et al., 2011; Palmer et al., 1998; Patton et al., 2003; Schretlen et al., 2008). Furthermore, our overall base rates of scores at or below the second percentile are very similar to those presented in a normative study for ANAM4 TBI-MIL using a large sample of healthy U.S. military service members (Vincent et al., 2012). Our findings suggest that ANAM4 TBI-MIL normative data should be stratified into more categories including race/ethnicity. To our knowledge, existing ANAM4 TBI-MIL normative data are not stratified by this category. However, it should be possible to do this considering that 876,507 service members have completed pre-deployment ANAM4 TBI-MIL baseline assessments from June 2007 through March 2013 (U.S. Army Neurocognitive Assessment Branch Office, 2013).

This study also shows how base rates of low scores can be used to help interpret results from ANAM4 TBI-MIL in the context of mTBI. By comparing the proportions of soldiers with mTBIs who had various numbers of low scores at different cutoffs to the proportions of healthy soldiers with various numbers of low scores at those cutoffs, we were able to provide evidence of an association between mTBI and low ANAM scores using RR, 95% CIs, and effect size guidelines. Mild TBI was moderately to strongly associated with having at least two, three, or four low ANAM scores at four different cutoffs: <16th percentile, <10th percentile, ≤5th percentile, and ≤2nd percentile and this association remained when controlling for the effects of race/ethnicity, education, and military rank (Table 6). Furthermore, despite the fact the time since injury varied greatly among those in the mTBI group and that many were tested long after the expected recovery period for mTBI, time since injury was not related to the likelihood of obtaining low scores. This is probably due to the fact that many of those in the mTBI sample were informed about the study when they sought care at the local TBI clinic. We believe that these soldiers were from the small minority of mTBI patients who do not recover within the expected recovery time and were seeking treatment for persistent symptoms and problems.

We also found that when there was a moderate to strong association between mTBI and low ANAM scores, only a minority of those with mTBI accounted for this statistical association. For three of the cutoffs, <16th percentile, <10th percentile, and ≤5th percentile, the moderate-to-strong associations occurred when no more than about one-third of those with mTBI had multiple low scores (Table 6). When the cutoff was ≤2nd percentile, the strong association occurred when 21.4% of those with mTBI had 2 or more scores at this criterion (Table 6). Furthermore, when the associations between mTBI and low ANAM scores were moderate to strong, the proportion of healthy soldiers with low scores was always <5% (Table 6). These results provide an empirical basis for identifying cutoffs at which ANAM4 TBI-MIL might reasonably indicate cognitive impairment following mTBI.

The results of the base rate analysis proved useful for interpreting results from the comparison of mean ANAM4 TBI-MIL throughput scores from the healthy and mTBI groups. As expected, only a minority of soldiers from the mTBI group accounted for most if not all of the differences in mean throughput scores between the entire mTBI group and the healthy group. This suggests that comparing means alone is often not sufficient enough to adequately characterize the level of cognitive functioning in two different groups of individuals.

Although it is tempting to make inferences about the sensitivity and specificity of ANAM to mTBI based on the results of this analysis doing so is inappropriate because ANAM4 TBI-MIL is not intended to diagnose any specific condition (Center for the Study of Human Operator Performance, 2008). The sensitivity and specificity of ANAM need to be viewed within the context of identifying cognitive impairment. Results from ANAM4 TBI-MIL would need to be compared with results from a different cognitive test battery in order to evaluate its sensitivity and specificity.

This study has several limitations. Although the soldiers in the mTBI group had sought clinical treatment for a suspected TBI, a TBI diagnosis was not confirmed and their TBI status was determined using self-reported data. Another limitation was that the mean time since injury for those in the mTBI group was 23 days. Many individuals who sustain an mTBI recover functionally sooner than that and this might explain why only a minority of those in the mTBI group had cognitive impairment. If those in the mTBI group had been tested within a day or so following their injury, we likely would have found that a much higher proportion of those with mTBIs were impaired. This study is also limited by its exclusion of females, which limits the generalizability of its findings. Additionally, soldiers in the healthy group completed ANAM in a group setting but the soldiers in the mTBI group completed ANAM individually. Research has shown that athletes tend to perform worse on computerized cognitive testing when it is administered in a group setting than athletes who completed the test individually (Moser, Schatz, Neidzwski, & Ott, 2011). If the soldiers in the healthy group had been tested individually, then it is possible that our effect sizes would have been different. Finally, we did not screen the soldiers in our study for PTSD, which may have contributed to the differences in the proportions of soldiers with low ANAM scores between the healthy and mTBI groups (O'neil et al., 2012).

Conclusion

Some healthy active duty male soldiers will have one or more low scores on ANAM4 TBI-MI and the probability of these soldiers obtaining low scores appears to be influenced considerably by race/ethnicity. After performing base rate analyses, ANAM4 TBI-MIL appears to best identify possible mTBI-related cognitive impairment in male soldiers when fewer than 5% of the healthy soldiers have two or more low scores at selected cutoffs, as indicated by RR of moderate to strong effect size. When this was the case, a minority of soldiers with mTBIs (no more than one-third) had multiple low scores at these cutoffs—consistent with the literature suggesting that cognitive recovery from mTBI in most people occurs swiftly and abnormalities were not likely to be permanent among those in our study who were tested relatively close to the time of their injury. A subsequent analysis of mean throughput scores from each of the ANAM4 TBI-MIL's seven cognitive tests found that the minority of those with mTBI who had multiple low scores accounted for most if not all of the effect of mTBI on differences between scores from individual ANAM4 TBI-MIL tests when the mTBI group as a whole was compared with the healthy group. We also found that low scores occurred on every ANAM4 TBI-MIL test.

The results of this study suggest that base rates of low scores should be determined using the large amount of baseline ANAM4 TBI-MIL data that the U.S. military currently possesses. The number of service members that have already been tested (>800,000) is large enough to allow for the development of base rates for multiple racial and ethnic groups, various levels of education, and for both males and females. Such base rates would be useful for military clinicians who use ANAM4 TBI-MIL to assess service members following mTBI, perhaps resulting in more accurate identification of cognitive impairment. They would also be useful for researchers who use ANAM4 TBI-MIL by providing additional information for interpreting study results, such as when group means are compared.

Conflict of Interest

The views, opinions and/or findings contained in this report are those of the author(s) and should not be construed as an official Department of Defense or Veterans Affairs position, policy or decision unless so designated by other documentation.

Funding

This research was funded by the Defense and Veterans Brain Injury Center, in part, through contract support provided by General Dynamics Information Technology (GDIT). GLI acknowledges support from the INTRuST Posttraumatic Stress Disorder and Traumatic Brain Injury Clinical Consortium funded by the Department of Defense Psychological Health/Traumatic Brain Injury Research Program (X81XWH-07-CC-CSDoD).

Acknowledgements

The views, opinions, and/or findings contained in this report are those of the author(s) and should not be construed as an official Department of Defense or Veterans Affairs position, policy or decision unless so designated by other documentation.

References

Axelrod
B. N.
Wall
J. R.
Expectancy of impaired neuropsychological test scores in a non-clinical sample
The International Journal of Neuroscience
 
2007
117
1591
1602
Beck
J. R.
Pyle
K. I.
Lusted
L. B.
A citation analysis of the field of medical decision making 1959–1982
Medical Decision Making
 
1984
4
449
468
Benjamini
Y.
Hochberg
Y.
Controlling the false discovery rate: A practical and powerful approach to multiple testing
Journal of the Royal Statistical Society. Series B
 
1995
57
289
300
Binder
L. M.
Iverson
G. L.
Brooks
B. L.
To err is human: “Abnormal” neuropsychological scores and variability are common in healthy adults
Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists
 
2009
24
31
46
Brooks
B. L.
Iverson
G. L.
Feldman
H. H.
Holdnack
J. A.
Minimizing misdiagnosis: Psychometric criteria for possible or probable memory impairment
Dementia
 
2009
27
439
450
Brooks
B. L.
Iverson
G. L.
Holdnack
J. A.
Feldman
H. H.
Potential for misclassification of mild cognitive impairment: A study of memory scores on the Wechsler Memory Scale-III in healthy older adults
Journal of the International Neuropsychological Society
 
2008
14
463
478
Brooks
B. L.
Iverson
G. L.
Sherman
E. M. S.
Holdnack
J. A.
Healthy children and adolescents obtain some low scores across a battery of memory tests
Journal of the International Neuropsychological Society
 
2009
15
613
617
Brooks
B. L.
Iverson
G. L.
White
T.
Substantial risk of “accidental MCI” in healthy older adults: Base rates of low memory scores in neuropsychological assessment
Journal of the International Neuropsychological Society
 
2007
13
490
500
Center for the Study of Human Operator Performance
Automated Neuropsychological Assessment Metrics (Version 4) [Computer software]
2007
Norman, OK
Center for the Study of Human Operator Performance (C-SHOP), University of Oklahoma
Center for the Study of Human Operator Performance
APR: ANAM performance Report Quict Start Guide
2008
Norman, OK
Center for the Study of Human Operator Performance (C-SHOP), University of Oklahoma
Chelune
G. J.
Evidence-based research and practice in clinical neuropsychology
The Clinical Neuropsychologist
 
2010
24
454
467
Cognitive Science Research Center
Military Normative/Reference Data: Military 2009
2009
Norman, OK
Cognitive Science Research Center, University of Oklahoma
Cognitive Science Research Center
ANAM4 Effort Measure: Quick Start Guide
2011
Norman, OK
Cognitive Science Research Center, University of Oklahoma
Cohen
J.
Statistical power analysis for the behavioral sciences
 
1988
2nd ed.)
Mahwah, NJ
Crawford
J. R.
Garthwaite
P. H.
Gault
C. B.
Estimating the percentage of the population with abnormally low scores (or abnormally large score differences) on standardized neuropsychological test batteries: A generic method with applications
Neuropsychology
 
2007
21
4
419
430
Diaz-Asper
C. M.
Schretlen
D. J.
Pearlson
G. D.
How well does IQ predict neuropsychological test performance in normal adults?
Journal of the International Neuropsychological Society
 
2004
10
82
90
Deddens
J. A.
Peterson
M. R.
Approaches for estimating prevalance ratios
Occupational and Environmental Medicine
 
2008
65
501
506
Ferguson
C. J.
An effect size primer: A guide for clinicians and researchers
Professional Psychology Research Practice
 
2009
40
3
532
538
Ingraham
L. J.
Aiken
C. B.
An empirical approach to determining criteria for abnormality in test batteries with multiple measures
Neuropsychology
 
1996
10
1
120
124
Iverson
G. L.
Webbe
F. M.
Evidence-based neuropsychological assessment in sports-related concussion
The handbook of sport neuropsychology
 
2011
New York
Springer
Iverson
G. L.
Brooks
B. L.
Holdnack
J. A.
Bush
S. S.
Iverson
G. L.
Evidence based neuropsychological assessment following work-related injury
Neuropsychological assessment of workplace injuries
 
2012
New York
Guilford Press
360
400
Iverson
G. L.
Brooks
B. L.
Langenecker
S. A.
Young
A. H.
Identifying a cognitive impairment subgroup in adults with mood disorders
Journal of Affective Disorders
 
2011
132
360
367
Kianifard
F.
Gallo
P. P.
Poisson regression analysis in clinical research
Journal of Biopharmaceutical Statistics
 
1995
5
1
115
129
Ledley
R. S.
Lusted
L. B.
Reasoning foundations of medical diagnosis
Science
 
1959
130
9
21
Lusted
L. B.
The clearing “haze:” a view from my window
Medical Decision Making
 
1991
11
76
88
Moser
R. S.
Schatz
P.
Neidzwski
K.
Ott
S. D.
Group versus individual administration affects baseline neurocognitive test performance
The American Journal of Sports Medicine
 
2011
39
11
2325
2330
O'neil
M. E.
Carlson
K. F.
Storzbach
K. F.
Brenner
L. A.
Freeman
M.
Quinones
A.
et al.  
Complications of Mild Traumatic Brain Injury in Veterans and Military Personnel: A Systematic Review
2012
Palmer
B. W.
Boone
K. B.
Lesser
I. M.
Wohl
M.
Base rates of “impaired” neuropsychological test performance among healthy older adults
Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists
 
1998
13
6
503
511
Patton
D. E.
Duff
K.
Schoenberg
M. R.
Mold
J.
Scott
J. G.
Adams
R. L.
Performance of cognitively normal African Americans on the RBANS in community dwelling older adults
The Clinical Neuropsychologist
 
2003
17
4
515
530
Reeves
D. L.
Winter
K. P.
Bleiberg
J.
Kane
R. L.
ANAM genogram: Historical perspectives, description, and current endeavors
Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists
 
2007
22S
S15
S37
Roebuck-Spencer
T. M.
Vincent
A. S.
Gilliland
K.
Johnson
D. R.
Cooper
D. B.
Initial clinical validation of an embedded performance validity measure within the Automated Neuropsychological Assessment Metrics (ANAM)
Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists
 
2013
28
700
710
Sahai
H.
Khurshid
A.
Statistics in epidemiology: Methods, techniques, and applications
 
1996
Boca Raton, FL
CRC Press
Schretlen
D. J.
Testa
S. M.
Winicki
J. M.
Pearlson
G. D.
Gordon
B.
Frequency and bases of abnormal performance by healthy adults on neuropsychological testing
Journal of the International Neuropsychological Society
 
2008
14
436
445
Short
P.
Cernich
A.
Wilken
J. A.
Kane
R. L.
Initial construct validation of frequently employed ANAM measures through structural equation modeling
Archives of Clinical Neuropsychology: The Official Journal of the National Academy of Neuropsychologists
 
2007
22S
S63
S77
Sox
H. C.
Probability theory in the use of diagnostic tests
Annals of Internal Medicine
 
1986
104
60
66
SPSS, Inc
IBM SPSS Statistics, Version 19
2010
Chicago, IL
SPSS, Inc
Thorne
D. R.
Throughput: A simple performance index with desirable characteristics
Behavior Research Methods
 
2006
38
569
573
US Army Neurocognitive Assessment Branch Office
Brooke Army Medical Center
2013
San Antonio, TX
Vincent
A. S.
Roebuck-Spencer
T.
Gilliland
K.
Schlegel
R.
Automated Neuropsychological Assessment Metrics (v4) Traumatic Brain Injury: Military normative data
Military Medicine
 
2012
177
3
256
269