Abstract

Despite the prevalence of concussion in soldiers deployed to Iraq and Afghanistan, neuropsychological tests used to assist in concussion management have not been validated on the battlefield. This study evaluated the validity of the Automated Neuropsychological Assessment Metrics (ANAM) in the combat environment. Cases meeting criteria for concussion, healthy controls, and injured controls were assessed. Soldiers were administered the ANAM, traditional neuropsychological tests, and a background questionnaire. Cases were enrolled within 72 h of concussion. Cases exhibited poorer performance than controls on all ANAM subtests, with significant differences on simple reaction time (SRT), procedural reaction time (PRT), code substitution, and matching to sample (p < .001). Discriminant ability of scores on SRT and PRT subtests was 71%, which improved to 76% when pre-deployment baseline scores were available. An exploratory clinical decision tool incorporating ANAM scores and symptoms improved discriminant ability to 81%. Results provide initial validation of the ANAM for detecting acute effects of battlefield concussion.

Assessment of Acute Concussion in the Combat Environment

Traumatic brain injury (TBI) is common in the U.S. military. The Department of Defense TBI surveillance system, conducted by the Defense and Veterans Brain Injury Center (DVBIC) and based on the clinician-confirmed diagnosis of TBI, indicates that, between 2000 and mid-2011, 220,430 service members had sustained TBI with 169,209 classified as concussion/mild TBI (concussion and mild TBI are considered interchangeable, but per the recommendation of Barth, Isler, Helmick, Wingler, and Jaffee (2010), the term concussion will be used throughout this paper ; DVBIC, 2011). Information from post-deployment screening questionnaires suggests that concussion is particularly widespread in service members deployed to Iraq and Afghanistan. These post-deployment surveys suggest a TBI history in ∼15%–23% of service members, with most categorized as concussion (Hoge et al., 2008; MacGregor, Schaffer, & Dougherty, 2010; Terrio, Brenner, & Ivins, 2009). Diagnosis and need for prompt evacuation is relatively clear-cut in moderate-to-severe TBI even in the austere combat environment (Iverson, Langlois, McCrea, & Kelly, 2009; Ling, Bandak, Armonda, Grant, & Ecklund, 2009). In military and civilian settings, concussions may be difficult to identify and challenging to manage as they exhibit subtle signs and symptoms (Iverson et al., 2009; Ling et al., 2009) and often demonstrate no abnormalities on standard imaging (Culotta, Sementilli, Gerold, & Watts, 1996; Huang, Theilmann, & Robb, 2009).

Common effects of concussion include impaired cognition, poor balance, and subjective post-concussive symptoms (Barr, McCrea, & Randolph, 2008; Belanger, Curtiss, Demery, Lebowitz, & Vanderploeg, 2005). For instance, one study of college football players with concussion found 85.2% experienced headache, 77% balance difficulties, and 69.4% feeling cognitively slowed down at the time of injury (Guskiewicz et al., 2003). This same research group (McCrea et al., 2003) found that college football players with elevated levels of subjective post-concussive complaints as well as objectively measured cognitive impairment and balance problems immediately after concussion tended to recover rapidly; balance deficits resolved within 3–5 days after injury, whereas post-concussive symptoms and objectively measured cognition gradually resolved by 7 days after injury.

Early identification of concussion in warriors is critical for appropriate medical management and prevention of premature return to hazardous duty. Mounting evidence from sports medicine (Guskiewicz et al., 2003; McCrea, Guskiewicz, & Randolph, 2009) and animal (Giza & Hovda, 2001) studies supports a period of increased vulnerability for repeat concussion within 10 days of initial injury. Patients with multiple concussions may experience prolonged recovery and be at elevated risk for second-impact syndrome, a rare disorder that can result in severe neurological disability (Guskiewicz et al., 2003; Kelly, Nichols, & Filley, 1991; Ling et al., 2009; Saunders & Harbaugh, 1984). Although guidelines from sports medicine vary in specifics, all concur that return to at-risk activities following concussion should not occur until all indicators of concussion have resolved completely (Barr et al., 2008). Moreover, providers cannot rely solely on patient self-report of post-concussive complaints in clinical decision-making as some patients may minimize post-concussive symptoms (Lovell & Collins, 1998), and subjective post-concussive symptoms are not specific to concussion but are common in other clinical conditions as well as in the general population (Gasquoine, 2000; Iverson, 2006; Machulda, Berquist, Ito, & Chew, 1998).

Neuropsychological testing has been successfully employed in sports-related concussion evaluation as a clinical tool to identify initial effects, track recovery, and assist clinicians in return to play decisions (Aubry et al., 2002; Barr et al., 2008). Multiple testing batteries demonstrate sensitivity to the effects of concussion in athletes (Barr et al., 2008; Barth et al., 1989; Belanger & Vanderploeg, 2005; Bleiberg et al., 2004; Broglio, Macciocchi, & Ferrara, 2007a, 2007b; Erlanger et al., 2003; Macciocchi, Barth, Alves, Rimel, & Jane, 1996; Schatz, Pardini, Lovell, Collins, & Podell, 2006; Van Kampen, Lovell, Pardini, Collins, & Fu, 2006; Warden et al., 2001), including those otherwise asymptomatic (Broglio et al., 2007a; Van Kampen et al., 2006). Among the validated neuropsychological test batteries is the Automated Neuropsychological Assessment Metrics (ANAM), a tool developed by the U.S. military and adapted for sports medicine (Barr et al., 2008; Bleiberg et al., 2004; Cernich, Reeves, Sun, & Bleiberg, 2007; Warden et al., 2001).

If validated, neuropsychological tests could aid identification, evaluation, and management of concussion on the battlefield and assist in fitness for duty decisions. However, neuropsychological test performance may be affected by factors including stress (McNeil & Morgan, 2010; Vasterling, MacDonald, Ulloa, & Rodier, 2010), sleep deprivation (Wesensten & Balkin, 2010), malingering (Greiffenstein, 2010), and deployment (Vasterling et al., 2006), all of which are considerations in combat, potentially limiting testing utility. Additionally, many battlefield concussions result from blast injury (Hoge et al., 2008; MacGregor et al., 2010). There are indications that blast TBI may have different biomechanics, pathobiology, and patterns of associated injury to other organs than non-blast TBI (Bhattacharjee, 2008; Cernak & Noble-Haeusslein, 2010; DePalma, Burris, Champion, & Hodgson, 2005; Ling et al., 2009; Wilk et al., 2010), though the few neuropsychological studies to date have not detected differences between blast- and non-blast-induced TBI (Belanger, Kretzmer, Yoash-Gantz, Pickett, & Tupler, 2009; Luethcke, Bryan, Morrow, & Isler, 2011).

This study prospectively examined validity and clinical utility of the ANAM in assessment of acute (within 72 h post-injury) concussion sustained in the combat environment. To our knowledge, this is the first controlled validation study of neuropsychological tests in theater of war.

Methods

Study Design

The study team deployed to Iraq from January to April 2009. Neurocognitive functioning of U.S. Army soldiers presenting for medical care within 72 h of a concussive event was assessed by traditional neuropsychological testing measures and ANAM with results compared with those of comparable controls. Participants were enrolled at Victory Base Complex, Joint Base Balad, and Mosul. The majority of the cognitive measures employed in this project were expressed as T-scores (M = 50, SD = 10). The authors considered a pre-determined discrepancy of five T-score points to be clinically significant. Sample size calculations determined that 60 cases and 120 controls would achieve a power of 0.89 to detect this clinically significant discrepancy with a Type I error rate of 0.05 using a two-tailed t-test and assuming a mean T-score of 50 and a standard deviation of 10 for control subjects. This sample size has been used in similar published studies (e.g., Bleiberg, Kane, Reeves, Garmoe & Halpern, 2000; Kabat, Kane, Jefferson, & DiPino, 2001).

This study was approved by the institutional review board of Brooke Army Medical Center. All participants provided written informed consent before enrollment.

Participants

Cases included all U.S. Army soldiers meeting eligibility criteria presenting to an outpatient medical facility within 72h of a concussion. The 72-h window was chosen as it has successfully been used previously in validation of neuropsychological measures in sports concussion (e.g., Broglio et al., 2007a; Pellman, Lovell, Viano, & Casson, 2006) and outpatient civilian (Hugenholtz, Stuss, Stetham, & Richard, 1988) settings and was estimated to be a reasonable time frame in which to enroll participants in view of combat operations considerations. Eligible participants had to be 18–50 years old, meet Department of Defense criteria for concussion (Department of Defense, 2007), be free of cognition altering medication, have no severe psychiatric diagnosis requiring ongoing therapy (i.e., ongoing medication management by a psychiatrist), report pain not more than 7 of 10, and give consent. Individuals with prior severe TBI, moderate TBI within the previous 3 years, or any concussion within the previous 90 days were excluded. Although multiple concussion grading systems exist, none is currently utilized by the Department of Defense to manage concussion in the deployed setting (Barth et al., 2010) and hence no grading system was employed in this project. To minimize fatigue effects participants had to have a night's rest before testing. Two control groups were enrolled. The first control group was a healthy group of U.S. Army soldiers from deployed units volunteering for participation. The second group consisted of acutely injured (within 72 h) U.S. Army soldiers presenting for outpatient care who were neither head-injured nor exposed to a blast. This group was recruited to control for negative neurocognitive effects of non-concussive injury. In addition to the inclusion/exclusion criteria for cases, controls could not have suffered a concussion during this deployment.

Questionnaire and Neuropsychological Testing

Participants were administered a questionnaire with items assessing demographics (including self-identified race, per Common Data Elements recommendations [Maas, Harrison-Felix, & Menon, 2010]), physical health, mental health, sleep, and other service-related factors. Physical and mental health were rated on a 1 (“poor”) to 5 (“excellent”) scale and pain on a 0 (“none”) to 10 (“severe”) scale. Cases were asked about the presence and duration of loss of consciousness (LOC) or post-traumatic amnesia (PTA) and time since injury. The study team had originally planned to obtain information from cases and/or collateral sources allowing distinction between LOC and PTA, but the circumstances under which many injuries occurred (e.g., blast injuries) often made this distinction impossible. Therefore, for purposes of analysis, cases reporting a period of time for which they had no memory post-injury (and may possibly have been unconscious) were combined with those with known or suspected LOC.

The neuropsychological tests administered include the Military Acute Concussion Evaluation (MACE), the ANAM-IV-TBI, and a battery of traditional neuropsychological tests used previously in the study of sports-related concussion including the Hopkins Verbal Learning Test-Revised, Symbol Digit Modalities Test, Stroop Test, Trail Making Test, and Controlled Oral Word Association Test (Barr et al., 2008). The Test of Memory Malingering (TOMM) was administered to control for poor effort. ANAM-IV-TBI results are reported as T-scores.

The ANAM-IV-TBI (hereafter referred to as the ANAM) includes six subtests: Simple reaction time (SRT), procedural reaction time (PRT), code substitution (CDS), CDS delayed (CDD), mathematical processing (MTH), and matching to sample (MSP). The ANAM subtests have been fully described in a previous article in this journal (Reeves, Winter, Bleiberg, & Kane, 2007). Briefly, the SRT provides a measure of simple reaction time. PRT is a choice reaction time measure that requires the participant to choose between sets of numbers and is included to assess the processing speed. CDS is a symbol-digit pairing task included to tap learning, whereas CDD evaluates delayed memory for the CDS associations. MTH involves performing single-digit multistep arithmetic operations and is designed to evaluate the working memory capacity. MSP assesses recognition memory for a 4 × 4 colored matrix and is included as a measure of spatial memory. The ANAM and similar reaction time-based measures have been shown in multiple studies to be sensitive to the effects of concussion in the early post-injury period and to provide an objective indicator of recovery of brain function in days to weeks post-injury (e.g., Bleiberg et al., 2004; Broglio et al., 2007a; Schatz et al., 2006; Van Kampen et al., 2006; Warden et al., 2001). Throughput scores, reflecting composite speed and accuracy, were analyzed. When available, pre-deployment baseline results were retrieved from the Army Neurocognitive Assessment Branch.

Data Analysis

Statistical analysis was performed using Stata v11.1 (Stata Corp., College Station, TX, USA). As the data were not normally distributed, proportions were compared using Fisher's exact test, and continuous data with the Mann–Whitney U-test, the Wilcoxon signed-rank test, or/and the Kruskal–Wallis test. A p-value of <.05 is considered statistically significant. To address the issue of possible Type I error, the Bonferroni corrections were calculated for all sets of comparisons.

Covariates are initially reported in accordance with Common Data Elements recommendations (Maas et al., 2010). However, for statistical analysis, many of these were collapsed.

To compare the tests' ability to discriminate cases versus controls, we examined the area under the receiver operating characteristic (ROC) curve and the discriminant ability, defined as the average of the sensitivity and specificity (Riegelman, 2004).

Results

Demographics

Initially, 71 cases and 166 controls were enrolled. Two cases were excluded after demonstrating poor effort on the TOMM (the first case obtained a score of 29 of 50 on Trial 1 and 18 of 50 on Trial 2; the second case obtained a score of 37 of 50 on Trial 1 and testing was aborted after he began behaving too inappropriately to continue). Only three cases were women. Because there are indications that gender differences may exist in outcomes from concussions (Dick, 2009) and the number of female cases was too small to permit the assessment of potential gender effects, analysis was limited to men, excluding 3 female cases and 20 female controls. The final group consists of 212 participants: 66 cases and 146 controls (86 healthy and 60 injured).

Healthy and injured controls were combined for analysis, although these were somewhat different groups. The unit controls were younger than the injured controls (median age 24 vs. 27, p = .0011), more often enlisted (74% vs. 52%, p = .011), had fewer years of service (median 3 vs. 5.25, p = .0064), and fewer had more than two combat tours (2% vs. 20%, p = .002). There was no statistical difference in other factors examined. None of these covariates correlated significantly with altered test performance in later analysis. Later analysis revealed no statistically significant difference between these groups' neuropsychological testing performance. Therefore, combining these groups is considered valid and acceptable to improve power.

Study group demographics, symptoms, and blast/concussion exposure history are presented in Table 1. Continuous data are reported as medians and interquartile ranges, dichotomous data as number and percentage answering yes, and categorical data as number and percent in each category.

Table 1.

Study population characteristics

Characteristic Cases (n = 66) Controls (n = 146) p-value 
Location (n [%]) 
 Baghdad 11 (17) 90 (62)  
 Balad 18 (27) 34 (23) <.001 
 Mosul 37 (56) 22 (15)  
Age (years, median [IQR]) 25 (22, 30) 25 (22, 31) .71 
Marital status (n [%]) 
 Never married 23 (35) 50 (34)  
 Married 36 (55) 80 (55) .99 
 Separated 3 (5) 6 (4)  
 Divorced 4 (6) 9 (6)  
Education (n [%]) 
 <High school graduate 6 (9) 5 (3)  
 High school graduate 32 (48) 80 (55) .03 
 Some college 25 (39) 43 (29)  
 College graduate 2 (3) 18 (12)  
Race (n [%]) 
 Caucasian 49 (74) 94 (70)  
 Black 4 (6) 22 (16)  
 Hispanic 11 (17) 11 (8)  
 Asian 0 (0) 2 (1) .11 
 American Indian 0 (0) 1 (1)  
 Pacific Islander 0 (0) 2 (1)  
 Other 2 (3) 2 (1)  
English as second language (n [%]) 5 (8) 9 (6) .56 
Military rank (n [%]) 
 Enlisted 45 (68) 95 (65)  
 Non-Commissioned Officer 21 (32) 38 (26) .08 
 Warrant Officer 0 (0) 3 (2)  
 Commissioned Officer 0 (0) 10 (7)  
Physical Health (median [IQR]) 4 (4, 5) 4 (4, 5) .63 
Presence of symptoms (n [%]) 
 Pain 29 (44) 38 (26) .01 
 Headache 36 (55) 12 (8) <.001 
 Blackouts 14 (21) 0 (0) <.001 
 Confusion 38 (58) 5 (3) <.001 
 Flashbacks 11 (17) 4 (3) .001 
 Impulsiveness 8 (12) 10 (7) .29 
Mental Health (median [IQR]) 4 (3, 5) 5 (4, 5) .01 
Learning disability/ADHD (n [%]) 4 (7) 27 (19) .03 
Current counseling (n [%]) 4 (6) 5 (3) .46 
Current mental health medication (n [%]) 3 (5) 8 (6) .99 
Alcohol problems (n [%]) 7 (12) 0 (0) <.001 
Nightly sleep (h; median [IQR]) 6 (5, 7) 6 (5, 7) .22 
Sleep change (h; median [IQR]) 0 (−2.5, 0) 0 (0, 0) <.001 
Length of service (years; median [IQR]) 4 (2, 6) 4 (2.5, 8) .56 
Months in Iraq (months; median [IQR]) 4 (2, 6) 6 (4, 9) <.001 
Number of tours (n [%]) 
 1 37 (56) 89 (61)  
 2 21 (32) 43 (29) .86 
 >2 7 (12) 14 (10)  
Total IED exposures (n [%]) 
 0 16 (24) 107 (74)  
 1 32 (48) 16 (11) <.001 
 2 9 (14) 9 (6)  
 >2 9 (14) 12 (9)  
Prior concussions (n [%]) 
 0 44 (68) 115 (86)  
 1 13 (20) 14 (10) <.01 
 2 6 (9) 1 (1)  
 >2 2 (3) 4 (3)  
LOC/PTA (n [%]) 35 (56)   
Mechanism of concussion (n [%]) 
 Blast 35 (53)   
 Blow 18 (27)   
 Mixed 6 (9)   
 Unknown 7 (11)   
Location of other injury (n [%]) 
 Lower extremity  30 (50)  
 Upper extremity  17 (28)  
 Back  7 (12)  
 Other  4 (7)  
 Unknown  2 (3)  
Type of other injury (n [%]) 
 Fracture/sprain/strain  38 (63)  
 Crush  6 (10)  
 Laceration  3 (5)  
 Shrapnel/gunshot  3 (5)  
 Burn  1 (2)  
 Unknown  9 (15)  
Characteristic Cases (n = 66) Controls (n = 146) p-value 
Location (n [%]) 
 Baghdad 11 (17) 90 (62)  
 Balad 18 (27) 34 (23) <.001 
 Mosul 37 (56) 22 (15)  
Age (years, median [IQR]) 25 (22, 30) 25 (22, 31) .71 
Marital status (n [%]) 
 Never married 23 (35) 50 (34)  
 Married 36 (55) 80 (55) .99 
 Separated 3 (5) 6 (4)  
 Divorced 4 (6) 9 (6)  
Education (n [%]) 
 <High school graduate 6 (9) 5 (3)  
 High school graduate 32 (48) 80 (55) .03 
 Some college 25 (39) 43 (29)  
 College graduate 2 (3) 18 (12)  
Race (n [%]) 
 Caucasian 49 (74) 94 (70)  
 Black 4 (6) 22 (16)  
 Hispanic 11 (17) 11 (8)  
 Asian 0 (0) 2 (1) .11 
 American Indian 0 (0) 1 (1)  
 Pacific Islander 0 (0) 2 (1)  
 Other 2 (3) 2 (1)  
English as second language (n [%]) 5 (8) 9 (6) .56 
Military rank (n [%]) 
 Enlisted 45 (68) 95 (65)  
 Non-Commissioned Officer 21 (32) 38 (26) .08 
 Warrant Officer 0 (0) 3 (2)  
 Commissioned Officer 0 (0) 10 (7)  
Physical Health (median [IQR]) 4 (4, 5) 4 (4, 5) .63 
Presence of symptoms (n [%]) 
 Pain 29 (44) 38 (26) .01 
 Headache 36 (55) 12 (8) <.001 
 Blackouts 14 (21) 0 (0) <.001 
 Confusion 38 (58) 5 (3) <.001 
 Flashbacks 11 (17) 4 (3) .001 
 Impulsiveness 8 (12) 10 (7) .29 
Mental Health (median [IQR]) 4 (3, 5) 5 (4, 5) .01 
Learning disability/ADHD (n [%]) 4 (7) 27 (19) .03 
Current counseling (n [%]) 4 (6) 5 (3) .46 
Current mental health medication (n [%]) 3 (5) 8 (6) .99 
Alcohol problems (n [%]) 7 (12) 0 (0) <.001 
Nightly sleep (h; median [IQR]) 6 (5, 7) 6 (5, 7) .22 
Sleep change (h; median [IQR]) 0 (−2.5, 0) 0 (0, 0) <.001 
Length of service (years; median [IQR]) 4 (2, 6) 4 (2.5, 8) .56 
Months in Iraq (months; median [IQR]) 4 (2, 6) 6 (4, 9) <.001 
Number of tours (n [%]) 
 1 37 (56) 89 (61)  
 2 21 (32) 43 (29) .86 
 >2 7 (12) 14 (10)  
Total IED exposures (n [%]) 
 0 16 (24) 107 (74)  
 1 32 (48) 16 (11) <.001 
 2 9 (14) 9 (6)  
 >2 9 (14) 12 (9)  
Prior concussions (n [%]) 
 0 44 (68) 115 (86)  
 1 13 (20) 14 (10) <.01 
 2 6 (9) 1 (1)  
 >2 2 (3) 4 (3)  
LOC/PTA (n [%]) 35 (56)   
Mechanism of concussion (n [%]) 
 Blast 35 (53)   
 Blow 18 (27)   
 Mixed 6 (9)   
 Unknown 7 (11)   
Location of other injury (n [%]) 
 Lower extremity  30 (50)  
 Upper extremity  17 (28)  
 Back  7 (12)  
 Other  4 (7)  
 Unknown  2 (3)  
Type of other injury (n [%]) 
 Fracture/sprain/strain  38 (63)  
 Crush  6 (10)  
 Laceration  3 (5)  
 Shrapnel/gunshot  3 (5)  
 Burn  1 (2)  
 Unknown  9 (15)  

Notes: p-values in italics are not significant with the Bonferroni correction. IRQ = interquartile range; LOC = loss of consciousness; PTA = post-traumatic amnesia.

The median age of cases and controls was 25. Cases had a larger proportion of participants with less than high school education, though with the Bonferroni correction applied this difference was no longer significant. There was no significant difference in marital status, race, or English as a second language. One of the cases reported having a moderate TBI previously, but the injury occurred >3 years prior to the study and he was therefore included in the sample, per the previously described inclusion criteria.

Physical Health, Mental Health, Sleep Patterns, and Other Service-Related Variables

Cases reported more headaches, blackouts, confusion, and flashbacks. Cases also endorsed more frequent alcohol problems.

Both groups reported a median of 6 h of nightly sleep. There was a statistically significant difference in sleep change, with cases reporting more sleep loss.

Cases had been in Iraq fewer months and more often reported previous improvised explosive device exposures.

Neuropsychological Testing Scores

Results of the MACE have been reported previously (Coldren, Kelly, Parish, Dretsch, & Russell, 2010) and detailed results of the traditional neuropsychological testing are beyond the scope of this manuscript.

Table 2 presents the ANAM results, comparing cases and controls.

Table 2.

ANAM scores comparing cases and controls

 Baseline group median
 
Enrollment group median
 
Individual change median
 
 Cases (n = 34) Controls (n = 45) p-value Cases (n = 66) Controls (n = 146) p-value Cases (n = 34) Controls (n = 45) p-value 
SRT 53 (48, 55) 52 (47, 55) .24 44.5 (34.5, 52) 52 (47, 55.5) <.001 −5 (−19, −1) 2 (−1.25, 4) <.001 
PRT 52 (43, 53) 50 (47, 58) .60 46 (34.5, 54.5) 52.5 (46.5, 59) <.001 −4.75 (−16, 4) 2 (−4.5, 6.5) .02 
CDS 44 (37, 50) 48 (43, 55) .02 44 (38.5, 51) 52 (45, 57) <.001 1 (−5, 6) 6 (2, 10) <.01 
CDD 42.5 (35, 50) 45 (38, 52) .20 42 (35, 49.5) 44.75 (39, 51) .04 −2.5 (−7, 10) 2 (−3.5, 7) .25 
MTH 47 (42, 57) 52 (42, 55) .57 46.7(40.5, 52.5) 50 (44, 56.6) .03 −3 (−4.5, 2.5) 3 (−1, 6.5) <.01 
MSP 51 (43, 53) 50 (45, 55) .57 44 (38.5, 51) 50 (44, 56) <.001 −5.25 (−9, 1.5) 3 (−2.5, 9) <.001 
 Baseline group median
 
Enrollment group median
 
Individual change median
 
 Cases (n = 34) Controls (n = 45) p-value Cases (n = 66) Controls (n = 146) p-value Cases (n = 34) Controls (n = 45) p-value 
SRT 53 (48, 55) 52 (47, 55) .24 44.5 (34.5, 52) 52 (47, 55.5) <.001 −5 (−19, −1) 2 (−1.25, 4) <.001 
PRT 52 (43, 53) 50 (47, 58) .60 46 (34.5, 54.5) 52.5 (46.5, 59) <.001 −4.75 (−16, 4) 2 (−4.5, 6.5) .02 
CDS 44 (37, 50) 48 (43, 55) .02 44 (38.5, 51) 52 (45, 57) <.001 1 (−5, 6) 6 (2, 10) <.01 
CDD 42.5 (35, 50) 45 (38, 52) .20 42 (35, 49.5) 44.75 (39, 51) .04 −2.5 (−7, 10) 2 (−3.5, 7) .25 
MTH 47 (42, 57) 52 (42, 55) .57 46.7(40.5, 52.5) 50 (44, 56.6) .03 −3 (−4.5, 2.5) 3 (−1, 6.5) <.01 
MSP 51 (43, 53) 50 (45, 55) .57 44 (38.5, 51) 50 (44, 56) <.001 −5.25 (−9, 1.5) 3 (−2.5, 9) <.001 

Notes: Values are given as the median (interquartile range). ANAM = Automated Neuropsychological Assessment Metrics; SRT = simple reaction time; PRT = procedural reaction time; CDS = code substitution; CDD = code substitution delayed; MTH = mathematical processing; MSP = matching to sample. p-values in italics are not significant with the Bonferroni correction.

We examined differences in group medians between cases and controls (see “Baseline group median” and “Enrollment group median” columns of Table 2) as well as differences in median individual change (see “Individual change median” column of Table 2). Although the examination of median individual change is more clinically meaningful than group change, as this inherently corrects for biases and for individual baseline variation, the number of baseline scores available for analysis was relatively low, reducing power. Therefore, to increase the power of our analysis, we analyzed differences in group medians at enrollment (“Enrollment group median” column of Table 2). We included the analysis of group medians at baseline (see “Baseline group median” of Table 2) to ensure that any differences in group medians detected at enrollment were not due to differences at baseline. An additional advantage of analyzing scores at enrollment is that this may be the only data point available to the military clinician at the time of initial evaluation in theater.

Pre-deployment baseline ANAM results were available for 34 cases and 45 controls (“Baseline group median” column of Table 2). There was no difference between cases and controls at the pre-deployment baseline for any ANAM subtest.

At the time of enrollment, cases performed more poorly than controls on all ANAM subtests, with the differences reaching statistical significance on SRT, PRT, CDS, and MSP (“Enrollment group median” column of Table 2). Examining median individual changes from the pre-deployment baseline to enrollment demonstrated that for cases, SRT (p = .001), PRT (p = .0106), and MSP (p = .0183) performance declined significantly. MTH also declined but the change did not reach statistical significance. CDS improved for cases while CDD declined, but neither reached significance. Controls improved on all subtests with SRT (p = .0207), MTH (p = .0030), and MSP (p = .013) reaching significance. As described in Table 2 (“Individual change median” column), there were significant differences in ANAM subtest score changes between cases and controls for all subtests except CDD and PRT (“Individual change median” column of Table 2).

ROC, Discriminant Ability, Sensitivity, and Specificity

Table 3 reports the area under the ROC curve, discriminant ability, and the sensitivity and specificity of the tests at the score maximizing discriminant ability. At enrollment, SRT performed best with an area under the ROC curve of 0.71 and discriminant ability of 69%. Adding SRT and PRT scores (SRT + PRT) yields an improved area under the ROC curve of 0.73 and discriminant ability of 71%. No other single test or test combination improves these scores.

Table 3.

ANAM performance

 Area under the ROC curve Discriminant ability (%) Sensitivity (%)/specificity (%) 
At enrollment 
 SRT 0.71 69 52/86 
 PRT 0.68 65 53/77 
 CDS 0.68 66 58/74 
 CDD 0.59 55 24/82 
 MTH 0.59 58 67/49 
 MSP 0.65 65 55/75 
 SRT + PRT 0.73 71 59/82 
Change from baseline 
 SRT 0.78 76 71/82 
 PRT 0.66 64 59/69 
 CDS 0.69 67 38/96 
 CDD 0.58 62 56/69 
 MTH 0.70 68 59/78 
 MSP 0.72 70 53/87 
 SRT + MTH + MSP 0.79 75 53/98 
 Area under the ROC curve Discriminant ability (%) Sensitivity (%)/specificity (%) 
At enrollment 
 SRT 0.71 69 52/86 
 PRT 0.68 65 53/77 
 CDS 0.68 66 58/74 
 CDD 0.59 55 24/82 
 MTH 0.59 58 67/49 
 MSP 0.65 65 55/75 
 SRT + PRT 0.73 71 59/82 
Change from baseline 
 SRT 0.78 76 71/82 
 PRT 0.66 64 59/69 
 CDS 0.69 67 38/96 
 CDD 0.58 62 56/69 
 MTH 0.70 68 59/78 
 MSP 0.72 70 53/87 
 SRT + MTH + MSP 0.79 75 53/98 

Notes: SRT = simple reaction time; PRT = procedural reaction time; CDS = code substitution; CDD = code substitution delayed; MTH = mathematical processing; MSP = matching to sample. At enrollment, 66 cases and 146 controls; for change from baseline analysis, 34 cases and 45 controls.

If baselines are available, the area under the ROC curve, discriminant ability, sensitivity, and specificity improve. The highest discriminant ability observed is the change in SRT, with a discriminant ability of 76% and an area under the ROC curve of 0.78. Combining the change in SRT, MTH, and MSP improves the area under the ROC curve to 0.79 but yields slightly lower discriminant ability of 75%. Again, no other single test or test combination improves these scores.

Impact of Covariates on ANAM Subtest Performance

Results are in Table 4 . Covariate analysis presented is limited to enrollment SRT alone as SRT + PRT was derived post hoc and provides only minimally improved results. Sample size was insufficient to allow analysis of covariates on change in SRT.

Table 4.

Median SRT by covariate at enrollment

 Cases (n = 66) (median [IQR]) p-value Controls (n = 146; median [IQR]) p-value 
Age (years) 
 18–20 47 (20.5, 54.5)  54 (50.5, 57.75)  
 21–25 47 (38, 54.75)  51 (48, 53.5)  
 26–30 38 (32, 44.5) .17 52 (47, 54.5) .07 
 31–40 49.75 (35.25, 52)  53.5 (48.5, 56.5)  
 40–55 49.5 (40.5, 56.5)  46 (37, 53.5)  
Marital status 
 Never married 47 (30.5, 53.5)  51 (48, 53.5)  
 Married 45.5 (38, 50) .72 52 (47, 56) .10 
 Separated 40 (28, 44.5)  45.5 (42, 50.5)  
 Divorced 44.25 (32.25, 61.5)  53.5 (46.5, 56)  
Education 
 <High school graduate 41.25 (32, 48)  52 (48.5, 53.5)  
 High school graduate 47.5 (34.5, 54.75) .64 51 (47.5, 54) .54 
 Some college 41 (34.5, 50)  53 (46.5, 56.5)  
 College graduate 48.25 (47, 49.5)  52.25 (46.5, 56.6)  
Race 
 Caucasian 44.5 (34, 52)  52 (48.5, 56)  
 Black 52.25 (46, 55) .24 52.25 (46, 55) .13 
 Hispanic 42 (28, 48)  46 (37, 54.5)  
English as second language 
 No 47 (34, 54) .36 52 (48, 55.5) .18 
 Yes 42 (38, 47.5)  46 (40, 53.5)  
Military rank 
 Enlisted 47 (35, 53.5)  52 (48, 55.5)  
 Non-Commissioned Officer 44.5 (32.5, 49.5) .48 52 (46, 56) .34 
 Warrant/Commissioned Officer   56.6 (46.5, 59)  
Physical health (1–5) 
 <4 41.5 (32, 48.5) .37 52 (48.5, 57) .57 
 4–5 47 (34.5, 52)  52 (47, 55.5)  
Pain 
 No 48 (40.5, 54.5) .03 52 (47, 55.5) .84 
 Yes 38 (32, 49.5)  51 (47, 55.5)  
Headache 
 No 48.25 (44, 54.5) <.01 52 (47, 55) .99 
 Yes 39 (29.25, 49.75)  51.25 (47, 56.25)  
Blackouts 
 No 47 (38, 52) .04   
 Yes 32.25 (15, 48.5)    
Confusion 
 No 49 (43, 54.75) <.01 52 (47, 55) .48 
 Yes 38.5 (28, 48.5)  52.5 (52, 56)  
Flashbacks 
 No 47 (35, 53.5) .20 52 (47, 55.5) .48 
 Yes 40.5 (32, 46.5)  49 (45.25, 54)  
Impulsiveness 
 No 45.75 (35, 52) .34 52 (47, 56) .43 
 Yes 42.5 (29.25, 48)  50 (46, 53.5)  
Mental health (1–5) 
 <4 40.5 (32, 46.5) .03 51.25 (46, 53.5) .23 
 4–5 48 (38, 54.5)  52 (48, 56)  
Learning disability/ADHD 
 No 44.5 (32.25, 52) .11 52 (47, 55) .60 
 Yes 51.5 (48.25, 56.5)  52 (48.5, 53.5)  
Current counseling 
 No 45.5 (34.5, 52) .77 52 (47, 55) .35 
 Yes 43 (37.75, 47.5)  53.5 (52.5, 56.5)  
Current mental health medication 
 No 46.5 (35, 52) .09 52 (47, 55) .16 
 Yes 34 (13.5, 41.5)  54.75 (52.25, 56.25)  
Alcohol problems 
 No 44.5 (32, 53.5) .14   
 Yes 48.5 (47, 52)    
Nightly sleep (h) 
 <4 38 (27, 44.5)  47.5 (42, 53.5)  
 4–6 47.75 (32.25, 54.5) .19 51 (46.5, 55.5) .07 
 >6 44.5 (38, 50)  52.5 (49.5, 56)  
Sleep change (h) 
 ≤−2 46.5 (28.5, 52)  51 (49.5, 53.5)  
 −2 to <0 46.25 (46.5, 53.5) .18 50.25 (44.5, 56) .93 
 0 48.5 (38, 54)  52 (49.5, 55)  
 >0 24.5 (20.5, 44.5)  50.25 (46, 55.5)  
Length of service (years) 
 ≤1 47.25 (38,52)  50 (48, 52.75)  
 >1–3 47.5 (38, 56)  52.25 (49, 54.25)  
 >3–5 44.25 (35, 53.5) .41  50.5 (46.5, 54) .78 
 >5–10 36.25 (23.5, 48.25)  52 (46, 56.6)  
 >10 49.5 (40, 50.5)  51 (46, 57.5)  
Months in Iraq (months) 
 <6 44.5 (34, 49.5) .23 52 (47, 54.25) .27 
 ≥6 48.5 (39, 54.5)  52 (47, 56.5)  
Number of tours 
 1 48 (39, 55) .02 52 (48.5, 54.5) .63 
 >1 41.25 (31.5, 49)  52 (46, 56)  
Total IED exposures 
 0 44.5 (38, 48.25)  52 (48, 54.5)  
 1 48.25 (36.5, 53.25) .17 48.25 (45.5, 54.5) .43 
 2 52 (30.5, 54.5)  50.5 (47, 53.5)  
 >2 38 (28.5, 44)  53.5 (46.5, 57)  
Prior concussions 
 0 46.75 (36.5, 52)  52 (48.5, 55)  
 1 40 (30.5, 44.5) .37 48.25 (44.5, 54) .33 
 >1 49 (37.5, 54.75)  50.5 (48, 53)  
LOC/PTA 
 No 48.5 (39, 54) .04   
 Yes 44 (32, 48.5)    
Total time without memory (min) 
 0 48.5 (39, 54)    
 ≤1 44.5 (31.5, 49.5) .16   
 >1–5 38 (35, 47)    
 >5 41.75 (20.5, 44.5)    
Days since injury 
 ≤1 47 (32, 54)    
 2 47 (39, 50.5) .32   
 3 42 (40, 48)    
 Cases (n = 66) (median [IQR]) p-value Controls (n = 146; median [IQR]) p-value 
Age (years) 
 18–20 47 (20.5, 54.5)  54 (50.5, 57.75)  
 21–25 47 (38, 54.75)  51 (48, 53.5)  
 26–30 38 (32, 44.5) .17 52 (47, 54.5) .07 
 31–40 49.75 (35.25, 52)  53.5 (48.5, 56.5)  
 40–55 49.5 (40.5, 56.5)  46 (37, 53.5)  
Marital status 
 Never married 47 (30.5, 53.5)  51 (48, 53.5)  
 Married 45.5 (38, 50) .72 52 (47, 56) .10 
 Separated 40 (28, 44.5)  45.5 (42, 50.5)  
 Divorced 44.25 (32.25, 61.5)  53.5 (46.5, 56)  
Education 
 <High school graduate 41.25 (32, 48)  52 (48.5, 53.5)  
 High school graduate 47.5 (34.5, 54.75) .64 51 (47.5, 54) .54 
 Some college 41 (34.5, 50)  53 (46.5, 56.5)  
 College graduate 48.25 (47, 49.5)  52.25 (46.5, 56.6)  
Race 
 Caucasian 44.5 (34, 52)  52 (48.5, 56)  
 Black 52.25 (46, 55) .24 52.25 (46, 55) .13 
 Hispanic 42 (28, 48)  46 (37, 54.5)  
English as second language 
 No 47 (34, 54) .36 52 (48, 55.5) .18 
 Yes 42 (38, 47.5)  46 (40, 53.5)  
Military rank 
 Enlisted 47 (35, 53.5)  52 (48, 55.5)  
 Non-Commissioned Officer 44.5 (32.5, 49.5) .48 52 (46, 56) .34 
 Warrant/Commissioned Officer   56.6 (46.5, 59)  
Physical health (1–5) 
 <4 41.5 (32, 48.5) .37 52 (48.5, 57) .57 
 4–5 47 (34.5, 52)  52 (47, 55.5)  
Pain 
 No 48 (40.5, 54.5) .03 52 (47, 55.5) .84 
 Yes 38 (32, 49.5)  51 (47, 55.5)  
Headache 
 No 48.25 (44, 54.5) <.01 52 (47, 55) .99 
 Yes 39 (29.25, 49.75)  51.25 (47, 56.25)  
Blackouts 
 No 47 (38, 52) .04   
 Yes 32.25 (15, 48.5)    
Confusion 
 No 49 (43, 54.75) <.01 52 (47, 55) .48 
 Yes 38.5 (28, 48.5)  52.5 (52, 56)  
Flashbacks 
 No 47 (35, 53.5) .20 52 (47, 55.5) .48 
 Yes 40.5 (32, 46.5)  49 (45.25, 54)  
Impulsiveness 
 No 45.75 (35, 52) .34 52 (47, 56) .43 
 Yes 42.5 (29.25, 48)  50 (46, 53.5)  
Mental health (1–5) 
 <4 40.5 (32, 46.5) .03 51.25 (46, 53.5) .23 
 4–5 48 (38, 54.5)  52 (48, 56)  
Learning disability/ADHD 
 No 44.5 (32.25, 52) .11 52 (47, 55) .60 
 Yes 51.5 (48.25, 56.5)  52 (48.5, 53.5)  
Current counseling 
 No 45.5 (34.5, 52) .77 52 (47, 55) .35 
 Yes 43 (37.75, 47.5)  53.5 (52.5, 56.5)  
Current mental health medication 
 No 46.5 (35, 52) .09 52 (47, 55) .16 
 Yes 34 (13.5, 41.5)  54.75 (52.25, 56.25)  
Alcohol problems 
 No 44.5 (32, 53.5) .14   
 Yes 48.5 (47, 52)    
Nightly sleep (h) 
 <4 38 (27, 44.5)  47.5 (42, 53.5)  
 4–6 47.75 (32.25, 54.5) .19 51 (46.5, 55.5) .07 
 >6 44.5 (38, 50)  52.5 (49.5, 56)  
Sleep change (h) 
 ≤−2 46.5 (28.5, 52)  51 (49.5, 53.5)  
 −2 to <0 46.25 (46.5, 53.5) .18 50.25 (44.5, 56) .93 
 0 48.5 (38, 54)  52 (49.5, 55)  
 >0 24.5 (20.5, 44.5)  50.25 (46, 55.5)  
Length of service (years) 
 ≤1 47.25 (38,52)  50 (48, 52.75)  
 >1–3 47.5 (38, 56)  52.25 (49, 54.25)  
 >3–5 44.25 (35, 53.5) .41  50.5 (46.5, 54) .78 
 >5–10 36.25 (23.5, 48.25)  52 (46, 56.6)  
 >10 49.5 (40, 50.5)  51 (46, 57.5)  
Months in Iraq (months) 
 <6 44.5 (34, 49.5) .23 52 (47, 54.25) .27 
 ≥6 48.5 (39, 54.5)  52 (47, 56.5)  
Number of tours 
 1 48 (39, 55) .02 52 (48.5, 54.5) .63 
 >1 41.25 (31.5, 49)  52 (46, 56)  
Total IED exposures 
 0 44.5 (38, 48.25)  52 (48, 54.5)  
 1 48.25 (36.5, 53.25) .17 48.25 (45.5, 54.5) .43 
 2 52 (30.5, 54.5)  50.5 (47, 53.5)  
 >2 38 (28.5, 44)  53.5 (46.5, 57)  
Prior concussions 
 0 46.75 (36.5, 52)  52 (48.5, 55)  
 1 40 (30.5, 44.5) .37 48.25 (44.5, 54) .33 
 >1 49 (37.5, 54.75)  50.5 (48, 53)  
LOC/PTA 
 No 48.5 (39, 54) .04   
 Yes 44 (32, 48.5)    
Total time without memory (min) 
 0 48.5 (39, 54)    
 ≤1 44.5 (31.5, 49.5) .16   
 >1–5 38 (35, 47)    
 >5 41.75 (20.5, 44.5)    
Days since injury 
 ≤1 47 (32, 54)    
 2 47 (39, 50.5) .32   
 3 42 (40, 48)    

Notes: IRQ = interquartile range; LOC = loss of consciousness; PTA = post-traumatic amnesia.

In controls, no covariate examined demonstrated a significant impact. Likewise, after the Bonferroni correction, median enrollment SRT did not vary significantly by any demographic factor, physical health parameter, mental health parameter, or deployment related factor in cases.

Impact of Using a Clinical Decision Tool

Results are shown in Table 5.

Table 5.

Clinical decision tool

 All cases (n = 66) LOC/PTA or 3+ symptoms (n = 23) 
Area under the ROC curve 
 SRT (at enrollment) 0.71 0.80 
 SRT + PRT (at enrollment) 0.73 0.84 
 Change in SRT (at enrollment) 0.78 0.84 
Discriminant ability (%) 
 SRT (at enrollment) 69 75 
 SRT + PRT (at enrollment) 71 76 
 Change in SRT (at enrollment) 76 81 
Sensitivity (%)/specificity (%) 
 SRT (at enrollment) 52/86 63/86 
 SRT + PRT (at enrollment) 59/82 71/82 
 Change in SRT (at enrollment) 71/82 81/82 
 All cases (n = 66) LOC/PTA or 3+ symptoms (n = 23) 
Area under the ROC curve 
 SRT (at enrollment) 0.71 0.80 
 SRT + PRT (at enrollment) 0.73 0.84 
 Change in SRT (at enrollment) 0.78 0.84 
Discriminant ability (%) 
 SRT (at enrollment) 69 75 
 SRT + PRT (at enrollment) 71 76 
 Change in SRT (at enrollment) 76 81 
Sensitivity (%)/specificity (%) 
 SRT (at enrollment) 52/86 63/86 
 SRT + PRT (at enrollment) 59/82 71/82 
 Change in SRT (at enrollment) 71/82 81/82 

Notes: ROC=receiver operating characteristic; SRT = simple reaction time; PRT = procedural reaction time.

Performance of these tests' can be improved with the use of a clinical decision tool. Based on covariate analyses, a decision tool was created examining five symptoms: Pain, headache, blackouts, confusion, and mental health <4 on a scale of 1–5. All combinations of LOC/PTA and symptoms listed above were analyzed. The largest area under the ROC curve and discriminant ability were obtained by examining cases with either LOC/PTA or at least three symptoms. Using this tool, the discriminant ability for SRT at enrollment increased from 69% to 75% and change in SRT from baseline increased from 76% to 81%. Enrollment SRT + PRT improved from 71% to 76%. Although specificity was not altered by the clinical decision tool, remaining constant at above 80% for all, sensitivity increased using the tool, from 52% to 63% for SRT at enrollment, from 59% to 71% for SRT + PRT at enrollment, and from 71% to 81% for change in SRT from baseline.

Concurrent Validity

Correlation of ANAM subtests with traditional neuropsychological testing was poor. No combination of traditional neuropsychological test results achieved a kappa >0.2 when compared with SRT in differentiating cases and controls. This does not imply that the ANAM lacks concurrent validity as SRT demonstrated significantly higher discriminant ability than any combination of traditional tests. No combination of traditional tests achieved discriminant ability greater than 62% for the entire case group or 70% using the clinical decision tool. Results show inferiority of the traditional tests in this setting, making their use as a gold standard for concurrent validity inappropriate.

Discussion

Results clearly demonstrate that ANAM, and particularly SRT, is more effective than the traditional brief sports medicine neuropsychological battery in differentiating concussed from non-concussed participants in the combat environment when administered within 72 h of injury.

Equally important, limiting testing to SRT provides higher sensitivity and specificity than administering the entire ANAM. The current research replicates prior studies (Luethcke et al., 2011; McCrea, Prichep, Powell, Chabot, & Barr, 2010; Warden et al., 2001) supporting the sensitivity of SRT to concussion relative to other ANAM subtests, though findings have not been entirely consistent: In contrast to other reports, in a study of U.S. Military Academy athletes, Bleiberg and colleagues (2004) found that while SRT declined following concussion, SRT also declined in controls and they speculated the variance in their results with other studies may have reflected the overarching stressors occurring in military academy life. In any event, implications of the sensitivity of SRT to concussion for the development of a neuropsychological testing platform that is portable and readily adaptable to the field environment are of great importance for the Department of Defense. As ANAM is currently administered on a laptop and the SRT subtest is scored based on two administrations, one at the beginning and one at the end of the battery, these results need to be validated in an environment where only SRT or SRT and PRT are tested. However, if results are cross-validated, a shortened testing battery could be developed.

Another key finding of this study is that the establishment of baseline SRT is valuable, as changes in SRT are more sensitive and specific than one-time testing after a concussion. Because neurocognitive performance varies in healthy individuals by age, ethnic background, education, and developmental disorders such as learning disability (Collins, Grindel, & Lovell, 1999; Heaton, Miller, Taylor, & Grant, 1997) comparing post-concussion scores to an individualized baseline would be expected to result in greater diagnostic accuracy than comparison with a normative reference group. Evidence from sports medicine concussion research supports the value of baseline testing (Warden et al., 2001) and it is accepted as a component of comprehensive concussion management programs (Aubry et al., 2002; Barr et al., 2008). The authors did not obtain in-theater baseline scores for concussed individuals, but one area of possible future exploration is whether changes in these neuropsychological test results versus in-theater baselines would be even more sensitive and specific.

The addition of a clinical decision tool incorporating subjective post-concussive symptoms improved sensitivity. Although a decrease in SRT of more than 1.5 points is 71% sensitive and 82% specific in differentiating concussions from controls, adding the clinical decision tool improves sensitivity to 81%, comparable with other neuropsychological tools described in sports medicine concussion literature (Barr et al., 2008; Broglio et al., 2007b; Schatz et al., 2006) and supporting evidence indicating algorithms combining neuropsychological results with the measurement of self-reported post-concussive symptoms increases diagnostic accuracy (Broglio et al., 2007a; Van Kampen et al., 2006). The authors stress, though, that this tool was developed in post hoc analysis and is presented only as an example of how a clinical decision tool could enhance test performance, not to suggest that this tool be adopted.

Analysis of the impact of covariates on SRT in controls demonstrates the insensitivity of this test to non-concussive factors. No demographic, physical health, sleep, or service-related factor was significantly associated with SRT performance at enrollment at the p < .05 level.

In order to address the possibility of Type I error, the Bonferroni corrections were calculated and significance is presented with this adjustment. It should be noted that adjustment for multiple comparisons, while commonly used in neuropsychological studies, is not universally recommended and may increase the possibility of Type II error (Rothman, 1990). Type II error may be a particular concern for studies of techniques designed to detect effects of concussion in soldiers being evaluated for return to hazardous duty.

Cernich, Brennana, Barker, and Bleiberg (2007) provided an overview of technology centered concerns related to the equivalence of reaction time-based computerized neuropsychological tests (with specific reference to the ANAM battery) when the measures were utilized across different hardware configurations and operating systems. Cernich and colleagues (2007) indicate that there are several advantages of ANAM as a computerized neuropsychological test: The test has been verified as compatible with multiple operating systems (including the operating system employed in this study), timing accuracy has been verified through oscilloscope timing, the ANAM program detects screen settings and adjusts display size so that the display is full screen on all platforms, presentation of stimuli in ANAM coincides with the refresh rate of the display (promoting accurate timing of the display of test stimuli), and during test administration the ANAM disables all other programs in order to capture CPU resources. Our project made specific efforts to address other concerns raised by Cernich and colleagues (2007). Although data for our study were collected in three locations, all used the same hardware (Dell D630), operating system (Windows XP), and a USB optical mouse as recommended by the test vendor. In addition, pre-deployment baseline ANAM assessments used in this project were collected on Dell D630 computers running Windows XP and using the recommended USB optical mouse as well. Because our ANAM testing was completed on freestanding computers without internet connectivity, sources of error related to assessments conducted in real time were eliminated.

Results from this study suggest that the ANAM, and particularly SRT, can detect changes in cognition following a concussion incurred in the combat environment that are both “statistically significant” and “clinically significant.” The term statistical significance in clinical neuropsychology tends to be used in the context of testing the null hypothesis. Findings indicating that cases performed more poorly than controls on multiple ANAM subtests at the time of enrollment, and that there was deterioration in ANAM performance from pre-deployment baseline to enrollment in cases when compared with controls, support the statistical significance of our results (Table 2). However, not all differences that are statistically significant can be considered practically clinically significant. For example, differences in function that are small in magnitude are not likely to be practically relevant. Likewise, differences in function common in a healthy population are also not likely clinically significant as they may merely be an indication of normal variation. One measure that has been proposed as a marker of practical clinical significance in clinical neuropsychology is effect size (Peterson, 2008). Additional evaluation of our data set found that when individual change from baseline to enrollment was calculated for cases, effect sizes for most ANAM subtests (the exception was CDD) ranged from .29 to .46 with SRT having the largest effect size. These effect sizes are in the small to medium range but comparable to effect sizes seen in cognitive studies of other disorders including Attention-Deficit/Hyperactivity Disorder (ADHD), depression, and benzodiazepine withdrawal (Iverson, 2005). Perhaps more importantly, SRT, particularly when used as a component of a clinical tool, provided encouraging discriminant ability results (Table 5).

This study has several limitations. The first is that sample size, while sufficient to assess sensitivity and specificity of the tests under study, was insufficient to fully analyze the impact of cofactors potentially impacting neurocognitive performance. This study was underpowered to detect the impact of these cofactors on change in SRT from baseline.

Another limitation is that pre-deployment baseline scores were often not available, and there were no in-theater baseline scores. Although in the present study covariates did not significantly impact controls' SRT scores, prior work based on the comparison of neuropsychological testing done prior to deployment and roughly 3 months after deployment (Vasterling et al., 2006) has suggested that deployment itself could alter cognition, potentially confounding results (though it is possible that the Vasterling and colleagues (2006) data could have been affected by events occurring in the time interval between the end of deployment and post-deployment testing). In any case, in-theater baselines would potentially improve ANAM discriminant ability by limiting such a confound.

Lack of a detailed post-concussion symptom questionnaire administered at pre-deployment baseline and at enrollment is another limitation, as prior work from sports medicine has suggested combining structured symptom report with neuropsychological data increases discriminant ability (Broglio et al., 2007a; Van Kampen et al., 2006).

It is possible, based on known natural history (Bleiberg et al., 2004; McCrea et al., 2009; Pellman, Lovell, Viano, Casson, & Tucker, 2004; Pellman et al., 2006), that some cases had recovered from concussion by the time ANAM was administered. As such, results may underestimate initial ANAM sensitivity as testing was conducted no earlier than the day after concussion.

Neuropsychological testing is not a substitute for thorough examination by medical professionals or neuroimaging. Rather, testing is a tool to aid in clinical decision-making. Further validation of these findings is required. Particularly promising would be prospective validation of the use of SRT alone, either at enrollment or in comparison with baseline scores. The addition of other tests, such as PRT or a clinical decision tool, will also require prospective validation before general use can be recommended. Associations found in post hoc analysis must be viewed with healthy skepticism.

Findings from the present study have significant implications for military clinicians using the ANAM to assist in the evaluation and management of acute concussion in the combat zone, where the ANAM continues to be used by the Department of Defense as the primary computerized neuropsychological tool. When originally packaged for use in Iraq and Afghanistan, interpretation of the test relied on a “stoplight” system based on a large normative data set obtained from healthy soldiers tested in the USA; validation in a clinical population of soldiers suffering concussion was not yet available. In the stoplight system, scores are characterized as “average or above” (green), “below average” (yellow), or “clearly below average” (red) based on the standard score cutoffs in the healthy soldier normative group. Prior related work by our research team (Coldren, Russell, Kelly, Parish, & Dretsch, 2011) found that the ANAM stoplight method of interpretation was neither sensitive nor specific enough for clinical use: For soldiers tested in Iraq within 72 h of a concussion, the stoplight system yielded a sensitivity of only 52% and specificity of 75%. Data from the present study make available valuable preliminary clinical information regarding ANAM subtests most vulnerable to the effects of concussion within 72 h after injury and provide clinicians with an alternative approach to the stoplight for interpreting ANAM data. In addition, our findings provide initial support for the value of pre-deployment baseline evaluation in improving diagnostic accuracy (Tables 2, 3, and 5) in acute concussion. Over time, the availability of baseline deployment scores has been growing, and data from our project should encourage clinicians evaluating patients early in the course of recovery from concussion to make efforts to obtain baseline scores for comparison.

We must also stress that the support for clinical use of the ANAM generated by this study is limited to clinical contexts involving the assessment of soldiers with acute (i.e., within 72 h of injury) concussion. Our findings do not provide support for the use of the ANAM as a population screening tool for concussion, as effects of concussion on objective measures of cognition typically resolve over a period of days to weeks (Bleiberg et al., 2004; Hugenholtz et al., 1988; McCrea et al., 2003) and many disorders other than concussion can impair performance on the ANAM and other computerized neuropsychological tests (Erlanger et al., 2002; Kane, Roebuck-Spencer, Short, Kabat, & Wilken, 2007).

In summary, ANAM SRT within 72 h of a concussion, particularly if compared with baseline results, is a relatively sensitive and specific method to differentiate concussed from non-concussed individuals in the combat environment. It demonstrated superiority to the traditional neuropsychological tests and to the full ANAM-IV-TBI battery. The addition of PRT or a clinical decision tool may improve the discriminant ability of this testing. Findings provide initial support for the use of neuropsychological testing to assist in acute management of concussion on the battlefield and for the utility of pre-deployment baseline testing.

Funding

This work was supported in part by the U.S. Army Medical Research Acquisition Activity, 820 Chandler Street, Fort Detrick, Maryland 21702-5014, Project W81XWIH-09-2-0057.

Conflict of Interest

The views expressed in this manuscript are those of the authors and do not necessarily reflect the official policy or position of the Department of the Army, Department of Defense, or the United States Government.

Acknowledgements

The authors acknowledge the assistance of numerous individuals whose efforts were critical for the performance of this study. We thank SSG David Lopez and SGT Pedro Cruz of the U.S. Army Aeromedical Research Laboratory for their diligent data collection efforts, Cara Olsen of Uniformed Services University for the Health Sciences for assistance in the statistical analysis, and April Korbel of the United States Army Medical Command at Brooke Army Medical Center for her assistance with manuscript preparation.

References

Aubry
M.
Cantu
R.
Dvorak
J.
Graf-Bauman
T.
Johnston
K.
Kelly
J.
, et al.  . 
Summary and agreement statement of the first International Conference on Concussion in Sport, Vienna, 2001
British Journal of Sports Medicine
 , 
2002
, vol. 
3
 (pg. 
6
-
10
)
Barr
W.
McCrea
M.
Randolph
C.
Morgan
J.
Ricker
J.
Neuropsychology of sports related injuries
Textbook of clinical neuropsychology
 , 
2008
New York
Taylor and Francis
(pg. 
660
-
678
)
Barth
J.
Alves
W.
Ryan
T.
Macciocchi
S.
Rimel
R.
Jane
J.
, et al.  . 
Levin
H.
Eisenberg
H.
Benton
A.
Mild head injury in sports: Neuropsychological sequelae and recovery of function
Mild head injury
 , 
1989
New York
Oxford University Press
(pg. 
257
-
275
)
Barth
J.
Isler
W.
Helmick
K.
Wingler
I.
Jaffee
M.
Kennedy
C.
Moore
J.
Acute battlefield assessment of concussion/mild TBI and return to duty evaluations
Military neuropsychology
 , 
2010
New York
Springer
(pg. 
127
-
174
)
Belanger
H.
Curtiss
G.
Demery
J.
Lebowitz
B.
Vanderploeg
R.
Factors moderating neuropsychological outcomes following mild brain injury: A meta-analysis
Journal of the International Neuropsychological Society
 , 
2005
, vol. 
11
 (pg. 
215
-
227
)
Belanger
H.
Kretzmer
T.
Yoash-Gantz
R.
Pickett
T.
Tupler
L.
Cognitive sequelae of blast related versus other mechanisms of brain trauma
Journal of the International Neuropsychological Society
 , 
2009
, vol. 
15
 (pg. 
1
-
8
)
Belanger
H.
Vanderploeg
R.
The neuropsycological impact of sports-related concussion: A meta-analysis
Journal of the International Neuropsychological Society
 , 
2005
, vol. 
11
 (pg. 
345
-
357
)
Bhattacharjee
Y.
Shell shock revisited: Solving the puzzle of blast trauma
Science
 , 
2008
, vol. 
319
 (pg. 
406
-
408
)
Bleiberg
J.
Cernich
A.
Cameron
K.
Sun
W.
Peck
K.
Ecklund
J.
, et al.  . 
Duration of cognitive impairment after sports concussion
Neurosurgery
 , 
2004
, vol. 
54
 (pg. 
1073
-
1080
)
Bleiberg
J.
Kane
R.
Reeves
D.
Garmoe
W.
Halpern
E.
Factor analysis of computerized and traditional tests used in mild traumatic brain injury
The Clinical Neuropsychologist
 , 
2000
, vol. 
14
 (pg. 
287
-
294
)
Broglio
S.
Macciocchi
S.
Ferrara
M.
Neurocognitive performance of concussed athletes when symptom free
Journal of Athletic Training
 , 
2007
, vol. 
42
 
4
(pg. 
504
-
508
)
Broglio
S.
Macciocchi
S.
Ferrara
M.
Sensitivity of the concussion assessment battery
Neurosurgery
 , 
2007
, vol. 
60
 
6
(pg. 
1050
-
1058
)
Cernak
I.
Noble-Haeusslein
L.
Traumatic brain injury: An overview with emphasis on military populations
Journal of Cerebral Blood Flow and Metabolism
 , 
2010
, vol. 
30
 (pg. 
255
-
266
)
Cernich
A.
Brennana
D.
Barker
L.
Bleiberg
J.
Sources of error in computerized neuropsychological assessment
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22S
 (pg. 
S39
-
S48
)
Cernich
A.
Reeves
D.
Sun
W.
Bleiberg
J.
Automated Neuropsychological Assessment Metrics sports medicine battery
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22S
 (pg. 
S101
-
S114
)
Coldren
R.
Kelly
M.
Parish
R.
Dretsch
M.
Russell
M.
Evaluation of the Military Acute Concussion Evaluation for use in combat operations more than 12 hours after injury
Military Medicine
 , 
2010
, vol. 
175
 
7
(pg. 
477
-
481
)
Coldren
R.
Russell
M.
Kelly
M.
Parish
R.
Dretsch
M.
ANAM stoplight
Military Medicine
 , 
2011
, vol. 
176
 
9
(pg. 
iv
-
iv
)
Collins
M.
Grindel
S.
Lovell
M.
Relationship between concussion and neuropsychological performance in college football players
Journal of the American Medical Association
 , 
1999
, vol. 
282
 
10
(pg. 
964
-
970
)
Culotta
V.
Sementilli
M.
Gerold
K.
Watts
C.
Clinicopathological heterogeneity in the classification of mild head injury
Neurosurgery
 , 
1996
, vol. 
38
 
2
(pg. 
245
-
250
)
DePalma
R.
Burris
D.
Champion
H.
Hodgson
M.
Blast injuries
New England Journal of Medicine
 , 
2005
, vol. 
352
 
13
(pg. 
1335
-
1342
)
Defense and Veterans Brain Injury Center.
DoD worldwide numbers for traumatic brain injury
2011
 
Retrieved November 14, 2011, from DVBIC.org: http://www.dvbic.org/TBI-Numbers.aspx
Department of Defense
Health Affairs Memorandum 07-030. Traumatic Brain Injury: Definition and Reporting, 2007
2007
 
Dick
R.
Is there a gender difference in concussion incidence and outcomes?
British Journal of Sports Medicine
 , 
2009
, vol. 
43
 
Issue Supplement
(pg. 
i46
-
i50
)
Erlanger
D.
Feldman
D.
Kutner
K.
Kaushik
T.
Kroger
H.
Festa
J.
, et al.  . 
Development and validation of a web-based neuropsychological test protocol for sports-related return-to-play decision-making
Archives of Clinical Neuropsychology
 , 
2003
, vol. 
18
 (pg. 
293
-
316
)
Erlanger
D.
Kaushik
T.
Broshek
D.
Freeman
J.
Feldman
D.
Festa
J.
Development and validation of a web-based screening tool for monitoring cognitive status
Journal of Head Trauma Rehabilitation
 , 
2002
, vol. 
17
 
5
(pg. 
458
-
476
)
Gasquoine
P.
Postconcussion symptoms in chronic back pain
Applied Neuropsychology
 , 
2000
, vol. 
7
 
2
(pg. 
83
-
89
)
Giza
C.
Hovda
D.
The neurometabolic cascade of concussion
Journal of Athletic Training
 , 
2001
, vol. 
36
 (pg. 
228
-
235
)
Greiffenstein
M.
Kennedy
C.
Moore
J.
Noncredible neuropsychological presentation in service members and veterans
Military neuropsychology
 , 
2010
New York
Springer
(pg. 
81
-
100
)
Guskiewicz
K.
McCrea
M.
Marshall
S.
Cantu
R.
Randolph
C.
Barr
W.
, et al.  . 
Cumulative effects associated ith recurrent concussion in collegiate football players
Journal of the American Medical Association
 , 
2003
, vol. 
290
 
19
(pg. 
2549
-
2555
)
Heaton
R.
Miller
S.
Taylor
M.
Grant
I.
Revised comprehensive norms for an expanded Halstead-Reitan Battery
 , 
1997
Lutz, FL
Psychological Assessment Resources
Hoge
C.
McGurk
D.
Thomas
J.
Cox
A.
Engel
C.
Castro
C.
Mild traumatic brain injury in U.S. soldiers returning from Iraq
New England Journal of Medicine
 , 
2008
, vol. 
358
 (pg. 
453
-
463
)
Huang
M.
Theilmann
R.
Robb
A.
Integrated imaging approach with MEG and DTI to detect mild traumatic brain injury in military and civilian patients
Journal of Neurotrauma
 , 
2009
, vol. 
26
 (pg. 
1213
-
1226
)
Hugenholtz
H.
Stuss
D.
Stethem
L.
Richard
M.
How long does it take to recover from mild concussion?
Neurosurgery
 , 
1988
, vol. 
22
 
5
(pg. 
853
-
858
)
Iverson
G.
Outcome from mild traumatic brain injury
Current Opinion in Psychiatry
 , 
2005
, vol. 
18
 (pg. 
301
-
317
)
Iverson
G.
Misdiagnosis of the persistent postconcussion syndrome in patients with depression
Archives of Clinical Neuropsychology
 , 
2006
, vol. 
21
 (pg. 
303
-
310
)
Iverson
G.
Langlois
J.
McCrea
M.
Kelly
J.
Challenges associated with post-deployment screening for mild traumatic brain injury in military personnel
The Clinical Neuropsychologist
 , 
2009
, vol. 
23
 (pg. 
1299
-
1314
)
Kabat
M.
Kane
R.
Jefferson
A.
DiPino
P.
Construct validity of selected Automated Neuropsychological Assessment Metrics (ANAM) battery measures
The Clinical Neuropsychologist
 , 
2001
, vol. 
15
 (pg. 
498
-
507
)
Kane
R.
Roebuck-Spencer
T.
Short
P.
Kabat
M.
Wilken
J.
Identifying and monitoring cognitive deficits using Automated Neuropsychological Assessment Metrics (ANAM) tests
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22S
 (pg. 
S115
-
S126
)
Kelly
J.
Nichols
J.
Filley
C.
Concussion in sports: Guidelines for prevention of catastrophic outcome
Journal of the American Medical Association
 , 
1991
, vol. 
266
 
20
(pg. 
2867
-
2869
)
Ling
G.
Bandak
F.
Armonda
R.
Grant
G.
Ecklund
J.
Explosive blast neurotrauma
Journal of Neurotrauma
 , 
2009
, vol. 
26
 (pg. 
815
-
825
)
Lovell
M.
Collins
M.
Neuropsychological assessment of the college football player
Journal of Head Trauma Rehabilitation
 , 
1998
, vol. 
13
 
2
(pg. 
9
-
26
)
Luethcke
C.
Bryan
C.
Morrow
C.
Isler
W.
Comparison of concussive symptoms, cognitive performance, and psychological symptoms between acute blast-versus nonblast-induced mild traumatic brain injury
Journal of the International Neuropsychological Society
 , 
2011
, vol. 
17
 (pg. 
36
-
45
)
Maas
A.
Harrison-Felix
C.
Menon
D.
Common data elements for traumatic brain injury: Recommendations from the interagency working group on demographics and clinical assessment
Archives of Physical Medicine and Rehabilitation
 , 
2010
, vol. 
91
 (pg. 
1641
-
1649
)
Macciocchi
S.
Barth
J.
Alves
W.
Rimel
R.
Jane
J.
Neuropsychological functioning and recovery after mild head injury in college athletes
Neurosurgery
 , 
1996
, vol. 
39
 
3
(pg. 
510
-
514
)
MacGregor
A.
Schaffer
R.
Dougherty
A.
Prevalence and psychological correlates of traumatic brain injury in Operation Iraqi Freedom
Journal of Head Trauma Rehabilitation
 , 
2010
, vol. 
25
 (pg. 
1
-
8
)
Machulda
M.
Berquist
T.
Ito
U.
Chew
S.
Relationship between stress, coping, and postconcussion symptoms in a healthy adult population
Archives of Clinical Neuropsychology
 , 
1998
, vol. 
13
 (pg. 
415
-
424
)
McCrea
M.
Guskiewicz
K.
Marshall
S.
Barr
W.
Randolph
C.
Cantu
R.
, et al.  . 
Acute effects and recovery time following concussion in college football players
Journal of the American Medical Association
 , 
2003
, vol. 
290
 
19
(pg. 
2556
-
2563
)
McCrea
M.
Guskiewicz
K.
Randolph
C.
Effects of symptom free waiting period on clinical outcome and risk of reinjury after sport-related concussion
Neurosurgery
 , 
2009
, vol. 
65
 
5
(pg. 
876
-
883
)
McCrea
M.
Prichep
L.
Powell
M.
Chabot
R.
Barr
W.
Acute effects and recovery after sport-related concussion: A neurocognitive and quantitative brain electrical activity study
Journal of Head Trauma Rehabilitation
 , 
2010
, vol. 
25
 
4
(pg. 
283
-
292
)
McNeil
J.
Morgan
C.
Kennedy
C.
Moore
J.
Cognition and decision making in extreme environments
Military neuropsychology
 , 
2010
New York
Springer
(pg. 
361
-
382
)
Pellman
E.
Lovell
M.
Viano
D.
Casson
J.
Concussion in professional football: Recovery of NFL and high school athletes assessed by computerized neuropsychological testing-Part 12
Neurosurgery
 , 
2006
, vol. 
58
 
2
(pg. 
262
-
274
)
Pellman
E.
Lovell
M.
Viano
D.
Casson
J.
Tucker
A.
Concussion in professional football: Neuropsychological testing-Part 6
Neurosurgery
 , 
2004
, vol. 
55
 
6
(pg. 
1290
-
1293
)
Peterson
L.
“Clinical” significance and “practical” significance are NOT the same things
2008
Paper presented at the Annual Meeting of the Southwest Educational Research Association
New Orleans, LA
Reeves
D.
Winter
K.
Bleiberg
J.
Kane
R.
ANAM genogram: Historical perspectives, description, and current endeavors
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22S
 (pg. 
S15
-
S37
)
Riegelman
R.
Studying a study and testing a test: How to read the medical evidence
 , 
2004
5th ed.
Philadelphia
Lippincott Williams and Wilkins
Rothman
K.
No adjustments are needed for multiple comparisons
Epidemiology
 , 
1990
, vol. 
1
 
1
(pg. 
43
-
46
)
Saunders
R.
Harbaugh
R.
The second impact in catastrophic contact-sports head trauma
Journal of the American Medical Association
 , 
1984
, vol. 
252
 
4
(pg. 
538
-
539
)
Schatz
P.
Pardini
J.
Lovell
M.
Collins
M.
Podell
K.
Sensitivity and specificity of the ImPACT test battery for concussion in athletes
Archives of Clinical Neuropsychology
 , 
2006
, vol. 
21
 (pg. 
91
-
99
)
Terrio
H.
Brenner
L.
Ivins
B.
Traumatic brain injury screening: Preliminary findings in a U.S. Army brigade combat team
Journal of Head Trauma Rehabilitation
 , 
2009
, vol. 
24
 (pg. 
14
-
23
)
Van Kampen
D.
Lovell
M.
Pardini
J.
Collins
M.
Fu
F.
The “value added” of neurocognitive testing after sports-related concussion
American Journal of Sports Medicine
 , 
2006
, vol. 
34
 
10
(pg. 
1630
-
1635
)
Vasterling
J.
MacDonald
H.
Ulloa
E.
Rodier
N.
Kennedy
C.
Moore
J.
Neuropsychological correlates of PTSD: A military perspective
Military neuropsychology
 , 
2010
New York
Springer
(pg. 
321
-
360
)
Vasterling
J.
Proctor
S.
Amoroso
P.
Kane
R.
Heeren
T.
White
R.
Neuropsychological outcomes of army personnel following deployment to the Iraq war
Journal of the American Medical Association
 , 
2006
, vol. 
296
 
5
(pg. 
519
-
529
)
Warden
D.
Bleiberg
J.
Cameron
K.
Ecklund
J.
Walter
J.
Sparling
B.
, et al.  . 
Persistent prolongation of simple reaction time in sports concussion
Neurology
 , 
2001
, vol. 
57
 (pg. 
524
-
526
)
Wesensten
N.
Balkin
T.
Kennedy
C.
Moore
J.
Cognitive sequelae of sustained operations
Military neuropsychology
 , 
2010
New York
Springer
(pg. 
297
-
320
)
Wilk
J.
Thomas
J.
McGurk
D.
Riviere
L.
Castro
C.
Hoge
C.
Mild traumatic brain injury (concussion) during combat: Lack of association of blast mechanism with persistent postconcussive symptoms
Journal of Head Trauma Rehabilitation
 , 
2010
, vol. 
25
 
1
(pg. 
9
-
14
)

Author notes

5
Present address: Psychology Department, Walter Reed National Military Medical Center, Bethesda, MD 20889, USA.
6
Present address: Department of Preventive Medicine and Biometrics, Uniformed Services University of the Health Sciences, Bethesda, MD 20814, USA.
7
Present address: Neuropsychology Service, Landstuhl Regional Medical Center, APO, AE 09180, USA.
8
Present address: Neurocognitive Assessment Branch, Rehabilitation and Reintegration Division, Office of the United States Army Surgeon General, Alexandria, VA 22031, USA.