Abstract

We present the initial validation of a two-factor structure of Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT) using ImPACT composite scores and document the reliability and validity of this factor structure. Factor analyses were conducted for baseline (N = 21,537) and post-concussion (N = 560) data, yielding “Memory” (Verbal and Visual) and “Speed” (Visual Motor Speed and Reaction Time) Factors; inclusion of Total Symptom Scores resulted in a third discrete factor. Speed and Memory z-scores were calculated, and test–retest reliability (using intra-class correlation coefficients) at 1 month (0.88/0.81), 1 year (0.85/0.75), and 2 years (0.76/0.74) were higher than published data using Composite scores. Speed and Memory scores yielded 89% sensitivity and 70% specificity, which was higher than composites (80%/62%) and comparable with subscales (91%/69%). This emergent two-factor structure has improved test–retest reliability with no loss of sensitivity/specificity and may improve understanding and interpretability of ImPACT test results.

Introduction

The reliability (Broglio, Ferrara, Macciocchi, Baumgartner, & Elliott, 2007; Elbin, Schatz, & Covassin, 2011; Iverson, Lovell, & Collins, 2003; Resch et al., 2013; Schatz, 2010; Schatz & Ferris, 2013), validity (Allen & Gfeller, 2011; Iverson, Gaetz, Lovell, & Collins, 2005; Maerlender et al., 2010, 2013; Schatz, Pardini, Lovell, Collins, & Podell, 2006; Schatz & Sandel, 2012), and utility (Lau, Collins, & Lovell, 2012; Van Kampen, Lovell, Pardini, Collins, & Fu, 2006) of the ImPACT computer-based neuropsychological test, for the purpose of diagnosis, assessment, and management of sports-related concussion, has been documented and debated (Lovell, 2006; Mayers & Redick, 2012; Randolph, 2006, 2011; Randolph, Lovell, & Laker, 2011; Randolph, McCrea, & Barr, 2005, 2006; Schatz, Kontos, & Elbin, 2012) in the literature. Similarly, the advantages and disadvantages of computer-based testing have been discussed (APA, 1986; Schatz & Zillmer, 2003), including the inherent inability to systematically assess receptive auditory modalities or verbal production (Schatz & Browndyke, 2002).

The ImPACT test has evolved since its inception (circa 1999), from an executable file on Windows machines, to its current form as an online, Flash-based application accessed through a web browser. At the time it was introduced (Version 1.2), ImPACT generated only three composites scores for use in clinical interpretation; Memory (i.e., Verbal Memory), Reaction Time, and Processing/Visual Motor Speed, along with Impulse Control (for evaluating test validity); Visual Memory was introduced as a composite score in Version 2.0, but for non-clinical, research purposes (Lovell, 2004), at which time the “Memory” composite was renamed “Verbal Memory.” The current (online) version, as well as the most recent stand-alone version (i.e., desktop, Version 6), generate four composite scores, all of which are documented in clinical reports and research exports: Verbal Memory, Visual Memory, Visual Motor Speed, Reaction Time; Impulse Control remains as a non-clinical composite score for evaluating test validity (Lovell, 2007). Verbal Memory uses a word-recognition paradigm using stimuli (e.g., words) that can be verbally encoded, whereas Visual Memory uses a design-recognition paradigm using stimuli (i.e., ambiguous designs) that cannot be easily encoded verbally. However, despite their names, all stimuli are presented visually, and there is no systematic evaluation regarding mechanisms or strategies for encoding. As a result, the resultant composite scores may create confusion, as test administrators and clinicians are often left with the task of deciphering differences between Verbal and Visual Memory, Reaction Time and Motor Speed, and Visual Memory and Visual Motor Speed.

The initial factor analytic studies, included in the “ImPACT Clinical Interpretation Manual, Version 2.0” (Lovell, 2004) generated several factor structures, many of which were comprised by only two factors (a combined Verbal Memory/Visual Memory and a combined Reaction Time/Processing Speed factor). In an attempt to validate ImPACT as a measure of processing speed (e.g., by including the Symbol Digit Modalities Test along with ImPACT composite scores), exploratory factor analysis identified a “speed/reaction time” factor and a “memory” factor (Iverson et al., 2005). Subsequent exploratory factor analysis of ImPACT subscales yielded a five-factor theory (Allen & Gfeller, 2011), measuring “forced-choice efficiency,” “verbal and visual memory,” “inhibitory cognitive abilities,” “visual processing abilities,” and a lone subscale of “color matching.” Given that subsequent manuals released after the introduction of the Visual Memory composite score did not include updated factor structures, and there appears to be evidence for shared variance among Processing Speed and Reaction Time scores as well as Verbal Memory and Visual Memory scores, we sought to (a) validate the existence of a two-factor structure of ImPACT (Memory, Speed) using the current composite scores, and (b) evaluate the utility of this emergent two-factor with respect to ImPACT's reliability and validity.

Study 1: Factor Analyses

Participants and Methods

Baseline and post-concussion ImPACT data were extracted from larger data sets provided by the lead programmer at ImPACT Inc., who was blind to the purposes of this study. More specifically, all cases within the age ranges 13–22 were extracted, whereas cases outside this age range were excluded. Institutional Review Board approval was obtained for secondary analysis of de-identified data.

Baseline sample

The Baseline sample was composed of 21,357 middle school, high school, and collegiate athletes, with a mean age of 15.5 (SD = 1.90), who completed preseason baseline assessments. Participants were nearly equal with respect to gender (51.7% men, 48.3% women) and had no previous history of concussion, history attention deficit disorder (ADD), learning disability (LD), or previous treatment for headaches, migraines, seizures, or other neurologic or psychiatric illness. All participants completed the computerized ImPACT baseline test (online version) in groups of approximately 10–20 athletes, supervised by Certified Athletic Trainers or Sports Medicine Personnel. Participants with country of origin other than the USA, first language other than English, or invalid baselines, as identified by built-in cut-offs on specific ImPACT subscales (denoted by “Baseline ++”) were excluded prior to the provision of the data.

Concussion sample

The concussion sample was composed of 560 middle school, high school, and collegiate athletes, with a mean age of 15.6 (SD = 1.95), who completed preseason baseline assessments, sustained a concussion, and were evaluated within 7 days of reported injury. Participants were, by majority, men (58.2% versus 41.8% women) and had no previous history of concussion, history of ADD, LD, or previous treatment for headaches, migraines, seizures, or other neurologic or psychiatric illness. All participants completed the computerized ImPACT baseline test (online version) in an individual setting, supervised by Certified Athletic Trainers or Sports Medicine Personnel. Participants with country of origin other than the USA, first language other than English, were excluded from analyses.

Analyses

Confirmatory factor analyses were conducted for a two-factor solution, with ImPACT Composite scores (Verbal Memory, Visual Memory, Visual Motor Speed, Reaction Time), using Statistical Package for Social Sciences (SPSS), Version 19, with Varimax rotation. Kaiser–Meyer–Olkin Measure of Sampling Adequacy was above 0.600 for all analyses, and Eigenvalues (EVs) were all ≥1.0. Additional confirmatory factor analyses were conducted for a three-factor solution, with the above-listed ImPACT composite scores as well as Total Symptom Scores.

Results

Verbal Memory and Visual Memory composite scores formed a unique factor (“Memory”), as did Visual Motor Speed and Reaction Time scores (“Speed”). Within the Baseline sample, the factor analysis accounted for 72.5% of the variance, with the “Speed” factor accounting for 37.8% of the variance (EV = 1.51) and “Memory” accounting for 34.7% of the variance (EV = 1.39). Within the Concussion sample, the factor analysis accounted for 78.8% of the variance, with the “Memory” factor accounting for 40.2% of the variance (EV = 1.51) and the “Speed” factor accounting for 38.6% of the variance (EV = 1.54; Table 1).

Table 1.

Factor analysis results

Composite Baseline samplea
 
Concussion sampleb
 
Factor 1 Factor 2 Factor 1 Factor 2 
Verbal Memory 0.120 0.824 0.876 0.212 
Visual Memory 0.143 0.808 0.823 0.307 
Visual Motor Speed 0.831 0.231 0.183 0.893 
Reaction Time −0.887 0.062 0.362 −0.772 
Composite Baseline samplea
 
Concussion sampleb
 
Factor 1 Factor 2 Factor 1 Factor 2 
Verbal Memory 0.120 0.824 0.876 0.212 
Visual Memory 0.143 0.808 0.823 0.307 
Visual Motor Speed 0.831 0.231 0.183 0.893 
Reaction Time −0.887 0.062 0.362 −0.772 

Bold indicates p-values not provided for factor analysis results.

aBaseline sample: N = 21,357.

bConcussion sample: N = 560.

Inclusion of Total Symptom Scores yielded a three-factor solution, for both the Baseline and Concussion samples. Within the Baseline sample, the factor analysis accounted for 78% of the variance, with the “Speed” factor accounting for 30.3% of the variance (EV = 1.51), “Memory” accounting for 27.8% of the variance (EV = 1.39), and “Symptoms” accounting for 20% of the variance (EV = 1.00). Within the Concussion sample, the factor analysis accounted for 83% of the variance, with the “Memory” factor accounting for 31.9% of the variance (EV = 1.59), “Speed” accounting for 30.8% of the variance (EV = 1.54), and “Symptoms” accounting for 20.5% of the variance (EV = 1.03; Table 2).

Table 2.

Factor analysis results with symptoms

Composite Baseline samplea
 
Concussion sampleb
 
Factor 1 Factor 2 Factor 3 Factor 1 Factor 2 Factor 3 
Verbal Memory 0.118 0.825 0.019 0.834 0.306 −0.052 
Visual Memory 0.144 0.806 −0.039 0.851 0.204 −0.195 
Visual Motor Speed 0.831 0.231 0.007 0.189 0.891 −0.045 
Reaction Time 0.887 −0.062 −0.002 0.339 0.773 0.162 
Symptom Total 0.007 −0.015 0.999 −0.150 0.115 0.978 
Composite Baseline samplea
 
Concussion sampleb
 
Factor 1 Factor 2 Factor 3 Factor 1 Factor 2 Factor 3 
Verbal Memory 0.118 0.825 0.019 0.834 0.306 −0.052 
Visual Memory 0.144 0.806 −0.039 0.851 0.204 −0.195 
Visual Motor Speed 0.831 0.231 0.007 0.189 0.891 −0.045 
Reaction Time 0.887 −0.062 −0.002 0.339 0.773 0.162 
Symptom Total 0.007 −0.015 0.999 −0.150 0.115 0.978 

aBaseline sample: N = 21,357.

bConcussion sample: N = 560.

Study 2: Utility of Two-Factor Approach

Participants and Methods

Data from previously published studies were reanalyzed to evaluate the 1-month (Schatz & Ferris, 2013), 1-year (Elbin et al., 2011), and 2-year (Schatz, 2010) test–retest reliability, as well as the sensitivity and specificity (Schatz & Sandel, 2012) of Speed and Memory scores in comparison to ImPACT composite scores:

  1. One-month test–retest reliability. Participants were 25 undergraduate student volunteers who were not varsity athletes, had no previous exposure to the ImPACT test, previous history of concussion, invalid performance on the test, history of ADD, LD, or previous treatment for headaches, migraines, seizures, or other neurologic or psychiatric illness. All participants completed the computerized ImPACT baseline test (online version) and returned 4 weeks later for the second assessment. Testing took place in a quiet laboratory setting and was conducted either individually, or in pairs, supervised by the same individual. Participants completed the “Baseline” version of ImPACT on the first testing session, and then the “Post-Injury 1” version on the second. These test versions are essentially identical, but incorporate different stimuli (for memory tasks) or placement of stimuli (for visual-motor tasks).

  2. One-year test–retest reliability. Participants were 369 varsity high school athletes who had no invalid performance on the test (at either assessment) or previous treatment for headaches, migraines, seizures, or other neurologic or psychiatric illness. Athletes ages 13–18 (mean 14.8, SD = 0.9) completed two mandatory preseason baseline cognitive assessments as required by their athletics program. Assessments were administered in high school computer laboratories in groups of up to 20–25 athletes and were supervised by either an athletic trainer or a member of the school's medical staff. All athletes completed preseason baseline assessments approximately 1.2 years apart (SD= 0.4). Of note, athletes with a history of ADD, LD, or previous concussion were not excluded from this sample.

  3. Two-year test–retest reliability. Participants were 95 varsity collegiate athletes who had no invalid performance on the test (at either assessment) or previous treatment for headaches, migraines, seizures, or other neurologic or psychiatric illness. Athletes ages 18–22 (mean 20.8, SD = 0.9) completed two mandatory preseason baseline cognitive assessments as required by their athletics program. Assessments were administered in high school computer laboratories in groups of up to 20–25 athletes and were supervised by either an athletic trainer or a member of the school's medical staff. All athletes completed pre-season baseline assessments approximately 1.9 years apart (SD = 0.6). Of note, athletes with a history of ADD, LD, or previous concussion were not excluded from this sample.

  4. Sensitivity and Specificity. Participants were 81 high school and collegiate athletes who had either (a) completed preseason baseline testing, or (b) sustained a concussion and were tested within 72 h of sustaining a concussion, with no invalid performance on baseline testing, history of ADD, LD, or previous treatment for headaches, migraines, seizures, or other neurologic or psychiatric illness. All concussed athletes had sustained a concussion which was witnessed by a Certified Athletic Trainer or Team Physician, who subsequently documented qualitative data describing the nature of the concussive injury; these data were verified by athletes’ self-report of concussion-related symptoms at the time of testing. Athletes completing baseline and post-concussion testing were matched on: Age (range = 13–22, mean 16.0, SD = 1.6), sex, sport, and previous history of concussion.

Analyses

Test–retest reliability

Means and standard deviations from initial baseline ImPACT data were used to calculate z-scores for both Baseline assessments. For each factor (i.e., Memory and Speed), z-scores were calculated by subtracting the athletes' score from the mean of the baseline sample, and dividing by the standard deviation (SD) of that sample. The z-score for Memory was obtained by taking the average of the combined z-scores for the Verbal Memory and the Visual Memory composite scores, and the z-score for Speed was obtained by taking the average of the combined z-scores for the Visual Motor Speed and the Reaction Time composite scores.

Intra-class correlation coefficients (ICCs) were calculated as an indicator of test–retest reliability. ICC can distinguish those sets of scores that are merely ranked in the same order from test to retest from those that are not only ranked in the same order but are in low, moderate, or complete agreement (Chicchetti, 1994); ICC model “Two-Way Mixed” type “Consistency,” were used, using “Average Measures” (Weir, 2005).

Sensitivity and specificity

Means and standard deviations from the baseline ImPACT sample data were used to calculate z-scores for both Baseline and Post-concussion assessments. For each factor (i.e., Memory and Speed), z-scores were calculated by subtracting the score from the mean of the baseline sample and dividing by the standard deviation of that sample. As above, the z-score for the Memory factor was obtained by taking the average of the combined z-scores for the Verbal Memory and the Visual Memory composite scores, and the z-score for the Speed factor was obtained by taking the average of the combined z-scores for the Verbal Motor Speed and the Reaction Time composite scores. Stepwise discriminant analysis was performed to determine the ability of Memory and Speed scores to discriminate between non-concussed and concussed athletes.

Results

Test–retest reliability

ICCs for Memory factor scores were higher than individual Verbal Memory or Visual Memory composite scores, for each of the three samples analyzed (30 days, 1 year, 2 years). Similarly, ICCs for Speed factor scores were equal to or higher than Visual Motor Speed and Reaction Time composite scores for each of the three samples analyzed (Table 3).

Table 3.

Two-factor versus composite score test–retest reliability data

Variable Interval between assessments
 
30 daya 1 yearb 2 yearsc 
Memory (ICC) 0.81 0.76 0.74 
Verbal Memory 0.79 0.62 0.46 
Visual Memory 0.60 0.70 0.65 
Speed (ICC) 0.88 0.85 0.76 
Visual Motor Speed 0.88 0.82 0.74 
Reaction Time 0.77 0.71 0.68 
Variable Interval between assessments
 
30 daya 1 yearb 2 yearsc 
Memory (ICC) 0.81 0.76 0.74 
Verbal Memory 0.79 0.62 0.46 
Visual Memory 0.60 0.70 0.65 
Speed (ICC) 0.88 0.85 0.76 
Visual Motor Speed 0.88 0.82 0.74 
Reaction Time 0.77 0.71 0.68 

Sensitivity and specificity

The sensitivity of Memory and Speed scores, in discriminating between concussed and non-concussed athletes, was 89%, in comparison with 91% using ImPACT subscales and 80% using ImPACT composite scores (as published in Schatz & Sandel, 2012). The specificity of Memory and Speed factor scores in discriminating between concussed and non-concussed athletes was 70%, in comparison with 69% using ImPACT subscales (as published: Schatz & Sandel, 2012) and 62% using ImPACT composite scores (Table 4). The positive predictive value (the proportion of positive test results that are true positives) of Memory and Speed factor scores was 70.2%, in comparison with 59.8% using ImPACT subscales (as published: Schatz & Sandel, 2012) and 59.8% using ImPACT composite scores. The negative predictive value (the proportion of subjects with a negative test result who are correctly diagnosed) of Memory and Speed factor scores was 89.3%, in comparison with 91.4% using ImPACT subscales (as published: Schatz & Sandel, 2012) and 79.8% using ImPACT composite scores.

Table 4.

Two-factor versus composite score and subscale score sensitivity and specificity

Approach Sensitivity (%) Specificity (%) 
Two-factor 89 70 
Composites 80 62 
Sub-scalesa 91 69 
Approach Sensitivity (%) Specificity (%) 
Two-factor 89 70 
Composites 80 62 
Sub-scalesa 91 69 

aOnly subscale scores were used in the published analyses.

General Discussion

This study of a two-factor structure of ImPACT composites scores yielded “Memory” and “Speed” factor scores with improved reliability over the use of individual composite scores. In addition, “Memory” and “Speed” factor scores yielded similar sensitivity and specificity, when compared with ImPACT subscale scores and improved sensitivity and specificity when compared with ImPACT composite scores. Factor analyses on large samples of baseline and post-concussion data yielded similar two-factor results, with discrete factors for “speed” and “memory”; symptom scores represented a third factor in both samples and accounted for the least amount of variance. With respect to psychometric data, Speed and Memory factors showed higher test–retest reliability as well as higher sensitivity and specificity than use of ImPACT composite scores alone.

Factor-analytic studies have previously identified 2 (Iverson et al., 2005), 2–4 (Lovell, 2004), and 5 (Allen & Gfeller, 2011) factors in ImPACT data, with “Speed” factors comprised of the combination of Reaction Time and Processing Speed composite scores in two of these studies (Iverson et al., 2005; Lovell, 2005), and separate factors when ImPACT subscales were used (Allen & Gfeller, 2011); all three previous factor analytic studies identified a combined factor for Memory, whether Verbal and Visual Memory composite scores were used, or the subscales comprising these composites. Symptom scores represented the lone contributor to a third factor, which is consistent with the only factor analytic study to include symptoms (Iverson et al., 2005). That symptoms scores contributed the least amount of variance may seem counter-intuitive, especially at the time of post-concussion assessments, given that diagnosis of acute concussion typically involves the assessment of clinical symptoms, and resolution of symptoms is a key factor in return-to-play decision-making (McCrory et al., 2013). However, the current results suggest that “shared” variance is lower for concussion-based symptoms than for Speed and Memory, at both baseline and post-concussion. These data may represent a decreased level of symptoms at baseline, and an increased level post-concussion, both of which are relatively consistent within baseline and post-concussion samples. In other words, concussed athletes may all share similarly higher levels of post-concussion symptoms, thus representing decreased variance relative to a wider range of post-concussion scores in the domains of Memory and Speed.

Inclusion of Speed and Memory scores may improve the understanding and communicability of baseline and post-concussion test results, for clinicians, athletes, sports medicine professionals, parents, and coaches. Currently, the semantic boundaries between “verbal” and “visual” memory, as well as “processing speed” and “reaction time” are quite vague. The lack of discrimination between composite scores in ImPACT has been documented (Maerlender et al., 2013), and the reliability of the ImPACT Verbal Memory composite has consistently been documented below 0.80 (Elbin et al., 2011; Iverson et al., 2003; Schatz, 2010; Schatz & Ferris, 2013), and reliability coefficients for ImPACT composite scores (other than Visual Motor Speed) also show tremendous variability in the published literature (Table 5), raising questions regarding the test's clinical utility (Randolph, 2011; Randolph et al., 2005). This new factor structure represents fewer, more tangible constructs, with better reliability, and similar-to-improved validity, which may translate to increased clinical utility.

Table 5.

Comparison of Pearson's and ICCs from other published studies

Variable Interval between assessments
 
7 daysa 30 daysb 45 daysc 45 daysd 45 dayse 1 yearf 2 yearsg 
Verbal Memory (ICC) (r— .79 .23 .56/.45 .77 .62 .46 
 .70 .66 — — .63 .45 .30 
Visual Memory (ICC) (r— .60 .32 .26/.52 .72 .70 .65 
 .67 .43 — — .58 .55 .49 
Visual Motor Sp (ICC) (r— .88 .38 .78/.76 .86 .82 .74 
 .86 .78 — — .74 .74 .60 
Reaction Time (ICC) (r— .77 .39 .84/.57 .71 .71 .68 
 .79 .63 — — .52 .62 .52 
Variable Interval between assessments
 
7 daysa 30 daysb 45 daysc 45 daysd 45 dayse 1 yearf 2 yearsg 
Verbal Memory (ICC) (r— .79 .23 .56/.45 .77 .62 .46 
 .70 .66 — — .63 .45 .30 
Visual Memory (ICC) (r— .60 .32 .26/.52 .72 .70 .65 
 .67 .43 — — .58 .55 .49 
Visual Motor Sp (ICC) (r— .88 .38 .78/.76 .86 .82 .74 
 .86 .78 — — .74 .74 .60 
Reaction Time (ICC) (r— .77 .39 .84/.57 .71 .71 .68 
 .79 .63 — — .52 .62 .52 

aIverson colleagues (2006); N = 56.

dResch and colleagues (2013); N = 46 (Group 1), N = 45 (Group 2).

fElbin, Schatz, and Covassin; N = 369.

gSchatz (2010); N = 95.

Conflict of Interest

P.S. has received funding to study the effects of concussion in high school and collegiate athletes from the International Brain Research Foundation, the Sports Concussion Center of New Jersey. He has also served as a consultant to ImPACT Applications Inc. A.M. has no conflict to declare.

References

Allen
B. J.
Gfeller
J. D.
The Immediate Post-Concussion Assessment and Cognitive Testing battery and traditional neuropsychological measures: A construct and concurrent validity study [Validation Studies]
Brain Injury
 , 
2011
, vol. 
25
 
2
(pg. 
179
-
191
)
APA
Guidelines for computer-based tests and interpretations
 , 
1986
Washington, DC
Author
Broglio
S. P.
Ferrara
M. S.
Macciocchi
S. N.
Baumgartner
T. A.
Elliott
R.
Test-retest reliability of computerized concussion assessment programs
Journal of Athletic Training
 , 
2007
, vol. 
42
 
4
(pg. 
509
-
514
)
Chicchetti
D. V.
Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology
Psychological Assessment
 , 
1994
, vol. 
6
 
4
(pg. 
294
-
290
)
Elbin
R. J.
Schatz
P.
Covassin
T.
One-year test-retest reliability of the online version of ImPACT in high school athletes
American Journal of Sports Medicine
 , 
2011
Iverson
G. L.
Gaetz
M.
Lovell
M.
Collins
M.
Validity of ImPACT for measuring processing speed following sports-related concussion
Journal of Clinical and Experimental Neuropsychology
 , 
2005
, vol. 
27
 (pg. 
683
-
689
)
Iverson
G. L.
Lovell
M. R.
Collins
M. W.
Interpreting change on ImPACT following sport concussion [Comparative Study]
The Clinical Neuropsychologist
 , 
2003
, vol. 
17
 
4
(pg. 
460
-
467
)
Lau
B. C.
Collins
M. W.
Lovell
M. R.
Cutoff scores in neurocognitive testing and symptom clusters that predict protracted recovery from concussions in high school athletes
Neurosurgery
 , 
2012
, vol. 
70
 
2
(pg. 
371
-
379
discussion 379.
Lovell
M.
ImPACT Version 2.0 Clinical User's Manual
2004
 
Retrieved February 8, 2005, from http://www.impacttest.com/clients.htm
Lovell
M. R.
ImPACT Version 4.0 Clinical User's Manual
2005
 
Lovell
M. R.
Letters to the Editor
Journal of Athletic Training
 , 
2006
, vol. 
41
 
2
(pg. 
137
-
140
)
Lovell
M. R.
ImPACT Version 6.0 Clinical User's Manual
2007
 
Maerlender
A.
Flashman
L.
Kessler
A.
Kumbhani
S.
Greenwald
R.
Tosteson
T.
, et al.  . 
Examination of the construct validity of ImPACT computerized test, traditional, and experimental neuropsychological measures [Research Support, N.I.H., Extramural]
The Clinical Neuropsychologist
 , 
2010
, vol. 
24
 
8
(pg. 
1309
-
1325
)
Maerlender
A.
Flashman
L.
Kessler
A.
Kumbhani
S.
Greenwald
R.
Tosteson
T.
, et al.  . 
Discriminant construct validity of ImPACT: A companion study [Research Support, N.I.H., Extramural]
The Clinical Neuropsychologist
 , 
2013
, vol. 
27
 
2
(pg. 
290
-
299
)
Mayers
L. B.
Redick
T. S.
Clinical utility of ImPACT assessment for postconcussion return-to-play counseling: Psychometric issues
Journal of Clinical and Experimental Neuropsychology
 , 
2012
, vol. 
34
 
3
(pg. 
235
-
242
)
McCrory
P.
Meeuwisse
W. H.
Aubry
M.
Cantu
B.
Dvorak
J.
Echemendia
R. J.
, et al.  . 
Consensus statement on concussion in sport: The 4th International Conference on Concussion in Sport held in Zurich, November 2012
British Journal of Sports Medicine
 , 
2013
, vol. 
47
 
5
(pg. 
250
-
258
)
Nakayama
Y.
Examination of test-retest reliability of a computerized neurocognitive test battery
 , 
2013
 
Unpublished doctoral dissertation, Michigan State University
Randolph
C.
Letters to the Editor
Journal of Athletic Training
 , 
2006
, vol. 
41
 
2
(pg. 
138
-
140
)
Randolph
C.
Baseline neuropsychological testing in managing sport-related concussion: Does it modify risk?
Current Sports Medicine Reports
 , 
2011
, vol. 
10
 
1
(pg. 
21
-
26
doi:10.1249/JSR.0b013e318207831d00149619-201101000-00009 [pii]
Randolph
C.
Lovell
M.
Laker
S. R.
Neuropsychological testing point/counterpoint
PM & R: The Journal of Injury, Function, and Rehabilitation
 , 
2011
, vol. 
3
 
10 Suppl 2
(pg. 
S433
-
S439
)
Randolph
C.
McCrea
M.
Barr
W.
Letters to the Editor
Journal of Athletic Training
 , 
2006
, vol. 
41
 
2
(pg. 
137
-
140
)
Randolph
C.
McCrea
M.
Barr
W. B.
Is neuropsychological testing useful in the management of sport-related concussion?
Journal of Athletic Training
 , 
2005
, vol. 
40
 
3
(pg. 
139
-
152
)
Resch
J.
Driscoll
A.
McCaffrey
N.
Brown
C.
Ferrara
M. S.
Macciocchi
S.
, et al.  . 
ImPACT test-retest reliability: Reliably unreliable?
Journal of Athletic Training
 , 
2013
, vol. 
48
 
4
(pg. 
506
-
511
)
Schatz
P.
Long-term test-retest reliability of baseline cognitive assessments using ImPACT
American Journal of Sports Medicine
 , 
2010
, vol. 
38
 
1
(pg. 
47
-
53
doi:0363546509343805 [pii]10.1177/0363546509343805
Schatz
P.
Browndyke
J. N.
Applications of computer-based neuropsychological assessment
Journal of Head Trauma Rehabilitation
 , 
2002
, vol. 
17
 
5
(pg. 
395
-
410
)
Schatz
P.
Ferris
C. S.
One-Month Test-Retest Reliability of the ImPACT Test Battery
Archives of Clinical Neuropsychology
 , 
2013
, vol. 
28
 
5
(pg. 
499
-
504
)
Schatz
P.
Kontos
A.
Elbin
R.
Response to Mayers and Redick: Clinical utility of ImPACT assessment for postconcussion return-to-play counseling: Psychometric issues
Journal of Clinical and Experimental Neuropsychology
 , 
2012
, vol. 
34
 
4
(pg. 
428
-
434
discussion 435–442.
Schatz
P.
Pardini
J. E.
Lovell
M. R.
Collins
M. W.
Podell
K.
Sensitivity and specificity of the ImPACT Test Battery for concussion in athletes
Archives of Clinical Neuropsychology
 , 
2006
, vol. 
21
 
1
(pg. 
91
-
99
)
Schatz
P.
Sandel
N.
Sensitivity and specificity of the online version of ImPACT in high school and collegiate athletes
American Journal of Sports Medicine
 , 
2012
Schatz
P.
Zillmer
E. A.
Computer-based assessment of sports-related concussion
Applied Neuropsychology
 , 
2003
, vol. 
10
 
1
(pg. 
42
-
47
)
Van Kampen
D. A.
Lovell
M. R.
Pardini
J. E.
Collins
M. W.
Fu
F. H.
The ‘value added’ of neurocognitive testing after sports-related concussion
American Journal of Sports Medicine
 , 
2006
, vol. 
34
 
10
(pg. 
1630
-
1635
)
Weir
J. P.
Quantifying test-retest reliability using the intraclass correlation coefficient and the SEM [Review]
Journal of Strength and Conditioning Research
 , 
2005
, vol. 
19
 
1
(pg. 
231
-
240
)