Abstract

The Auditory Consonant Trigrams (ACT) test was developed to evaluate immediate memory in the absence of rehearsal. There are few psychometric studies of the measure and a lack of normative data using samples from the United States or Veterans. ACT data were examined for 184 participants who passed the Word Memory Test, denied a history of moderate to severe traumatic brain injury (TBI), and consented for research purposes only. Reliability and construct validity were examined and normative data developed using a healthy subsample. Cronbach’s α for the ACT total score was 0.79. Regression analyses suggested that years of education, estimated premorbid IQ, psychomotor speed, working memory, and impulsivity had the strongest relationships with performance on the ACT. Performance was unrelated to posttraumatic stress disorder and remote mild TBI, but the presence of major depressive disorder was associated with lower total scores. These results demonstrate the ACT has adequate psychometric properties.

Auditory Consonant Trigrams: A Psychometric Update

The Auditory Consonant Trigrams (ACT) test is an adaptation of the Brown–Peterson technique, which was originally developed as a method for studying memory decay over a brief period of time, referred to as immediate memory (Brown, 1958). By using a distractor task following presentation of a stimulus, Brown (1958) suggested that rehearsal is prevented and the retention of information could be measured. A year later, Peterson and Peterson (1959) demonstrated that a brief period of rehearsal prior to the distractor task significantly improved performance on the task, suggesting that immediate memory performance could not be fully explained by decay of a stimulus trace over time. Since its inception, theories about the exact cognitive processes measured by the ACT have evolved, but no clear consensus has been reached.

Several versions of the Brown–Peterson technique have been studied, with the Stuss version (Stuss, Stethem, & Poirier, 1987; Stuss, Stethem, Hugenholtz, & Richard, 1989) one of the most prominent in clinical use. Stuss and colleagues standardized the task to use three consonants as stimulus items, use of an oral serial threes distractor task, and five trials across 9-, 18-, and 36-s intervals. In the initial analysis, 60 Canadian subjects aged 16–69 were examined, with means and standard deviations provided either across gender and education groupings or across age groupings (Stuss et al., 1987). Stuss and colleagues (1987) found no evidence of age or education effects on ACT performance in that study. However, subjects were tested at two visits 1 week apart using alternate forms, and a significant practice effect was noted. A year later, a second paper was published extending the norms by presenting data on 90 subjects in three age groups, again tested at two time points 1 week apart (Stuss, Stethem, & Pelchat, 1988). The 0-s trial was not administered as part of the protocol in either of those studies, but was added to administration (but not scoring) in a third study by the group examining sensitivity of the test in patients with a history of traumatic brain injury (TBI) (Stuss et al., 1989). This was the only study of the three to evaluate a total score.

Since the Stuss version of ACT was published, few studies have evaluated the measure psychometrically. A study by Boone, Miller, Lesser, Hill, and D'Elia (1990) found no differences in performance due to age in healthy individuals between 50 and 79 years old using a version of the ACT with 3-, 9-, and 18-s intervals. Mertens, Gagnon, Coulombe, and Messier (2006) found test–retest reliabilities of a modified version of the task to be adequate: r = .63 in younger individuals (mean age = 20.83 years) and r = .71 in older individuals (mean age = 70.14). Most notably, a study of the ACT using a Turkish version (Anil et al., 2003) reported a Cronbach's α of 0.85, though it is not reported which items were included in that calculation (i.e., if the 0-s trial was included). This is the only reliability estimate listed in the Strauss, Sherman, and Spreen (2006)Compendium. Additionally, the Turkish sample performed significantly lower than Stuss' Canadian sample across intervals and education levels on all but one group (the greater than high school education performance on the 18-s trials). Although every attempt was made to create a Turkish version equivalent to the English version, the authors noted that the English letters W, X, and Q do not exist in the Turkish language, and application of the study's normative and psychometric data might not translate to the English version. To date, there have been no normative data reported on U.S. subjects, and no studies with Veterans.

Validity studies of the ACT have been equivocal. Two exploratory factor analyses (EFA) evaluated the construct validity of the ACT. Using a version of the ACT composed of 3-, 9-, and 18-s intervals, Boone, Pontón, Gorsuch, González, and Miller (1998) conducted an EFA on 250 clinical patients and controls (mean age: 55.50) using a wide array of neuropsychological tests, including the ACT, to evaluate the relationships among several tests purported to evaluate prefrontal lobe function. Three factors were found: a cognitive flexibility factor, a speeded processing factor, and a basic/divided attention and short-term memory factor. Of note, all three factors loaded onto a higher-order frontal factor, and Wechsler Memory Scales (WMS) scores did not load onto any factor. ACT variables loaded on a factor including Wechsler Adult Intelligence Scale-Revised (WAIS-R) VIQ index score, PIQ index score, Digit Span scaled score, Digit-Symbol-scaled score, as well as percent retention on the Rey–Osterrieth Complex Figure Test. The results suggest that the ACT might be heavily influenced by general education, due to the relationship with WAIS-R IQ scores, and even though the ACT and Rey–Osterrieth Complex Figure Test retention loaded onto the same factor, WMS scores did not load onto that factor, contradictory to the historic notion of a heavy memory component to the ACT performance.

In contrast, an EFA by Mertens and colleagues (2006) used a modified version of the ACT to evaluate a sample of 200 participants aged 18–88. The EFA resulted in two factors: Factor 1 included WAIS-III processing speed subtests and WMS-III verbal memory variables, and Factor 2 included the ACT variable and WAIS-III working memory subtests. The ACT variable used in this study was total number of stimuli for all trials, which involved a 20-s serial threes distractor task, and the participants wrote the answers down as opposed to saying them out loud. Collectively, these studies do not reach consensus regarding construct validity of the ACT, with attention, processing speed, intelligence, memory, and working memory all showing some relationship to the measure. A notable limitation to these studies is the varying forms of the ACT used, which limits the generalizability to the Stuss version if used clinically.

Finally, the ACT has been shown to be sensitive to a number of neurological populations, which might render the test highly useful in the clinical setting. A study of adult Attention Deficit Hyperactivity Disorder (ADHD) found moderate to large effect sizes for differences between subjects with ADHD (n = 48) and controls (n = 48) on all ACT variables (Dige & Wik, 2005). A study on cognitive performance in 158 Turkish patients with multiple sclerosis (MS) found significant differences on performance on ACT between the relapsing-remitting MS and the secondary progressive MS groups (Ozakbas, Ormeci, Akdede, Alptekin, & Idiman, 2004). Additionally, the study found a high correlation between ACT total scores (which included the 0-s trial) and Paced Auditory Serial Attention Test total scores (r = .83). A study on 47 manganese-exposed welders found 47.7%–63.6% scored in the impaired range on the 3-, 9-, and 18-s trials of the Boone version of the ACT (Bowler et al., 2006). The ACT was also found to be sensitive to TBI for recent (acute stage) mild TBI (mTBI) as well as a “More Severe-Chronic” TBI group, though the sample sizes were small (Stuss et al., 1989, p. 146). More recently, a study on patients with severe TBI found a moderate effect size (0.45) difference between the 12 TBI patients and 18 controls on the ACT total score of 3-, 9-, and 18-s delay trials (Merkley, Larson, Bigler, Good, & Perlstein, 2013).

Although there is variability regarding administration of the ACT in these studies, the results suggest that the ACT is sensitive to a variety of neurological conditions and cerebral insult; thus, inclusion into neuropsychological batteries and screenings for neurological disorders appears warranted. A survey of neuropsychologists (Rabin, Barr, & Burton, 2005) listed the ACT among the top 40 tests of attention used by neuropsychologists, further stressing the need for solid normative data and an updated psychometric evaluation of the test. The purpose of this study is to provide psychometric data on the ACT including reliability estimates, analysis of construct validity, and means/standard deviations for use in calculating z-scores for normative purposes.

Method

The prospective studies from which these data were drawn were reviewed and approved by the affiliated Institutional Review Board. The welfare and privacy of human subjects was protected and maintained. Voluntary verbal and written informed consent was obtained prior to initiation of any study activities. Statistical analyses were completed using SPSS 21 and GPower 3.1. For all analyses, α was set a priori at the 0.05 level; standardized β are reported for regression results. All statistical assumptions were met for the analyses completed, and an a priori power analysis was completed for the regression model. Raw scores were used for all cognitive measures except Wechsler Test of Adult Reading (WTAR), which was converted to age-based standard scores (M = 100, SD = 15).

Participants

Participants for the current study were pulled from a larger database of Operation Enduring Freedom/Operation Iraqi Freedom/Operation New Dawn (OEF/OIF/OND) postdeployment Veterans participating in a multisite study of postdeployment mental health and cognitive functioning. Inclusion criteria for the larger study included military service since January 10, 2001. Exclusion criteria included combat exposure prior to 1985, completion of a neuropsychological evaluation in the prior 6 months, acute psychosis, current substance abuse or dependence (per the Diagnostic and Statistical Manual of Mental Disorders-IV [DSM-IV]; American Psychiatric Association, 2000), diagnosed posttraumatic stress disorder (PTSD) related to predeployment nonmilitary trauma, and history of moderate to severe TBI prior to or since deployment. Test results were kept separate from the participants' clinical records except in cases where dual consent was provided. For the current study, data from a single site were analyzed (n = 250). Those who were missing ACT data (n = 7), were dually consented for clinical purposes (n = 10), reported a history of moderate to severe TBI during deployment (n = 11), or failed the performance validity measure (n = 38) were additionally excluded to eliminate the possibility of secondary gain issues, frank neurological impairment, or deficient performance validity. The remaining participants (n = 184) were included in the current analyses. For those endorsing any history of mTBI at any point in the lifespan, the date of most recent injury was compared with the date tested. All injuries were remote (n = 57; years since most recent mTBI: range = 1–42, M = 8.68, SD = 10.25, median = 5, mode = 2). Tests were administered by a master's or doctorate level clinician with training in neuropsychology and supervised by a board certified neuropsychologist. Tests were administered in a fixed order and in a standardized manner in accordance with the tests' manuals. Data were collected between June 2006 and March 2015.

Measures

ACT is a version of the Brown–Peterson Technique. The version used in this study (Stuss et al., 1989) involved presenting three consonants sometimes followed by a number (except in the 0-s trials). The respondent then counted backwards by threes from the provided number, and after an interval of 0, 9, 18, or 36 s, verbally reported the three consonants presented at the initiation of the trial. There are a total of 20 trials, with five items for each interval. A summary score for each interval is derived, and a total score created from all but the 0-s trials. Higher scores on all variables reflect better performance.

Cognitive measures for construct validity were selected from the available battery based on cognitive domains previously reported to be related to ACT performance (processing speed, attention, verbal learning, and executive functioning). Cognitive domains measured by tests were defined using Lezak, Howieson, Bigler, and Tranel (2012) and test manuals for classification (though it is recognized that other ability descriptors may be used for tests). Four variables were selected from each domain based on common use, variety of sub-ability measured, and psychometric properties. Table 1 lists all cognitive tests, categorizations of the tests, reliability estimates from established research, and correlations to the ACT total score.

Table 1.

List of selected tests, reliability estimates, and correlations to Auditory Consonant Trigrams (ACT) total score

Domaina Measure r ACT correlation 
Processing Speed WAIS-III DSC .84b .25** 
WAIS-III SS .77b .17* 
TMT A .79c −.27** 
CPT-II HRT .95d −.01 
Attention CPT-II OM .94d .01 
TMT B .89c −.22** 
PASAT .90e .32** 
WAIS-III LNS .82b .36** 
Verbal Learning CVLT-II 1–5 .82f .26** 
CVLT-II 1 .57f .19* 
CVLT-II B .61f .14 
CVLT-II INT .63f −.07 
Executive Stroop .73g .22** 
CPT-II COM .83d −.25** 
COWAT .83h .18* 
BDS-II .62i .32** 
Domaina Measure r ACT correlation 
Processing Speed WAIS-III DSC .84b .25** 
WAIS-III SS .77b .17* 
TMT A .79c −.27** 
CPT-II HRT .95d −.01 
Attention CPT-II OM .94d .01 
TMT B .89c −.22** 
PASAT .90e .32** 
WAIS-III LNS .82b .36** 
Verbal Learning CVLT-II 1–5 .82f .26** 
CVLT-II 1 .57f .19* 
CVLT-II B .61f .14 
CVLT-II INT .63f −.07 
Executive Stroop .73g .22** 
CPT-II COM .83d −.25** 
COWAT .83h .18* 
BDS-II .62i .32** 

Notes:r = reliability estimate per citation; ACT = Auditory Consonant Trigrams total score; WAIS = Wechsler Adult Intelligence Scales; DSC = Digit-Symbol Coding; SS = Symbol Search; TMT = Trail Making Test; CPT-II = Connors' Continuous Performance Test II; HRT = Hit Reaction Time; OM = Omission Errors; PASAT = Paced Auditory Serial Addition Test; LNS = Letter–Number Sequencing; CVLT = California Verbal Learning Test; 1–5 = Trials 1–5; 1 = Trial 1; B = Trial B; INT = Total Intrusion Errors; COM = Commission Errors; COWAT = Controlled Oral Word Association Test; BDS-II = Behavioral Dyscontrol Scale-II.

*p < .05. **p < .01.

aBased on domains as described in Lezak et al. (2012) and test manuals.

Descriptive

The Structured Clinical Interview for DSM-IV Axis I Disorders (SCID; First, Spitzer, Gibbon, & Williams, 1997) was used for the evaluation of psychiatric diagnoses. Variable of interest included meeting full DSM-IV criteria for current PTSD and major depressive disorder (MDD). Additionally, most included participants (n = 131, 71.2%) completed a semi-structured interview for lifetime TBI history based on the American Congress of Rehabilitation Medicine criteria (Menon, Schwab, Wright, & Maas, 2010). The interview was used to create a dichotomous (yes/no) mTBI during deployment variable. For those without the interview data, a self-report measure of TBI exposure was completed (based on Ivins et al., 2003) and used to create the mTBI variable. The Green Word Memory Test (WMT; Green, 2005) is a performance validity test based on forced-choice recognition of 16 word pairs. There are three validity subtests to the WMT: Immediate Recall, Delayed Recall, and Consistency. The Multiple Choice subtest was also administered, but not considered here. The Paired Associates, Free Recall, and Long Delayed Free Recall subtests were not administered as part of the standardized battery. Participants scoring 82.5% or less on one or more of the three validity subtests were considered to have failed that measure and were excluded from analyses. The WTAR (average internal consistency r = .93) is a test reflecting educational exposure based on the respondent's ability to read phonemically irregular words (The Psychological Corporation, 2001). The WTAR correlated with WAIS-III Full Scale IQ scores (r = .73; The Psychological Corporation, 2001). Age-corrected standard scores were used as an estimate of general intellectual ability.

Processing speed

The WAIS-III (Wechsler, 1997) is a clinician-administered battery of intellectual ability. Three subtests were used for this study. Digit-Symbol-Coding is a timed performance test of processing speed, visual scanning, and motor control. Symbol Search is a test of speed of visual search and processing. Higher scores on both reflect better performance. The Trail Making Test (TMT; Reitan & Wolfson, 1985) consists of two subtests. Part A involves connecting a series of 25 numbered circles in order as fast as possible, and evaluates psychomotor speed. Lower (faster) scores relate to better performance. The Conners' Continuous Performance Test-II Version 5 (CPT-II; Conners & MHS Staff, 2004) is a computer-administered test of simple attention, sustained attention, reaction time, and impulsivity. The respondent must quickly hit the space bar when a letter flashes on the screen but inhibit hitting the space bar if the letter is an X. Hit Reaction Time is a measure of simple reaction time based on the average latency of stimulus presentation to pressing the space bar, with lower scores related to better performance.

Attention

CPT-II Omissions errors occur when the space bar is not pressed when it should have been following presentation of a target stimulus, which is considered a measure of inattention (Conners & MHS Staff, 2004). TMT Part B involves connecting 25 circles with both numbers and letters in order and alternating, thus measuring set shifting or divided attention (Reitan & Wolfson, 1985). Scores represent time to complete each task, with lower (faster) scores indicating better performance. The Paced Auditory Serial Addition Test (Gronwall, 1977) is a measure of auditory working memory in which the respondent must track a list of single-digit numbers and add each number to the number immediately preceding. There are four trials, with each subsequent trial read at a faster rate. A total score was calculated by summing total correct across the first two trials (read at 2.4- and 2.0-s intervals). Per test instructions, the test is to be discontinued if the first two trials are not tolerated; thus, only those trials were used for this study. Higher scores represent better performance. WAIS-III Letter–Number Sequencing (LNS; Wechsler, 1997) is a verbal, alphanumeric measure of working memory and sequencing. Higher scores on all three measures indicate better performance.

Verbal learning

The California Verbal Learning Test-II (CVLT-II, Standard Form; Delis, Kramer, Kaplan, & Ober, 2000) is a list learning and memory test composed of five learning trials of a list of 16 words from four semantic categories, an interference list trial, four recall trials, and two recognition trials. Trial 1 measures single-trial learning and auditory attention span, and 1–5 Total is the sum of correct responses across all five learning trials, used as a measure of global learning. Trial B is a new list administered following the last trial of the original list, reflecting attention span with a proactive interference component. Total Intrusions is number of non-list words given as responses over all learning and memory trials. Higher scores on the CVLT-II variables represent better performance, except for the total intrusions measure.

Executive functioning

The Color/Word Inhibition trial from the Stroop Color and Word Test, Adult Version (Stroop; Golden & Freshwater, 2002) measures processing speed, inhibitory abilities, and resistance to interference. The subject must refrain from reading a colored word and instead identify the contrasting ink color of the text. Total number correct in 45 s is the outcome, with higher scores reflecting better performance. CPT-II Commissions (COM) are number of responses to non-target Xs, which is considered a measure of impulsivity. Higher scores reflect poorer performance. The Controlled Oral Word Association Test (COWAT; Ruff, Light, Parker, & Levin, 1996) is a measure of phonemic verbal fluency. Participants name as many words as possible in 60 s for the three letters (C, F, and L). The total number correct across the three trials was used, with higher scores reflecting better performance. The BDS-II is a nine-item executive functioning measure that primarily evaluates dynamic motor control, but also includes an auditory version of the TMT B task and an insight item (Grigsby & Kaye, 1996). The total score ranges from 0 to 27, with higher scores reflecting better performance.

Results

Sample Characteristics and Reliability

Table 2 presents demographic data for the total included sample. Means and standard deviations for ACT scores and reliabilities (Cronbach's α) are listed in Table 3. For general normative purposes, a subsample of healthy control participants was identified who passed the WMT, denied a history of TBI of any severity across the lifetime, and did not meet criteria for any DSM-IV diagnosis at the time of participation. Descriptive statistics of this subsample are presented in a separate column in Table 2. ACT data for this healthy group are presented in Table 3 for use in calculating z scores. Using the total included sample, only the ACT total score achieved an acceptable reliability (α = 0.79), and will thus be the focus of the rest of these results. Item level data are listed in Table 4. There is considerable variability for all items other than 0 s trials, though some items are observably easier than others. Correlations were run between total scores for all delay groups and the overall total, as shown in Table 5. The significant correlations among the 9, 18, and 36 s scores further support collapsing across delays into a total score. The 0 s total was not correlated with any other subscale, suggesting that those trials measure a different construct.

Table 2.

Descriptive statistics for total included group and healthy control group

Variable IncludedaM (SD; min–max) or n (%) ControlbM (SD; min–max) or n (%) 
Age 35.54 (9.42; 21–60) 36.41 (10.33; 21–60) 
Sex 
 Men 157 (85.3%) 49 (84.5%) 
 Women 27 (14.7%) 9 (15.5%) 
Ethnicity 
 White 136 (73.9%) 43 (74.1%) 
 Black 49 (26.6%) 15 (25.9%) 
 Hispanic 10 (5.4%) 2 (3.4%) 
 Other 7 (3.8%) 2 (3.4%) 
Education Level 
 GED 2 (1.1%) 0 (0.0%) 
 High School 75 (40.8%) 27 (46.6%) 
 Trade/Tech 16 (8.7%) 5 (8.6%) 
 Associate's 32 (17.4%) 8 (13.8%) 
 Bachelor's 34 (18.5%) 11 (19.0%) 
 Master's 13 (7.1%) 5 (8.6%) 
 Doctorate 1 (0.5%) 1 (1.7%) 
 Other 11 (6.0%) 1 (1.7%) 
Years of education 13.74 (1.97; 11–20) 13.81 (2.18; 12–20) 
WTAR 103.20 (12.26; 72–127) 107.90 (9.91; 84–123) 
Deployment mTBI 59 (32.1%) 
Any Dx present 81 (44.0%) 
 Current MDD 26 (14.1%) 
 Current PTSD 58 (31.5%) 
Service Connectedc 105 (57.1%) 28 (48.3%) 
SC Percentaged 43.40 (26.88; 0–100) 27.50 (18.58; 0–70) 
Plan to file SCe 20 (10.9%) 4 (17.4%) 
Variable IncludedaM (SD; min–max) or n (%) ControlbM (SD; min–max) or n (%) 
Age 35.54 (9.42; 21–60) 36.41 (10.33; 21–60) 
Sex 
 Men 157 (85.3%) 49 (84.5%) 
 Women 27 (14.7%) 9 (15.5%) 
Ethnicity 
 White 136 (73.9%) 43 (74.1%) 
 Black 49 (26.6%) 15 (25.9%) 
 Hispanic 10 (5.4%) 2 (3.4%) 
 Other 7 (3.8%) 2 (3.4%) 
Education Level 
 GED 2 (1.1%) 0 (0.0%) 
 High School 75 (40.8%) 27 (46.6%) 
 Trade/Tech 16 (8.7%) 5 (8.6%) 
 Associate's 32 (17.4%) 8 (13.8%) 
 Bachelor's 34 (18.5%) 11 (19.0%) 
 Master's 13 (7.1%) 5 (8.6%) 
 Doctorate 1 (0.5%) 1 (1.7%) 
 Other 11 (6.0%) 1 (1.7%) 
Years of education 13.74 (1.97; 11–20) 13.81 (2.18; 12–20) 
WTAR 103.20 (12.26; 72–127) 107.90 (9.91; 84–123) 
Deployment mTBI 59 (32.1%) 
Any Dx present 81 (44.0%) 
 Current MDD 26 (14.1%) 
 Current PTSD 58 (31.5%) 
Service Connectedc 105 (57.1%) 28 (48.3%) 
SC Percentaged 43.40 (26.88; 0–100) 27.50 (18.58; 0–70) 
Plan to file SCe 20 (10.9%) 4 (17.4%) 

Notes: Other Education Level encompassed some college for all who selected that option; WTAR = Wechsler Test of Adult Reading standard score; Dx = diagnosis per DSM-IV; mTBI = mild traumatic brain injury; MDD = major depressive disorder; PTSD = posttraumatic stress disorder; SC = service connected.

an = 184.

bn = 58.

cNumber who identified as service connected for any condition.

dPercentage is only for those who identified as service connected.

en = 59 is only asked for those who stated that they are not currently service connected.

Table 3.

ACT scores across total and control groups

ACT variable Totala Controlb α 
0-s 14.98 (0.13; 14–15) 14.98 (0.13; 14–15) — 
9-s 11.51 (2.66; 4–15) 11.90 (2.63; 6–15) .53 
18-s 10.13 (3.08; 1–15) 10.45 (3.18; 3–15) .58 
36-s 10.29 (3.32; 0–15) 10.52 (3.06; 3–15) .64 
9–36 total 31.93 (7.57; 9–45) 32.86 (7.10; 18–45) .79 
ACT variable Totala Controlb α 
0-s 14.98 (0.13; 14–15) 14.98 (0.13; 14–15) — 
9-s 11.51 (2.66; 4–15) 11.90 (2.63; 6–15) .53 
18-s 10.13 (3.08; 1–15) 10.45 (3.18; 3–15) .58 
36-s 10.29 (3.32; 0–15) 10.52 (3.06; 3–15) .64 
9–36 total 31.93 (7.57; 9–45) 32.86 (7.10; 18–45) .79 

Note: ACT = Auditory Consonant Trigrams. Reliabilities (α) calculated from the total group. The variable α could not be calculated for the 0-s scores due to violations in assumptions for reliability calculations.

an = 184.

bn = 58.

Table 4.

ACT item level descriptive data

Item Delay M (SDMin Max 
3.00 (0.00) 
2.99 (0.10) 
3.00 (0.00) 
2.99 (0.07) 
3.00 (0.00) 
18 1.91 (0.96) 
2.03 (0.99) 
36 1.85 (1.06) 
2.27 (0.95) 
10 18 1.99 (1.04) 
11 36 1.99 (1.05) 
12 18 2.37 (0.90) 
13 2.38 (0.85) 
14 36 2.09 (1.02) 
15 2.27 (0.92) 
16 36 2.20 (1.10) 
17 18 2.07 (0.98) 
18 2.57 (0.79) 
19 18 1.79 (1.16) 
20 36 2.16 (0.96) 
Item Delay M (SDMin Max 
3.00 (0.00) 
2.99 (0.10) 
3.00 (0.00) 
2.99 (0.07) 
3.00 (0.00) 
18 1.91 (0.96) 
2.03 (0.99) 
36 1.85 (1.06) 
2.27 (0.95) 
10 18 1.99 (1.04) 
11 36 1.99 (1.05) 
12 18 2.37 (0.90) 
13 2.38 (0.85) 
14 36 2.09 (1.02) 
15 2.27 (0.92) 
16 36 2.20 (1.10) 
17 18 2.07 (0.98) 
18 2.57 (0.79) 
19 18 1.79 (1.16) 
20 36 2.16 (0.96) 

Note:n = 184.

Table 5.

Intercorrelations among ACT variables

Variable 0 s 9 s 18 s 36 s Total 
0 s 1.00     
9 s 0.09 1.00    
18 s 0.08 0.45* 1.00   
36 s 0.10 0.52* 0.64* 1.00  
Total 0.12 0.76* 0.85* 0.88* 1.00 
Variable 0 s 9 s 18 s 36 s Total 
0 s 1.00     
9 s 0.09 1.00    
18 s 0.08 0.45* 1.00   
36 s 0.10 0.52* 0.64* 1.00  
Total 0.12 0.76* 0.85* 0.88* 1.00 

Notes:n = 184.

*p < .01, two tailed.

Considering the issues common to the post-deployed Veteran cohort, dichotomous groups were created based on presence versus absence of deployment mTBI, and current MDD and PTSD. Independent t-tests were run between the groups for each variable. ACT total scores were significantly different between those with and without current MDD (t = −2.04, p = .043, d = 0.43). ACT scores were not significantly different between those with and without a reported history of mTBI during deployment (t = 0.54, ns, d = 0.09) nor with and without current PTSD diagnosis (t = −1.85, ns, d = 0.30). Additionally, the ACT total score was not significantly correlated with number of mTBIs sustained during deployment (r = −.06, ns).

Effects of Education and Intellectual Ability on ACT Performance

Bivariate correlations and t-tests were used to examine the relationship between demographic characteristics and ACT total scores. Age (r = .03, ns), sex (t = 0.09, ns, d = 0.02), and minority status (t = −1.19, ns, d = 0.20) were not significantly related to ACT total scores. However, both years of education (r = .25, p < .001) and WTAR standard scores (r = .45, p < .001) were significantly correlated with ACT total scores. In order to more fully explore the relationship of education to ACT scores, level of attained education was defined by four groups: (1) Attained high school diploma or less (including GED; n = 77, M = 30.26, SD = 7.98); (2) Earned a degree from a technical college, a trade school, an associate's, or some college (n = 59, M = 31.76, SD = 7.03); (3) Earned a bachelor's degree (n = 34, M = 34.59, SD = 6.57), and (4) Earned a graduate degree (n = 14, M = 35.36, SD = 7.52). An ANOVA was then completed to examine significant differences across the categorical groups, which was significant, F (3, 180) = 3.78, p = .012, η2 = 0.06. Post hoc analyses revealed that the only significant difference was between the high school or less group and the bachelor's degree group (p = .026). As the bachelor's degree seemed to be driving the differences in education on ACT total score performance, a dichotomous variable defined as education less than completed bachelor's degree versus completed bachelor's degree or greater was completed. Those with a bachelor's degree or higher level of education performed significantly better on ACT total (n = 48, M = 34.81, SD = 6.79) than those with less than a bachelor's degree (n = 136, M = 30.91, SD = 7.59), t (182) = 3.14, p = .002, d = 0.53.

Next, due to the significant relationship between WTAR scores and ACT total scores, group data were calculated based on WTAR scores. Participants were divided into three groups: those with WTAR scores at least 1 SD below the mean (<85; n = 16, M = 25.13, SD = 7.97), those with scores in the average range (85–115; n = 133, M = 31.11, SD = 7.06), and those with scores above 1 SD (> 115; n = 35, M = 38.14, SD = 4.76). An ANOVA using ACT total score across these groups was significant, F (2, 181) = 23.77, p < .001, η2 = 0.21. Post hoc analyses revealed that all groups were significantly different from each other at p = .003 or less, suggesting a robust relationship between intellectual ability and ACT total scores.

Construct Validity

Correlations between the ACT total score and a number of cognitive variables related to attention, working memory, processing speed, and memory can be seen in Table 1. Variables significantly correlated with the ACT total score were retained for regression analysis. A hierarchical linear regression analysis was conducted predicting ACT total score. Because all raw scores were used, the first Step of the regression included demographic variables (see Table 6 for all variables used). All cognitive variables that were significantly correlated with the ACT total score at p < .05 were entered into Step 2. Power analysis using a medium effect size, α = 0.05, and 1 − β = 0.80 suggested a minimum sample size of 151 was needed for the hierarchical multiple regression, which was surpassed with the used sample of 184. Step 1 was significant, F (5, 170) = 11.26, p < .001 and explained 24.9% of the variance. Years of education (β = 0.19, p = .012) and WTAR (β = 0.45, p < .001) were significant predictors. The addition of cognitive variables significantly improved the model, F change (12, 158) = 2.21, p = .013, R2 change = .11. The final model was also significant, F (17, 158) = 5.16, p < .001 and explained 35.7% of the variance. In the final model, significant variables included years of education (p = .025), WTAR (p = .002), and Trails A (p = .013). Table 6 presents the final model statistics.

Table 6.

Regression results, final model

Variable β t p VIF 
Age −0.03 −0.40 .727 1.50 
Sex −0.13 −1.81 .072 1.27 
Minority −0.03 −0.39 .699 1.42 
Education 0.17 2.27 .025 1.40 
WTAR 0.27 3.17 .002 1.81 
WAIS-III DSC 0.07 0.69 .489 2.25 
WAIS-III SS −0.10 −1.01 .313 2.39 
WAIS-III LNS 0.14 1.68 .095 1.61 
TMT A −0.20 −2.50 .013 1.62 
TMT B 0.02 0.24 .809 1.97 
CPT-II COM −0.13 −1.79 .075 1.27 
BDS-II 0.02 0.26 .795 1.61 
CVLT-II 1 0.04 0.40 .687 2.10 
CVLT-II 10.08 0.83 .410 2.41 
PASAT 0.02 0.26 .794 1.97 
Stroop −0.02 −0.21 .834 1.54 
COWAT 0.05 0.71 .479 1.44 
Variable β t p VIF 
Age −0.03 −0.40 .727 1.50 
Sex −0.13 −1.81 .072 1.27 
Minority −0.03 −0.39 .699 1.42 
Education 0.17 2.27 .025 1.40 
WTAR 0.27 3.17 .002 1.81 
WAIS-III DSC 0.07 0.69 .489 2.25 
WAIS-III SS −0.10 −1.01 .313 2.39 
WAIS-III LNS 0.14 1.68 .095 1.61 
TMT A −0.20 −2.50 .013 1.62 
TMT B 0.02 0.24 .809 1.97 
CPT-II COM −0.13 −1.79 .075 1.27 
BDS-II 0.02 0.26 .795 1.61 
CVLT-II 1 0.04 0.40 .687 2.10 
CVLT-II 10.08 0.83 .410 2.41 
PASAT 0.02 0.26 .794 1.97 
Stroop −0.02 −0.21 .834 1.54 
COWAT 0.05 0.71 .479 1.44 

Note:n = 184. β are standardized β. Bold font highlights significant β. WTAR = Wechsler Test of Adult Reading standard score; WAIS = Wechsler Adult Intelligence Scales; DSC = Digit-Symbol Coding; LNS = Letter–Number Sequencing; TMT = Trail Making Test; CPT-II = Connors' Continuous Performance Test II; COM = Commission Errors; BDS-II = Behavioral Dyscontrol Scale –II; CVLT = California Verbal Learning Test; 1 = Trial 1; 1–5 = Trials 1–5; PASAT = Paced Auditory Serial Addition Test; COWAT = Controlled Oral Word Association Test.

Post Hoc Exploratory Analyses

The current sample performed near perfect on the 0-s trial, which resulted in poor reliability for the 0-s trial. Also, the sum of the 0-s scores was not significantly correlated with any other summed ACT score, again likely due to a lack of variance. Qualitatively, three participants (2%) committed errors on the trials, and all of the errors involved stating a like-sounding consonant to the stimulus (e.g., substituting T for P), suggesting that these errors might be related to hearing issues or poor sound discrimination. Because of the small number who did not score perfect on the 0-s trial and the tendency for errors to resemble auditory misperceptions, statistical evaluation was not conducted.

A follow-up backward regression was used to further examine significantly related variables and ACT total score. By eliminating nonsignificant variables from subsequent models, unique variance can be further explored while eliminating the noise of shared variance. The final model was significant, F (5, 170) = 16.56, p < .001, and explained 32.8% of the variance. The final model contained five variables; all significantly contributed unique variance: years of education (β = 0.14, p = .035), WTAR (β = 0.30, p < .001), WAIS-III LNS (β = 0.16, p = .025), TMT A (β = −0.18, p = .007), and CPT-II Commissions (β = −0.15, p = .027). The results provided further support of the relationships among age, education, and processing speed to ACT performance. Additionally, after removing nonsignificant variables, attention (auditory working memory), and executive functioning (impulsivity control) also appeared to contribute to ACT performance.

Discussion

Despite the fact that the ACT has been ranked among the top 40 tests of attention used by neuropsychologists (Rabin et al., 2005), few studies have examined its psychometric properties. Psychometrically, the 9-, 18-, and 36-s trials demonstrated generally poor reliabilities; however, correlations among the trials were moderate, supporting the use of the Total sum score. The Total score demonstrated good reliability (α = 0.79) and was grossly consistent with the reliability estimate using a Turkish sample (α = 0.85; Anil et al., 2003). Based on these results, future use of the ACT should focus on the Total score. Although Stuss and colleagues (1987, 1988) provided normative data based on age groups, they did not report statistics demonstrating a significant effect of age on performance. We provide normative data on a healthy group of U.S. Veterans without frank neurological insult or psychiatric diagnoses, who passed a stand-alone performance validity measure, and with an age range of 21–60. Additionally, data are presented for the ACT total score, as well as each delay-based sub-score total and per individual item. Contrary to Anil and colleagues (2003) results but consistent with Stuss and colleagues (1989), we found no relationship between age and ACT performance. Methodological differences may have contributed to this outcome: compared with Anil and colleagues (2003), the current study had a more restricted age range (2160) and higher levels of education (range 1022 years, 99% had at least a high school diploma). ACT performance was not related to minority status nor was it related to sex, consistent with previous research (Anil et al., 2003; Stuss et al., 1987). Similar to Stuss and colleagues (1989), ACT performance was significantly affected by educational level and estimated intellectual functioning.

Previous studies reported differences in ACT performance related to neurological or psychological conditions (e.g., Anil et al., 2003; Oral et al., 2012). The current study found no differences in ACT performance related to remote mTBI sustained during deployment. Others indicated that individuals with severe TBI (Merkley et al., 2013; Stuss et al., 1989) or recent mTBI (with 72 hr to 90 days post-injury; Stuss et al., 1989) were affected on the ACT. Our results suggest that more remote mTBI (>11 months since most recent injury, M = 8.68 years) in the presence of adequate performance validity is not related to significantly poorer performance on the ACT. Additionally, the presence of PTSD was not related to ACT total scores. However, ACT total scores were significantly lower for those with DSM-IV current MDD compared with those without MDD. Oral and colleagues (2012) also reported that individuals with MDD performed significantly worse on the ACT compared with healthy controls, though performance validity was not assessed in that study. The relationship between MDD and ACT performance might be explained by decreased processing speed in those with MDD, considering our finding that the ACT was significantly related to a measure of psychomotor speed. A meta-analysis on cognitive functioning and MDD (Snyder, 2013) found that individuals with MDD performed significantly slower on tests of psychomotor speed than those without (d = 0.33), with depression symptom severity moderating that relationship. One possible explanation might be the motor slowing hypothesis, which suggests motor slowing in depression irrespective of higher cognitive impairment; however, Snyder found that impaired executive functions were not fully explained by motor slowing. Thus, the lower ACT total scores in those with current MDD in our sample might be related to an interaction of motor slowing and impaired executive functioning, but such relationships would need further evaluation in future studies.

The ACT purports to measure memory decay, or rate at which perceived information degrades, and is conceptualized as related to frontal lobe functioning (e.g., Boone et al., 1998). In the present study, ACT performance was not related to tests of simple reaction time, basic attention, single-trial learning, learning after an interference task, or verbal memory intrusion errors. Although preliminary results suggested a relationship between ACT performance and measures across several cognitive domains including attention, verbal fluency, processing speed, set switching, working memory, and verbal learning, most of these relationships were not significant after accounting for educational level and estimated intelligence, highlighting the importance of accounting for these variables in statistical models. Ultimately, only education, intelligence, simple processing speed (Trails A), verbal working memory (LNS), and impulsivity (CPT-II Commission Errors) were related to ACT performance. These results bear some similarities to Boone and colleagues (1998) EFA which suggested factors representing cognitive flexibility, processing speed, and attention and short-term memory which included verbal and performance IQ. We did not find a significant relationship between cognitive flexibility and ACT above and beyond the effects of general intelligence, possibly due to differences in study design. Boone and colleagues (1998) primary factor contained variance attributed by the WCST which was not included in this study. Our results do not support the previous view that the ACT primarily assesses executive functioning or short-term memory decay; rather it seems to be measuring general intellectual ability and psychomotor speed.

Qualitatively, the few errors committed on the 0-s trial appeared to be related to auditory processing or hearing issues, and all involved phonemic substitutions (e.g., P for B). This could suggest that those participants have problems in auditory phonemic processing or general auditory impairment, but this hypothesis needs evaluation in future studies. Also, Veterans are at higher risk of tinnitus than non-Veterans (OR = 2.3, 95% CI = 1.9–2.9; 11.7%), which could negatively affect speech discrimination and help explain the findings (Folmer, McMillan, Austin, & Henry, 2011), but such data were not available with our sample. Furthermore, because stimuli were not prerecorded, it is also possible that these errors are actually related to poor stimulus presentation on the part of the assessor.

Limitations of the study included sample characteristics and analytics. Participants in the total sample included Veterans with current psychiatric diagnoses and a history of mTBI. Data used from this total sample for the primary analyses were also limited to individuals with a relatively high educational level (M = 13.74 years of education) and who were in early adulthood (M = 35.54 years of age). However, the presented normative data for the healthy subsample reflected lack of known neurological insult due to TBI, lack of current psychiatric diagnoses, and all had valid test results based on one stand-alone performance validity test (PVT). Nonetheless, the healthy sample was small (n = 58), and although education and intellectual functioning were shown to affect ACT performance, it was not possible to derive normative data accounting for those variables due to the small size of the healthy group. Additional research is needed to support these findings, especially since the larger body of results appears to be mixed. Further research on ACT performance across a broad age range is necessary to support findings that ACT performance is unrelated to age. Although a more thorough item error analysis was beyond the scope of the project, these preliminary findings suggest that errors on the 0-s trial are rare and possibly related to auditory processing. Future projects might further examine errors, particularly those related to phonemic substitution. Examining the effect of standardized stimulus presentation (i.e., prerecorded stimuli) would also further clarify the possible contributing factors. These findings may also suggest that future studies examining the effect of tinnitus on ACT performance would improve clinical utility of the test.

Future research may seek to improve general understanding of the ACT task more broadly, including qualities of the distractor task such as length of time between presentation of letters and counting (permitting possible rehearsal of letters) or engagement in counting, accuracy of counting, and use of the ACT with more cognitively impaired populations. Furthermore, the version of the ACT used in this study did not account for performance during the distractor task (e.g., number of errors or number correct). Also, the larger battery did not include measures allowing an evaluation of the contributions of math skills (e.g., math fluency) to ACT performance. Finally, a single PVT was used for excluding those with invalid performance (WMT). One study found that the WMT identified performance of more participants as invalid than another common PVT, the Test of Memory Malingering, and the false concordance rate was 27.9% (Greiffenstein, Greve, Bianchini, & Baker, 2008). This could either mean that the WMT is more sensitive to invalid performance (making it ideal for the psychometric purposes here) or more susceptible to false-positive identification of invalid performance (in which case, this study is only affected by lost power, which still remained acceptable). In either case, the test is most likely not 100% sensitive and some subjects performing below their full capacity may have been included in the analyses. Nevertheless, the current study represents a methodological improvement as most previous studies on the ACT did not consider performance validity (e.g., Anil et al., 2003; Stuss et al., 1987, 1988, 1989). Overall, the ACT total score demonstrated good psychometric properties and appears to be a viable assessment tool in the OEF/OIF/OND Veteran population.

Funding

This research was supported by resources of the W.G. “Bill” Hefner Veterans Affairs Medical Center, the Mid-Atlantic Mental Illness Research Education and Clinical Center, and the Department of Veterans Affairs Office of Academic Affiliations Advanced Fellowship Program in Mental Illness Research and Treatment.

Conflict of Interest

There are no conflicts of interest to disclose. The prospective studies from which these data were drawn were reviewed and approved by the W.G. (Bill) Hefner VA Medical Center (Hefner VAMC) Institutional Review Board. The welfare and privacy of human subjects was protected and maintained.

References

American Psychiatric Association
. (
2000
).
Diagnostic and statistical manual of mental disorders
  (
4th ed.
).
Washington, DC
:
Author
.
Anil
A. E.
,
Kivircik
B. B.
,
Batur
S.
,
Kabaka
E.
,
Kitis
A.
,
Guven
E.
et al
. (
2003
).
The Turkish version of the Auditory Consonant Trigram Test as a measure of working memory: A normative study
.
The Clinical Neuropsychologist
 ,
17
,
159
169
.
Boone
K. B.
,
Miller
B. L.
,
Lesser
I. M.
,
Hill
E.
,
D'Elia
L.
(
1990
).
Performance on frontal lobe tests in healthy, older individuals
.
Developmental Neuropsychology
 ,
6
,
215
223
.
Boone
K. B.
,
Pontón
M. O.
,
Gorsuch
R. L.
,
González
J. J.
,
Miller
B. L.
(
1998
).
Factor analysis of four measures of prefrontal lobe functioning
.
Archives of Clinical Neuropsychology
 ,
13
,
585
595
.
Bowler
R. M.
,
Gysens
S.
,
Diamond
E.
,
Nakagawa
S.
,
Drezgic
M.
,
Roels
H. A.
(
2006
).
Manganese exposure: Neuropsychological and neurological symptoms and effects in welders
.
NeuroToxicology
 ,
27
,
315
326
.
Brown
J.
(
1958
).
Some tests of the decay theory of immediate memory
.
Quarterly Journal of Experimental Psychology
 ,
10
,
12
21
.
Conners
C. K.
, &
MHS Staff
. (
2004
).
Conners’ continuous performance test II (CPT II) for windows: Technical and software manual
 .
NorthTonawanda, NY
:
MHS
.
Crawford
J. R.
,
Obonsawin
M. C.
,
Allan
K. M.
(
1998
).
PASAT and components of WAIS-R performance: Convergent and discriminant validity
.
Neuropsychological Rehabilitation
 ,
8
,
255
272
.
Delis
D. C.
,
Kramer
J. H.
,
Kaplan
E.
,
Ober
B. A.
(
2000
).
California verbal learning test
  (
2nd ed.
,
Adult Version
):
Manual
.
Bloomington, MN
:
NCS Pearson
.
Dige
N.
,
Wik
G.
(
2005
).
Adult attention deficit hyperactivity disorder identified by neuropsychological testing
.
International Journal of Neuroscience
 ,
115
,
169
183
.
Dikmen
S. S.
,
Heaton
R. K.
,
Grant
I.
,
Temkin
N. R.
(
1999
).
Test-retest reliability and practice effects of expanded Halstead-Reitan neuropsychological test battery
.
Journal of theInternational Neuropsychological Society
 ,
5
,
346
356
.
First
M. B.
,
Spitzer
R. L.
,
Gibbon
M.
,
Williams
J. B. W.
(
1997
).
Structured clinical interview for DSM-IV-TR axis I disorders, research version, patient edition (SCID-I/P)
 .
New York
:
Biometrics Research, New York State Psychiatric Institute
.
Folmer
R. L.
,
McMillan
G. P.
,
Austin
D. F.
,
Henry
J. A.
(
2011
).
Audiometric thresholds and prevalence of tinnitus among male veterans in the United States: Data from the National Health and Nutrition Examination Survey, 1999–2006
.
Journal of Rehabilitation Research & Development
 ,
48
,
503
516
.
Golden
C. J.
,
Freshwater
S. M.
(
2002
).
Stroop Color and Word Test: A manual for clinical and experimental uses
 .
Wood Dale, IL
:
Stoelting
.
Green
P.
(
2005
).
Greens Word Memory Test for Windows users manual
 .
Edmonton, Canada
:
Green's Publishing
.
Greiffenstein
M. F.
,
Greve
K. W.
,
Bianchini
K. J.
,
Baker
W. J.
(
2008
).
Test of Memory Malingering and Word Memory Test: A new comparison of failure concordance rates
.
Archives of Clinical Neuropsychology
 ,
23
,
801
807
.
Grigsby
J.
,
Kaye
K.
(
1996
).
The behavioral dyscontrol scale: Manual
  (
2nd ed.
).
Denver, CO
:
BDS
.
Gronwall
D. M. A.
(
1977
).
Paced Auditory Serial Addition Task: A measure of recovery from concussion
.
Perceptual and Motor Skills
 ,
44
,
367
373
.
Ivins
B. J.
,
Schwab
K. A.
,
Warden
D.
,
Harvey
S.
,
Hoilien
M.
,
Powell
J.
et al
. (
2003
).
Traumatic brain injury in U.S. Army Paratroopers: Prevalence and character
.
The Journal of Trauma Injury, Infection, and Critical Care
 ,
55
,
617
621
.
Lezak
M. D.
,
Howieson
D. B.
,
Bigler
E. D.
,
Tranel
D.
(
2012
).
Neuropsychological assessment
  (
5th ed.
).
Oxford
:
Oxford University Press
.
Menon
D. K.
,
Schwab
K.
,
Wright
D. W.
,
Maas
A. I.
(
2010
).
Position statement: Definition of traumatic brain injury
.
Archives of Physical Medicine and Rehabilitation
 ,
91
,
1637
1640
.
Merkley
T. L.
,
Larson
M. J.
,
Bigler
E. D.
,
Good
D. A.
,
Perlstein
W. M.
(
2013
).
Structural and functional changes of the cingulate gyrus following traumatic brain injury: Relation to attention and executive skills
.
Journal of the International Neuropsychological Society
 ,
19
,
899
910
.
Mertens
V. B.
,
Gagnon
M.
,
Coulombe
D.
,
Messier
C.
(
2006
).
Exploratory factor analysis of the neuropsychological tests and their relationship to the Brown-Peterson task
.
Archives of Clinical Neuropsychology
 ,
21
,
733
739
.
Oral
E.
,
Canpolat
S.
,
Yildirim
S.
,
Gulec
M.
,
Aliyev
E.
,
Aydin
N.
(
2012
).
Cognitive functions and serum levels of brain-derived neurotrophic factor in patients with major depressive disorder
.
Brain Research Bulletin
 ,
88
,
454
459
.
Ozakbas
S.
,
Ormeci
B.
,
Akdede
B. B. K.
,
Alptekin
K.
,
Idiman
E.
(
2004
).
Utilization of the Auditory Consonant Trigram Test to screen for cognitive impairment in patients with multiple sclerosis: Comparison with the Paced Auditory Serial Addition Test
.
Multiple Sclerosis
 ,
10
,
686
689
.
Peterson
L. R.
,
Peterson
M. J.
(
1959
).
Short-term retention of individual verbal items
.
Journal of Experimental Psychology
 ,
58
,
193
198
.
Rabin
L. A.
,
Barr
W. B.
,
Burton
L. A.
(
2005
).
Assessment practices of clinical neuropsychologists in the United States and Canada: A survey of INS, NAN, and APA Division 40 members
.
Archives of Clinical Neuropsychology
 ,
20
,
33
65
.
Reitan
R. M.
,
Wolfson
D.
(
1985
).
The Halstead-Reitan neuropsychological test battery: Theory and clinical interpretation
 .
Tuscon, AZ
:
Neuropsychology Press
.
Ruff
R. M.
,
Light
R. H.
,
Parker
S. B.
,
Levin
H. S.
(
1996
).
Benton Controlled Oral Word Association Test: Reliability and updated norms
.
Archives of Clinical Neuropsychology
 ,
11
,
329
338
.
Shura
R. D.
,
Rowland
J. A.
,
Yoash-Gantz
R. E.
(
2014
).
The Behavioral Dyscontrol Scale-II with non-elderly veterans
.
Archives of Clinical Neuropsychology
 ,
29
,
409
414
.
Snyder
H. R.
(
2013
).
Major depressive disorder is associated with broad impairments on neuropsychological measures of executive function: A meta-analysis and review
.
Psychological Bulletin
 ,
139
,
81
132
.
Strauss
E.
,
Sherman
E. M. S.
,
Spreen
O.
(
2006
).
A compendium of neuropsychological tests: Administration, norms, and commentary
  (
3rd ed.
).
New York, NY
:
Oxford University Press
.
Stuss
D. T.
,
Stethem
L. L.
,
Hugenholtz
H.
,
Richard
M. T.
(
1989
).
Traumatic brain injury: A comparison of three clinical tests, and analysis of recovery
.
The Clinical Neuropsychologist
 ,
3
,
145
156
.
Stuss
D. T.
,
Stethem
L. L.
,
Pelchat
G.
(
1988
).
Three tests of attention and rapid information processing: An extension
.
The Clinical Neuropsychologist
 ,
2
,
246
250
.
Stuss
D. T.
,
Stethem
L. L.
,
Poirier
C. A.
(
1987
).
Comparison of three tests of attention and rapid information processing across six age groups
.
The Clinical Neuropsychologist
 ,
1
,
139
152
.
The Psychological Corporation
. (
2001
).
Wechsler Test of Adult Reading: Manual
 .
San Antonio, TX
:
Author
.
Wechsler
D.
(
1997
).
Wechsler Adult Intelligence ScaleIII: Administration and scoring manual
 .
San Antonio
:
The Psychological Corporation
.

Author notes

The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs, the Department of Defense, or the U.S. government.