Abstract

Computerized screening measures can provide valuable information on cognition. However, determining the validity of obtained data is critical for interpretation. The purpose of this study was to examine the embedded validity indicators on the CNS Vital Signs battery in a sample of youth with neurological diagnoses. The sample included 275 children and adolescents (mean = 13.9, SD = 3.0) with neurological disorders. Six out of seven subtests and six of the nine domain scores on CNS Vital Signs had fewer than 5% of the sample flagged as invalid on the embedded indicators. However, the Shifting Attention Test and its derived domain scores had higher rates of being flagged. Patients with one or more flagged scores (18% of sample) were younger and had lower intellectual abilities, psychomotor speed, verbal memory, and performance on other validity tests. Compared to stand-alone validity tests, CNS Vital Signs embedded validity indicators had low sensitivity. More research is needed with the embedded indicators in youth.

Introduction

Determining the validity of obtained patient data is an important component of a neuropsychological assessment (Bush et al., 2005; Heilbronner, Sweet, Morgan, Larrabee, & Millis, 2009; Sherman & Brooks, 2012). Compared with adult studies, the literature on validity measurement in children is small, relatively underdeveloped, and slowly increasing over time (see Kirkwood, 2012). The majority of pediatric studies have focused on stand-alone performance validity tests (PVTs) and have demonstrated that they can be readily passed by healthy children and adolescents (Constantinou & McCaffrey, 2003; Rienstra, Spaan, & Schmand, 2010) and by youth with neurological disorders (Brooks, 2012; Brooks, Sherman, & Krol, 2012; Carone, 2008; Donders, 2005; Donders & Boonstra, 2007; Green & Flaro, 2003; Green, Flaro, & Courtney, 2009; Kirk et al., 2011; Kirkwood & Kirk, 2010; Loughan & Perna, 2014; Loughan, Perna, & Hertza, 2012; MacAllister, Nakhutina, Bender, Karantzoulis, & Carlson, 2009; Perna & Loughan, 2013), often at levels comparable with adults.

In comparison to stand-alone PVTs, there are very few studies on embedded validity indicators, which are typically indicators derived from within a cognitive measure and based on unlikely performance on the cognitive measure. Considering the Reliable Digit Span (RDS; Greiffenstein, Baker, & Gola, 1994), which is derived from the Digit Span subtest on Wechsler intelligence tests (Wechsler, 2004, 2008a), pass rates are very low in children when using the previously established adult cutoff (RDS ≤ 7). Blaskewitz, Merten, and Kathman (2008) reported very low pass rates in healthy children, including 30% for second graders, 38% for third graders, and 60% for fourth graders. In a sample of adolescents with mild traumatic brain injuries, Kirkwood, Hargrave, and Kirk (2011) reported a 31% false-positive rate using the adult RDS cutoff. Suggestions for pediatric RDS cutoffs that provide at least 90% specificity have been RDS ≤ 6 in patients with mTBI (sensitivity = 0.51; Kirkwood et al., 2011), RDS ≤ 4 in a mixed clinical sample (sensitivity = 0.43; Loughan et al., 2012), and RDS ≤ 3 in patients with epilepsy (sensitivity = 0.20; Welsh, Bender, Whitman, Vasserman, & MacAllister, 2012).

Some studies have examined embedded validity indicators for pediatric memory measures. Baker, Connery, Kirk, and Kirkwood (2014) reported that a recognition discriminability score from the California Verbal Learning Test, Children's Version (CVLT-C; Delis, Kramer, Kaplan, & Ober, 1994) of z = −0.5 could have 55% sensitivity for poor engagement (with a specificity of 91%) in youth referred clinically due to an mTBI. Perna, Loughan, Hertza, and Segraves (2014) suggested that a 20-point difference between the Children's Memory Scale (Cohen, 1997) Verbal Delayed Recall and Verbal Delayed Recognition could meet a recommended 90% specificity rate for children referred for academic or behavioral problems, but this level of discrepancy only provided an 11% sensitivity rate. Overall, these results suggest that embedded validity indicators may have potential use with pediatric patients, but considerable caution needs to be exercised due to the chances of higher false-positive rates and lack of strong supportive evidence.

A potential advantage of embedded validity indicators is that additional tests might not need to be administered to determine performance validity. This is most advantageous when faced with time constraints, such as when rapidly screening cognitive abilities, and for providing evidence of performance validity on a specific cognitive measure (rather than through a proxy measure). The CNS Vital Signs, a rapid computer-assisted battery of tests that provides a screen of cognitive abilities and has been used previously with pediatric patients (Brooks & Barlow, 2011; Brooks, Iverson, Sherman, & Roberge, 2010; Brooks & Sherman, 2012), contains embedded validity indicators that are automatically calculated by the program. According to the publishers of the CNS Vital Signs battery, the embedded validity indicators were developed to “ … indicate whether the patient gave poor effort or generated invalid results” (p. 5; CNS Vital Signs, 2014). However, despite the creation of validity indicators by the test publisher, there are no peer-reviewed manuscripts demonstrating the use and validity of these indicators with clinical populations. There have been four unpublished studies, presented at conferences that have examined performance validity on CNS Vital Signs. The first set of embedded validity indicators were introduced by Gualtieri, Vaishnavi, and Hervey (2009). These authors reported high specificity rates in healthy control participants and in multiple clinical samples. Rohling, Hill, Ploetz, Womble, and Shenessey (2011) introduced a different set of embedded validity indicators and showed that they could detect university students instructed to malinger. Two studies examined the rated of invalid test scores in adults with depression (Saffer, Iverson, Kjernisted, & McIntosh, 2013) and in older adults referred for a cognitive assessment (Saffer, Rosenblatt, Lanting, Koehle, & Iverson, 2013). For the study with adults with depression (Saffer, Iverson, et al., 2013), participants were 48 outpatients who were enrolled in a treatment study. They completed the CNS Vital Signs battery three times (at a 1-week interval and again after ∼8 weeks). Applying the published validity indicators, 19.6% were identified as having invalid scores at time 1, 13.0% at time 2, and 10.9% at time 3. For the second study (Saffer, Rosenblatt, et al., 2013), participants were 71 older adults who self-referred or were referred by their physicians for a cognitive screening evaluation. Based on the embedded validity indicators, six (8.5%) were identified as having invalid scores. For five of the six participants, their scores were out-of-range (i.e., standard score <40) and the examiner noted that the participant did not appear to do the test properly. These studies suggest that a minority of adults and older adults with no known incentive for underperformance on testing have invalid scores on CNS Vital Signs.

The ability to determine the validity of CNS Vital Signs performance is an asset of this battery, particularly in clinical situations where there is a desire to rapidly screen cognitive functioning. However, information is needed on the sensitivity and specificity of these embedded validity indicators so that researchers and clinicians can have confidence in their use. The purpose of this study was to examine the prevalence rates, sensitivity and specificity of the CNS Vital Signs embedded validity indicators in a sample of children and adolescents with neurological disorders. Although there is not a consensus on acceptable rates of sensitivity and specificity for embedded validity indicators, it was our a priori decision that desirable levels would be set at 0.5 for sensitivity and 0.9 for specificity based on the abovementioned existing studies with youth.

Methods

Participants

The participants for this study were children and adolescents between 7 and 18 years of age who were assessed through a neuropsychology service at a tertiary care pediatric hospital. Because the neuropsychological assessments were completed as part of a clinical service, not every patient who was seen was given all the tests in the standard battery (i.e., the determination of which tests to include was up to the individual neuropsychologist). Therefore, this represented a sample of convenience rather than a prospective, consecutive series of patients. It reflects a more typical patient population seen by neuropsychologists working in tertiary care centers. Patients were excluded, and therefore not administered the CNS Vital Signs, if they were unable to understand test directions, unable to read words printed on a screen, or had motor or vision impairments that prevented test administration. A portion of this sample was previously reported in Brooks and Sherman (2012). The University of Calgary Conjoint Health Research Ethics Board granted ethical clearance for the collection and use of these data (REB13-0515).

Measures

CNS Vital Signs battery

CNS Vital Signs is a computer-administered battery of seven neuropsychological tests that are designed to rapidly screen cognitive abilities. The Verbal Memory Test involves immediate and delayed recognition for 15 words presented on the screen (drawn from a reservoir of 100 words). The same paradigm is followed for the Visual Memory Test, with immediate and delayed recognition of 15 geometric figures (drawn from a reservoir of 45 designs). For the Finger Tapping Test, participants are asked to rapidly press the space bar with their index finger (separately for right and left hands). The Symbol Digit Coding Test involves typing numbers that correspond to eight different symbols presented on the screen (drawn from a reservoir of 32 symbols). The Stroop Test contains three parts that involve responding to words and colors. Part one involves pressing the space bar as soon as the word RED, YELLOW, BLUE, or GREEN is shown on the screen (printed in black). Part two involves pressing the space bar when the color of the word matches what the word says (e.g., the word RED in red ink) but not responding when the color of the word does not match what the word says (e.g., RED printed in blue ink). Part three involves pressing the space bar when the color of the word does not match what the word says (e.g., RED printed in blue ink) but not responding when the color of the word matches what the word says (e.g., RED printed in red ink). The Shifting Attention Test involves matching geometric objects either by shape or by color (the match “rule” is alternated). The Continuous Performance Test is a 5-min sustained attention test that involves responding to the target stimulus “B” but not to any other letter.

The CNS Vital Signs primary domain scores (mean = 100, SD = 15; index scores for this study had a maximum range of −4 to +4 SDs, with low scores truncated at 40) are derived from performance on the seven tests (which together generate 18 subtest scores). Correct responses from the verbal and visual memory tests provide Verbal Memory and Visual Memory domain scores, respectively, as well as a composite Memory domain score. The total of right and left taps from the Finger Tapping Test and the total correct responses on the Symbol Digit Coding Test generate a composite score for Psychomotor Speed. Averaging the two complex reaction time scores from the Stroop Test generates a domain score for Reaction Time, which can be considered as measuring information processing speed in a test of executive function. The number of correct responses on the Shifting Attention Test, minus the number of errors on the Shifting Attention Test and the Stroop Test, is used to create a domain score for Cognitive Flexibility. A domain score for Complex Attention is generated by adding the number of errors committed in the Continuous Performance Test, the Shifting Attention Test, and the Stroop Test. Additional domain scores include Processing Speed (derived from performance on the Symbol Digit Coding Test) and Executive Functions (derived from performance on the Shifting Attention Test). The overall summary score, called the Neurocognition Index, is the average of the five original domain scores (Memory, Psychomotor Speed, Reaction Time, Complex Attention, and Cognitive Flexibility).

Each CNS Vital Signs subtest has an embedded validity indicators designed to determine whether the performance and resultant test score is valid. If a subtest's embedded validity indicator is flagged as “invalid”, then domain scores derived from that subtest and the overall summary score are also flagged. The embedded validity indicators are automatically generated on the computer printout of the test results; therefore, they do not need to be calculated by the user. The calculations used by the computer program to determine each of the validity indicators are presented in Table 1. These formulas, which were confirmed by the first author with the test publisher, were derived by the CNS Vital Signs company based on internal analyses of normative and clinical samples. The preliminary work on the development of these embedded validity indicators was presented at a conference (Gualtieri et al., 2009).

Table 1.

Calculations for validity indicators on the CNS vital signs

CNS vital signs subtests Subtest score is valid if 
Verbal Memory Test (VERM) VERM Correct Hits Immediate + VERM Correct Passes Immediate + VERM Correct Hits Delay + VERM Correct Passes Delay >30 
Visual Memory Test (VISM) VISM Correct Hits Immediate + VISM Correct Passes Immediate + VISM Correct Hits Delay + VISM Correct Passes Delay >30 
Finger Tapping Test (FTT) FTT Right Taps Average + FTT Left Taps Average ≥ 40 
Symbol Digit Coding Test (SDC) SDC Correct Responses ≥ 20 AND
SDC Correct Responses > SDC Errors 
Stroop Test [Stroop Simple RT < (Stroop Complex RT Correct × 0.1) + Stroop Complex RT Correct] AND
[Stroop Complex RT Correct < (Stroop RT Correct × 0.1) + Stroop RT Correct] AND
(Stroop Complex Correct > Stroop Complex Errors) AND
(Stroop Correct > Stroop Errors) 
Shifting Attention Test (SAT) SAT Correct Responses > SAT Errors 
Continuous Performance Test (CPT) CPT Correct Responses ≥ 30 AND CPT Correct Responses > CPT Commission Errors 
CNS Vital Signs Domain Scores Domain score is valid if… 
 Neurocognition Index VERM, VISM, SDC, FTT, SAT, Stroop, and CPT are valid (all subtests) 
 Memory Domain VERM and VISM are valid 
 Verbal Memory Domain VERM is valid 
 Visual Memory Domain VISM is valid 
 Processing Speed Domain SDC is valid 
 Psychomotor Speed Domain FTT and SDC are valid 
 Executive Functions Domain SAT is valid 
 Reaction Time Domain Stroop is valid 
 Complex Attention Domain Stroop, SAT, and CPT are valid 
 Cognitive Flexibility Domain Stroop and SAT are valid 
CNS vital signs subtests Subtest score is valid if 
Verbal Memory Test (VERM) VERM Correct Hits Immediate + VERM Correct Passes Immediate + VERM Correct Hits Delay + VERM Correct Passes Delay >30 
Visual Memory Test (VISM) VISM Correct Hits Immediate + VISM Correct Passes Immediate + VISM Correct Hits Delay + VISM Correct Passes Delay >30 
Finger Tapping Test (FTT) FTT Right Taps Average + FTT Left Taps Average ≥ 40 
Symbol Digit Coding Test (SDC) SDC Correct Responses ≥ 20 AND
SDC Correct Responses > SDC Errors 
Stroop Test [Stroop Simple RT < (Stroop Complex RT Correct × 0.1) + Stroop Complex RT Correct] AND
[Stroop Complex RT Correct < (Stroop RT Correct × 0.1) + Stroop RT Correct] AND
(Stroop Complex Correct > Stroop Complex Errors) AND
(Stroop Correct > Stroop Errors) 
Shifting Attention Test (SAT) SAT Correct Responses > SAT Errors 
Continuous Performance Test (CPT) CPT Correct Responses ≥ 30 AND CPT Correct Responses > CPT Commission Errors 
CNS Vital Signs Domain Scores Domain score is valid if… 
 Neurocognition Index VERM, VISM, SDC, FTT, SAT, Stroop, and CPT are valid (all subtests) 
 Memory Domain VERM and VISM are valid 
 Verbal Memory Domain VERM is valid 
 Visual Memory Domain VISM is valid 
 Processing Speed Domain SDC is valid 
 Psychomotor Speed Domain FTT and SDC are valid 
 Executive Functions Domain SAT is valid 
 Reaction Time Domain Stroop is valid 
 Complex Attention Domain Stroop, SAT, and CPT are valid 
 Cognitive Flexibility Domain Stroop and SAT are valid 

Notes: RT, reaction time. Previous studies (Saffer, Iverson, et al., 2013; Saffer, Rosenblatt, et al., 2013) used the validity indicators that were published on the company website (www.cnsvs.com). There are some differences in the formulas in the table above, compared with the formulas published on the website in the past. The first author of the present study confirmed via email that the formulas in the table above are the correct formulas that were programmed into the software. Therefore, future researchers should use the formulas in the table above.

Stand-alone measures of performance validity

The criterion validity for the CNS Vital Signs embedded validity indicators was based on performance from two stand-alone PVTs—the Test of Memory Malingering (TOMM; Tombaugh, 1996) and the Victoria Symptom Validity Test (VSVT; Slick, Hopp, Strauss, & Thompson, 1997). The TOMM is a forced-choice visual recognition PVT that has a low-presumed false-positive rate in healthy children (Constantinou & McCaffrey, 2003; Rienstra et al., 2010) and children with neurological disorders (Brooks et al., 2012; Donders, 2005; Kirk et al., 2011; Loughan et al., 2012; MacAllister et al., 2009). For information on the test format, please refer to the TOMM manual. The VSVT is a forced-choice visual recognition PVT that also has a low-presumed false-positive rate in youth with neurological diagnoses (Brooks, 2012).

Neuropsychological measures

Additional neuropsychological tests were included in this study for the purpose of better describing the sample. The scores and tests included: General Ability Index from the Wechsler Intelligence Scale for Children, Fourth Edition (WISC-IV; Wechsler, 2003a) or the Wechsler Adult Intelligence Scale, Third/Fourth Edition (WAIS-III/IV; Wechsler, 1997, 2008b); the Processing Speed Index from the WISC-IV, WAIS-III, or WAIS-IV (Wechsler, 1997, 2003b, 2008a); the Continuous Attention Test (Seidel & Joschko, 1991); long delay free recall from the CVLT-C (Delis et al., 1994) or California Verbal Learning Test, Second Edition (Delis, Kramer, Kaplan, & Ober, 2000); and the general executive composite from the Behaviour Rating Inventory of Executive Function (Gioia, Isquith, Guy, & Kenworthy, 2000).

Analyses

To examine the prevalence rates of being flagged on the CNS Vital Signs embedded validity indicators, patients were grouped into those who “passed” all seven embedded validity indicators and those who were “flagged” on one or more embedded validity indicators. Descriptive methods were used for performance on the CNS Vital Signs and the percent of patients being flagged as having an invalid score on the seven subtests. Comparing performance on the neuropsychological tests and stand-alone PVTs between those who “pass” and those who are flagged was completed using one-way analyses of variance and further described using Cohen's d effect sizes (d = .20, small; d = .50, medium; and d = .80, large).

Participants were also categorized into either “pass PVTs” [adequate performance on (a) the TOMM based on a score above published cutoffs and (b) the VSVT based on the computer printout indicating “valid” on both the easy and the hard items] or “questionable PVTs” [questionable performance on either (a) the TOMM based on a score below published cutoffs or (b) the VSVT based on the computer printout indicating “questionable” or “invalid” on either the easy or the hard items]. If only one PVT was administered with a patient, then performance of “pass PVT” or “questionable PVT” was based on that single test. Using the two groups created by the stand-alone PVTs (i.e., this served as the criterion validity measure), sensitivity and specificity were computed for the CNS Vital Signs embedded validity indicators. Sensitivity, defined as the probability of a positive test or the ability of the CNS Vital Signs EVI to detect questionable (below cutoff) performance based on the stand-alone PVTs, was computed based on: number of true positives/(number of true positives + number of false negatives). Specificity, defined as the probability of a negative test or the ability of the CNS Vital Signs embedded validity indicator to detect adequate (above cutoff) performance based on the stand-alone PVTs, was computed based on: number of true negatives/(number of true negatives + number of false positives).

Results

Participant demographics are presented in Table 2 (N = 275). Participants in this study ranged in age from 7.0 to 18.9 years, with a mean age of 13.9 years (SD = 3.0) and the majority of the sample being adolescents (18.9% were 7–10 years, 38.9% were 11–14 years, and 42.2% were 15–18 years). The sample was split fairly evenly with boys and girls, with the majority being Caucasian. Their parents were relatively well educated, with the majority identifying at least some level of post-secondary schooling. This sample included a heterogeneous range of neurological diagnoses, including those youth with epilepsy, traumatic brain injury, stroke, and hydrocephalus. The mean Neurocognition Index, a summary score of overall performance on the CNS Vital Signs, was 88.6 (SD = 15.4) with 23.4% of the sample having a score 1.5 SD below the mean (note that this would be found in 7% of the standardization sample). The individual domain scores representing memory, processing and psychomotor speed, executive control, and reaction time ranged from 86.2 (SD = 23.0; Complex Attention) to 95.7 (SD = 18.5; Processing Speed). Between 16.9% and 31.8% of the sample had low domain scores when considering each domain individually. On the individual CNS Vital Signs subtests, a subtest score 1.5 SD below the mean was found in 7.0%–41.7% of the sample.

Table 2.

Descriptive information for the children and adolescents with neurological diagnoses

Descriptive variables N Percent Mean SD Range 
Age (years) 275 – 13.9 3.0 7.0–18.9 
Gender 
 Male (%) 131 47.6 — — — 
 Female (%) 144 52.4 — — — 
Ethnicity 
 Caucasian (%) 142 51.6 — — — 
 Asian (%) 2.2 — — — 
 Hispanic (%) 2.2 — — — 
 First nations (Native American) (%) 1.8 — — — 
 African American (%) 1.8 — — — 
 Other (%) 16 5.8 — — — 
 Unknown/not documented (%) 95 34.5 — — — 
Mother's education (years)   14.1 2.4 4–20 
 Less than High School (%) 16 5.8 — — — 
 High School (%) 66 24.0 — — — 
 Some Post-Secondary (%) 89 32.4 — — — 
 University Degree (%) 88 32.0 — — — 
 Not documented (%) 16 5.8 — — — 
Father's education (years)   14.2 2.6 4–20 
 Less than High School (%) 27 9.8 — — — 
 High School (%) 59 21.5 — — — 
 Some Post-Secondary (%) 56 20.4 — — — 
 University Degree (%) 104 37.8 — — — 
 Not documented (%) 29 10.5 — — — 
Diagnoses 
 Epilepsy (%) 90 32.7 — — — 
 Traumatic brain injury (%) 79 28.7 — — — 
 Stroke (%) 27 9.8 — — — 
 Hydrocephalus (%) 16 5.8 — — — 
 Other 63 22.9 — — — 
Descriptive variables N Percent Mean SD Range 
Age (years) 275 – 13.9 3.0 7.0–18.9 
Gender 
 Male (%) 131 47.6 — — — 
 Female (%) 144 52.4 — — — 
Ethnicity 
 Caucasian (%) 142 51.6 — — — 
 Asian (%) 2.2 — — — 
 Hispanic (%) 2.2 — — — 
 First nations (Native American) (%) 1.8 — — — 
 African American (%) 1.8 — — — 
 Other (%) 16 5.8 — — — 
 Unknown/not documented (%) 95 34.5 — — — 
Mother's education (years)   14.1 2.4 4–20 
 Less than High School (%) 16 5.8 — — — 
 High School (%) 66 24.0 — — — 
 Some Post-Secondary (%) 89 32.4 — — — 
 University Degree (%) 88 32.0 — — — 
 Not documented (%) 16 5.8 — — — 
Father's education (years)   14.2 2.6 4–20 
 Less than High School (%) 27 9.8 — — — 
 High School (%) 59 21.5 — — — 
 Some Post-Secondary (%) 56 20.4 — — — 
 University Degree (%) 104 37.8 — — — 
 Not documented (%) 29 10.5 — — — 
Diagnoses 
 Epilepsy (%) 90 32.7 — — — 
 Traumatic brain injury (%) 79 28.7 — — — 
 Stroke (%) 27 9.8 — — — 
 Hydrocephalus (%) 16 5.8 — — — 
 Other 63 22.9 — — — 

Notes: SD = standard deviation; % = percent.

The percentages of the sample flagged on the various CNS Vital Signs embedded validity indicators are presented in Table 3. With the exception of the Shifting Attention Test (12.6% of the sample flagged as “invalid” based on correct > errors) and the domain scores derived in part from this subtest (e.g., Executive Functions, 12.6% “invalid”; Complex Attention, 17.6% “invalid”; Cognitive Flexibility, 14.6% “invalid”), fewer than 5% of the sample were flagged on each of the other individual subtest and domain-embedded validity indicators. When considering all subtest embedded validity indicators simultaneously, 17.8% of the sample had one or more embedded validity indicators flagged by the program (this is also reflected with the same percent who are flagged on the Neurocognition Index summary score). In the present sample, having up to nine more errors than correct responses for the Shifting Attention Test embedded validity indicator would result in 4.6% of the sample being flagged as invalid (future research can help determine the best validity criterion for the Shifting Attention Test).

Table 3.

Percent of children and adolescents with neurological diagnoses above or below cutoffs for the CNS Vital Signs embedded validity indicators

CNS Vital Signs embedded validity indices
 
% Above cutoff on validity indicator (valid) % Below Cutoff on validity indicator (invalid) 
Subtests 
 Verbal Memory (VERM) 98.2 1.8 
 Visual Memory (VISM) 98.2 1.8 
 Finger Tapping (FTT) 99.2 0.8 
 Symbol Digit Coding (SDC) 95.3 4.7 
  Correct20 95.3 4.7 
  Correct > Errors 99.6 0.4 
 Stroop Test (STROOP) 96.0 4.0 
  Simple Reaction Time<Complex Reaction Time 98.9 1.1 
  Complex Reaction Time<Stroop Reaction Time 96.7 3.3 
  Complex Correct>Complex Errors 98.9 1.1 
  Stroop Correct>Stroop Errors 99.3 0.7 
 Shifting Attention Test (SAT) 87.4 12.6 
 Continuous Performance Test (CPT) 94.8 5.2 
  Correct30 95.7 4.3 
  Correct > Errors 99.1 0.7 
Domains Subtests that contribute to embedded validity indicator   
 Neurocognition Index All subtests 82.2 17.8 
 Memory Domain VERM, VISM 96.9 3.1 
 Verbal Memory Domain VSRM 98.2 1.8 
 Visual Memory Domain VISM 98.2 1.8 
 Processing Speed Domain SDC 95.3 4.7 
 Psychomotor Speed Domain FTT, SDC 95.6 4.4 
 Executive Functions Domain SAT 87.4 12.6 
 Reaction Time Domain STROOP 96.0 4.0 
 Complex Attention Domain STROOP, SAT, CPT 82.4 17.6 
 Cognitive Flexibility Domain STROOP, SAT 85.4 14.6 
CNS Vital Signs embedded validity indices
 
% Above cutoff on validity indicator (valid) % Below Cutoff on validity indicator (invalid) 
Subtests 
 Verbal Memory (VERM) 98.2 1.8 
 Visual Memory (VISM) 98.2 1.8 
 Finger Tapping (FTT) 99.2 0.8 
 Symbol Digit Coding (SDC) 95.3 4.7 
  Correct20 95.3 4.7 
  Correct > Errors 99.6 0.4 
 Stroop Test (STROOP) 96.0 4.0 
  Simple Reaction Time<Complex Reaction Time 98.9 1.1 
  Complex Reaction Time<Stroop Reaction Time 96.7 3.3 
  Complex Correct>Complex Errors 98.9 1.1 
  Stroop Correct>Stroop Errors 99.3 0.7 
 Shifting Attention Test (SAT) 87.4 12.6 
 Continuous Performance Test (CPT) 94.8 5.2 
  Correct30 95.7 4.3 
  Correct > Errors 99.1 0.7 
Domains Subtests that contribute to embedded validity indicator   
 Neurocognition Index All subtests 82.2 17.8 
 Memory Domain VERM, VISM 96.9 3.1 
 Verbal Memory Domain VSRM 98.2 1.8 
 Visual Memory Domain VISM 98.2 1.8 
 Processing Speed Domain SDC 95.3 4.7 
 Psychomotor Speed Domain FTT, SDC 95.6 4.4 
 Executive Functions Domain SAT 87.4 12.6 
 Reaction Time Domain STROOP 96.0 4.0 
 Complex Attention Domain STROOP, SAT, CPT 82.4 17.6 
 Cognitive Flexibility Domain STROOP, SAT 85.4 14.6 

Notes: Above or below the cutoff for an embedded validity indicator is based on the calculations provided in Table 1.

Group comparisons between those who were or were not flagged on the CNS Vital Signs embedded validity indicators revealed statistically significant differences and large to very large effect sizes on age, overall cognitive/intellectual skills, psychomotor speed, verbal memory, and performance on stand-alone PVTs. Those patients who were flagged on one or more CNS Vital Signs embedded validity indicators were younger [F(1,211) = 45.62, p < .001; Cohen's d = 1.25] and they had lower cognitive/intellectual abilities [F(1,149) = 20.50, p < .001; d = .78], worse psychomotor processing speed [F(1,169) = 8.38, p = .004; d = .61], worse verbal memory [F(1,156) = 12.36, p = .001; d = .83], and worse performance on both the TOMM [Trial 1; F(1,121) = 10.31, p = .002; d = .85] and VSVT [Easy Items; F(1,82) = 11.57, p = .001; d = .99: Difficult Items; F(1,82) = 20.20, p < .001; d = 1.28].

When using performance on the stand-alone PVTs as the criterion for determining adequate or questionable performance, the sensitivity (true positive detection of questionable compliance) and specificity (true negative detection of questionable compliance) of the CNS Vital Signs embedded validity indicators was examined (see Table 4). Across the subtests, the CNS Vital Signs embedded validity indicators have sensitivity ranging from 0.04 to 0.29 and specificity ranging from 0.94 to 0.99. The highest sensitivity was obtained on the Shifting Attention Test (sensitivity = 0.29), which improved marginally to 0.35 when considering an adjusted embedded validity indicator derived for the present sample (correct errors ≥ −9). It is worth noting that several domain scores are based on single subtests (e.g., Verbal Memory Domain, Visual Memory Domain, Processing Speed Domain, Executive Functions Domain, and Reaction Time Domain), so the sensitivity and specificity values are identical to those presented for the respective subtests. When considering the remaining domain scores that are based on multiple subtests, sensitivity ranged from 0.12 to 0.35 and specificity ranged from 0.88 to 0.99. For the Neurocognition Index, sensitivity was 0.39 and specificity was 0.89.

Table 4.

Sensitivity and specificity of CNS Vital Signs embedded validity indicators in children and adolescents with neurological diagnoses

Subtest embedded validity indicator TOMM/VSVT
 
Sensitivity Specificity 
Below cutoff Above cutoff 
Verbal Memory 
 Below Cutoff 2 1 0.08 0.99 
 Above Cutoff 23 122   
Visual Memory 
 Below Cutoff 1 1 0.04 0.99 
 Above Cutoff 25 123   
Symbol-Digit Coding 
 Below Cutoff 5 3 0.19 0.98 
 Above Cutoff 21 131   
Finger Tapping 
 Below Cutoff 2 9 0.06 0.95 
 Above Cutoff 30 158   
Shifting Attention 
 Below Cutoff 9 10 0.29 0.94 
 Above Cutoff 22 143   
Stroop Test 
 Below Cutoff 2 5 0.06 0.97 
 Above Cutoff 30 157   
Continuous Performance 
 Below Cutoff 4 6 0.16 0.95 
 Above Cutoff 21 123   
Domain-embedded validity indicator 
Neurocognition Index (overall) 
 Below Cutoff 9 13 0.39 0.89 
 Above Cutoff 14 101   
Memory Domain 
 Below Cutoff 3 1 0.12 0.99 
 Above Cutoff 22 120   
Verbal Memory Domain 
 Below Cutoff 2 1 0.08 0.99 
 Above Cutoff 23 122   
Visual Memory Domain 
 Below Cutoff 1 1 0.04 0.99 
 Above Cutoff 25 123   
Processing Speed Domain 
 Below Cutoff 5 3 0.19 0.98 
 Above Cutoff 21 131   
Psychomotor Speed Domain 
 Below Cutoff 5 3 0.19 0.98 
 Above Cutoff 20 126   
Executive Functions Domain 
 Below Cutoff 9 10 0.29 0.94 
 Above Cutoff 22 143   
Reaction Time Domain 
 Below Cutoff 2 5 0.06 0.97 
 Above Cutoff 30 157   
Complex Attention Domain 
 Below Cutoff 8 15 0.35 0.88 
 Above Cutoff 15 106   
Cognitive Flexibility Domain 
 Below Cutoff 10 13 0.32 0.92 
 Above Cutoff 21 140   
Subtest embedded validity indicator TOMM/VSVT
 
Sensitivity Specificity 
Below cutoff Above cutoff 
Verbal Memory 
 Below Cutoff 2 1 0.08 0.99 
 Above Cutoff 23 122   
Visual Memory 
 Below Cutoff 1 1 0.04 0.99 
 Above Cutoff 25 123   
Symbol-Digit Coding 
 Below Cutoff 5 3 0.19 0.98 
 Above Cutoff 21 131   
Finger Tapping 
 Below Cutoff 2 9 0.06 0.95 
 Above Cutoff 30 158   
Shifting Attention 
 Below Cutoff 9 10 0.29 0.94 
 Above Cutoff 22 143   
Stroop Test 
 Below Cutoff 2 5 0.06 0.97 
 Above Cutoff 30 157   
Continuous Performance 
 Below Cutoff 4 6 0.16 0.95 
 Above Cutoff 21 123   
Domain-embedded validity indicator 
Neurocognition Index (overall) 
 Below Cutoff 9 13 0.39 0.89 
 Above Cutoff 14 101   
Memory Domain 
 Below Cutoff 3 1 0.12 0.99 
 Above Cutoff 22 120   
Verbal Memory Domain 
 Below Cutoff 2 1 0.08 0.99 
 Above Cutoff 23 122   
Visual Memory Domain 
 Below Cutoff 1 1 0.04 0.99 
 Above Cutoff 25 123   
Processing Speed Domain 
 Below Cutoff 5 3 0.19 0.98 
 Above Cutoff 21 131   
Psychomotor Speed Domain 
 Below Cutoff 5 3 0.19 0.98 
 Above Cutoff 20 126   
Executive Functions Domain 
 Below Cutoff 9 10 0.29 0.94 
 Above Cutoff 22 143   
Reaction Time Domain 
 Below Cutoff 2 5 0.06 0.97 
 Above Cutoff 30 157   
Complex Attention Domain 
 Below Cutoff 8 15 0.35 0.88 
 Above Cutoff 15 106   
Cognitive Flexibility Domain 
 Below Cutoff 10 13 0.32 0.92 
 Above Cutoff 21 140   

Notes: Bolded numbers represent correct classifications (true positives and true negatives). Italicized numbers represent incorrect classifications (false positives and false negatives).

Discussion

There are very few published studies examining embedded performance validity indicators in children and adolescents. To our knowledge, this is the first study to do so using the CNS Vital Signs battery. In this large sample of youth with neurological diagnoses, the fail rate was very low across all but one of the individual tests (i.e., 5% or fewer were flagged as invalid on six of seven subtests). Generally, a >90% pass rate in children and adolescents on an individual PVT is considered clinically useful, especially when it is found in a sample with known cognitive impairment (Brooks, 2012; Brooks et al., 2012; Carone, 2008; Donders, 2005; Donders & Boonstra, 2007; Green & Flaro, 2003; Green et al., 2009; Kirk et al., 2011; Kirkwood & Kirk, 2010; Loughan & Perna, 2014; MacAllister et al., 2009; Perna & Loughan, 2013). The exceptions to the very high pass rate were the Shifting Attention Test (87.4%) and three domain scores (Executive Functions, 87.4% passed; Complex Attention, 82.4% passed; and Cognitive Flexibility, 85.4% passed).

When considering all of the embedded validity indicators from the seven subtests simultaneously, nearly 18% of the sample had at least one flagged validity indicator (this value is also reflected in the pass rate on the Neurocognition Index). Those children and adolescents with one or more flagged validity indicators were younger, and they had lower cognitive/intellectual abilities, worse psychomotor processing speed, worse verbal memory, and worse performance on the two stand-alone PVTs.

When the CNS Vital Signs embedded validity indicators were compared with external criterion stand-alone PVTs (i.e., the TOMM and VSVT), specificity was adequate-to-high but sensitivity for identifying possible poor effort was often low. On the seven subtests, specificity of the embedded validity indicators was very high (ranging from 0.94 to 0.99). Specificity for the domain scores was also quite high (ranging from 0.88 to 0.99), although the Complex Attention domain (specificity = 0.88) and Neurocognition Index (specificity = 0.89) were slightly under the desired 90% level of specificity. Sensitivity values of the CNS Vital Signs embedded validity indicators, however, were quite low for all of the subtests (sensitivity = 0.04–0.29) and most of the domains, suggesting that these indicators may not be able to detect invalid test scores (assuming that unusually low performance on the TOMM or VSVT would also be associated with questionably valid scores on CNS VS). In other words, the validity indicators might not detect the majority of children and adolescents who are identified as having invalid scores on stand-alone PVTs such as the TOMM and VSVT (i.e., they have a high miss rate). Given that the specificity is so high, it might be helpful to investigate embedded validity indicators in a pediatric sample that is instructed to simulate malingering. Simulation research might advance our understanding of deliberately poor effort in children and adolescents, and lay the foundation for the development of more sensitive embedded indicators on CNS Vital Signs.

Based on the information available on the company website, several unpublished conference presentations, and the current study, three conclusions can be made. First, the extent to which the CNS Vital Signs embedded validity indicators identify deliberately poor performance, misunderstanding regarding how to take the test, or both is unknown. We cannot assume that a score flagged as invalid represents volitional poor performance; it might reflect misunderstanding or even cognitive impairment in some children. Second, there is a pressing need for analog malingering studies in children, adolescents, and adults to better understand how people underperform on this test battery. Third, more research is needed on how confusion or misunderstanding on the part of the participant, regarding the test instructions or procedures, influences the probability of being flagged by a validity indicator.

The low sensitivity and high specificity of the CNS Vital Signs embedded validity indicators in this sample is not unique. Prior studies have reported high specificity (>90%) at the sacrifice of lower sensitivity on the RDS (ranging from 20% in patients with epilepsy to 51% in patients with mTBI; Kirkwood et al., 2011; Loughan et al., 2012; Welsh et al., 2012) and the CMS Verbal Delayed Memory measures (11% in children referred for behavioral and academic reasons; Perna et al., 2014).

There are some limitations of this study. The sample consists of mixed neurological diagnoses, representing those patients who are referred for a neuropsychological assessment within our tertiary care hospital; it is not representative of all children and adolescents with a neurological diagnosis. However, it is representative of the population assessed within this specific environment, where a rapid evaluation of cognitive abilities may be desired in order to screen functioning. Second, there is a limitation on the stand-alone PVTs used for comparison in this study. Although there is burgeoning evidence for the use of the TOMM in pediatric samples (see DeRight & Carone, in press; Kirkwood, 2012 for a review), there is currently only one study illustrating the use of the VSVT in pediatric patients (Brooks, 2012).

On six of seven subtests, the CNS Vital Signs embedded validity indicators have low rates of being flagged (i.e., the exception is the Shifting Attention Test). Users should bear in mind, however, that when all validity indicators were considered simultaneously in the present study, nearly one in five children with a neurological diagnosis had a flagged score. Despite adequate-to-high specificity, most of the CNS Vital Signs embedded validity indicators have low sensitivity when compared with stand-alone PVTs. The exceptions may be the Complex Attention (sensitivity = .35) and Cognitive Flexibility (sensitivity = .32) domains, although these are below our a priori desired level of 0.5. Further studies on the utility of the CNS Vital Signs embedded validity indicators are needed, with studies in healthy children, those ask to simulate malingering, other clinical groups, and additional adult samples before a conclusive determination on the sensitivity, specificity, and clinical usefulness of these indicators can be achieved.

Conflict of Interest

Collection of cognitive data was supported through in-kind support from the publisher of the CNS Vital Signs test (i.e., test credits). Drs Brooks and Sherman currently receive funding from another test publisher, Psychological Assessment Resources, Inc. Drs Brooks and Sherman receive book royalties from Oxford University Press. Dr Iverson has received past research support from CNS Vital Signs, ImPACT Applications Systems, and Psychological Assessment Resources, Inc. He has not received research support from a publishing company during the past 3 years. None of the authors, nor their families, have a financial interest in CNS Vital Signs.

Acknowledgements

The authors thank Dr Helen Carlson for her assistance with data entry, manuscript formatting, and data management, as well as (alphabetically) Hussain Daya, Courtney Habina, Andrea Jubinville, Christianne Laliberté, Emily Tam, and Julie Wershler for data entry.

References

Baker
D. A.
Connery
A. K.
Kirk
J. W.
Kirkwood
M. W.
Embedded performance validity indicators within the California Verbal Learning Test, Children's Version
The Clinical Neuropsychologist
 , 
2014
, vol. 
28
 
1
(pg. 
116
-
127
)
Blaskewitz
N.
Merten
T.
Kathmann
N.
Performance of children on symptom validity tests: TOMM, MSVT, and FIT
Archives of Clinical Neuropsychology
 , 
2008
, vol. 
23
 
4
(pg. 
379
-
391
)
Brooks
B. L.
Victoria Symptom Validity Test performance in children and adolescents with neurological disorders
Archives of Clinical Neuropsychology
 , 
2012
, vol. 
27
 
8
(pg. 
858
-
868
)
Brooks
B. L.
Barlow
K. M.
A methodology for assessing treatment response in Hashimoto's encephalopathy: A case study demonstrating repeated computerized neuropsychological testing
Journal of Child Neurology
 , 
2011
, vol. 
26
 
6
(pg. 
786
-
791
)
Brooks
B. L.
Sherman
E. M.
Computerized neuropsychological testing to rapidly evaluate cognition in pediatric patients with neurologic disorders
Journal of Child Neurology
 , 
2012
, vol. 
27
 
8
(pg. 
982
-
991
)
Brooks
B. L.
Iverson
G. L.
Sherman
E. M.
Roberge
M. C.
Identifying cognitive problems in children and adolescents with depression using computerized neuropsychological testing
Applied Neuropsychology
 , 
2010
, vol. 
17
 
1
(pg. 
37
-
43
)
Brooks
B. L.
Sherman
E. M.
Krol
A. L.
Utility of TOMM Trial 1 as an indicator of effort in children and adolescents
Archives of Clinical Neuropsychology
 , 
2012
, vol. 
27
 
1
(pg. 
23
-
29
)
Bush
S. S.
Ruff
R. M.
Troster
A. I.
Barth
J. T.
Koffler
S. P.
Pliskin
N. H.
, et al.  . 
Symptom validity assessment: Practice issues and medical necessity NAN policy & planning committee
Archives of Clinical Neuropsychology
 , 
2005
, vol. 
20
 
4
(pg. 
419
-
426
)
Carone
D. A.
Children with moderate/severe brain damage/dysfunction outperform adults with mild-to-no brain damage on the Medical Symptom Validity Test
Brain Injury
 , 
2008
, vol. 
22
 
12
(pg. 
960
-
971
)
CNS Vital Signs
CNS Vital Signs test interpretation guide
2014
 
Cohen
M. J.
Children's memory scale
 , 
1997
San Antonio, TX
The Psychological Corporation
Constantinou
M.
McCaffrey
R. J.
Using the TOMM for evaluating children's effort to perform optimally on neuropsychological measures
Child Neuropsychology
 , 
2003
, vol. 
9
 
2
(pg. 
81
-
90
)
Delis
D. C.
Kramer
J. H.
Kaplan
E.
Ober
B. A.
California verbal learning test-childrens version
 , 
1994
San Antonio, TX
Psychological Corporation
Delis
D. C.
Kramer
J. H.
Kaplan
E.
Ober
B. A.
California verbal learning test – Second edition
 , 
2000
San Antonio, TX
Psychological Corporation
DeRight
J.
Carone
D. A.
Assessment of effort in children: A systematic review
Child Neuropsychology
 , 
in press
Donders
J.
Performance on the test of memory malingering in a mixed pediatric sample
Child Neuropsychology
 , 
2005
, vol. 
11
 
2
(pg. 
221
-
227
)
Donders
J.
Boonstra
T.
Correlates of invalid neuropsychological test performance after traumatic brain injury
Brain Injury
 , 
2007
, vol. 
21
 
3
(pg. 
319
-
326
)
Gioia
G. A.
Isquith
P. K.
Guy
S. C.
Kenworthy
L.
Behavior rating inventory of executive function
Child Neuropsychology
 , 
2000
, vol. 
6
 
3
(pg. 
235
-
238
)
Green
P.
Flaro
L.
Word Memory Test performance in children
Child Neuropsychology
 , 
2003
, vol. 
9
 
3
(pg. 
189
-
207
)
Green
P.
Flaro
L.
Courtney
J.
Examining false positives on the Word Memory Test in adults with mild traumatic brain injury
Brain Injury
 , 
2009
, vol. 
23
 
9
(pg. 
741
-
750
)
Greiffenstein
M.
Baker
W.
Gola
T.
Validation of malingered amnesia measures with a large clinical sample
Psychological Assessment
 , 
1994
, vol. 
6
 (pg. 
218
-
224
)
Gualtieri
C. T.
Vaishnavi
S.
Hervey
A.
Differentiating malingerers and patients with conversion disorder using a computerized neurocognitive battery
2009
Paper presented at the 20th annual meeting of the American Neuropsychiatric Association
San Francisco, CA
Heilbronner
R. L.
Sweet
J. J.
Morgan
J. E.
Larrabee
G. J.
Millis
S. R.
American Academy of Clinical Neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias, and malingering
Clinical Neuropsychology
 , 
2009
, vol. 
23
 
7
(pg. 
1093
-
1129
)
Kirk
J. W.
Harris
B.
Hutaff-Lee
C. F.
Koelemay
S. W.
Dinkins
J. P.
Kirkwood
M. W.
Performance on the Test of Memory Malingering (TOMM) among a large clinic-referred pediatric sample
Child Neuropsychology
 , 
2011
, vol. 
17
 
3
(pg. 
242
-
254
)
Kirkwood
M. W.
Sherman,
E. M. S.
Brooks
B. L.
Overview of tests and techniques to detect negative response bias in children
Pediatric forensic neuropsychology
 , 
2012
New York
Oxford University Press
(pg. 
136
-
161
)
Kirkwood
M. W.
Kirk
J. W.
The base rate of suboptimal effort in a pediatric mild TBI sample: Performance on the Medical Symptom Validity Test
The Clinical Neuropsychologist
 , 
2010
, vol. 
24
 
5
(pg. 
860
-
872
)
Kirkwood
M. W.
Hargrave
D. D.
Kirk
J. W.
The value of the WISC-IV Digit Span subtest in detecting noncredible performance during pediatric neuropsychological examinations
Archives of Clinical Neuropsychology
 , 
2011
, vol. 
26
 
5
(pg. 
377
-
384
)
Loughan
A. R.
Perna
R.
Performance and specificity rates in the test of memory malingering: An investigation into pediatric clinical populations
Applied Neuropsychology: Child
 , 
2014
, vol. 
3
 
1
(pg. 
26
-
30
)
Loughan
A. R.
Perna
R.
Hertza
J.
The value of the Wechsler intelligence scale for children-fourth edition digit span as an embedded measure of effort: An investigation into children with dual diagnoses
Archives of Clinical Neuropsychology
 , 
2012
, vol. 
27
 
7
(pg. 
716
-
724
)
MacAllister
W. S.
Nakhutina
L.
Bender
H. A.
Karantzoulis
S.
Carlson
C.
Assessing effort during neuropsychological evaluation with the TOMM in children and adolescents with epilepsy
Child Neuropsychology
 , 
2009
, vol. 
15
 
6
(pg. 
521
-
531
)
Perna
R. B.
Loughan
A. R.
Children and the test of memory malingering: Is one trial enough?
Child Neuropsychology
 , 
2013
, vol. 
19
 
4
(pg. 
438
-
447
)
Perna
R.
Loughan
A. R.
Hertza
J.
Segraves
K.
The value of embedded measures in detecting suboptimal effort in children: An investigation into the WISC-IV Digit Span and CMS verbal memory subtests
Applied Neuropsychology: Child
 , 
2014
, vol. 
3
 
1
(pg. 
45
-
51
)
Rienstra
A.
Spaan
P. E.
Schmand
B.
Validation of symptom validity tests using a “child-model” of adult cognitive impairments
Archives of Clinical Neuropsychology
 , 
2010
, vol. 
25
 
5
(pg. 
371
-
382
)
Rohling
M.
Hill
B.
Ploetz
D.
Womble
M.
Shenessey
J.
Embedded validity measures for a computerized cognitive test battery
Archives of Clinical Neuropsychology
 , 
2011
, vol. 
26
 pg. 
563
 
Saffer
B. Y.
Iverson
G. I.
Kjernisted
K.
McIntosh
D.
Identifying invalid computerized cognitive test performance in patients with depression
Journal of the International Neuropsychological Society
 , 
2013
, vol. 
19
 
S1
pg. 
131
 
Saffer
B. Y.
Rosenblatt
C. K.
Lanting
S. C.
Koehle
M. S.
Iverson
G. L.
Identifying invalid computerized cognitive test performance in older adults
Journal of the International Neuropsychological Society
 , 
2013
, vol. 
19
 
S1
pg. 
131
 
Seidel
W. T.
Joschko
M.
Assessment of attention in children
The Clinical Neuropsychologist
 , 
1991
, vol. 
5
 
1
(pg. 
53
-
66
)
Sherman
E. M. S.
Brooks
B. L.
Pediatric forensic neuropsychology
 , 
2012
New York
Oxford University Press
Slick
D. J.
Hopp
G.
Strauss
E.
Thompson
G. B.
Victoria Symptom Validity Test
 , 
1997
Odessa, FL
Psychological Assessment Resources
Tombaugh
T.
Test of Memory of Malingering (TOMM)
 , 
1996
North Tonawanda, NY
Multi-Health Systems
Wechsler
D.
Wechsler Adult Intelligence Scale, third edition, administration and scoring manual
 , 
1997
San Antonio, TX
The Psychological Corporation
Wechsler
D.
Wechsler Intelligence Scale for Children, fourth edition
 , 
2003a
San Antonio, TX
The Psychological Corporation
Wechsler
D.
Wechsler Intelligence Scale for Children, fourth edition. Technical and interpretive manual
 , 
2003b
San Antonio, TX
The Psychological Corporation
Wechsler
D.
Wechsler Intelligence Scale for Children, fourth edition, Canadian
 , 
2004
San Antonio, TX
The Psychological Corporation
Wechsler
D.
Wechsler Adult Intelligence Scale, fourth edition
 , 
2008a
San Antonio, TX
Pearson
Wechsler
D.
Wechsler Adult Intelligence Scale, fourth edition, Canadian
 , 
2008b
San Antonio, TX
Pearson
Welsh
A. J.
Bender
H. A.
Whitman
L. A.
Vasserman
M.
MacAllister
W. S.
Clinical utility of reliable digit span in assessing effort in children and adolescents with epilepsy
Archives of Clinical Neuropsychology
 , 
2012
, vol. 
27
 
7
(pg. 
735
-
741
)