Abstract

Recently published practice standards recommend that multiple effort indicators be interspersed throughout neuropsychological evaluations to assess for response bias, which is most efficiently accomplished through use of effort indicators from standard cognitive tests already included in test batteries. The present study examined the utility of a timed recognition trial added to standard administration of the WAIS-III Digit Symbol subtest in a large sample of “real world” noncredible patients (n=82) as compared with credible neuropsychology clinic patients (n=89). Scores from the recognition trial were more sensitive in identifying poor effort than were standard Digit Symbol scores, and use of an equation incorporating Digit Symbol Age–Corrected Scaled Scores plus accuracy and time scores from the recognition trial was associated with nearly 80% sensitivity at 88.7% specificity. Thus, inclusion of a brief recognition trial to Digit Symbol administration has the potential to provide accurate assessment of response bias.

Introduction

Within the past 15 years there has been a groundswell of attention and research focusing on techniques to detect feigned cognitive performances in the context of medical legal, disability-seeking, and correctional neuropsychological evaluations. Many free-standing effort measures are available (Boone, 2007a; Larrabee, 2007), although there is an increasing interest in identifying indices of response bias that can be obtained from standard cognitive tests. The use of such indicators has two important advantages: (i) the tests serve “double duty” in terms of providing data on both effort and discrete cognitive skills and (ii) test-takers are less likely to be coached or educated regarding such indices because the tests do not have single identities as effort measures and are less likely to be accessed on internet searches of malingering tests.

Current practice standards within the field of clinical neuropsychology recommend that multiple measures of response bias be interspersed throughout a neuropsychological test battery (AACN, 2007; Bush et al., 2005; Heilbronner, Sweet, Morgan, Larrabee, & Millis, 2009) to continuously sample of effort. Ideally, every standard cognitive measure should have an embedded effort indicator. Unfortunately, decisions on whether response bias is present on many neuropsychological tasks must be extrapolated from performance on effort measures administered before or after the target test.

The Wechsler Adult Intelligence Scale-III (WAIS-III) has been a core component of most neuropsychological evaluations and may constitute a third or more of total battery administration time. However, effort indicators from only one subtest, Digit Span, have been adequately validated and are in common use, and unfortunately, sensitivity of Digit Span as a measure of response bias is only moderate (approximately 36%–57%; see Babikian & Boone, 2007, for review). Thus, while Digit Span effort indices can be used to “rule in” noncredible performance (i.e., failed performance is generally specific to suboptimal effort), they cannot be used to “rule out” feigning (i.e., a person passing the indices may simply be in the large subset of noncredible individuals who score within normal limits).

The question arises as to whether additional effort indices derived from other WAIS-III subtests can be used to supplement data from Digit Span and perhaps be shown to be more effective than Digit Span in identification of poor effort.

A few previous studies have been conducted using the Digit Symbol subtest as an effort indicator. Trueblood (1994) reported that a WAIS-R Digit Symbol scaled score of ≤4 achieved 100% specificity, yet only 33% sensitivity, in a sample of credible and noncredible mild head-injured patients. Inman and Berry (2002) also obtained an adequate specificity (100%) in normal and head-injured controls when applying a cut-off of ≤4 for WAIS-III Digit Symbol ACSS. However, their sensitivity rate was only 2% for college student simulators. Etherton, Bianchini, Heinly, and Greve (2006) examined WAIS-III Digit Symbol performance in chronic pain patients, moderate to severe brain injury patients, and patients with diagnosed memory disorders. When using an ACSS cut-off of ≤4, moderate sensitivity was obtained for a malingering chronic pain group (66%) and specificity was excellent in credible chronic pain patients (96%), but was inadequate in moderate to severe brain-injured patients (79%), and minimally adequate in memory-disordered patients (89%).

In summary, the studies cited above generally documented low sensitivity rates, and with regard to the Etherton and colleagues (2006) study, specificity rates above or equal to 90% were not obtained in some patient subgroups. Furthermore, the studies appear to have limited generalizability. Trueblood's (1994) mild brain injury control group subjects were probably functioning comparably with normal individuals, given literature showing no permanent cognitive sequelae from mild traumatic brain injury (Belanger, Curtiss, Demery, Lebowitz, & Vanderploeg, 2005; Belanger & Vanderploeg, 2005; Carroll, Cassidy, Holm, Kraus, & Coronado, 2004; Dikmen, Machamer, Winn, & Temkin, 1995; Frencham, Fox, & Maybery, 2005; Schretlen & Shapiro, 2003). Thus, this study cannot be used to determine specificity rates for individuals with actual cognitive dysfunction. Further, Inman and Berry's (2002) study employed college student simulators rather than actual noncredible subjects, and thus, their published sensitivity rates are probably unreliable (as suggested by the 2% hit rate). Also, the use of normals to determine specificity enabled them to select a higher cut-off score than could likely be used in patients with actual cognitive dysfunction.

Given the moderate to high sensitivity of recognition memory paradigms (63%–89%; Boone, Lu, & Wen, 2005; Boone, Salazar, Lu, Warner-Chacon, & Razani, 2002; Kim et al., in press; Lu, Boone, Cozolino, & Mitchell, 2003; Nitch, Boone, Wen, Arnold, & Alfano, 2006; Sherman, Boone, Lu, & Razani, 2002) and time scores (37%–79%; Arnold et al., 2005; Babikian, Boone, Lu, & Arnold, 2006; Boone, Lu, & Herzberg, 2002a, 2002b; Kim et al., 2010) in the identification of suspect effort, the question arises as to whether inclusion of a timed recognition trial after standard Digit Symbol administration has the potential to increase its effectiveness as an effort indicator. The purpose of the present study was to investigate the effectiveness of such a modification of the WAIS-III Digit Symbol test in a large sample of “real world” noncredible patients. In addition, given reports that Digit Symbol rotational errors may be specific to feigning (Binder, 1992), the decision was made to assess whether tabulation of 180° false-positive errors on the recognition task could enhance the recognition trial sensitivity.

Materials and Methods

Subjects

Subjects were referred for neuropsychological assessment to the Harbor-UCLA Medical Center Department of Psychiatry Outpatient Neuropsychology Service or the private practice of the second author. Patients evaluated in the former setting were primarily referred by treating psychiatrists or neurologists for diagnostic clarification, case management, and/or determination of appropriateness for disability compensation. Patients tested in the latter setting were either evaluated in the context of litigation or at the request of private disability carriers. IRB approval to examine the archival data was obtained from the hospital-affiliated research institute (Los Angeles Biomedical Institute). All participants were fluent in English and most were native English-speakers. Criteria for inclusion and exclusion within noncredible and credible groups are described below, including the use of performance on measures of response bias for group assignment. Owing to the clinical nature of the data and the evolving nature of the neuropsychological test battery, not all nine effort indicators were administered to all subjects (particularly for earlier evaluations). Therefore, potential subjects were excluded from the study if their files contained data for less than three effort indicators.

Patients with Suspect Effort

In total, 82 patients met the Slick, Sherman, and Iverson (1999) criteria for probable malingered neurocognitive dysfunction. Specifically, all were in litigation or seeking to obtain disability benefits for cognitive symptoms associated with alleged medical or psychiatric disorders; all failed two or more independent effort indicators (tests and cut-offs listed in Table 1) not due to other psychiatric, neurologic, or developmental disorders; and low standard cognitive scores were at variance with evidence of normal function in activities of daily living. Demographic information is contained in Table 2. Presenting diagnoses (i.e., the conditions claimed by the subjects at the time of evaluation) are provided in Table 3.

Table 1.

Effort indicators used for group assignment

Free standing effort indicators 
 Rey 15-Item Test plus recognition combination score (Boone et al., 2002<20 
 Dot Counting Test E-score (Boone, Lu, & Herzberg, 2002a≥17 
 Warrington Recognition Memory Test—words (Iverson & Franzen, 1994<33 
 b-test E-score (Boone, Lu, & Herzberg, 2002b≥150 
 Rey Word Recognition Test (total correctly recognized) 
  Men ≤5 
  Women (Nitch et al., 2006≤7 
Embedded effort indicators 
 WAIS-III Reliable Digit Span (Babikian et al., 2006≤6 
 Rey Auditory Verbal Learning Test Effort Equation (Boone, et al., 2005≤12 
 RO/RAVLT discriminant function (Sherman et al., 2002≤−.40 
 Finger Tapping Test (dominant hand average across three trials) 
  Men ≤35 
  Women (Arnold et al., 2005≤28 
Free standing effort indicators 
 Rey 15-Item Test plus recognition combination score (Boone et al., 2002<20 
 Dot Counting Test E-score (Boone, Lu, & Herzberg, 2002a≥17 
 Warrington Recognition Memory Test—words (Iverson & Franzen, 1994<33 
 b-test E-score (Boone, Lu, & Herzberg, 2002b≥150 
 Rey Word Recognition Test (total correctly recognized) 
  Men ≤5 
  Women (Nitch et al., 2006≤7 
Embedded effort indicators 
 WAIS-III Reliable Digit Span (Babikian et al., 2006≤6 
 Rey Auditory Verbal Learning Test Effort Equation (Boone, et al., 2005≤12 
 RO/RAVLT discriminant function (Sherman et al., 2002≤−.40 
 Finger Tapping Test (dominant hand average across three trials) 
  Men ≤35 
  Women (Arnold et al., 2005≤28 
Table 2.

Group comparisons on demographic and digit symbol variables

 Credible patients (N = 89) Noncredible patients (N = 82) t p 
Age 44.25 ± 13.53 42.95 ± 11.88 −.66 .51 
Education 13.07 ± 2.82 12.80 ± 3.48 −.54 .59 
Gender distribution m = 42, f = 47 m = 49, f = 33   
Ethnicity 
 African American 13 (15%) 33 (40%)   
 Asian 4 (5%) 4 (5%)   
 Caucasian 38 (42%) 27 (33%)   
 Hispanic 21 (24%) 9 (11%)   
 Middle Eastern 3 (3%) 3 (4%)   
 Native American 1 (1%) 1 (1%)   
 Other 9 (10%) 5 (6%)   
Digit symbol age corrected scaled score 7.60 ± 2.91 (n = 89) 4.70 ± 1.89 (n = 82) −7.87 <.0001 
Digit symbol raw score 55.26 ± 17.40 (n = 89) 37.88 ± 14.72 (n = 82) −7.10 <.0001 
Digit symbol recognition correct 7.26 ± 1.43 (n = 89) 4.66 ± 2.09 (n = 82) −9.42 <.0001 
Digit symbol recognition time (in seconds) 39.79 ± 20.79 (n = 71) 80.22 ± 36.45 (n = 59) 7.56 <.0001 
Digit symbol recognition 180° rotations 0.61 ± .78 (n = 89) 1.12 ± 1.07 (n = 82) 3.58 <.0001 
Digit symbol recognition equation 110.78 ± 48.9 (N = 71) 19.44 ± 54.4 (N = 59) −10.08 <.0001 
 Credible patients (N = 89) Noncredible patients (N = 82) t p 
Age 44.25 ± 13.53 42.95 ± 11.88 −.66 .51 
Education 13.07 ± 2.82 12.80 ± 3.48 −.54 .59 
Gender distribution m = 42, f = 47 m = 49, f = 33   
Ethnicity 
 African American 13 (15%) 33 (40%)   
 Asian 4 (5%) 4 (5%)   
 Caucasian 38 (42%) 27 (33%)   
 Hispanic 21 (24%) 9 (11%)   
 Middle Eastern 3 (3%) 3 (4%)   
 Native American 1 (1%) 1 (1%)   
 Other 9 (10%) 5 (6%)   
Digit symbol age corrected scaled score 7.60 ± 2.91 (n = 89) 4.70 ± 1.89 (n = 82) −7.87 <.0001 
Digit symbol raw score 55.26 ± 17.40 (n = 89) 37.88 ± 14.72 (n = 82) −7.10 <.0001 
Digit symbol recognition correct 7.26 ± 1.43 (n = 89) 4.66 ± 2.09 (n = 82) −9.42 <.0001 
Digit symbol recognition time (in seconds) 39.79 ± 20.79 (n = 71) 80.22 ± 36.45 (n = 59) 7.56 <.0001 
Digit symbol recognition 180° rotations 0.61 ± .78 (n = 89) 1.12 ± 1.07 (n = 82) 3.58 <.0001 
Digit symbol recognition equation 110.78 ± 48.9 (N = 71) 19.44 ± 54.4 (N = 59) −10.08 <.0001 
Table 3.

Frequency of diagnoses by group

Diagnosis Credible N Noncredible N 
Alcohol/substance abuse 
Aneurysm/stroke 
Anxiety disorder 
Asperger's disorder 
Attention deficit hyperactivity disorder 
Bipolar disorder 
Borderline personality disorder 
Brain tumor 
Chronic fatigue syndrome 
Cognitive disorder not otherwise specified 
Depressive disorder 18 10 
Factitious disorder 
Fibromyalgia 
HIV 
Learning disability 11 
Mild mental retardation 
Multiple sclerosis 
Narcolepsy 
Neurosyphilis 
Obsessive compulsive disorder 
Posttraumatic stress disorder 
Psychosis 10 
Psychotic depression 
Seizure disorder 
Somatoform disorder 
Toxic mold exposure 
Traumatic brain injury   
 Mild 28 
 Moderate 
 Severe 
Total 89 82 
Diagnosis Credible N Noncredible N 
Alcohol/substance abuse 
Aneurysm/stroke 
Anxiety disorder 
Asperger's disorder 
Attention deficit hyperactivity disorder 
Bipolar disorder 
Borderline personality disorder 
Brain tumor 
Chronic fatigue syndrome 
Cognitive disorder not otherwise specified 
Depressive disorder 18 10 
Factitious disorder 
Fibromyalgia 
HIV 
Learning disability 11 
Mild mental retardation 
Multiple sclerosis 
Narcolepsy 
Neurosyphilis 
Obsessive compulsive disorder 
Posttraumatic stress disorder 
Psychosis 10 
Psychotic depression 
Seizure disorder 
Somatoform disorder 
Toxic mold exposure 
Traumatic brain injury   
 Mild 28 
 Moderate 
 Severe 
Total 89 82 

Credible Patients

The 89 credible subjects were not in litigation or seeking to obtain disability benefits for cognitive symptoms and failed ≤1 effort indicator (tests and cut-offs listed in Table 1; patients who failed 1 effort test were retained in the sample because research shows that failure on a single effort indicator among several is not unusual in credible populations; Victor, Boone, Serpa, Beuhler, & Ziegler, 2009). Patients with a full-scale IQ (FSIQ) of <70 or a dementia or amnestic disorder diagnosis were excluded. Demographic data are provided in Table 2, and final diagnoses (i.e., determined by history and cognitive test results) are listed in Table 3.

Instruments/Procedures

The Digit Symbol subtest was administered in standard format as part of the WAIS-III; administration of the WAIS-III occurred within the first third of the test battery. The recognition trial, consisting of four multiple choice options for each of the nine numbers in the Digit Symbol task (contact the second author for recognition trial stimuli), was presented to patients immediately following administration of the Digit Symbol subtest (the optional Incidental Learning procedure was not administered). For five numbers, foils were included which represented a 180° rotation of the correct answer. Subjects were handed the recognition trial page and instructed:

On this page are nine numbers with a choice of four symbols for each number. I want you to circle the symbol that went with each number in the task you just completed.

Recognition trial performance was timed.

The following five scores were examined for their ability to differentiate credible versus noncredible effort: Digit Symbol Age Corrected Scaled Score (ACSS), Digit Symbol Raw Score, and three variables from the recognition trial (Recognition Correct, Recognition Time (in seconds), and Recognition 180° Rotations).

Results

As shown in Table 2, groups did not differ significantly in age or educational level.

Correlational analyses were conducted separately within each group to examine whether Digit Symbol scores were significantly related to age or education (data are reproduced in Table 4). In both the credible and noncredible groups, a significant negative correlation was observed between age and Digit Symbol Raw Score, however, no significant correlations were found between age and the other Digit Symbol scores. In both groups, education was significantly correlated with both the Digit Symbol Raw Score and ACSS, but was not significantly correlated with any of the variables from the Recognition trial.

Table 4.

Correlational analyses between digit symbol scores and demographic variables

Digit symbol scores Demographic data
 
Digit symbol data recognition scores
 
 Age Education ACSS Raw Correct Time 180 
Credible subjects 
 ACSS 0.08 0.41* — 0.83* 0.34* −0.34* .03 
 Raw score −0.33* 0.35* 0.83* — 0.33* −0.38* 0.04 
 Recognition correct −0.10 0.20 0.34* 0.33* — −0.46* −.52* 
 Recognition time 0.06 −0.15 −0.34* −0.38* −0.46* — .12 
 180° rotation recognition equation −0.02 0.07 0.03 0.04 −0.52* 0.12 — 
Noncredible subjects 
 ACSS −0.08 0.42* — 0.93* 0.40* −0.22 −0.15 
 Raw score −0.31* 0.38* 0.93* — 0.42* −0.25 −0.13 
 Recognition correct 0.01 0.04 0.40* 0.42* — −0.17 −0.40 
 Recognition time 0.02 0.10 −0.22 −0.25 −0.17 — 0.16 
 180° rotations −0.13 0.07 −0.15 −0.13 −0.40* 0.16 — 
Digit symbol scores Demographic data
 
Digit symbol data recognition scores
 
 Age Education ACSS Raw Correct Time 180 
Credible subjects 
 ACSS 0.08 0.41* — 0.83* 0.34* −0.34* .03 
 Raw score −0.33* 0.35* 0.83* — 0.33* −0.38* 0.04 
 Recognition correct −0.10 0.20 0.34* 0.33* — −0.46* −.52* 
 Recognition time 0.06 −0.15 −0.34* −0.38* −0.46* — .12 
 180° rotation recognition equation −0.02 0.07 0.03 0.04 −0.52* 0.12 — 
Noncredible subjects 
 ACSS −0.08 0.42* — 0.93* 0.40* −0.22 −0.15 
 Raw score −0.31* 0.38* 0.93* — 0.42* −0.25 −0.13 
 Recognition correct 0.01 0.04 0.40* 0.42* — −0.17 −0.40 
 Recognition time 0.02 0.10 −0.22 −0.25 −0.17 — 0.16 
 180° rotations −0.13 0.07 −0.15 −0.13 −0.40* 0.16 — 

*p<.01

Correlations were also computed between Digit Symbol scores to examine the extent of redundancy. In both groups, ACSS and Raw Scores were only modestly correlated with the Recognition Correct and Recognition Time scores, and were not significantly related to Recognition 180° Rotations. Recognition Correct and Recognition Time scores were modestly related to each other in the credible group, and lastly, Recognition Correct was found to be significantly correlated with Recognition 180° Rotations in both groups; no other significant correlations were observed.

As shown in Table 2, significant differences were observed between the two groups across all the Digit Symbol variables, with the noncredible group consistently performing worse than the credible group.

Score frequency counts were computed for the five scores for each group separately for the purposes of assessing sensitivity of cut-scores selected for ≥89% specificity. As can be seen from Table 5, the highest sensitivity values at adequate specificity (≥89%) were observed for recognition scores. The modest correlations among the recognition scores, and between the recognition scores and standard Digit Symbol scores, indicated that the data were not redundant, and suggested that by combining scores, sensitivity might be increased with no sacrifice to specificity. The following combination of scores (with accuracy data multiplied by 10 to provide comparable weight to time scores) achieved the highest sensitivity: 

formula

Table 5.

Sensitivity and specificity values for various digit symbol scores

Cut-off scores Sensitivity Specificity 
ACSS n = 82 n = 89 
≤2 8.5 100.0 
≤3 18.3 96.6 
≤4 57.3 84.3 
≤5 78.0 75.3 
≤6 90.2 59.6 
Raw score n = 82 n = 89 
≤26 18.3 100.0 
≤29 23.2 95.5 
≤30 25.6 93.3 
≤32 34.1 91.0 
≤33 40.2 89.9 
≤35 48.8 87.6 
Recognition correct n = 82 n = 89 
≤2 14.6 100.0 
≤3 32.9 98.9 
≤4 50.0 96.6 
≤5 58.5 88.8 
≤6 80.5 69.7 
≤7 91.5 48.3 
≤8 97.6 23.6 
Recognition time n = 59 n = 71 
≥97.00 33.9 100.0 
≥82.00 39.3 91.5 
≥72.00 49.2 88.7 
≥70.00 50.8 87.3 
≥65.00 54.1 85.9 
≥61.00 59.0 83.1 
Recognition 180° rotations n = 82 n = 89 
≥1 65.9 52.8 
≥2 32.9 89.9 
≥3 11.0 97.8 
≥4 1.2 98.9 
≥5 1.2 100.0 
Recognition equation n = 59 n = 71 
≤34.00 57.6 94.4 
≤35.00 57.6 93.0 
≤36.00 59.3 93.0 
≤37.00 61.0 93.0 
≤38.00 62.7 91.5 
≤43.00 64.4 91.5 
≤46.00 66.1 91.5 
≤47.00 67.8 91.5 
≤49.00 69.5 90.1 
≤50.00 72.9 90.1 
≤53.00 76.3 90.1 
≤55.00 78.0 90.1 
≤56.00 78.0 88.7 
≤57.00 79.9 88.7 
≤58.00 81.4 87.3 
≤60.00 83.1 87.3 
≤61.00 84.7 87.3 
≤64.00 84.7 85.9 
≤65.00 84.7 83.1 
≤67.00 86.4 83.1 
≤69.00 86.4 81.7 
≤71.00 86.4 80.3 
≤72.00 86.4 78.9 
≤73.00 86.4 77.5 
≤78.00 86.4 76.1 
≤79.00 88.1 74.6 
Cut-off scores Sensitivity Specificity 
ACSS n = 82 n = 89 
≤2 8.5 100.0 
≤3 18.3 96.6 
≤4 57.3 84.3 
≤5 78.0 75.3 
≤6 90.2 59.6 
Raw score n = 82 n = 89 
≤26 18.3 100.0 
≤29 23.2 95.5 
≤30 25.6 93.3 
≤32 34.1 91.0 
≤33 40.2 89.9 
≤35 48.8 87.6 
Recognition correct n = 82 n = 89 
≤2 14.6 100.0 
≤3 32.9 98.9 
≤4 50.0 96.6 
≤5 58.5 88.8 
≤6 80.5 69.7 
≤7 91.5 48.3 
≤8 97.6 23.6 
Recognition time n = 59 n = 71 
≥97.00 33.9 100.0 
≥82.00 39.3 91.5 
≥72.00 49.2 88.7 
≥70.00 50.8 87.3 
≥65.00 54.1 85.9 
≥61.00 59.0 83.1 
Recognition 180° rotations n = 82 n = 89 
≥1 65.9 52.8 
≥2 32.9 89.9 
≥3 11.0 97.8 
≥4 1.2 98.9 
≥5 1.2 100.0 
Recognition equation n = 59 n = 71 
≤34.00 57.6 94.4 
≤35.00 57.6 93.0 
≤36.00 59.3 93.0 
≤37.00 61.0 93.0 
≤38.00 62.7 91.5 
≤43.00 64.4 91.5 
≤46.00 66.1 91.5 
≤47.00 67.8 91.5 
≤49.00 69.5 90.1 
≤50.00 72.9 90.1 
≤53.00 76.3 90.1 
≤55.00 78.0 90.1 
≤56.00 78.0 88.7 
≤57.00 79.9 88.7 
≤58.00 81.4 87.3 
≤60.00 83.1 87.3 
≤61.00 84.7 87.3 
≤64.00 84.7 85.9 
≤65.00 84.7 83.1 
≤67.00 86.4 83.1 
≤69.00 86.4 81.7 
≤71.00 86.4 80.3 
≤72.00 86.4 78.9 
≤73.00 86.4 77.5 
≤78.00 86.4 76.1 
≤79.00 88.1 74.6 

As can be seen in Table 2, groups differed significantly in scores on the above equation. As shown in Table 5, an equation cut-off score of ≤57 was associated with 89% specificity in the credible group, while identifying more than three-fourths of the noncredible subjects (79.9% sensitivity; ROC area = 0.90, SE = .028). With the use of this equation, sensitivity was increased by nearly 20 percentage-points over the highest value for any individual score (i.e., a cut off score of ≤5 on Recognition Correct represented sensitivity of 58.5%). Examination of the noncredible mild traumatic brain injury patients in isolation (n = 28) showed that a cut-off of ≤57 identified 74% of this subgroup, showing that the detection rate was comparable with that of the larger heterogeneous noncredible population.

No significant correlations were obtained between age or education and equation scores in the noncredible group. In the credible group, age was not significantly related to the equation score, and while education was significantly correlated with equation performance, the amount of variance accounted for by education was <10%.

Positive and negative predictive values for the Digit Symbol recognition equation at base rates of 15%, 30%, and 50% noncredible subjects are reproduced in Table 6. These data document good negative predictive power (the likelihood that a subject passing the indicator is actually credible) for the equation, as well as high positive predictive power (the likelihood that a subject failing the indicator is actually noncredible), particularly at higher base rates.

Table 6.

Positive predictive values (PPV) and negative predictive values (NPV) for digit symbol recognition equation at base rates of 15%, 30%, and 50%

Cut-off scores 15% Base rate
 
30% Base rate
 
50% Base rate
 
 PPV (%) NPV (%) PPV (%) NPV (%) PPV (%) NPV (%) 
≤35 60.0 92.5 78.4 83.6 87.2 72.5 
≤38 57.1 93.0 76.2 85.3 86.0 74.7 
≤49 56.3 94.2 74.5 87.1 85.4 78.0 
≤55 58.2 95.9 77.2 90.5 88.7 80.4 
≤57 55.5 96.2 75.2 91.1 87.6 81.5 
≤65 46.8 96.8 68.3 92.6 80.6 86.8 
≤71 43.1 93.5 64.7 92.3 78.5 87.7 
≤79 38.3 97.3 60.0 93.8 74.3 88.3 
Cut-off scores 15% Base rate
 
30% Base rate
 
50% Base rate
 
 PPV (%) NPV (%) PPV (%) NPV (%) PPV (%) NPV (%) 
≤35 60.0 92.5 78.4 83.6 87.2 72.5 
≤38 57.1 93.0 76.2 85.3 86.0 74.7 
≤49 56.3 94.2 74.5 87.1 85.4 78.0 
≤55 58.2 95.9 77.2 90.5 88.7 80.4 
≤57 55.5 96.2 75.2 91.1 87.6 81.5 
≤65 46.8 96.8 68.3 92.6 80.6 86.8 
≤71 43.1 93.5 64.7 92.3 78.5 87.7 
≤79 38.3 97.3 60.0 93.8 74.3 88.3 

Recommended cut-off in bold.

Discussion

In the current study, a large sample of “real world” noncredible subjects performed significantly worse than a large sample of credible patients on Digit Symbol ACSS and raw score, and on variables from a specially constructed Digit Symbol recognition trial (total correct, time, and number of 180° false-positive errors). Digit Symbol ACSS and raw score cut-scores were associated with 18% and 40% sensitivity, respectively, at minimally adequate specificity (≥89%), while recognition cut-scores for total correct and time were associated with higher sensitivity rates (58.5% and 49.2%, respectively) at comparable specificity levels. Combining recognition correct, ACSS, and recognition time into a weighted equation increased sensitivity to 79.9% with no sacrifice in specificity. Examination of performance in the subset of noncredible patients claiming cognitive symptoms in the context of a mild traumatic brain injury showed generally comparable sensitivity (74%) when compared with the noncredible group as a whole.

A quarter of the credible patients (two of eight) who fell below cut-offs for the recognition equation carried diagnoses of somatoform disorder (a total of five subjects in the credible group were diagnosed with this condition). It is not unexpected that some patients with somatoform disorder would fail effort indicators given that somatoform conditions involve symptom fabrication, albeit nonconscious; nonconscious creation of nonorganic cognitive symptoms should not be any more accurate than conscious creation (malingering) of such deficits (Boone, 2007b). When somatoform subjects were excluded from the credible sample, the recognition equation cut-off score could be made more stringent (cut-off of ≤58), increasing sensitivity to 81.4% (while still maintaining specificity of ≥89%).

Three credible subjects who failed the equation had psychiatric disorders involving psychosis (out of a total of six in the credible group with this condition and in whom equation scores were available; 50.0%); in fact, a subset of psychotic individuals (particularly those who are low functioning as defined by low MMSE score and/or low education) typically fail effort indicators (Back et al., 1996; Goldberg, Back-Madruga, & Boone, 2007). Lowering the recognition equation cut-off to 0 correctly assigns 83% of subjects in this subgroup (five of six) while still identifying 83% of noncredible individuals presenting with psychotic symptoms. One of the six credible severe brain injury patients fell beyond cut-offs for the recognition equation (16.7%); lowering the cut-off to ≤49 provides 100% specificity in this subsample while still identifying 90% of noncredible moderate-to-severe brain-injured subjects.

One credible patient with major depression obtained a recognition equation score below the cut-off of ≤57, but given that there were 14 credible subjects with this diagnosis and who had data available to compute the recognition equation, this cut-off has adequate specificity (93%) in depressed individuals. In fact, the cut-off can be raised to ≤64 and still maintain 93% specificity. Both cut-off scores identified 75% of the noncredible depressed subjects who had recognition equation data available.

The remaining credible subject who fell beyond the cut-score for the recognition equation was diagnosed with cognitive disorder not otherwise specified.

Demographic variables did not appear to predict failed performance in credible subjects; in credible subjects falling beyond the equation cut-off, mean age was 46.9 years, mean educational level was 12.9 years, and 63% were female (when compared with a mean of 44.3 years of age, a mean of 13.1 years of education, and 53% female composition for the credible sample as a whole). Credible subjects with dementia, amnestic disorder, or FSIQ of <70 were excluded from the current study, given high rates of effort test failures in these populations despite adequate effort (Dean, Victor, Boone, & Arnold, 2008; Dean et al., 2009), and therefore Digit Symbol scores should not be used as a measure of response bias in these groups until further investigation and validation has been completed.

The Digit Symbol recognition stimuli included five foils that were 180° rotations of correct answers; these stimuli allowed investigation of the hypothesis that individuals feigning cognitive symptoms might overselect these incorrect options. Although group differences in number of 180° rotation errors were statistically significant, this variable in isolation was associated with a low sensitivity rate (32.9% at ≥90% specificity), and it did not contribute to an increased sensitivity when attempts were made to incorporate it in a recognition trial equation. Credible subjects committed on average less than one 180° rotational error, whereas noncredible patients made on average only slightly more than one 180° rotational error.

In conclusion, the available data suggest that scores from the standard Digit Symbol subtest are inadequate in identifying response bias, but that the addition of a brief recognition trial following standard administration of Digit Symbol has the potential to be a highly effective means of documenting noncredible effort. The recognition trial was administered directly after completion of Digit Symbol (the Incidential Learning procedure was not administered); thus, the recognition trial should only be used in those clinical situations that employ the same Digit Symbol administration procedures. In 2008, the WAIS-IV was published and contains a version of the Digit Symbol (Coding) subtest that differs from that in the WAIS-III. As a result, the recognition paradigm described in this manuscript cannot be imported for use with the WAIS-IV Coding subtest. Future research is needed to develop a recognition trial specific to the WAIS-IV Coding version.

Conflict of Interest

None declared.

References

American Board of Clinical Neuropsychology.
American Academy of Clinical Neuropsychology (AACN) practice guidelines for neuropsychological assessment and consultation
The Clinical Neuropsychologist
 , 
2007
, vol. 
21
 (pg. 
209
-
231
)
Arnold
G.
Boone
K.
Lu
P.
Dean
A.
Wen
J.
Nitch
S.
Sensitivity and specificity of Finger Tapping Test scores for the detection of suspect effort
The Clinical Neuropsychologist
 , 
2005
, vol. 
19
 (pg. 
105
-
120
)
Babikian
T.
Boone
K.
Boone
K.
Intelligence tests as measures of effort
Assessment of feigned cognitive impairment: A neuropsychological perspective.
 , 
2007
New York
Guilford Press
Babikian
T.
Boone
K. B.
Lu
P.
Arnold
G.
Sensitivity and specificity of various Digit Span scores in the detection of suspect effort
The Clinical Neuropsychologist
 , 
2006
, vol. 
20
 (pg. 
145
-
159
)
Back
C.
Boone
K. B.
Edwards
C.
Parks
C.
Burgoyne
K.
Silver
B.
The performance of schizophrenics on three cognitive tests of malingering: Rey 15-item, Rey Dot Counting, and Forced Choice method
Assessment
 , 
1996
, vol. 
3
 (pg. 
449
-
458
)
Belanger
H. G.
Curtiss
G.
Demery
J. A.
Lebowitz
B. K.
Vanderploeg
R. D.
Factors moderating neuropsychological outcomes following mild traumatic brain injury: A meta-analysis
Journal of the International Neuropsychological Society
 , 
2005
, vol. 
11
 (pg. 
215
-
227
)
[PubMed]
Belanger
H. G.
Vanderploeg
R. D.
The neuropsychological impact of sports-related concussion: A meta-analysis
Journal of the International Neuropsychological Society
 , 
2005
, vol. 
11
 (pg. 
345
-
357
)
[PubMed]
Binder
L. M.
Puente
A.
McCaffrey
R.
Deception and malingering
Handbook of neuropsychological assessment: A biopsychosocial approach
 , 
1992
New York
Plenum
(pg. 
353
-
372
)
Boone
K. B.
Assessment of feigned cognitive impairment: A neuropsychological perspective.
 , 
2007
New York
Guilford Press
Boone
K. B.
Boone
K. B.
A reconsideration of the Slick et al. (1999) criteria for malingered neurocognitive dysfunction
Assessment of Feigned Cognitive Impairment: A neuropsychological perspective.
 , 
2007
New York
Guilford Press
Boone
K.
Lu
P.
Herzberg
D.
Dot counting test.
 , 
2002
Los Angeles
Western Psychological Services
Boone
K.
Lu
P.
Herzberg
D.
The b test.
 , 
2002
Los Angeles
Western Psychological Services
Boone
K.
Lu
P.
Wen
J.
Comparison of various RAVLT scores in the detection of noncredible memory performance
Archives of Clinical Neuropsychology
 , 
2005
, vol. 
20
 (pg. 
301
-
319
)
Boone
K.
Salazar
X.
Lu
P.
Warner-Chacon
I.
Razani
J.
The Rey-item Recognition Trial: A technique to enhance sensitivity of the Rey 15-item memorization test
Journal of Clinical and Experimental Neuropsychology
 , 
2002
, vol. 
24
 (pg. 
561
-
573
)
[PubMed]
Bush
S. S.
Ruff
R. M.
Troster
A. I.
Barth
J. T.
Koffler
S. P.
Pliskin
N. H.
, et al.  . 
Symptom validity assessment: Practice issues and medical necessity (NAN Policy and Planning Committees)
Archives of Clinical Neuropsychology
 , 
2005
, vol. 
20
 (pg. 
419
-
426
)
Carroll
L. J.
Cassidy
J. D.
Holm
L.
Kraus
J.
Coronado
V. G.
Methodological issues and research recommendations for mild traumatic brain injury: The WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury
Journal of Rehabilitation Medicine
 , 
2004
, vol. 
37
 (pg. 
61
-
62
)
Dean
A. C.
Victor
T. L.
Boone
K. B.
Arnold
G.
The relationship of IQ to effort test performance
The Clinical Neuropsychologist
 , 
2008
, vol. 
22
 (pg. 
705
-
722
)
Dean
A. D.
Victor
T. L.
Boone
K. B.
Philpott
L. M.
Hess
R. A.
The relationship of IQ to effort test performance
The Clinical Neuropsychologist
 , 
2008
, vol. 
22
 (pg. 
705
-
722
)
Dean
A. D.
Victor
T. L.
Boone
K. B.
Philpott
L. M.
Hess
R. A.
Dementia and effort test performance
The Clinical Neuropsychologist
 , 
2009
, vol. 
23
 (pg. 
133
-
152
)
Dikmen
S. S.
Machamer
J. E.
Winn
H. R.
Temkin
N. R.
Neuropsychological outcome at 1-year post head injury
Neuropsychology
 , 
1995
, vol. 
9
 (pg. 
80
-
90
)
Etherton
J. L.
Bianchini
K. J.
Heinly
M. T.
Greve
K. W.
Pain, malingering and performance on the WAIS-III Processing Speed Index
Journal of Clinical and Experimental Neuropsychology
 , 
2006
, vol. 
28
 (pg. 
1218
-
1237
)
Frencham
K. A.
Fox
A. M.
Maybery
M. T.
Neuropsychological studies of mild traumatic brain injury: a meta-analytic review of the research since 1995
Journal of Clinical and Experimental Neuropsychology
 , 
2005
, vol. 
27
 (pg. 
334
-
351
)
Goldberg
H. B.
Back-Madruga
C.
Boone
K. B.
Boone
K. B.
The impact of psychiatric disorders on cognitive symptom validity test scores
Assessment of feigned cognitive impairment: A neuropsychological perspective
 , 
2007
New York
Guilford Press
Heilbronner
R. L.
Sweet
J. J.
Morgan
J. E.
Larrabee
G.
Millis
S.
conference participants
American Academy of Clinical Neuropsychology Consensus Conference Statement on the neuropsychological assessment of effort, response bias, and malingering
The Clinical Neuropsychologist
 , 
2009
, vol. 
23
 (pg. 
1093
-
1129
)
Inman
T. H.
Berry
D. T. R.
Cross-validation of indicators of malingering: A comparison of nine neuropsychological tests, four tests of malingering and behavioral observations
Archives of Clinical Neuropsychology
 , 
2002
, vol. 
17
 (pg. 
1
-
23
)
[PubMed]
Iverson
G. L.
Franzen
M. D.
The Recognition Memory Test, Digit Span and Knox Cube Test as markers of malingered memory impairment
Assessment
 , 
1994
, vol. 
1
 (pg. 
323
-
3334
)
Kim
M. S.
Boone
K.
Victor
T.
Marion
S.
Amano
S.
Cottingham
M.
, et al.  . 
The Warrington Recognition Memory Test for words as a measure of response bias: Total score and response time cutoffs developed on “real world” credible and noncredible subjects
Archives of Clinical Neuropsychology
 , 
2010
, vol. 
25
 (pg. 
60
-
70
)
Larrabee
G. L.
Assessment of malingered neuropsychological deficits.
 , 
2007
New York
Oxford University Press
Lu
P. H.
Boone
K. B.
Cozolino
L.
Mitchell
C.
Effectiveness of the Rey–Osterrieth Complex Figure Test and the Meyers and Meyers recognition trial in the detection of suspect effort
The Clinical Neuropsychologist
 , 
2003
, vol. 
17
 (pg. 
426
-
440
)
[PubMed]
Nitch
S.
Boone
K. B.
Wen
J.
Arnold
G.
Alfano
K.
The utility of the Rey Word Recognition Test in the detection of suspect effort
The Clinical Neuropsychologist
 , 
2006
, vol. 
20
 (pg. 
873
-
887
)
Schretlen
D. J.
Shapiro
A. M.
A quantitative review of the effects of traumatic brain injury on cognitive functioning
International Review of Psychiatry
 , 
2003
, vol. 
15
 (pg. 
341
-
349
)
Sherman
D. S.
Boone
K. B.
Lu
P.
Razani
J.
Re-examination of a Rey Auditory Verbal Learning Test/Rey Complex Figure discriminant function to detect suboptimal effort
The Clinical Neuropsychologist
 , 
2002
, vol. 
16
 (pg. 
242
-
250
)
[PubMed]
Slick
D. J.
Sherman
E. M.
Iverson
G. L.
Diagnostic criteria for malingered neurocognitive dysfunction: Proposed standards for clinical practice and research
The Clinical Neuropsychologist
 , 
1999
, vol. 
13
 (pg. 
545
-
561
)
[PubMed]
Trueblood
W.
Qualitative and quantitative characteristics of malingered and other invalid WAIS-R and clinical memory data
Journal of Clinical and Experimental Neuropsychology
 , 
1994
, vol. 
16
 (pg. 
597
-
607
)
Victor
T.
Boone
K. B.
Serpa
G.
Beuhler
J.
Ziegler
E. A.
Interpreting the meaning of multiple effort test failure
The Clinical Neuropsychologist
 , 
2009
, vol. 
23
 (pg. 
297
-
313
)