Abstract

Several studies have examined the usefulness of the Warrington Recognition Memory Test–Words as a measure to detect suspect effort, although samples have generally been small and/or comprised of simulators rather than “real world” credible and noncredible patients. The current study examined the Warrington Recognition Memory Test–Words total score and response time of “real world” noncredible patients (as determined by motive to feign, failure on ≥2 independent measures of response bias, low cognitive scores inconsistent with normal ADLs; n = 190) versus credible patients (as determined by no motive to feign, failure of ≤1 measure of response bias; n = 124) derived from an archival database of individuals from the Harbor-UCLA Medical Center, Department of Psychiatry, Outpatient Neuropsychology Service, and the private practice of the second author. Noncredible patients obtained significantly lower total scores and longer times to complete the task. A total correct cutoff of ≤42 was found to have excellent specificity (91.9%) and sensitivity (88.9%), whereas a time cutoff of ≥207″ was associated with 65.5% sensitivity at 90.7% specificity, and when the time cut-score was used in combination with the total score cutoff, an additional 5% of the noncredible participants were captured, raising overall sensitivity to 93.7% (at 87.1% specificity). Thus, the Warrington Recognition Memory Test–Words, although not originally created for the purposes of measuring suspect effort, appears to be an excellent measure for detecting response bias on neuropsychological testing.

Introduction

There have recently been growing demands in the legal system to determine the credibility of an individual's claims of cognitive impairment, creating a need for expert neuropsychological testimony in cases of personal injury, criminal prosecution, employability, disability, worker's compensation, and medical malpractice (Larrabee, 2007; Nelson et al., 2003; Nies & Sweet, 1994; Slick, Sherman, & Iverson, 1999). Survey estimates suggest that approximately 30% of disability seeking, 29% of personal injury, and 19% of criminal cases display symptom exaggeration and probable malingering (Mittenberg, Patton, Canyock, & Condit, 2002).

The Warrington Recognition Memory Test–Words (Warrington, 1984), originally created to assess recognition memory, has more recently been used as a measure to detect suspect effort (Goldberg, Back-Madruga, & Boone, 2007; Iverson & Franzen, 1994, 1998; Millis, 1992, 1994; Millis & Putnam, 1994; Ross, Putnam, & Adams, 2006; Tardif, Barry, Fox, & Johnstone, 2000). On this test, individuals are presented with a series of 50 words and then administered trials in which they are shown pairs of words—a target and a foil—and instructed to select the item to which they were previously exposed. Owing to its forced-choice format, the Warrington Recognition Memory Test–Words is well-suited for detection of non-optimal performance and feigned memory impairment. Advantages of the test include administration brevity (single learning and recognition trials completed in ∼5 min), simplicity and portability of the test stimuli (one stimulus booklet and one stimulus page), and ease of scoring (i.e., tabulation of correct responses out of a possible of 50). In addition, the test shows only modest correlations with most other effort indicators (i.e., Rey 15-item, Dot Counting Test, Rey Word Recognition Test, Rey Auditory Verbal Learning Test recognition trial, Rey–Osterrieth effort equation, Digit Span, and b Test; Nelson et al., 2003) with the exception of the Word Memory Test (r = .731; unpublished data reported in Boone, 2009), indicating that it generally provides nonredundant information.

The few previous studies of the Warrington Recognition Memory Test–Words as a measure of effort have supported its utility. Millis (1992) compared performance in 10 mild traumatic brain injury (TBI) subjects pursuing compensation with that of 20 moderate to severe non-compensation-seeking TBI patients and found that the former group performed significantly worse. A cut score of 31 captured 70% of the mild TBI patients while misidentifying ≤10% of the moderate–severe TBI patients; lowering the cut score to 29 still identified half of the compensation seekers and did not misidentify any of the noncompensation seekers. Subsequently, Millis and Putnam (1994) observed that mild head trauma patients seeking compensation (n = 19) performed significantly worse on the Warrington Recognition Memory Test–Words than their counterparts who returned to work (n = 12), and poorer than patients with moderate and severe head injuries (n = 32; litigation status unknown). Using a cut score of <25, 29% of the compensation-seeking patients were captured, with misclassification of only 6% of the moderate/severe head-injured group, and none of the mild TBI patients who had returned to work.

Other studies have examined Warrington Recognition Memory Test–Words performance in individuals instructed to feign cognitive symptoms when compared with controls, TBI patients, and neuropsychology clinic patients with and without memory impairment (Cato, Brewster, Ryan, & Giuliano, 2002; Iverson & Franzen, 1994, 1998; Suhr & Gunstad, 2000; Wogar, van den Broek, Bradshaw, & Szabadi, 1998). Iverson and Franzen (1994, 1998) found that use of a cutoff of: (a) <33 did not misidentify any TBI patients (n = 20; 80% with severe head injury), (b) <38 did not misidentify any neuropsychology clinic referrals with (n = 20) or without (n = 20) memory impairment, and (c) <40 was associated with 90% specificity in TBI patients (n = 20; 80% severe) and patients with memory difficulties (n = 20). In contrast, a cutoff score of <38 correctly classified 95% of the experimental malingerers (n = 20).

In a study examining the effects of coaching on effort indicators, Suhr and Gunstad (2000) observed that undergraduate students simulating cognitive impairment (n = 73) performed significantly worse on the Warrington Recognition Memory Test–Words than participants providing their best effort (n = 33). Simulators who were told to malinger (half of whom were given information about head injury; n = 38) averaged 36.7, whereas those who were asked to malinger and were provided information about head injuries, as well as warnings about the presence of effort testing (n = 35), averaged 40.5. The researchers concluded that when individuals are warned about effort testing, sensitivity of forced-choice measures is lowered, but that they still remain effective.

In contrast, Cato and colleagues (2002) reported that non-litigating mild TBI patients (n = 21) performed near or below chance on the Warrington Recognition Memory Test–Words. The authors also administered the test to undergraduate students asked to perform with best effort (n = 26), to feign impairment with no instruction (n = 28), to feign impairment after being provided strategies (n = 27), or to feign impairment after being provided both strategies and examples (n = 28). The best effort group outperformed the remaining groups, whereas the malingering group with no instruction was the only group that performed comparably with the mild TBI group. Both groups that were provided strategies performed comparably on the task, scoring higher than the mild TBI group. The authors concluded that the Warrington Recognition Memory Test–Words is not an effective measure in the detection of feigned effort, but their findings regarding the performance of mild TBI patients are highly anomalous and raise questions regarding the integrity of the study.

Thus, previous studies have overall shown the usefulness of the Warrington Recognition Memory Test–Words in detecting feigned cognitive impairments, but the data are inconclusive due to methodological issues. The studies generally had small sample sizes (10–20 subjects in the simulating/malingering groups and 20–42 subjects in the comparison groups), raising questions regarding the reliability of the findings. In addition, the use of mild TBI patients seeking compensation by Millis (1992) and Millis and Putnam (1994) likely resulted in an underestimate of Warrington Recognition Memory Test–Words sensitivity in identifying suspect effort given that not all compensation-seeking mild TBI patients are malingering. Furthermore, the use of simulators in the remaining studies (Cato et al., 2002; Iverson & Franzen, 1994, 1998; Suhr & Gunstad, 2000; Wogar et al., 1998) leads to concerns regarding the generalizability of findings to real-world populations and also is likely to underestimate test effectiveness given that many subjects instructed to malinger do not do so (cf. Heaton, Smith, Lehman, & Vogt, 1978; Suhr & Gunstad, 2000; Tenhula & Sweet, 1996) or do so in a manner inconsistent with that shown by “real world” noncredible patients (cf. Boone et al., 2001; Boone, Lu, & Wen, 2005).

The purpose of the present study was to examine the effectiveness of the Warrington Recognition Memory Test–Words in a large sample of “real world” noncredible subjects when compared with performance of neuropsychological clinic patients without motive to feign.

Materials and Methods

Participants

Subjects were referred for neuropsychological assessment to the Harbor-UCLA Medical Center, Department of Psychiatry, Outpatient Neuropsychology Service, or the private practice of the second author. Patients evaluated in the former setting were primarily referred by treating psychiatrists or neurologists for diagnostic clarification, case management, and/or determination of appropriateness for disability compensation. Patients tested in the latter setting were either evaluated in the context of litigation or at the request of private disability carriers. IRB approval to examine archival data was obtained from the hospital-affiliated research institute (Los Angeles Biomedical Institute). All participants were fluent in English and most were native English-speakers. Criteria for inclusion and exclusion within noncredible and credible groups are described below.

Patients with suspect effort

One hundred and ninety patients met Slick and colleagues' (1999) criteria for probable malingered neurocognitive dysfunction; that is, all were in litigation or seeking to obtain disability benefits for cognitive symptoms associated with alleged medical or psychiatric disorders, failed ≥2 independent effort indicators (tests and cutoffs, and number and percentage of noncredible subjects failing each test, are listed in Table 1), and neuropsychological impairments were inconsistent with normal and independent function in activities of daily living. Demographic information is contained in Table 2. Presenting diagnoses are provided in Table 3.

Table 1.

Criteria for Inclusion in the Suspect Effort Group and Percent of Noncredible Subjects Failing Each Test

To be included in the suspect effort group, each participant had to show observations of noncredible cognitive symptoms drawn from ≥ two of the following nine tests:
 
(1) Rey 15 plus Recognition (Boone, Salazar, Lu, Warner-Chacon, & Razani, 2002):
  • Combination score <20; 56.2% failed (95 of 169 administered the test)

 
(2) Dot Counting Test (Boone et al., 2002b):
  • E-score ≥17; 62.6% failed (107 of 171 administered the test)

 
(3) b Test (Boone et al., 2002a):
  • E-score ≥155; 46.6% failed (69 of 148 administered the test)

 
(4) Digit Span (Babikian, Boone, Lu, & Arnold, 2006):
  • Age-Corrected Scaled Score ≤5; 43.3% failed (81 of 187 administered)

  • Reliable Digit Span ≤6; 42.8% failed (80 of 187 administered)

  • Average time to repeat 3 digits forward ≥3 s; 29.1% failed (44 of 151 administered)

  • Average time to repeat 4 digits forward ≥5 s; 24.8% failed (36 of 145 administered)

 
(5) Rey Word Recognition (Nitch, Boone, Wen, Arnold, & Alfano, 2006):
  • Total recognized (without subtracting false positives) for men ≤5; 50.5% failed (47 of 93 administered)

  • Total recognized (without subtracting false positives) for women ≤7; 75.0% failed (54 of 72 administered)

  • Combination equation ≤9; 62.8% failed (103 of 164 administered)

 
(6) Rey Auditory Verbal Learning Test (RAVLT) Effort Equation (Boone et al., 2005) and Rey–Osterrieth (RO)/RAVLT Discriminant Function (Sherman, Boone, Lu, & Razani, 2002):
  • Effort Equation Score ≤12; 93.3% failed (154 of 165 administered)

  • RO/RAVLT discriminant function ≤−0.40; 59.3% failed (73 out of 123 administered)

 
(7) Finger Tapping dominant hand (Arnold et al., 2005):
  • ≤35 for male dominant hand; 52.1% failed (50 of 96 administered

  • ≤28 for female dominant hand; 47.1% failed (32 of 68 administered)

 
(8) Test of Memory Malingering Trial 2 (Tombaugh, 1997):
  • <45; 66.7% failed (32 of 48 administered)

 
(9) Rey–Osterrieth effort equation (Lu, Boone, Cozolino, & Mitchell, 2003):
  • combination score <47; 65.2% failed (86 of 132 administered)

 
To be included in the suspect effort group, each participant had to show observations of noncredible cognitive symptoms drawn from ≥ two of the following nine tests:
 
(1) Rey 15 plus Recognition (Boone, Salazar, Lu, Warner-Chacon, & Razani, 2002):
  • Combination score <20; 56.2% failed (95 of 169 administered the test)

 
(2) Dot Counting Test (Boone et al., 2002b):
  • E-score ≥17; 62.6% failed (107 of 171 administered the test)

 
(3) b Test (Boone et al., 2002a):
  • E-score ≥155; 46.6% failed (69 of 148 administered the test)

 
(4) Digit Span (Babikian, Boone, Lu, & Arnold, 2006):
  • Age-Corrected Scaled Score ≤5; 43.3% failed (81 of 187 administered)

  • Reliable Digit Span ≤6; 42.8% failed (80 of 187 administered)

  • Average time to repeat 3 digits forward ≥3 s; 29.1% failed (44 of 151 administered)

  • Average time to repeat 4 digits forward ≥5 s; 24.8% failed (36 of 145 administered)

 
(5) Rey Word Recognition (Nitch, Boone, Wen, Arnold, & Alfano, 2006):
  • Total recognized (without subtracting false positives) for men ≤5; 50.5% failed (47 of 93 administered)

  • Total recognized (without subtracting false positives) for women ≤7; 75.0% failed (54 of 72 administered)

  • Combination equation ≤9; 62.8% failed (103 of 164 administered)

 
(6) Rey Auditory Verbal Learning Test (RAVLT) Effort Equation (Boone et al., 2005) and Rey–Osterrieth (RO)/RAVLT Discriminant Function (Sherman, Boone, Lu, & Razani, 2002):
  • Effort Equation Score ≤12; 93.3% failed (154 of 165 administered)

  • RO/RAVLT discriminant function ≤−0.40; 59.3% failed (73 out of 123 administered)

 
(7) Finger Tapping dominant hand (Arnold et al., 2005):
  • ≤35 for male dominant hand; 52.1% failed (50 of 96 administered

  • ≤28 for female dominant hand; 47.1% failed (32 of 68 administered)

 
(8) Test of Memory Malingering Trial 2 (Tombaugh, 1997):
  • <45; 66.7% failed (32 of 48 administered)

 
(9) Rey–Osterrieth effort equation (Lu, Boone, Cozolino, & Mitchell, 2003):
  • combination score <47; 65.2% failed (86 of 132 administered)

 
Table 2.

Means, standard deviations, ranges, frequencies, and group comparisons for demographic variables and Warrington Recognition Memory Test–Words scores

 Credible patients (n = 124) Noncredible patients (n = 190) t/Mann–Whitney U-test p-value 
Age 43.62 ± 13.72 (17–75) 44.61 ± 12.93 (17–77) −.69 .49 
Education 13.31 ± 2.75 (3–20) 12.93 ± 3.03 (4–21) 1.11 .27 
Gender (men/women) 65/59 107/83  .50 
Warrington–Words Accuracy Score 47.19 ± 3.24 (29–50) 31.86 ± 9.17 (5–50) 1189.00 <.001 
Warrington–Words Time Score 131.66 ± 56.52 (44–314), N = 107 296.94 ± 170.21 (100–1126), N = 145 1878.00 <.001 
Ethnicity n (%) n (%) Total (%)  
Caucasian 64 (51.6) 76 (40.0) 140 (44.6)  
African American 13 (10.5) 64 (33.7) 77 (24.5)  
Hispanic 24 (19.4) 20 (10.5) 44 (14.0)  
Asian American 4 (3.2) 15 (7.9) 19 (6.0)  
Middle Eastern 4 (3.2) 5 (2.6) 9 (2.9)  
Native American 2 (1.6) 3 (1.6) 5 (1.6)  
Mixed Ethnicity 13 (10.5) 7 (3.7) 20 (6.4)  
English as a second language 20 (16.1) 39 (20.5) 59 (18.8)  
 Credible patients (n = 124) Noncredible patients (n = 190) t/Mann–Whitney U-test p-value 
Age 43.62 ± 13.72 (17–75) 44.61 ± 12.93 (17–77) −.69 .49 
Education 13.31 ± 2.75 (3–20) 12.93 ± 3.03 (4–21) 1.11 .27 
Gender (men/women) 65/59 107/83  .50 
Warrington–Words Accuracy Score 47.19 ± 3.24 (29–50) 31.86 ± 9.17 (5–50) 1189.00 <.001 
Warrington–Words Time Score 131.66 ± 56.52 (44–314), N = 107 296.94 ± 170.21 (100–1126), N = 145 1878.00 <.001 
Ethnicity n (%) n (%) Total (%)  
Caucasian 64 (51.6) 76 (40.0) 140 (44.6)  
African American 13 (10.5) 64 (33.7) 77 (24.5)  
Hispanic 24 (19.4) 20 (10.5) 44 (14.0)  
Asian American 4 (3.2) 15 (7.9) 19 (6.0)  
Middle Eastern 4 (3.2) 5 (2.6) 9 (2.9)  
Native American 2 (1.6) 3 (1.6) 5 (1.6)  
Mixed Ethnicity 13 (10.5) 7 (3.7) 20 (6.4)  
English as a second language 20 (16.1) 39 (20.5) 59 (18.8)  
Table 3.

Diagnoses of credible and noncredible patients

Diagnosis Credible Noncredible 
Abscess — 
Anoxia 
Anxiety disorder 
Attention deficit hyperactivity disorder — 
Bipolar disorder 
Brain tumor 
Cerebral hemorrhage — 
Chronic fatigue syndrome — 
Cognitive disorder, not otherwise specified 
Depression 20 17 
Depression with psychotic features 
Dementia, not otherwise specified — 
Electrocution — 
Encephalopathy — 
Fibromyalgia — 
Human immunodeficiency virus — 
Klinefelter syndrome — 
Learning disability 16 
Liver disease — 
Meningitis — 
Mild to moderate mental retardation — 
Mild traumatic brain injury 55 
Moderate traumatic brain injury 
Multiple sclerosis — 
Obsessive-compulsive disorder — 
Pain disorder — 
Panic disorder — 
Psychosis 17 
Seizure disorder 
Severe traumatic brain injury 15 
Somatoform disorder 
Stroke 10 
Substance abuse 
Syncopal episode — 
Toxic exposure — 10 
Vascular dementia — 
Rule out attention deficit hyperactivity disorder 
Rule out anoxia — 
Rule out Asperger disorder — 
Rule out frontotemporal dementia — 
Rule out dementia, not otherwise specified 
Rule out learning disability — 
Rule out major depressive disorder 
Rule out pain vs. conversion disorder — 
Rule out psychotic disorder — 
Rule out somatoform disorder 
N 124 190 
Diagnosis Credible Noncredible 
Abscess — 
Anoxia 
Anxiety disorder 
Attention deficit hyperactivity disorder — 
Bipolar disorder 
Brain tumor 
Cerebral hemorrhage — 
Chronic fatigue syndrome — 
Cognitive disorder, not otherwise specified 
Depression 20 17 
Depression with psychotic features 
Dementia, not otherwise specified — 
Electrocution — 
Encephalopathy — 
Fibromyalgia — 
Human immunodeficiency virus — 
Klinefelter syndrome — 
Learning disability 16 
Liver disease — 
Meningitis — 
Mild to moderate mental retardation — 
Mild traumatic brain injury 55 
Moderate traumatic brain injury 
Multiple sclerosis — 
Obsessive-compulsive disorder — 
Pain disorder — 
Panic disorder — 
Psychosis 17 
Seizure disorder 
Severe traumatic brain injury 15 
Somatoform disorder 
Stroke 10 
Substance abuse 
Syncopal episode — 
Toxic exposure — 10 
Vascular dementia — 
Rule out attention deficit hyperactivity disorder 
Rule out anoxia — 
Rule out Asperger disorder — 
Rule out frontotemporal dementia — 
Rule out dementia, not otherwise specified 
Rule out learning disability — 
Rule out major depressive disorder 
Rule out pain vs. conversion disorder — 
Rule out psychotic disorder — 
Rule out somatoform disorder 
N 124 190 

“Rule out” diagnoses are those in which the patient met/reported some but not all symptoms required for the disorder.

Credible patients

The 124 credible subjects were not in litigation or seeking to obtain disability benefits for cognitive symptoms and failed ≤1 effort indicators (tests and cutoffs listed in Table 1). Patients with a FSIQ lower than 70 or a dementia or amnestic disorder diagnosis were excluded. Demographic data are provided in Table 2, and final diagnoses are listed in Table 3.

Procedures

The Warrington Recognition Memory Test–Words was administered as part of a neuropsychological battery to all patients in the current study. It consists of a booklet with a series of 50 words shown individually at 3-s intervals. Instructions were adapted from the Warrington Recognition Memory Test manual (Warrington, 1984). The word “neutral” was added to the original instructions as many patients claimed difficulty determining whether they liked or disliked a word.

Instructions for the exposure trial were as follows:

This is a test of memory for words. I am going to show you some words, one at a time. I want you to read each word out loud and then tell me if you like the word, don't like the word, or feel neutral about it. There is no right or wrong answer but I do want you to try and make a decision about each word. Here is the first word.

The first word in the stimulus booklet was then shown.

Do you like it, not like it, or feel neutral about it?

The recognition task consists of a page containing 50 pairs of words; one word of each pair is a word previously shown in the series of 50 words presented in the booklet and the other word a foil. Patients were instructed:

Now I am going to test your memory for the words I have just shown you.

The card with the word pairs was presented and the first pair was pointed with the following instruction:

Which of these two words have you just read – was it XXX or YYY? Take a guess if you are not sure.

Timing began at this point and ended when the patient had completed the remaining 49 pairs of words.

The highest possible score was 50. The time score was the amount of seconds to complete the recognition task.

Results

As shown in Table 2, the credible and noncredible groups did not significantly differ in age or education. Additional chi-square analyses indicated that there were no significant differences in gender distribution between the two groups (p = .50).

Warrington Recognition Memory Test–Words scores were not normally distributed, necessitating use of nonparametric statistics. In the credible group, no significant correlations were obtained between Warrington Recognition Memory Test–Words total correct and age or education (Rs = − .004 and Rs = .106, respectively), and while age was not significantly related to time scores (Rs = .033), a modest relationship was detected with education (Rs = − .195, p = .04), although it accounted for minimal test score variance (<4%). Within the noncredible group, no significant relationships were observed between accuracy scores and age (Rs = −.019), although a minor relationship was found with education (Rs = .189, p = .01), although again accounting for <4% score variance. No significant relationships were documented between time scores and age or education (Rs = .043 and Rs = − .062, respectively).

As reproduced in Table 2, nonparametric analyses showed that noncredible patients performed significantly worse on total correct than their credible counterparts and obtained significantly longer response times. Nonparametric comparisons of total correct and time scores in male and female subjects in each group separately revealed no significant differences (U = 1349.00–4308.50, p ≥ .10).

Cutoff scores for total correct and time scores, based on visual inspection of the frequency data to maximize sensitivity while maintaining specificity of at least 90% (Baker, Donders, & Thompson, 2000), are shown in Tables 4 and 5. A total correct cutoff of ≤42 was associated with 91.9% specificity (95% confidence interval = 87.10–96.70; SE = 2.5%) while detecting 88.9% of the noncredible patients (95% confidence interval = 84.43–93.37; SE = 2.3%). A cut score of <29 was associated with 100% specificity, but only correctly identified 36% of the noncredible patients. Less than 10% of noncredible subjects scored significantly below chance (i.e., ≤19/50; n = 18, 9.5%).

Table 4.

Sensitivity and specificity values for Warrington Recognition Memory Test–Words accuracy scores

Cutoff score Sensitivity (% noncredible, n = 190) Specificity (% credible, n = 124) 
≤19 9.5  
≤20 10.5  
≤21 11.6  
≤22 13.7  
≤23 16.8  
≤24 18.4  
≤25 24.2  
≤26 27.4  
≤27 32.1  
≤28 35.8 100.0 
≤29 38.4 99.2 
≤30 40.5 99.2 
≤31 46.8 99.2 
≤32 53.2 99.2 
≤33 57.9 99.2 
≤34 60.5 98.4 
≤35 63.7 98.4 
≤36 66.8 98.4 
≤37 70.0 98.4 
≤38 73.2 98.4 
≤39 77.4 96.8 
≤40 82.1 96.8 
≤41 85.8 96.8 
≤42 88.9 91.9 
≤43 90.0 89.5 
≤44 92.1 84.7 
≤45 93.7 77.4 
≤46 94.7 72.6 
≤47 95.8 56.5 
≤48 97.9 44.4 
≤49 98.4 23.4 
≤50 100.0 0.0 
Cutoff score Sensitivity (% noncredible, n = 190) Specificity (% credible, n = 124) 
≤19 9.5  
≤20 10.5  
≤21 11.6  
≤22 13.7  
≤23 16.8  
≤24 18.4  
≤25 24.2  
≤26 27.4  
≤27 32.1  
≤28 35.8 100.0 
≤29 38.4 99.2 
≤30 40.5 99.2 
≤31 46.8 99.2 
≤32 53.2 99.2 
≤33 57.9 99.2 
≤34 60.5 98.4 
≤35 63.7 98.4 
≤36 66.8 98.4 
≤37 70.0 98.4 
≤38 73.2 98.4 
≤39 77.4 96.8 
≤40 82.1 96.8 
≤41 85.8 96.8 
≤42 88.9 91.9 
≤43 90.0 89.5 
≤44 92.1 84.7 
≤45 93.7 77.4 
≤46 94.7 72.6 
≤47 95.8 56.5 
≤48 97.9 44.4 
≤49 98.4 23.4 
≤50 100.0 0.0 
Table 5.

Sensitivity and specificity values for Warrington Recognition Memory Test–Words time scores

Cutoff (in seconds) Sensitivity (% noncredible, n = 190) Specificity (% credible, n = 124) 
≥44  0.0 
≥45  0.9 
≥100 100.0 32.7 
≥119 93.8 49.5 
≥144 85.5 66.4 
≥155 82.8 74.8 
≥172 80.0 80.4 
≥196 68.3 85.0 
≥199 68.3 86.9 
≥202 68.3 87.9 
≥203 66.9 87.9 
≥205 65.5 87.9 
≥206 65.5 88.8 
≥207 65.5 90.7 
≥208 65.5 90.7 
≥211 64.8 90.7 
≥212 64.1 90.7 
≥213 63.4 90.7 
≥214 62.8 90.7 
≥217 62.1 90.7 
≥220 60.7 90.7 
≥222 60.7 91.6 
≥277 41.4 95.3 
≥315 32.4 100.0 
Cutoff (in seconds) Sensitivity (% noncredible, n = 190) Specificity (% credible, n = 124) 
≥44  0.0 
≥45  0.9 
≥100 100.0 32.7 
≥119 93.8 49.5 
≥144 85.5 66.4 
≥155 82.8 74.8 
≥172 80.0 80.4 
≥196 68.3 85.0 
≥199 68.3 86.9 
≥202 68.3 87.9 
≥203 66.9 87.9 
≥205 65.5 87.9 
≥206 65.5 88.8 
≥207 65.5 90.7 
≥208 65.5 90.7 
≥211 64.8 90.7 
≥212 64.1 90.7 
≥213 63.4 90.7 
≥214 62.8 90.7 
≥217 62.1 90.7 
≥220 60.7 90.7 
≥222 60.7 91.6 
≥277 41.4 95.3 
≥315 32.4 100.0 

A cutoff of ≥207″ for the recognition trial was associated with 90.7% specificity (95% confidence interval = 85.59–95.81; SE = 2.6%), while detecting 65.5% of the noncredible patients (95% confidence interval = 59.79–73.21; SE = 3.5%). One third of the credible patients (32.7%) were able to complete the task in <100″; in contrast, none of the noncredible patients completed the task within that amount of time.

Significant although modest correlations were obtained between accuracy and time scores in both credible (Rs = −.449, p = .0001) and noncredible groups (Rs = −.239, p = .004). Use of the time cut-score of ≥207″ identified an additional 5% of the noncredible patients not detected by the total correct cutoff of ≤42, leading to a combined sensitivity of 93.7% (95% confidence interval = 90.25–97.15; SE = 1.8%) while maintaining 87.1% specificity (95% confidence interval = 81.20–93.00; SE = 3.0%).

In Table 6 are reproduced positive and negative predictive values for the optimal accuracy and time scores, and the combination of both, at base rates of 15% noncredible, 40% noncredible, and 50% noncredible subjects.

Table 6.

Classification rates for Warrington–Words total score, time to complete and combination of the two for the detection of noncredible performance

Warrington—Words Cutoff Sensitivity Specificity BR = 15%
 
BR = 40%
 
BR = 50%
 
    PPP NPP PPP NPP PPP NPP 
Accuracy score ≤42 88.9 91.9 65.9 97.9 88.0 92.5 91.6 89.2 
Time in seconds ≥207″ 65.5 90.7 55.4 93.7 82.4 79.8 87.6 72.4 
Combination of above  93.7 87.1 56.2 98.7 82.9 95.4 87.9 93.3 
Warrington—Words Cutoff Sensitivity Specificity BR = 15%
 
BR = 40%
 
BR = 50%
 
    PPP NPP PPP NPP PPP NPP 
Accuracy score ≤42 88.9 91.9 65.9 97.9 88.0 92.5 91.6 89.2 
Time in seconds ≥207″ 65.5 90.7 55.4 93.7 82.4 79.8 87.6 72.4 
Combination of above  93.7 87.1 56.2 98.7 82.9 95.4 87.9 93.3 

Notes: BR = base rate, percentage of individuals in sample who were malingering; Sensitivity = percentage of malingering group falling below cutoff; Specificity = percentage of credible group falling above cutoff; PPP = Positive Predictive Power, percentage of those with positive test sign who were malingering; NPP = Negative Predictive Power, percentage of those with negative test sign who were not malingering.

In Table 7 are shown demographic and diagnostic data for the 17 credible subjects who fell beyond cutoffs on accuracy and/or time.

Table 7.

Demographic and diagnosis information on test false-positive identifications

Tests failed Gender Education (years) Age Ethnicity/native language Diagnosis 
Accuracy: 29, Time: 234″ Men 12 61 Hispanic/ESL Cognitive Disorder NOS 
Accuracy: 34 Men 13 71 Caucasian Depression 
Accuracy: 39, Time: 287″ Men 16 60 Hispanic/ESL Depression 
Accuracy: 39 Men 18 44 Caucasian Klinefelter Syndrome 
Accuracy: 42, Time: 218″ Women 11 32 Native American Depression 
Accuracy: 42 Men 10 50 Caucasian Severe TBI 
Accuracy: 42 Men 18 49 Caucasian Anoxia/seizures 
Accuracy: 42 Men 16 56 Caucasian Klinefelter Syndrome 
Accuracy: 42 Women 11 30 Hispanic/ESL R/o Asperger and/or Learning Disability 
Accuracy: 42 Women 13 23 Hispanic/ESL Encephalitis 
Time: 220″ Men 13 71 Caucasian Depression 
Time: 222″ Men 14 22 Caucasian Severe TBI 
Time: 250″ Men 13/spec. ed 43 African American Learning Disability 
Time: 277″ Women 13 26 Hispanic/ESL Seizures 
Time: 238″ Women 16 41 Caucasian ADHD 
Time: 314″ Women 12 30 Hispanic Severe TBI 
Time: 296″ Men 50 Hispanic/ESL Stroke 
Tests failed Gender Education (years) Age Ethnicity/native language Diagnosis 
Accuracy: 29, Time: 234″ Men 12 61 Hispanic/ESL Cognitive Disorder NOS 
Accuracy: 34 Men 13 71 Caucasian Depression 
Accuracy: 39, Time: 287″ Men 16 60 Hispanic/ESL Depression 
Accuracy: 39 Men 18 44 Caucasian Klinefelter Syndrome 
Accuracy: 42, Time: 218″ Women 11 32 Native American Depression 
Accuracy: 42 Men 10 50 Caucasian Severe TBI 
Accuracy: 42 Men 18 49 Caucasian Anoxia/seizures 
Accuracy: 42 Men 16 56 Caucasian Klinefelter Syndrome 
Accuracy: 42 Women 11 30 Hispanic/ESL R/o Asperger and/or Learning Disability 
Accuracy: 42 Women 13 23 Hispanic/ESL Encephalitis 
Time: 220″ Men 13 71 Caucasian Depression 
Time: 222″ Men 14 22 Caucasian Severe TBI 
Time: 250″ Men 13/spec. ed 43 African American Learning Disability 
Time: 277″ Women 13 26 Hispanic/ESL Seizures 
Time: 238″ Women 16 41 Caucasian ADHD 
Time: 314″ Women 12 30 Hispanic Severe TBI 
Time: 296″ Men 50 Hispanic/ESL Stroke 

Notes: ESL = English as a second language; TBI = Traumatic brain injury; NOS = Not otherwise specified; ADHD = Attention deficit hyperactivity disorder; spec. ed = special education placement.

Discussion

The Warrington Recognition Memory Test–Words, although not originally created for the purposes of measuring suspect effort, appears to be an excellent measure for detecting response bias on neuropsychological testing. The present study, utilizing a large sample of “real world” noncredible subjects (n = 190) and heterogeneous credible neuropsychological clinic patients (n = 124), found that a total correct cut-score of ≤42 was associated with 91.9% specificity while detecting 88.9% of the noncredible patients. A cut score of <29 was associated with 100% specificity, as was also observed by Millis (1992), but this cutoff only correctly identified 36% of the noncredible patients. Less than 10% of noncredible subjects scored significantly below chance (i.e., ≤19/50; n = 18, 9.5%).

The current study is the first to examine recognition trial completion time in addition to total correct. Use of a cut score of ≥207″ was associated with 90.7% specificity, with sensitivity of 65.5%. Use of the time cut-score captured an additional 5% of subjects not identified as noncredible by the accuracy score, raising the total sensitivity of the Warrington Recognition Memory Test–Words to 93.7% (at 87.1% specificity). Although one third of the credible patients completed the recognition trial in <100″, none of the noncredible patients finished the task within that time frame. These data are consistent with findings from other studies showing that noncredible individuals are slower to complete tasks (b Test, Boone, Lu, & Herzberg, 2002a; Dot Counting Test, Boone, Lu, & Herzberg, 2002b; Finger Tapping, Arnold et al., 2005).

Men and women did not differ in Warrington Recognition Memory Test–Words performance in either credible or noncredible groups. However, examination of mean performances between men and women showed that women consistently scored better in both groups, although not significantly. Of note, there were nearly twice as many credible men than women who fell beyond test cut-scores (11 men vs. 6 women). The accuracy cut-score of ≤42 was associated with 89.2% specificity in credible men and 89.7% sensitivity in male subjects; however, the score could be raised in female subjects to ≤43 while still maintaining 91.5% specificity in credible women (89% sensitivity in noncredible women). In men, a time cutoff of ≥207″ was associated with 89.3% specificity and 65.9% sensitivity, whereas in women, the cut-score could be lowered to 197″ and still achieve 90.2% specificity, with a slight raise in sensitivity to 68.3%. Thus, the cutoffs generated for the entire credible sample are appropriate for use with male subjects, but it would be acceptable to adopt slightly more stringent cutoffs for women subjects.

Mild TBI was the most common presenting diagnosis in the noncredible group (n = 55; 29%) when compared with only two subjects with this diagnosis in the credible group (<1%). In fact, no doubt due to the prevalence of mild TBI patients in noncredible samples, most validation studies of cognitive measures of response bias have focused on this population. To allow comparison of current findings to the persistent post-concussion literature, the above Warrington Recognition Memory Test–Words cutoffs were applied to the noncredible mild TBI subsample separately. Sensitivity was 87.3% for total correct and 62.8% for time, indicating that performance of noncredible mild TBI subjects was only slightly lower than that of the noncredible group as a whole and suggesting that mild TBI individuals perform comparably to individuals feigning symptoms in the context of other claimed conditions.

Of the 17 credible subjects who fell beyond cutoffs for Warrington Recognition Memory Test–Words accuracy or time, seven were Hispanic including six with English as a second language (ESL). This represented slightly less than one third of the credible ESL sample and suggests that this demographic subgroup may be at risk for increased false-positive rates on the test. In Table 8 are shown cut scores that would achieve ≥90% and 100% specificity in this population. One of the two Native Americans in the credible sample exceeded both time and accuracy cut-scores. Only one credible African American subject (out of 13; 7.7%) exceeded the time cutoff (none exceeded accuracy scores) and had a condition which appears to have placed him at higher risk for false-positive identification (i.e., learning disability; see subsequently). O'Bryant, Hilsabeck, McCaffrey, and Gouvier (2003) observed no difference in Warrington Recognition Memory Test–Words performance between Caucasian (n = 30) and African American (n = 30) college students (means of 48.4 and 47.8, respectively). Thus, there appear to be no concerns regarding the use of the Warrington Recognition Memory Test–Words as a measure of effort in this population.

Table 8.

Warrington Recognition Memory Test–Words cutoffs in credible groups at “high risk” for false-positive identifications

 Cutoff specificity
 
Cutoffs associated with
 
 Accuracy Time ≥90% specificity
 
100% specificity
 
 ≤42 ≥207″ Accuracy Time Accuracy Time 
Learning disability (including R/0; n = 24) 12.5% 95.8 ≤41 ≥207″ ≤38 ≥251″ 
English as a second language (n = 20) 90.0% 80% ≤42 ≥239″ ≤41 ≥297″ 
Severe traumatic brain injury (n = 7) 85.7% 71.4% n/a n/a ≤41 ≥315″ 
Age ≥70 (n = 4) 75.0% 75.0% n/a n/a ≤33 ≥221″ 
Late life depressiona (n = 108) 82.5% n/a ≤40 n/a ≤24 n/a 
Late life psychosisa (n = 23) 52.5% n/a ≤36 n/a ≤26 n/a 
Late-onset psychotic depressiona (n = 14) 68.6% n/a ≤38 n/a ≤28 n/a 
 Cutoff specificity
 
Cutoffs associated with
 
 Accuracy Time ≥90% specificity
 
100% specificity
 
 ≤42 ≥207″ Accuracy Time Accuracy Time 
Learning disability (including R/0; n = 24) 12.5% 95.8 ≤41 ≥207″ ≤38 ≥251″ 
English as a second language (n = 20) 90.0% 80% ≤42 ≥239″ ≤41 ≥297″ 
Severe traumatic brain injury (n = 7) 85.7% 71.4% n/a n/a ≤41 ≥315″ 
Age ≥70 (n = 4) 75.0% 75.0% n/a n/a ≤33 ≥221″ 
Late life depressiona (n = 108) 82.5% n/a ≤40 n/a ≤24 n/a 
Late life psychosisa (n = 23) 52.5% n/a ≤36 n/a ≤26 n/a 
Late-onset psychotic depressiona (n = 14) 68.6% n/a ≤38 n/a ≤28 n/a 

Age was not significantly associated with Warrington Recognition Memory Test–Words performance, suggesting that in general cut scores do not require adjustment for this variable. However, of the four credible subjects of age 70 or higher, one surpassed time cutoffs and another fell below accuracy cutoffs, suggesting that cutoffs require adjustment for individuals in this age range; cutoffs associated with 100% specificity in this age group are shown in Table 8.

Modest relationships with education were detected for some Warrington Recognition Memory Test–Words scores, but the amount of test score variance was negligible, suggesting that this variable does not require consideration in interpretation of test scores. The average educational level of the 17 credible subjects who fell beyond either accuracy and/or time scores was 13.35, which is comparable to the 13.31 average years of education for the credible sample as a whole. However, four of the credible subjects who fell beyond time or accuracy cutoffs either had likely learning disabilities or conditions associated with learning disabilities (e.g., Klinefelter syndrome, Asperger syndrome), and represented 17% of the credible subjects with learning disability/rule out learning disability (n = 24). Thus, these data suggest that this population may require adjustment of test scores; in Table 8 are shown cut-scores associated with ≥90% and 100% specificity in this subgroup.

Nearly, half of the credible patients with severe brain injury (three of seven; 43%) scored beyond cutoffs on either time or accuracy cut-scores. Use of the accuracy cut-score associated with 100% specificity in this subgroup (≤41; Table 8) still identified 86.7% of noncredible severe TBI patients (n = 15; 100% were identified with the original cut-score). However, adopting the time cut-score associated with 100% specificity in credible severe TBI patients resulted in sensitivity of only 30.8% in noncredible severe TBI (down from 53.8% with the original cut-score). Thus, in examination of effort in severe TBI samples, accuracy scores should be emphasized.

Four of the 17 credible subjects who failed Warrington Recognition Memory Test–Words cutoffs had diagnoses of depression, which represented 20% of the depressed credible sample (n = 20). However, of the four, two were above age 70, and two were ethnic minorities and/or ESL (Hispanic/ESL and Native American), suggesting that factors aside from diagnosis may have been responsible for the failures. Rohling, Green, Allen, and Iverson (2002) found no difference on the Warrington Recognition Memory Test–Words in disability seekers reporting low (n = 81) or high (n = 90) levels of depression and who had been screened for poor effort (45.7 vs. 45.4, respectively). However, as shown in Table 8, Goldberg and colleagues (2007) provided Warrington Recognition Memory Test–Words frequency data for 108 middle aged and older patients with major depression that show that the accuracy cut-score requires slight lowering (to ≤40) to maintain ≥90% specificity in this population. Use of this latter cut-score still achieved 70.6% sensitivity in identifying the 17 noncredible patients in the current study claiming depression (down from 100% sensitivity using the original cut-score); the time cut-score of ≥207″ detected 64.3% of noncredible depressed patients.

In the current sample, none of the six credible subjects who carried a psychotic diagnosis exceeded test cutoffs. Egeland and colleagues (2003) in fact reported no significant difference in Warrington Recognition Memory Test–Words performance in patients with schizophrenia (n = 53), depressed patients (n = 50), and controls (n = 50) (means of 45.9, 48.1, and 49.0, respectively). However, as reproduced in Table 8, additional frequency data from Goldberg and colleagues (2007) show that in 23 patients with late life psychosis and 14 subjects with late-onset psychotic depression, specificity was unacceptable using the recommended cut-score of ≤42. Thus, older psychotic patients in particular appear to require adjustment to Warrington Recognition Memory Test–Words cutoffs. However, lowering the cutoffs to ≤36 to ≤38 required to achieve ≥90% specificity in these populations still captured 76.5% and 64.7% of the 17 noncredible subjects in the current study reporting psychotic symptoms (down from 88.2% using the cut-score of ≤42); 53.8% were identified with a time cutoff of ≥207″.

Patients with diagnoses of dementia or amnestic disorder and with full scale IQ <70 were excluded from the credible group, and thus findings from this study cannot be applied to these populations. Recent research has shown that Warrington Recognition Memory Test–Words has to be lowered to <32 to maintain ≥90% specificity for patients with IQ <70 (Dean, Victor, Boone, & Arnold, 2008), confirming that cutoffs require adjustment for individuals of very low intellectual level. Further, in an examination of effort test performance in patients with dementia (Dean, Victor, Boone, Philpott, & Hess, 2009), the Warrington Recognition Memory Test–Words accuracy cutoff had to be adjusted to chance-level performance (<26) to maintain specificity of ≥90%. Thus, these findings indicate that the Warrington Recognition Memory Test–Words is not appropriate for use in the differential between actual versus feigned dementia.

In conclusion, results from the present study suggest that the Warrington Recognition Memory Test–Words is an effective measure of response bias in most clinical neuropsychological assessment settings, although current findings should only be applied in those situations in which the adapted test instructions described in this study have been employed.

Conflict of Interest

None declared

References

Arnold
G.
Boone
K. B.
Lu
P.
Dean
A.
Wen
J.
Nitch
S.
, et al.  . 
Sensitivity and specificity of finger tapping test scores for the detection of suspect effort
The Clinical Neuropsychologist
 , 
2005
, vol. 
19
 (pg. 
105
-
120
)
Babikian
T.
Boone
K. B.
Lu
P.
Arnold
G.
Sensitivity and specificity of various Digit Span scores in the detection of suspect effort
The Clinical Neuropsychologist
 , 
2006
, vol. 
20
 (pg. 
145
-
159
)
Baker
R.
Donders
J.
Thompson
E.
Assessment of incomplete effort with the California Verbal Learning Test
Applied Neuropsychology
 , 
2000
, vol. 
7
 (pg. 
111
-
114
)
Boone
K. B.
The need for continuous and comprehensive sampling of effort/response bias during neuropsychological examinations
The Clinical Neuropsychologist
 , 
2009
Boone
K. B.
Lu
P.
Back
C.
King
C.
Lee
A.
Philpott
L.
, et al.  . 
Sensitivity and specificity of the Rey Dot Counting Test in patients with suspect effort and various clinical samples
Archives of Clinical Neuropsychology
 , 
2001
, vol. 
17
 (pg. 
625
-
642
)
Boone
K. B.
Lu
P.
Herzberg
D.
The b test.
 , 
2002
Los Angeles
Western Psychological Services
Boone
K. B.
Lu
P.
Herzberg
D.
The Dot Counting Test
 , 
2002
Los Angeles
Western Psychological Services
Boone
K. B.
Lu
P.
Wen
J.
Comparison of various RAVLT scores in the detection of noncredible memory performance
Archives of Clinical Neuropsychology
 , 
2005
, vol. 
20
 (pg. 
310
-
319
)
Boone
K. B.
Salazar
X.
Lu
P.
Warner-Chacon
K.
Razani
J.
The Rey 15-Item Recognition Trial: A technique to enhance sensitivity of the Rey 15-Item Memorization Test
Journal of Clinical and Experimental Neuropsychology
 , 
2002
, vol. 
24
 (pg. 
561
-
573
)
Cato
M. A.
Brewster
J.
Ryan
T.
Giuliano
A. J.
Coaching and the ability to simulate mild traumatic brain injury symptoms
The Clinical Neuropsychologist
 , 
2002
, vol. 
16
 (pg. 
524
-
535
)
Dean
A. C.
Victor
T. L.
Boone
K. B.
Arnold
G.
The relationship of IQ to effort test performance
The Clinical Neuropsychologist
 , 
2008
, vol. 
22
 (pg. 
705
-
722
)
Dean
A. C.
Victor
T. L.
Boone
K. B.
Philpott
L.
Hess
R. A.
Dementia and effort test performance
The Clinical Neuropsychologist
 , 
2009
, vol. 
23
 (pg. 
133
-
152
)
Egeland
J.
Sundet
K.
Rund
B. R.
Asbjornsen
A.
Hugdahl
K.
Landro
N. I.
, et al.  . 
Sensitivity and specificity of memory dysfunction in schizophrenia: A comparison with major depression
Journal of Clinical and Experimental Neuropsychology
 , 
2003
, vol. 
25
 (pg. 
79
-
93
)
Goldberg
H. E.
Back-Madruga
C.
Boone
K. B.
Boone
K.
The impact of psychiatric disorders on cognitive symptom validity test scores
Assessment of feigned cognitive impairment: A neuropsychological perspective.
 , 
2007
New York
Guilford Press
Heaton
R. K.
Smith
H. H.
Lehman
R. A.
Vogt
A. T.
Prospects for faking believable deficits on neuropsychological testing
Journal of Consulting and Clinical Psychology
 , 
1978
, vol. 
46
 (pg. 
892
-
900
)
Iverson
G. L.
Franzen
M. D.
The Recognition Memory Test, Digit Span, and Knox Cube Test as markers of malingered memory impairment
Assessment
 , 
1994
, vol. 
1
 (pg. 
323
-
334
)
Iverson
G. L.
Franzen
M. D.
Detecting malingered memory deficits with the Recognition Memory Test
Brain Injury
 , 
1998
, vol. 
12
 (pg. 
275
-
282
)
Larrabee
G. J.
Larrabee
G. J.
Malingering, research designs and base rates
Assessment of Malingered Neuropsychological Deficits
 , 
2007
New York
Oxford University
(pg. 
3
-
13
)
Lu
P. H.
Boone
K. B.
Cozolino
L.
Mitchell
C.
Effectiveness of the Rey–Osterrieth Complex Figure Test and the Meyers and Meyers recognition trial in the detection of suspect effort
The Clinical Neuropsychologist
 , 
2003
, vol. 
17
 (pg. 
426
-
440
)
Millis
S. R.
The recognition memory test in the detection of malingered and exaggerated memory deficits
The Clinical Neuropsychologist
 , 
1992
, vol. 
6
 (pg. 
406
-
414
)
Millis
S. R.
Assessment of motivation and memory with the recognition memory test after financially compensable mild head injury
Journal of Clinical Psychology
 , 
1994
, vol. 
50
 (pg. 
601
-
605
)
Millis
S. R.
Putnam
S. J.
The Recognition Memory Test in the assessment of memory impairment after financially compensable mild head injury: A replication
Perceptual and Motor Skills
 , 
1994
, vol. 
79
 (pg. 
384
-
386
)
Mittenberg
W.
Patton
C.
Canyock
E. M.
Condit
D. C.
Base rates of malingering and symptom exaggeration
Journal of Clinical and Experimental Neuropsychology
 , 
2002
, vol. 
24
 (pg. 
1094
-
1102
)
Nelson
N. W.
Boone
K.
Dueck
A.
Wagener
L.
Lu
P.
Grills
C.
Relationship between eight measures of suspect effort
The Clinical Neuropsychologist
 , 
2003
, vol. 
17
 (pg. 
263
-
272
)
Nies
K. J.
Sweet
J. J.
Neuropsychological assessment of malingering: A critical review of past and present strategies
Archives of Clinical Neuropsychology
 , 
1994
, vol. 
9
 (pg. 
501
-
552
)
Nitch
S.
Boone
K. B.
Wen
J.
Arnold
G.
Alfano
K.
The utility of the Rey Word Recognition Test in the detection of suspect effort
The Clinical Neuropsychologist
 , 
2006
, vol. 
20
 (pg. 
873
-
887
)
O'Bryant
S. E.
Hilsabeck
R. C.
McCaffrey
R. J.
Gouvier
W. D.
The Recognition Memory Test: Examination of ethnic differences and norm validity
Archives of Clinical Neuropsychology
 , 
2003
, vol. 
18
 (pg. 
135
-
143
)
Rohling
M. L.
Green
P.
Allen
L. M.
Iverson
G. L.
Depressive symptoms and neurocognitive test score sin patients passing symptom validity tests
Archives of Clinical Neuropsychology
 , 
2002
, vol. 
17
 (pg. 
205
-
222
)
Ross
S. R.
Putnam
S. H.
Adams
K. M.
Psychological disturbance, incomplete effort, and compensation-seeking status as predictors of neuropsychological test performance in head injury
Journal of Clinical and Experimental Neuropsychology
 , 
2006
, vol. 
28
 (pg. 
111
-
125
)
Sherman
D. S.
Boone
K. B.
Lu
P.
Razani
J.
Re-examination of a Rey Auditory Verbal Learning Test/Rey Complex Figure discriminant function to detect suboptimal effort
The Clinical Neuropsychologist
 , 
2002
, vol. 
16
 (pg. 
242
-
250
)
Slick
D. J.
Sherman
E. M.
Iverson
G. L.
Diagnostic criteria for malingered neurocognitive dysfunction: Proposed standards for clinical practice and research
The Clinical Neuropsychologist
 , 
1999
, vol. 
13
 (pg. 
545
-
561
)
Suhr
J. A.
Gunstad
J.
The effects of coaching on the sensitivity and specificity of malingering measures
Archives of Clinical Neuropsychology
 , 
2000
, vol. 
15
 (pg. 
415
-
424
)
Tardif
H. P.
Barry
R. J.
Fox
A. M.
Johnstone
S. J.
Detection of feigned recognition memory impairment using the old/new effect of the event-related potential
International Journal of Psychophysiology
 , 
2000
, vol. 
36
 (pg. 
1
-
9
)
Tenhula
W. N.
Sweet
J. J.
Double cross-validation of the Booklet Category Test in detecting malingered traumatic brain injury
Clinical Neuropsychologist
 , 
1996
, vol. 
10
 (pg. 
104
-
116
)
Tombaugh
T. N.
The Test of Memory Malingering (TOMM): Normative data from cognitively intact and cognitively impaired individuals
Psychological Assessment
 , 
1997
, vol. 
9
 (pg. 
260
-
268
)
Warrington
E. K.
Recognition Memory Test: Manual.
 , 
1984
Berkshire, UK
NFER-Nelson
Wogar
M. A.
van den Broek
M. D.
Bradshaw
C. M.
Szabadi
E.
A new performance-curve method for the detection of simulated cognitive impairment
British Journal of Clinical Psychology
 , 
1998
, vol. 
37
 (pg. 
327
-
339
)