Abstract

Neuropsychological research frequently uses non-clinical undergraduate participants to evaluate neuropsychological tests. However, a recent study by An and colleagues (2012, Archives of Clinical Neuropsychology, 27, 849–857) called into question that the extent to which the interpretation of these participants' performance on neuropsychological tests is valid. This study found that in a sample of 36 participants, 55.6% exhibited performance invalidity at an initial session and 30.8% exhibited performance invalidity at a follow-up session. The current study attempted to replicate these findings in a larger, more representative sample using a more rigorous methodology. Archival data from 133 non-clinical undergraduate research participants were analyzed. Participants were classified as performance invalid if they failed any one PVT. In the current sample, only 2.26% of participants exhibited performance invalidity. Thus, concerns regarding insufficient effort and performance invalidity when using undergraduate research participants appear to be overstated.

The use of undergraduate students as research participants is a common practice in psychological research. According to the most recent estimates, 68% of psychological research is conducted using undergraduate students, with a little less than half of all research participants recruited from first-year introductory psychology classes. This practice appears to be relatively stable over time, as 70% and 68% of psychological research used undergraduate research participants in 1975 and 1995, respectively (Gallander Wintre, North, & Sugar, 2001). Although not specifically surveyed, the use of undergraduate research participants appears to be a fairly common practice in neuropsychological research in which cognitive measures are evaluated and examined.

Several concerns have been raised regarding the use of undergraduate student samples in neuropsychology research. Chief among them is that undergraduate research participants may not be putting forth sufficient effort, and therefore, that the interpretation of their performance on neuropsychological tests as representative of brain function may not be valid. Several aspects of the nature of undergraduate student sample-based research contribute to this concern. For one, students often participate in research for course credit and are awarded credit regardless of how much effort they apply. Thus, they have no incentive to exert effort. For another, research testing may be lengthy and tiresome. Combined with the lack of personal commitment to the research outcome, the tedious nature of psychological research may cause undergraduate student participants to mentally fatigue and apply less effort over time (Kato, Endo, & Kizuka, 2009; Mackworth, 1968).

Data obtained from undergraduate research participants exhibiting insufficient effort are problematic as it may lead to unreliable scores on psychological and neuropsychological tests and result in invalid interpretation of those scores (Fox, 2011; Green, Rohling, Lees-Haley, & Allen, 2001). Thus, a sufficient level of effort is a prerequisite for the valid interpretation of neuropsychological test scores. In the absence of sufficient effort, the test scores obtained may be unreliable (Heilbronner et al., 2009; Iverson & Binder, 2000; Moss, Jones, Fokias, & Quin, 2003).

Several methods have been proposed to detect performance invalidity, including the use of specialized performance validity tests (PVTs). PVTs are tests that are designed to detect performance invalidity and may be constructed out of already-existing neuropsychological tests (embedded PVTs) or specifically designed to detect performance invalidity as stand-alone tests (free-standing PVTs). Although maintaining face validity as tests of cognitive abilities, PVTs are in fact easy enough for patients with a broad range of psychiatric, neurological, and developmental problems to pass (Heilbronner et al., 2009). Ultimately, failure of free-standing PVTs has been associated with poorer performance on neuropsychological tests across multiple cognitive domains (Constantinou, Bauer, Ashendorf, & McCaffrey, 2005; Silk-Eglit, Stenclik, Miele, Lynch, & McCaffrey, 2013).

In a recent study, An and colleagues (2012) found high rates of performance invalidity in a sample of non-clinical undergraduate research participants. Specifically, An and colleagues recruited a sample of 36 undergraduate participants from an introductory psychology class and tested them at two time-points. Participants were administered three free-standing PVTs, the Test of Memory Malingering (TOMM; Tombaugh, 1996), the Victoria Symptom Validity Test (VSVT; Slick, Hopp, Strauss, & Thompson, 1997), and the Dot Counting Test (DCT; Boone, Lu, & Herzberg, 2002). Table 1 presents a description of the cutoffs implemented by An and colleagues on each of these three administered PVTs.

Table 1.

PVT cutoffs implemented in An and colleagues (2012)

PVT Cutoff 
TOMM <45 on Trial 2 or the Retention Trial (Tombaugh, 1996) 
VSVT ≤20 correct on difficult items, ≤23 correct on easy items, ≤44 correct on all items, >2.95 second response latency on difficult items, >2.00 second response latency on easy items, or >2.43 second response latency on all items (An et al., 2012
DCT ≥14 E score (Boone et al., 2002
PVT Cutoff 
TOMM <45 on Trial 2 or the Retention Trial (Tombaugh, 1996) 
VSVT ≤20 correct on difficult items, ≤23 correct on easy items, ≤44 correct on all items, >2.95 second response latency on difficult items, >2.00 second response latency on easy items, or >2.43 second response latency on all items (An et al., 2012
DCT ≥14 E score (Boone et al., 2002

Notes: TOMM = Test of Memory Malingering; VSVT = Victoria Symptom Validity Test; DCT = Dot Counting Test.

In their sample, An and colleagues found that 55.6% (n = 20) of participants exhibited performance invalidity in Session 1, with the majority of examinees failing the VSVT (65%; n = 13) or the DCT (45%; n = 9). At follow-up, 30.8% (n = 4) of participants were classified as performances invalid, the majority of whom failed the DCT (75%; n = 3) followed by the VSVT (25%; n = 1). No participants failed the TOMM during either session.

There may be several problems with An and colleagues (2012) study that limit the validity, generalizability and reliability of their findings. First, the authors used the VSVT in a manner that is not well-validated and/or recommended by the test manual. Specifically, An and colleagues created new cutoffs for their study on the VSVT for both total items correct and response latency scores. These cutoffs were established by examining distributions of scores in the initial validation sample of this test and setting performance invalidity cutoffs at the score that corresponded with the fifth percentile among non-clinical examinees. Thus, none of these cutoffs have been validated or recommended for use by the test authors. An and colleagues use of response latency scores as primary measures of performance validity may be especially problematic. In the initial validation sample for the VSVT, Slick and colleagues found that there was significant overlap between Easy and Difficult Item Response Latency scores for individuals exhibiting a valid performance, a questionably valid performance, and an invalid performance. Thus, Slick and colleagues recommend that, “clinical decisions regarding malingering should not be made on the basis of Easy and Difficult Items Response Latency scores alone. Rather, Easy and Difficult Items Response Latency scores should be used only in an ancillary manner for providing additional context for interpreting Items Correct scores” (p. 33). Secondly, An and colleagues sample may not generalize to most studies using undergraduate students as research participants. Specifically, the majority of An and colleagues sample was Asian and English was not their first language. Thirdly, the sample size of An and colleagues study was small (n = 36). Given this small sample size, the reliability of their findings is questionable.

Given these potential problems with An and colleagues (2012) study, the current study was conducted to examine the replicability of their findings and to expand on their study. To evaluate the reliability and generalizability of their findings, a larger and more representative sample was recruited. In addition, to assess the validity of their performance invalidity classifications, we used performance validity cutoffs that have been well established. The current study expanded on An and colleagues by evaluating six PVTs in total, four of which were not used in their study. We hypothesized that the high rate of performance invalidity reported by An and colleagues might have been a sampling artifact that would not hold up to replication in a larger sample using well-accepted cutoff scores for identifying performance invalidity.

Methods

Participants

The current study used archival data from two recent studies. Following institutional review board approval, 153 undergraduate research participants were recruited into the first study and 100 undergraduate research participants were recruited into the second study. All students were recruited through a University-based online recruitment system and all were enrolled in the University through which they were recruited. Participation in research was either mandated for course credit or students received course extra credit for their participation.

Only the control conditions in both of these studies were analyzed. The first study was a multi-site study in which participants were recruited through two separate universities, University at Albany, State University of New York, and University of Colorado Colorado Springs. In this sample (Sample 1; n = 83), mean age was 21.61 (SD = 5.61) years and mean education was 13.56 (SD = 1.65) years. The majority of participants were female (59%), right-handed (93%), and non-Hispanic Caucasian (72%). In the second study, participants were recruited through University at Albany only. Participants in this sample (Sample 2; n = 50) had a mean age of 19.10 (SD = 1.25) years and mean education of 12.68 (SD = 0.98) years. Again, the majority of participants were female (58%), right-handed (88%), and non-Hispanic Caucasian (64%). After combining both samples (n = 133), mean age was 20.67 (SD = 4.65) years and mean years of education was 13.22 (SD = 1.49). The majority of participants were female (59%), right-handed (91%), and non-Hispanic Caucasian (69%). See Table 2 for a full listing of demographic variables in Sample 1, Sample 2, and both samples combined.

Table 2.

Summary of demographic variables in Sample 1, Sample 2, and both samples combined

  Sample 1 (n = 83) Sample 2 (n = 50) Both samples combined (n = 133) 
Mean (SD) Mean (SD) Mean (SD) 
Age 21.61 (5.61) 19.10 (1.25) 20.67 (4.65) 
Education 13.56 (1.65) 12.68 (0.98) 13.22 (1.49) 
Gender 
 Female 49 (59%) 29 (58%) 78 (59%) 
 Male 34 (41%) 21 (42%) 55 (41%) 
Handedness 
 Right 77 (93%) 44 (88%) 121 (91%) 
 Left 6 (7%) 6 (12%) 12 (9%) 
Race/ethnicity 
 Caucasian 58 (72%) 32 (64%) 90 (69%) 
 African-American 13 (16%) 11 (22%) 24 (18%) 
 Asian 8 (10%) 4 (8%) 12 (9%) 
 Hispanic 2 (2%) 3 (6%) 5 (4%) 
  Sample 1 (n = 83) Sample 2 (n = 50) Both samples combined (n = 133) 
Mean (SD) Mean (SD) Mean (SD) 
Age 21.61 (5.61) 19.10 (1.25) 20.67 (4.65) 
Education 13.56 (1.65) 12.68 (0.98) 13.22 (1.49) 
Gender 
 Female 49 (59%) 29 (58%) 78 (59%) 
 Male 34 (41%) 21 (42%) 55 (41%) 
Handedness 
 Right 77 (93%) 44 (88%) 121 (91%) 
 Left 6 (7%) 6 (12%) 12 (9%) 
Race/ethnicity 
 Caucasian 58 (72%) 32 (64%) 90 (69%) 
 African-American 13 (16%) 11 (22%) 24 (18%) 
 Asian 8 (10%) 4 (8%) 12 (9%) 
 Hispanic 2 (2%) 3 (6%) 5 (4%) 

Materials

Participants were recruited through two separate studies and were thus administered two different batteries of tests. The control condition of the first study (Sample 1) was administered three PVTs and two cognitive tests. The PVTs were the Word Memory Test (WMT; Green, 2003), the VSVT (Slick et al., 1997), and the TOMM (Tombaugh, 1996). The two cognitive tests were the Seashore Rhythm Test (Reitan & Wolfson, 1993) and the Speech Sounds Perception Test (Reitan & Wolfson, 1993). Although embedded measures of performance validity have been developed for the Speech Sounds Perception Test and Seashore Rhythm Test, we omitted them from the current study for two reasons. First, previous research on these embedded measures has found that they do not have adequate classification accuracy (Miele, Gunner, Lynch, & McCaffrey, 2012). Specifically, Miele and colleagues found that the embedded measure for the Speech Sounds Perception Test had a specificity of 0.93 and sensitivity of 0.33 and that the embedded measure for the Seashore Rhythm Test had a specificity of 0.83 and a sensitivity of 0.33. Second, aggregating too many measures of performance validity will likely increase the false-positive rate and overestimate the base rate of performance invalidity (Berthelson, Mulchan, Odland, Miller, & Mittenberg, 2013). Not all examinees were administered every test due to computer malfunction or time limitations. Eighty-two participants were administered the WMT, 79 were administered the VSVT, and 78 examinees were administered the TOMM. In addition, 81 were administered the Seashore Rhythm Test and the Speech Sounds Perception Test.

The participants in the control condition of the second study (Sample 2) were administered three PVTs and four cognitive tests. The PVTs included the Medical Symptom Validity Test (MSVT; Green, 2004), Reliable Digit Span (RDS; Greiffenstein, Baker, & Gola, 1994), and the Force Choice Recognition (FCR) trial of the California Verbal Learning Test-II (CVLT-II; Moore & Donders, 2004). The cognitive tests included the Trail Making Test A and B (Trails A and Trails B; Reitan, 1955), Finger Tapping Test (FTT) Dominant and NonDominant hand (Halstead, 1947), the computer version of the Category Test (CT; Halstead, 1947; Hom, 2011; Reitan & Wolfson, 1993), and the CVLT-II (Delis, Kaplan, Kramer, & Ober, 2000; Moore & Donders, 2004). All examinees completed all of these tests.

Procedure

In the first sample, after providing informed consent, participants were administered five tests: TOMM, WMT, VSVT, Seashore Rhythm Test, and Speech Sounds Perception Test. To protect against order effects, the order of test administration was counterbalanced. Half of all participants were administered the WMT, Seashore Rhythm Test, and Speech Sounds Perception Test first and the TOMM and VSVT second, whereas the other half of participants were administered the TOMM and VSVT first and the WMT, Seashore Rhythm Test, and Speech Sounds Perception Test second. Administration of each test followed standardized administration procedures as set forth by each respective test manual. Cutoffs implemented for PVTs were those commonly used in clinical practice (see Table 3).

Table 3.

PVT cutoffs implemented in Sample 1 and Sample 2

PVT Cutoff 
Sample 1 
 TOMM <45 on Trial 2 or the Retention Trial (Tombaugh, 1996) 
 VSVT ≤17 correct on difficult items (Grote et al., 2000; Loring, Lee, & Meador, 2005
 WMT ≤82.5% correct on Immediate Recognition, Delayed Recognition or Consistency Index (Green, 2003) 
Sample 2 
 MSVT <85% correct on Immediate Recognition, Delayed Recognition or Consistency Index (Green, 2004) 
 FCR Score of <15 correct (Moore & Donders, 2004
 RDS Score of <7 correct (Babikian, Boone, Lu, & Arnold, 2006; Schroeder, Twumasi-Andkrah, Baade, & Marshall, 2012
PVT Cutoff 
Sample 1 
 TOMM <45 on Trial 2 or the Retention Trial (Tombaugh, 1996) 
 VSVT ≤17 correct on difficult items (Grote et al., 2000; Loring, Lee, & Meador, 2005
 WMT ≤82.5% correct on Immediate Recognition, Delayed Recognition or Consistency Index (Green, 2003) 
Sample 2 
 MSVT <85% correct on Immediate Recognition, Delayed Recognition or Consistency Index (Green, 2004) 
 FCR Score of <15 correct (Moore & Donders, 2004
 RDS Score of <7 correct (Babikian, Boone, Lu, & Arnold, 2006; Schroeder, Twumasi-Andkrah, Baade, & Marshall, 2012

Notes: TOMM = Test of Memory Malingering; VSVT = Victoria Symptom Validity Test; WMT = Word Memory Test; MSVT = Medical Symptom Validity Test; FCR = Forced Choice Recognition from CVLT-II; RDS = Reliable Digit Span.

In the second sample, again, following informed consent, standardized administration of tests occurred in the following order: MSVT, Trails A, Trails B, Digit Span, FTT, MSVT delay, CVLT-II, CT, and delayed-recall and recognition of the CVLT-II. Examinees were then given a demographic information questionnaire and subsequently completed the CVLT-II FCR. The MSVT, CVLT-II FCR, and RDS were utilized as measures of performance validity. Again, cutoffs used for these PVTs were those commonly used in clinical practice (see Table 3).

Results

Analyses focused on documenting the base rate of performance invalidity in each sample. Table 4 presents the frequency and percentages of performance invalidity in Sample 1 (n = 83), Sample 2 (n = 50), and in both samples combined (n = 133). As can be seen in this table, rates of PVT failure were quite low. In Sample 1, only 3.75% (n = 3) of the sample failed any one of the three PVTs administered. Broken down by specific PVT, the majority of examinees failed the VSVT (n = 2). Only one participant failed the WMT and no participants failed the TOMM. In Sample 2, no examinees were classified as exhibiting performance invalidity on any of the measures. After combining both samples, only three participants failed any PVT. This led to an estimated base rate of performance invalidity across all PVTs of 2.26% (CI0.95 = 0.00%–5.26%).

Table 4.

Frequency and percentage of PVT failures in sample 1, sample 2, and both samples combined

Test Pass Fail 
Sample 1 (n = 83) 
WMT 81 (99%) 1 (1%) 
TOMM 78 (100%) 0 (0%) 
VSVT 77 (97%) 2 (3%) 
All PVTs combined 80 (96%) 3 (4%) 
Sample 2 (n = 50) 
MSVT 50 (100%) 0 (0%) 
RDS 50 (100%) 0 (0%) 
FCR 50 (100%) 0 (0%) 
All PVTs combined 50 (100%) 0 (0%) 
Both samples combined (n = 133) 
All PVTs combined 130 (98%) 3 (2%) 
Test Pass Fail 
Sample 1 (n = 83) 
WMT 81 (99%) 1 (1%) 
TOMM 78 (100%) 0 (0%) 
VSVT 77 (97%) 2 (3%) 
All PVTs combined 80 (96%) 3 (4%) 
Sample 2 (n = 50) 
MSVT 50 (100%) 0 (0%) 
RDS 50 (100%) 0 (0%) 
FCR 50 (100%) 0 (0%) 
All PVTs combined 50 (100%) 0 (0%) 
Both samples combined (n = 133) 
All PVTs combined 130 (98%) 3 (2%) 

Notes: WMT = Word Memory Test; TOMM = Test of Memory Malingering; VSVT = Victoria Symptom Validity Test; RDS = Reliable Digit Span; MSVT = Medical Symptom Validity Test; FCR = CVLT-II Forced Choice Recognition.

Owing to the low base rate of performance invalidity and consequent low power, the impact of performance validity classification on cognitive test scores was not analyzed.

Discussion

The current study evaluated the base rate of performance invalidity in two samples of non-clinical undergraduate research participants. Results suggested that very few participants exhibited performance invalidity. Specifically, when combining the results of all six PVTs across both samples, only 2.26% (CI0.95 = 0.00%–5.26%) of participants were identified as exhibiting performance invalidity.

These results suggest that performance invalidity is relatively rare in non-clinical undergraduate research samples. Inspection of descriptive statistics of raw scores on all six PVTs further demonstrates the high level of performance exhibited by participants (see Table 5 for descriptive statistics on PVTs in Sample 1 and Sample 2). Specifically, for each PVT, mean performance was at or near the ceiling. Moreover, in all cases, mean performance was >2 SDs above performance invalidity cutoffs. In addition, it should be noted that using multiple PVTs to classify performance invalidity likely inflates estimated base rates relative to using a single PVT (Berthelson et al., 2013). Thus, the true base rate of performance invalidity may be even <2.26% documented in the current study.

Table 5.

Mean, SD, 95% confidence interval, and ranges for each PVT in Sample 1 and Sample 2

PVT Mean SD 95% confidence interval Range 
Sample 1 
 WMT     
  Immediate recognition (%) 98.48 2.33 97.97–98.98 90–100 
  Delayed recognition (%) 98.80 2.18 98.33–99.27 90–100 
  Consistency (%) 97.68 3.17 96.99–98.36 82.5–100 
 TOMM 
  Trial 1 total 48.08 3.01 47.41–48.75 36–50 
  Trial 2 total 49.92 0.31 49.85–49.99 48–50 
  Retention total 49.81 0.49 49.70–49.92 47–50 
 VSVT 
  Easy items total 23.92 0.31 23.85–23.99 22–24 
  Difficult items total 23.16 1.35 22.86–23.46 17–24 
  Total 47.09 1.55 46.75–47.43 40–48 
Sample 2 
 RDS total 10.18 1.89 9.66–10.70 7–15 
 MSVT 
  Immediate recognition (%) 99.70 1.20 99.37–100 95–100 
  Delayed recognition (%) 99.90 0.71 99.70–100 95–100 
  Consistency (%) 99.60 1.70 99.13–100 90–100 
FCR total 16.00 0.00 NA NA 
PVT Mean SD 95% confidence interval Range 
Sample 1 
 WMT     
  Immediate recognition (%) 98.48 2.33 97.97–98.98 90–100 
  Delayed recognition (%) 98.80 2.18 98.33–99.27 90–100 
  Consistency (%) 97.68 3.17 96.99–98.36 82.5–100 
 TOMM 
  Trial 1 total 48.08 3.01 47.41–48.75 36–50 
  Trial 2 total 49.92 0.31 49.85–49.99 48–50 
  Retention total 49.81 0.49 49.70–49.92 47–50 
 VSVT 
  Easy items total 23.92 0.31 23.85–23.99 22–24 
  Difficult items total 23.16 1.35 22.86–23.46 17–24 
  Total 47.09 1.55 46.75–47.43 40–48 
Sample 2 
 RDS total 10.18 1.89 9.66–10.70 7–15 
 MSVT 
  Immediate recognition (%) 99.70 1.20 99.37–100 95–100 
  Delayed recognition (%) 99.90 0.71 99.70–100 95–100 
  Consistency (%) 99.60 1.70 99.13–100 90–100 
FCR total 16.00 0.00 NA NA 

Notes: WMT = Word Memory Test; TOMM = Test of Memory Malingering; VSVT = Victoria Symptom Validity Test; RDS = Reliable Digit Span; MSVT = Medical Symptom Validity Test; FCR = Forced Choice Recognition test of the CVLT-II.

The findings of the current study are in stark contrast to those of An and colleagues (2012), in which 55.6% of their sample of non-clinical undergraduate research participants exhibited an invalid performance at initial assessment and 30.8% at a follow-up session. There are several important differences between the current study and An and colleagues that might account for this discrepancy in findings. First, the current study implemented different cutoffs on the VSVT than An and colleagues In particular, An and colleagues created new cutoffs for non-clinical examinees based on raw scores corresponding to the fifth percentile in the initial validation sample for both items correct and response latency scores. These cutoffs were omitted from the current study as the VSVT manual recommends against using response latency scores as primary measures of performance invalidity (Slick et al., 1997) and because these cutoffs have never been validated for use with non-clinical participants. As a result, implementing these cutoffs may have led to false-positive misclassification of participants as performance invalid and thus inflated the base rate of performance invalidity in An and colleagues's study. Second, An and colleagues used a different set of PVTs than the current study. Specifically, An and colleagues administered the VSVT, TOMM, and DCT. In contrast, in the current study, participants were administered the VSVT, TOMM, and WMT in Sample 1 and the MSVT, RDS, and FCR in Sample 2. Third, the composition of the sample in An and colleagues differed from the composition of the samples analyzed in the current study. Of particular importance, 77% of An and colleagues sample was Asian and half of their sample did not speak English as their first language. In contrast, the current samples were composed mostly of non-Hispanic Caucasian participants with only 10% and 8% of Sample 1 and Sample 2 consisting of Asian participants, respectively.

The results of the current study have significant implications for neuropsychological research using non-clinical undergraduate participants. Contrary to the claims of An and colleagues, performance invalidity does not appear to be a rampant problem in this mode of research. Thus, previous neuropsychological research using undergraduate research participants should not be rejected wholesale due to presumed high rates of performance invalidity or for failure to administer PVTs.

The current study is subject to a few limitations. First, although we used the VSVT and TOMM as in An and colleagues, the current study did not administer the DCT. Instead, the current study administered four different PVTs—the WMT, RDS, MSVT, and FCR—which enabled us to investigate the generalizability of An and colleagues findings. It could be argued that replacing the DCT with these four different PVTs may have reduced the base rate of performance invalidity in our sample. However, Strauss and colleagues (2002) found that when comparing the effectiveness of the VSVT, RDS, and computerized DCT with clinical examinees, the VSVT was best at detecting invalid performance, followed by the RDS then computerized DCT. Second, the current study was not able to evaluate whether performance invalidity was associated with poorer performance on cognitive tests due to the low base rate of performance invalidity and consequent low power of those comparisons. Third, the current study did not evaluate performance validity classification longitudinally as in An and colleagues Thus, we were unable to determine the temporal stability of performance invalidity.

In summary, the current study documented a dramatically lower base rate of performance invalidity than previous research and did so using a much larger, more representative sample and a more rigorous methodology. Therefore, concerns regarding insufficient effort and performance invalidity when using undergraduate research participants appear to be overstated.

Funding

The authors would like to thank Multi-Health Systems (MHS) for their generosity in providing free Test of Memory Malingering (TOMM) uses.

Acknowledgements

The authors are grateful to Cecil R. Reynolds, former editor of Archives of Clinical Neuropsychology and the current editor of Psychological Assessment, who served as the guest action editor. This manuscript was submitted to the blind peer review.

References

An
K. Y.
Zakzanis
K. K.
Joordens
S.
Conducting research with non-clinical healthy undergraduates: Does effort play a role in neuropsychological test performance?
Archives of Clinical Neuropsychology
 , 
2012
, vol. 
27
 (pg. 
849
-
857
)
Babikian
T.
Boone
K. B.
Lu
P.
Arnold
G.
Sensitivity and specificity of various digit span scores in the detection of suspect effort
The Clinical Neuropsychologist
 , 
2006
, vol. 
20
 (pg. 
145
-
159
)
Berthelson
L.
Mulchan
S. S.
Odland
A. P.
Miller
L. J.
Mittenberg
W.
False positive diagnosis of malingering due to the use of multiple effort tests
Brain Injury
 , 
2013
, vol. 
27
 (pg. 
909
-
916
)
Boone
K. B.
Lu
P.
Herzberg
D. S.
The dot counting test manual
 , 
2002
Los Angeles, CA
Western Psychological Services
Constantinou
M.
Bauer
L.
Ashendorf
L.
McCaffrey
R. J.
Is poor performance on recognition memory effort measures indicative of generalized poor performance on neuropsychological tests?
Archives of Clinical Neuropsychology
 , 
2005
, vol. 
20
 (pg. 
191
-
198
)
Delis
D.
Kaplan
E.
Kramer
J.
Ober
B.
California Verbal Learning Test–II
 , 
2000
San Antonio, TX
The Psychological Corporation
Fox
D.
Symptom validity test failure indicates invalidity of neuropsychological tests
The Clinical Neuropsychologist
 , 
2011
, vol. 
25
 (pg. 
488
-
495
)
Gallander Wintre
M.
North
C.
Sugar
L. A.
Psychologists’ response to criticisms about research based on undergraduate participants: A developmental perspective
Canadian Psychology/Psychologie Canadienne
 , 
2001
, vol. 
42
 (pg. 
216
-
225
)
Green
P.
Greens Word Memory Test for Microsoft Windows
 , 
2003
Edmonton, AB
Green's Publishing
Green
P.
Greens Medical Symptom Validity Test (MSVT) for Microsoft Windows: Users manual
 , 
2004
Edmonton,AB
Green's Publishing
Green
P.
Rohling
M. L.
Lees-Haley
P. R.
Allen
L. M.
Effort has a greater effect on test scores than severe brain injury in compensation claimants
Brain Injury
 , 
2001
, vol. 
15
 (pg. 
1045
-
1060
)
Greiffenstein
M. F.
Baker
W. J.
Gola
T.
Validation of malingered amnesia measures with a large clinical sample
Psychological Assessment
 , 
1994
, vol. 
6
 (pg. 
218
-
224
)
Grote
C. L.
Kooker
E. K.
Garron
D. C.
Nyenhuis
D. L.
Smith
C. L.
Mattingly
M. L.
Performance of compensation seeking and non-compensation seeking samples on the Victoria Symptom Validity Test: Cross-validation and extension of a standardization study
Journal of Clinical and Experimental Neuropsychology
 , 
2000
, vol. 
22
 
6
(pg. 
709
-
719
)
Halstead
W. C.
Brain and intelligence
 , 
1947
Chicago
University of Chicago Press
Heilbronner
R. L.
Sweet
J. J.
Morgan
J. E.
Larrabee
G. L.
Millis
S. R.
Conference Participants
American academy of clinical neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias and malingering
The Clinical Neuropsychologist
 , 
2009
, vol. 
23
 (pg. 
1093
-
1129
)
Hom
Computer Version 1.1 of the Category Test
 , 
2011
Dallas, TX
The Neuropsychology Center
Iverson
G. L.
Binder
L. M.
Detecting exaggeration and malingering in neuropsychological assessment
Journal of Head Trauma Rehabilitation
 , 
2000
, vol. 
15
 (pg. 
829
-
858
)
Kato
Y.
Endo
H.
Kizuka
T.
Metal fatigue and impaired response processes: Event-related brain potentials in a Go/NoGo task
International Journal of Psychology
 , 
2009
, vol. 
72
 (pg. 
204
-
211
)
Loring
D. W.
Lee
G. P.
Meador
K. J.
Victoria Symptom Validity Test performance in non-litigating epilepsy surgery candidates
Journal of Clinical and Experimental Neuropsychology
 , 
2005
, vol. 
27
 (pg. 
610
-
617
)
Mackworth
J. F.
Vigilance, arousal, and habituation
Psychological Review
 , 
1968
, vol. 
75
 (pg. 
308
-
322
)
Miele
A. S.
Gunner
J. H.
Lynch
J. K.
McCaffrey
R. J.
Are embedded validity indices equivalent to free-standing symptom validity tests?
Archives of Clinical Neuropsychology
 , 
2012
, vol. 
27
 (pg. 
10
-
22
)
Moore
B. A.
Donders
J.
Predictors of invalid neuropsychological test performance after traumatic brain injury
Brain Injury
 , 
2004
, vol. 
18
 (pg. 
975
-
984
)
Moss
A.
Jones
C.
Fokias
D.
Quin
D.
The mediating effects of effort upon the relationship between head injury severity and cognitive functioning
Brain Injury
 , 
2003
, vol. 
17
 (pg. 
377
-
387
)
Reitan
R. M.
The relation of the Trail Making Test to organic brain damage
Journal of Consulting Psychology
 , 
1955
, vol. 
19
 (pg. 
393
-
394
)
Reitan
R. M.
Wolfson
D.
neuropsychological test battery. Theory and clinical interpretation
The Halstead-Reitan
 , 
1993
2nd ed.
Tucson, AZ
Neuropsychology Press
Schroeder
R. W.
Twumasi-Andkrah
P.
Baade
L. E.
Marshall
P. S.
Reliable Digit Span: A systematic review and cross-validation study
Assessment
 , 
2012
, vol. 
19
 (pg. 
21
-
30
)
Silk-Eglit
G. M.
Stenclik
J. H.
Miele
A. S.
Lynch
J. K.
McCaffrey
R. J.
The degree of conation on neuropsychological tests does not account for performance invalidity among litigants
Archives of Clinical Neuropsychology
 , 
2013
, vol. 
28
 (pg. 
213
-
221
)
Slick
D.
Hopp
G.
Strauss
E.
Thompson
G. B.
Victoria Symptom Validity Test
 , 
1997
Odessa, FL
Psychological Assessment Resources
Strauss
E.
Slick
D. J.
Levy-Bencheton
J.
Hunter
B.
MacDonald
S. W. S.
Hultsch
D. F.
Intraindividual variability as an indicator of malingering in head injury
Archives in Clinical Neuropsychology
 , 
2002
, vol. 
17
 (pg. 
423
-
444
)
Tombaugh
T. N.
The Test of Memory Malingering (TOMM)
 , 
1996
North Tonawanda, NY
Multi-Health Systems