Abstract

This investigation adds to the burgeoning body of research concerned with discriminating performance and symptom validity tests (SVTs) through examination of their differential relationships with cognitive performance and symptom self-report measures. To the authors' current knowledge, prior studies have not assessed differences between participants who fail either a performance validity test (PVT) or an SVT but not both. As part of their neuropsychological evaluations at four Veterans Affairs medical centers across the United States, participants were administered a fixed, standardized battery that consisted of performance validity, symptom validity, cognitive performance, and symptom self-report measures. Compared with participants who failed a PVT and an SVT, participants who passed both and participants who only passed a PVT demonstrated better cognitive performance and self-reported fewer symptoms. Results support differential clinical utility of performance validity and SVTs when assessing cognitive performance and symptom self-report.

Introduction

The increasing availability of symptom self-report validity measures and Larrabee's (2012) distinction between performance validity tests (PVTs) and symptom validity tests (SVTs)—respectively, used to assess the validity of cognitive performance scores and symptom self-report responses—have recently spurred interest in further discrimination. Divergent validity between PVTs and SVTs is supported by studies demonstrating their differential predictive validity (e.g., Whitney, Davis, Shepard, & Herman, 2008), relationships (e.g., Lange, Iverson, Brooks, & Rennison, 2010), and factor loadings (Van Dyke, Millis, Axelrod, & Hanks, 2013) with cognitive performance and symptom self-report measures. However, positive relationships between PVTs and SVTs have also been found, suggesting some convergence. For example, Lange, Iverson, Brooks, and Rennison (2010) found that while individuals evaluated for mild traumatic brain injury (mTBI) who failed a commonly used PVT, the Test of Memory Malingering (TOMM; Tombaugh, 1996), performed more poorly than individuals who passed the PVT on cognitive performance measures, they were also more likely to fail two SVTs, the Post-Concussive Scale (Lovell et al., 2003) and the British Columbia Cognitive Complaints Inventory (Iverson, 2003). Likewise, Lippa and colleagues (2014) found that individuals who failed a different PVT, the Word Memory Test (Green, 2003), were more likely to fail an SVT currently known as the Symptom Validity Scale and previously known as the Fake Bad Scale (FBS; Lees-Haley, English, & Glenn, 1991). To the present authors' knowledge, no prior studies have examined differences in cognitive performance and symptom self-report between individuals who passed either a PVT or an SVT but not both.

Given various suggestions of convergence and divergence between PVTs and SVTs, the purpose of the current investigation was to add to the growing body of research concerned with their discrimination by examining relationships between cognitive performance and symptom self-report for groups of individuals who passed: a PVT and an SVT (Both-Pass), a PVT but not an SVT (PVT-Pass), an SVT but not a PVT (SVT-Pass), and neither a PVT nor an SVT (No-Pass). It was hypothesized that Both-Pass and PVT-Pass would demonstrate better cognitive performance than No-Pass and SVT-Pass. In contrast, it was also hypothesized that Both-Pass and SVT-Pass would self-report fewer symptoms than No-Pass and PVT-Pass. Confirmed hypotheses would support differential clinical utility of PVTs and SVTs, while disconfirmation would support their concurrent use for cognitive performance and symptom self-report assessment.

Materials and Methods

Participants and Procedure

After removal of 10 cases due to invalid MMPI-2 profiles (VRIN T-Score ≥80 and/or TRIN ≥80 T/F), data from 179 Operation Enduring Freedom and Operation Iraqi Freedom U.S. military veterans were accessed from the Functional Outcomes Research Team (FORT) database. FORT consists of four Veterans Affairs (VA) traumatic brain injury (TBI) clinics in northern, western, and southern U.S.' regions. Participants, referred for neuropsychological evaluations at FORT sites subsequent to a positive mTBI screening (Belanger, Uomoto, & Vanderploeg, 2009) and a comprehensive TBI evaluation, were administered standardized test batteries. All participants met criteria for mTBI, defined by the Centers for Disease Control and Prevention (2003) as an occurrence of injury to the head with at least one of the following: Any period of confusion, disorientation, or impaired consciousness; any period of memory dysfunction around the time of the injury; loss of consciousness lasting ≤30 min; or neurological or neuropsychological dysfunction. Consistent with overall demographic characteristics of the U.S. military population (Institute of Medicine of the National Academies, 2009), the sample consisted of mostly male (n = 167), Caucasian (n = 121; Hispanic n = 31, African American n = 18, Asian = 3, Other n = 6) veterans with a mean age of 31.41 (SD = 6.88). Most participants reported head injuries sustained from blast exposure (n = 125; fall n = 47, vehicular n = 31, assault n = 18, sports-related n = 5, bullet n = 3, missing n = 8; total exceeds total sample size due to multiple injury type endorsement). About half of the sample sought increased financial compensation for medical conditions at the time of their evaluations (n = 91). FORT sites complied with respective committees for protection of human subjects and institutional review. Participants consented to participate after they were informed about minimal risks, confidentiality, and the voluntary nature of participation that could be revoked at any time without consequence.

Analyses

Statistical analyses were conducted with SPSS software (version 19.0). The Pearson bivariate correlations indicated relationships between variables. Independent samples T-tests with the Bonferroni α correction (α = 0.05/2 = 0.025) indicated differences between compensation- and non-compensation-seeking participants on the two dependent variables in this study: cognitive performance and symptom self-report. Compensation-seeking participants were defined as individuals seeking increases in service connection rating that would result in increased financial compensation for medical conditions. A multivariate analysis of variance (MANOVA) indicated validity test group differences on the combined dependent variables. Follow-up, univariate analyses of variance (ANOVAs) with the Bonferroni α correction (α = 0.05/2 = 0.025) indicated differences between each dependent variable for combined validity test groups. Partial η2p2) indicated effect-size estimates for ANOVAs, and Cohen's d indicated effect-size estimates for between-group comparisons. Tukey's HSD post hoc tests, commonly used to maintain accurate α levels when all pairs of means are compared, were used to clarify validity test group differences on dependent variables.

Measures

Performance validity test

The TOMM (Tombaugh, 1996) is a variably sensitive and highly specific test of memory impairment feigning that consists of two learning trials and an optional retention trial, each of which consists of 50 target pictures of common objects. Scores denote total correct per trial. Tombaugh's suggested cutoff score for TOMM trial 2 was used for PVT classification in the current study.

Symptom validity test

Formerly known as the Fake Bad Scale, the Symptom Validity Scale (FBS; Lees-Haley, English, & Glenn, 1991) is a 43-item, self-report measure of symptom over-reporting on the second edition of the Minnesota Multiphasic Personality Inventory (MMPI-2; Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer, 1989). Participants rate each item as either true or false. Raw score cutoffs for men and women were, respectively, above 24 and 26 (Lees-Haley, 1992; Nelson, Hoelzle, Sweet, Arbisi, & Demakis, 2010) in the current study.

Cognitive performance

The second edition of the California Verbal Learning Test (CVLT-II; Delis, Kramer, Kaplan, & Ober, 2000) is a 16-item, verbal learning and memory measure validated with demonstration of good construct validity and test–retest reliability over a mean interval of 21 days. Participants recite words spoken to them several times. The raw total number of words recalled across five trials (Trials 1–5 Total) denoted cognitive performance.

Symptom self-report

The Neurobehavioral Symptom Inventory (NSI; Cicerone & Kalmar, 1995) is a 22-item survey of somatic/sensory, affective, and cognitive postconcussive symptoms as derived from a relatively recent factor-analytic, construct validity study (Caplan et al., 2010). Participants rate symptom presence and severity on a 5-point scale. Symptom self-report was measured with NSI total raw scores.

Results

Cognitive performance and symptom self-report summary scores by validity test group are presented in Table 1. A correlation matrix and summary scores for the entire sample are presented in Table 2. Cognitive performance on the CVLT-II between compensation-seeking participants (M = 47.67, SD = 11.00) and non-compensation-seeking participants (M = 49.76, SD = 10.28) did not significantly differ, t(172) = 1.29, p = .198. Similarly, symptom self-report on the FBS between compensation-seeking participants (M = 52.53, SD = 21.77) and non-compensation-seeking participants (M = 49.23, SD = 22.58) did not significantly differ, t(172) = 0.98, p = .328. Preliminary MANOVA assumption checks revealed: no univariate or multivariate outliers as, respectively, assessed by boxplot and Mahalanobis distance (p > .001); multivariate normality as indicated by Shapiro–Wilk's test (p > .05); no multicollinearity, r(177) = −0.29, p < .01; linearity as assessed by scatterplots; and homogeneity of variance–covariance matrices as assessed by Box's test of equality of covariance matrices (p = .335).

Table 1.

Means and (standard deviations) for cognitive performance and symptom self-report scores among performance (PVT) and symptom validity test (SVT) groups (n = 179)

 PVT-Pass (n = 57) SVT-Pass (n = 11) Both-Pass (n = 91) No-Pass (n = 20) 
Symptom self-report (NSI Total) 52.25 (14.28) 51.73 (14.36) 34.75 (15.11) 63.50 (13.50) 
Cognitive performance (CVLT-II Trials 1–5 Total) 49.07 (10.22) 42.91 (10.96) 51.69 (9.85) 39.00 (9.94) 
PVT (TOMM Trial 2) 49.18 (1.42) 33.91 (10.57) 49.56 (1.63) 36.70 (7.71) 
SVT (FBS) 28.89 (8.29) 21.00 (1.73) 19.00 (4.04) 29.40 (3.42) 
 PVT-Pass (n = 57) SVT-Pass (n = 11) Both-Pass (n = 91) No-Pass (n = 20) 
Symptom self-report (NSI Total) 52.25 (14.28) 51.73 (14.36) 34.75 (15.11) 63.50 (13.50) 
Cognitive performance (CVLT-II Trials 1–5 Total) 49.07 (10.22) 42.91 (10.96) 51.69 (9.85) 39.00 (9.94) 
PVT (TOMM Trial 2) 49.18 (1.42) 33.91 (10.57) 49.56 (1.63) 36.70 (7.71) 
SVT (FBS) 28.89 (8.29) 21.00 (1.73) 19.00 (4.04) 29.40 (3.42) 

Notes: NSI = Neurobehavioral Symptom Inventory. CVLT-II = California Verbal Learning Test—Second Edition. FBS = Symptom Validity Scale. TOMM = Test of Memory Malingering. Raw scores were used for each test.

Table 2.

Means, standard deviations, and intercorrelations for the entire sample (n = 179)

Variable M SD 
1. NSI Total 44.58 17.95 — −0.29** 0.68** −0.39** 
2. CVLT-II Trials 1–5 Total 48.90 10.79  — −0.23* 0.39** 
3. FBS 23.44 6.09   — −0.20* 
4. TOMM Trial 2 47.04 6.48    — 
Variable M SD 
1. NSI Total 44.58 17.95 — −0.29** 0.68** −0.39** 
2. CVLT-II Trials 1–5 Total 48.90 10.79  — −0.23* 0.39** 
3. FBS 23.44 6.09   — −0.20* 
4. TOMM Trial 2 47.04 6.48    — 

Notes: NSI = Neurobehavioral Symptom Inventory. CVLT-II = California Verbal Learning Test—Second Edition. FBS = Symptom Validity Scale. TOMM = Test of Memory Malingering. Raw scores were used for each test.

*p < 0.01. **p < 0.001.

There was a statistically significant difference between validity test groups on the combined cognitive performance and symptom self-report variables, F(6, 348) = 17.03, p < .001; Wilks’ Λ = 0.598; ηp2 = 0.23. Follow-up ANOVAs revealed statistically significant differences in cognitive performance, F(3, 175) = 10.13, p < .001; ηp2 = 0.15, and symptom self-report, F(3, 175) = 30.92, p < .001; ηp2 = 0.35. On the cognitive performance measure, Tukey's post hoc tests indicated that Both-Pass performed significantly better than No-Pass (p < .001, d = 2.08) and SVT-Pass (p = .034, d = 1.60), while PVT-Pass performed significantly better than No-Pass (p = .001, d = 1.00). There were no statistically significant differences in cognitive performance between Both-Pass and PVT-Pass (p = .413, d = 1.05), PVT-Pass and SVT-Pass (p = .248, d = 0.58), or SVT-Pass and No-Pass (p = .728, d = 0.37). On the symptom self-report measure, Tukey's post hoc tests indicated that Both-Pass endorsed fewer symptoms than No-Pass (p < .001, d = 1.96), PVT-Pass (p < .001, d = 1.23), SVT-Pass (p < .002, d = 1.15), and PVT-Pass endorsed fewer symptoms than No-Pass (p = .018, d = 0.81). There were no statistically significant differences in symptom self-report between PVT-Pass and SVT-Pass (p = 1.000, d = 0.04) or SVT-Pass and No-Pass (p = .144, d = 0.84).

Discussion

Results support differential PVT and SVT use as cognitive performance and symptom report differed by validity test classification. In the present sample, participants who passed a PVT and an SVT, as well as those who only passed a PVT, demonstrated better cognitive performance and self-reported fewer symptoms than those who failed both validity tests. This present conclusion is limited, however, by findings that individuals who only passed an SVT did not report fewer symptoms than those who passed neither validity test. Furthermore, there were no differences in cognitive performance or symptom self-report between participants who only passed a PVT or an SVT. Small samples sizes for these groups likely account for the present failure to find statistically significant differences between them. At present, it is unknown if these groups would perform differently on cognitive performance and symptom self-report measures.

Present conclusions are also limited by possibly decreased sensitivity of TOMM Trial 2 use without the Retention Trial, as well as sample and measurement representativeness. These findings may not apply to different populations, treatment settings, presenting problems, or tests. There are of course many cognitive performance, symptom self-report, and validity measures beyond those used from the CVLT-II, NSI, TOMM, and FBS. Conclusions do not equally apply to all represented tests. Furthermore, while external validity was improved through recruitment of participants from various national regions, internal validity was diminished by the use of different test administrators. Threats to internal validity were minimized through the use of a standardized, fixed battery. Despite these limitations, statistically significant differences in cognitive performance and symptom self-report that were found between validity test groups support differential clinical utility of PVTs and SVTs. In addition to the inclusion of populations, presenting problems, and measures beyond those used in the current study, future investigations would especially increase confidence in present conclusions through comparison of larger participant groups consisting of individuals who pass a PVT or an SVT but not both.

Funding

No funding was collected or used in the conduct of this study or preparation of this manuscript.

Conflict of Interest

None declared.

References

Belanger
H. G.
,
Uomoto
J. M.
,
Vanderploeg
R. D.
(
2009
).
The Veterans Health Administration system of care for mild traumatic brain injury: Costs, benefits, and controversies
.
Journal of Head Trauma Rehabilitation
 ,
24
,
4
13
.
Butcher
J. N.
,
Dahlstrom
W. G.
,
Graham
J. R.
,
Tellegen
A.
,
Kaemmer
B.
(
1989
).
Manual for the administration and scoring of the MMPI-2
 .
Minneapolis
:
University of Minnesota Press
.
Caplan
L. J.
,
Ivins
B.
,
Poole
J. H.
,
Vanderploeg
R. D.
,
Jaffee
M. S.
,
Schwab
K.
(
2010
).
The structure of postconcussive symptoms in 3 US military samples
.
Journal of Head Trauma Rehabilitation
 ,
25
,
447
458
.
Centers for Disease Control and Prevention.
(
2003
, September).
Report to congress on mild traumatic brain injury in the United States: Steps to prevent a serious public health problem
. Retrieved from .
Cicerone
K.
,
Kalmar
K.
(
1995
).
Persistent postconcussion syndrome: The structure of subjective complaints after mild traumatic brain injury
.
Journal of Head Trauma Rehabilitation
 ,
10
,
1
17
.
Delis
D. C.
,
Kramer
J. H.
,
Kaplan
E.
,
Ober
B. A.
(
2000
).
California verbal learning test—Second edition, adult version
 .
San Antonio, TX
:
The Psychological Corporation
.
Green
P.
(
2003
).
Green's word memory test for windows: User's manual
 .
Edmonton, Canada
:
Green's Publishing
.
Institute of Medicine of the National Academies
. (
2009
).
Returning home from Iraq and Afghanistan: Preliminary assessment of readjustment needs of veterans, service members, and their families
. .
Iverson
G. L.
(
2003
, August).
Internal consistency and factor structure of the BC Cognitive Complaints Inventory
.
Paper presented at the Canadian Psychiatric Association
,
Halifax, Nova Scotia, Canada
.
Lange
R. T.
,
Iverson
G. L.
,
Brooks
B. L.
,
Rennison
V. L.
(
2010
).
Influence of poor effort on self-reported symptoms and neurocognitive test performance following mild traumatic brain injury
.
Journal of Clinical and Experimental Neuropsychology
 ,
32
,
961
972
.
Larrabee
G. J.
(
2012
).
Performance validity and symptom validity in neuropsychological assessment
.
Journal of the International Neuropsychological Society
 ,
18
,
1
7
.
Lees-Haley
P. R.
(
1992
).
Efficacy of MMPI-2 validity scales and MCMI-II modifier scales for detecting spurious PTSD claims: F, F-K, Fake Bad Scale, ego strength, subtle-obvious subscales, DIS, and DEB
.
Journal of Clinical Psychology
 ,
48
,
681
689
.
Lees-Haley
P. R.
,
English
L. T.
,
Glenn
W. J.
(
1991
).
A Fake Bad Scale on the MMPI-2 for personal injury claimants
.
Psychological Reports
 ,
68
,
203
210
.
Lippa
S. M.
,
Pastorek
N. J.
,
Romesser
J.
,
Linck
J.
,
Sim
A. H.
et al
. (
2014
).
Ecological validity of performance validity testing
.
Archives of Clinical Neuropsychology
 ,
29
,
236
244
.
Lovell
M. R.
,
Collins
M. W.
,
Iverson
G. L.
,
Field
M.
,
Maroon
J. C.
,
Cantu
R.
et al
. (
2003
).
Recovery from mild concussion in high school athletes
.
Journal of Neurosurgery
 ,
98
,
295
301
.
Nelson
N. W.
,
Hoelzle
J. B.
,
Sweet
J. J.
,
Arbisi
P. A.
,
Demakis
G. J.
(
2010
).
Updated meta-analysis of the MMPI-2 symptom validity scale (FBS): Verified utility in forensic practice
.
The Clinical Neuropsychologist
 ,
24
,
701
724
.
Tombaugh
T. N.
(
1996
).
Test of memory malingering (TOMM)
 .
North Tonawanda, NY
:
Multi Health Systems
.
Van Dyke
S. A.
,
Millis
S. R.
,
Axelrod
B. N.
,
Hanks
R. A.
(
2013
).
Assessing effort: Differentiating performance and symptom validity
.
Clinical Neuropsychologist
 ,
27
,
1234
1246
.
Whitney
K. A.
,
Davis
J. J.
,
Shepard
P. H.
,
Hermans
S. M.
(
2008
).
Utility of the Response Bias Scale (RBS) and other MMPI-2 validity scales in predicting TOMM performance
.
Archives of Clinical Neuropsychology
 ,
23
,
777
786
.

Author notes

6
Present address of J.J.M: Neurocognitive Assessment Laboratory, University of Virginia School of Medicine, Charlottesville, VA 22908, USA.
7
Present address of C.K.B: Department of Physical Medicine and Rehabilitation, TIRR Memorial Hermann Hospital, Houston, TX 77030, USA.
8
Present address of J.F.L: Oklahoma University Health Sciences Center, Department of Psychiatry and Behavioral Sciences, Oklahoma City, OK 73162, USA.