Abstract

While the Validity Indicator Profile (VIP) and the Test of Memory Malingering (TOMM) are designed to limit the influence of actual cognitive impairment on successful performance, the extent to which cognitive dysfunction does play a role in the assessment of effort should be verified in distinct clinical groups. To date, little research has been conducted on VIP performance in individuals diagnosed with a psychotic disorder. Fifty-four patients with either schizophrenia or schizoaffective disorder were administered the VIP, TOMM, Short Test of Mental Status, and the Wide Range Achievement Test-4 Reading subtest. Specificity rates were compared between tests, with normative data, and with published specificity rates in psychiatric samples. Results indicate that the use of the VIP with psychotic-disordered individuals will generate increased invalid performance profiles, whereas the TOMM is more resilient in this population. Significantly, mental status and estimated intellectual ability were predictive of classifications on the VIP Verbal subtest and the TOMM.

Introduction

Tests of effort and motivation have been developed to detect feigned cognitive impairment within forensic settings; unfortunately, there is only limited research on the performance of psychiatric samples on these measures. It is imperative to understand how clinical populations perform on tests of effort and motivation to examine whether the assumptions of each measure are valid in the clinical group assessed, and so that performance validity can be effectively measured in these groups. The cognitive performance of patients with psychotic disorders on effort tests was identified early on as an area in need of future research; however, few empirical studies have been conducted on this population to date (Goldberg, Back-Madruga, & Boone, 2007).

Schizophrenia, in particular, affects ∼1% of the population and is characterized by cognitive deficits in memory (Lee & Park, 2005), attention (Sharma & Antonova, 2003), executive functions such as reasoning and problem solving (Kuperberg & Heckers, 2000), and motivation/mental effort (Gorissen, Sanz, & Schmand, 2005). Furthermore, these deficits appear to be relatively stable in patients from the time of their first hospitalization up to intervals of 10 years or more (Hoff, Svetina, Shields, Stewart, & DeLisi, 2005). In light of the fact that patients with schizophrenia and psychotic disorders more generally may be expected to exhibit cognitive deficits in neuropsychological testing, the effects of these deficits on formal effort testing remains an important issue to address.

Previous studies have examined individuals with schizophrenia on neuropsychological measures of effort and motivation (Goldberg et al., 2007, for review). In general, results of studies on effort testing in individuals with psychotic disorders suggest higher failure rates in this group compared with normative data, although rates vary with the specific measure studied. Failure rates of 10% or higher have been found on a subset of effort tests in samples of individuals diagnosed with a psychotic disorder, including Digit Span measures (Lesser et al., 1991; Miller et al., 1991), The Digit Memory Test/Hiscock Forced Choice Test (Back et al., 1996), the Dot Counting Test (Back et al., 1996), the Rey 15-Item Test (Back et al., 1996), the Finger Tapping Test (Arnold et al., 2005), the B test (Back et al., 1996), and the Word Memory Test (Gorissen et al., 2005). Relatively preserved specificity rates (≥90%) were found for the Warrington Recognition Memory Test-Words (Egeland et al., 2003; Lesser et al., 1991; Miller et al., 1991), the Test of Memory Malingering (TOMM; Duncan, 2005), and the Victoria Symptom Validity Test (Egeland et al., 2003).

Further research on the TOMM (Tombaugh, 1996) found results consistent with those reported by Duncan (2005). In a sample of 20 honest-responding, psychiatric inpatients, which included individuals diagnosed with schizophrenia (Gierok, Dickson, & Cole, 2005), 95% of participants exceeded the cutoff score. In a sample primarily of individuals diagnosed with schizophrenia (Pivovarova, Rosenfeld, Dole, & Green, 2010), 93% of participants exceeded the cutoff score on Trial 2 of the TOMM (Retention Trial not given). In a sample of 26 individuals, in which over half of subjects were diagnosed with a psychotic spectrum disorder (Weinborn, Orr, Woods, Conover, & Feix, 2003), 88% of participants exceeded the cutoff score. In a sample of 104 individuals with a psychotic-spectrum disorder (Hubbard, 2007), 84% of participants exceeded the cutoff score.

To date, performance characteristics on concurrent Verbal and Non-verbal subtests of the Validity Indicator Profile (VIP; Frederick, 2003) in a non-litigating sample of individuals with a psychotic-spectrum disorder have not been reported. Frederick, Crosby, and Wynkoop (2000) reported the performance of a forensic sample, believed to be performing with sufficient effort and diagnosed with a psychotic disorder, on the VIP Nonverbal subtest. In that sample, psychotic disorder, affective disorder, and non-psychiatric groups were matched on adjusted score of the VIP Non-verbal subtest. Results indicated that of 60 subjects in the psychotic disorder group, 55% generated valid profiles on the VIP Non-verbal subtest, with this rate not significantly different from non-psychiatric pretrial defendants (48%). That the psychotic disorder group consisted solely of pretrial defendants suggests that either inadequate effort or a motivation to perform poorly cannot be entirely ruled out in this group, and the influence of cognitive dysfunction associated with psychotic disorders cannot be specifically identified. That poor effort or motivation to perform poorly may have been possible in this forensic sample is suggested by the fact that nearly one-half of the non-psychiatric sample generated invalid profiles. In an effort to clarify the effects of actual cognitive dysfunction associated with psychosis on VIP performance in a non-forensic sample, Pivovarova and colleagues (2010) examined VIP Verbal subtest performance in a psychiatric sample composed primarily of individuals with a psychotic disorder and a community sample instructed to feign cognitive impairment. Of 81 psychiatric subjects, 70% generated valid profiles on the Verbal subtest. Of 26 simulators, 27% generated valid profiles.

As the above review suggests, little research exists on performance profiles of psychotic-disordered individuals on the VIP specifically in the clinical setting, and no data exist on the performance of a clinical group on either concurrently administered Verbal and Non-verbal subtests or on the VIP Non-verbal subtest alone. Although few published reports are available that have examined the performance of individuals with psychosis on the VIP and TOMM, available evidence suggests that results of effort assessment in these individuals may be confounded with cognitive deficits associated with psychosis depending on the measure used, with the TOMM being relatively resilient to cognitive dysfunction, and the VIP potentially less so. This may be partly due to the level of cognitive demand required by each measure and the specific cognitive deficits typically associated with psychosis. Verified cognitive impairment has been found to affect effort test performance; previous research suggests that both intellectual ability (Dean, Victor, Boone, & Arnold, 2008), as well as mental status as determined by mental status screening measures (Dean, Victor, Boone, Philpott, & Hess, 2009), are associated with performance on effort measures.

Given developmental, educational, and cognitive issues typically present in individuals diagnosed with schizophrenia, this cohort may be at higher risk for both lower general intellectual ability as well as lower mental status performance on formal measures. Additionally, specific cognitive difficulties associated with psychosis may interact with task demands inherent in each measure. Cognitive deficits associated with schizophrenia such as inattention, distractibility, and difficulties in conceptualization and abstraction may be exploited by the different formats, difficulty levels, and task demands of the TOMM and VIP. As an example, with the exception of task instructions that are given by the clinician, the VIP is largely self-administered, lengthy, and varies in item difficulty. In contrast, the TOMM administration requires clinician-patient contact throughout, is shorter, and is less demanding in regard to item difficulty. As a result, disagreement in classifications between the VIP and the TOMM may be common given the difference in task demands and the ways in which these interact with cognitive difficulties associated with schizophrenia.

The research reported here had three main aims: (a) to establish performance profiles on both the VIP Non-verbal and Verbal subtests in a group of non-forensic psychotic-disordered individuals; (b) to examine the concordance of VIP classifications with performance on the TOMM; and (c) to examine the effects of both premorbid estimated intellectual ability and mental status on VIP and TOMM classifications. We administered the VIP and the TOMM to 54 consecutive residential-dwelling patients diagnosed with a psychotic disorder (schizophrenia; schizoaffective disorder) together with the Short Test of Mental Status (STMS) and the Reading subtest of the Wide Range Achievement Test-4th Edition (WRAT-4). Given previous research (Pivovarova et al., 2010), we hypothesized that the rate of Invalid classifications on both VIP Verbal and Non-verbal subtests would be >10%, with the majority of these classifications falling in the Inconsistent category. Given previous research suggesting that the TOMM is relatively resilient to actual cognitive impairment associated with schizophrenia, we hypothesized that the rate of scores falling below the TOMM cutoff would be significantly lower than that of the VIP. Finally, given research indicating that general intellectual ability and current mental status may influence performance on measures of effort and motivation, we hypothesized that lower performance on the STMS and the WRAT-4 Reading subtest would be related to lower performance on the VIP and the TOMM.

Methods

Subjects

The study sample consisted of 58 individuals diagnosed with schizophrenia or schizoaffective disorder residing in a community residential facility (Table 1). Of the original sample, four individuals were excluded from the final analyses because they discontinued testing prior to the completion of the test battery and one individual did not complete the VIP Verbal subtest. The final participant pool consisted of 54 individuals (34 schizophrenia; 20 schizoaffective disorder). Of the schizophrenia-diagnosed participants, 67% (23) were classified as Undifferentiated and 33% (11) were classified as Paranoid. All patients were previously diagnosed by board-certified psychiatrists using criteria set forth by the Diagnostic and Statistical Manual of Mental Disorders: Fourth Edition-Text Revision (American Psychiatric Association, 2000). Record of first episode/first psychiatric hospitalization for all participants was obtained from patient charts and is as follows: Ages 12–16 (26%), ages 17–21 (42%), over 21 (16%), unknown (16%).

Table 1.

Demographic characteristics

Demographics Mean (SD) or n (%) 
Age (year) 51 (11.91), range = 26–72 
Gender (F) 28 (51%) 
Education (year) 10 (2.47), range = 4–16 
Race 
 Caucasian 34 (63%) 
 Asian 9 (16%) 
 African American 7 (13%) 
 Hispanic 4 (7%) 
Estimated Intellectual Ability (Standard Score) 89.6 (11.50) 
Diagnosis 
 Schizophrenia 34 (63%) 
 Undifferentiated 23 (67%) 
 Paranoid 11 (33%) 
 Schizoaffective 20 (37%) 
Age at First Episode 
 12–16 14 (26%) 
 17–21 22 (42%) 
 21 or over 9 (16%) 
 Unknown 9 (16%) 
Demographics Mean (SD) or n (%) 
Age (year) 51 (11.91), range = 26–72 
Gender (F) 28 (51%) 
Education (year) 10 (2.47), range = 4–16 
Race 
 Caucasian 34 (63%) 
 Asian 9 (16%) 
 African American 7 (13%) 
 Hispanic 4 (7%) 
Estimated Intellectual Ability (Standard Score) 89.6 (11.50) 
Diagnosis 
 Schizophrenia 34 (63%) 
 Undifferentiated 23 (67%) 
 Paranoid 11 (33%) 
 Schizoaffective 20 (37%) 
Age at First Episode 
 12–16 14 (26%) 
 17–21 22 (42%) 
 21 or over 9 (16%) 
 Unknown 9 (16%) 

All participants were residents of a 200-bed assisted living facility that provides independent living and supportive care including housing, meals, psychiatric treatment, medical services, and other support staff to patients with chronic mental illness. Approximately 50% of residents in the facility are diagnosed with psychotic disorders and treated with antipsychotic medications, psychotherapy twice per month, and psychiatric services every 2 months or as needed. For this study, participants were excluded if they had dual diagnoses of substance abuse/dependence, neurological disorders, and/or mental retardation by medical chart review.

Participants were recruited by means of sign-up sheets placed around the residential facility. Participants were also alerted to the study through staff members at the facility. Participants were compensated $5.00 for their time and were assured that they would receive payment regardless of whether they completed the study. All tests were administered by the primary investigator, a clinical neuropsychologist, and testing was completed in two sessions.

Measures

Subjects were administered the VIP Verbal and Non-verbal subtests (Frederick, 2003), the TOMM (Tombaugh, 1996), the STMS (Kokmen, Naessens, & Offord, 1987), and the Reading subtest of the WRAT-4 (Wilkinson & Robertson, 2006). The VIP is a self-administered, paper-and-pencil test consisting of a 78-item verbal subtest and a 100-item non-verbal subtest designed to detect suboptimal effort or malingering in cognitive assessment. Performance on the VIP Verbal and Non-verbal subtests is classified as either Compliant, or as one of four Invalid classifications: Inconsistent—performance is inconsistent; Irrelevant—performance may reflect random responding or little effort to answer correctly; and Suppressed—performance may have been affected by an intentional effort to respond incorrectly. The TOMM is a visual recognition task consisting of two learning trials (50 items each) and a retention trial (50 items) specifically designed for the assessment of insufficient effort defined as “likely malingering” during neuropsychological evaluations. The STMS was developed and validated as a brief screening mental status test to be used for detecting mild to severe cognitive difficulties, with a cutoff of ≤29 indicating a greater likelihood of impairment (Kokmen et al., 1987). Cognitive functions assessed include orientation, attention, learning/memory, calculation, abstraction, information, and construction. The WRAT-4 Reading subtest was used to estimate intellectual ability; scores on this subtest were transformed into standard scores with a mean of 100 and standard deviation of 15.

The design and requirements of this study were reviewed and approved by the Institutional Review Board.

Analysis

For all comparisons, the z-test for equality of proportions was used to assess differential passing rates and classification agreement; because a subset of cell counts contained inadequate numbers, Fisher's Exact Test was used to confirm significance in affected comparisons.

Passing rates on the VIP Verbal and Non-verbal subtests were calculated and compared with normative values for honest responders (Frederick & Crosby, 2000). Observed passing rates were then compared with published data for non-litigating psychiatric samples in which the majority of patients were diagnosed with a psychotic disorder (Non-verbal subtest [Frederick et al., 2000]; Verbal subtest [Pivovarova et al., 2010]).

Passing rates for the Retention Trial of the TOMM were calculated and compared with normative data (Tombaugh, 1997) of cognitively intact and cognitively impaired compliant responders (no cognitive impairment; cognitive impairment; traumatic brain injury; aphasia). Observed passing rates on the Retention Trial of the TOMM were then compared with individual and pooled published data for non-litigating psychiatric samples in which the majority of patients were diagnosed with a psychotic disorder and a Retention classification was available (Duncan, 2005; Hubbard, 2007; Weinborn et al., 2003). For analysis of the influence of general intellectual ability on effort test performance, WRAT-4 Reading standardized scores were dichotomized into two groups (≤79; ≥80). For analysis of the influence of mental status on effort test performance, STMS raw scores were dichotomized into two groups (≤29; ≥30).

Results

VIP Classification Accuracy

The VIP Verbal subtest exhibited a passing rate of 40%, with 21 of 53 subjects generating a Compliant classification. Invalid profiles were predominantly classified as Inconsistent (n = 21), with fewer Irrelevant classifications (n = 11), and no Suppressed classifications. The VIP Non-verbal subtest exhibited a passing rate of 17%, with 9 of 54 subjects generating a Compliant classification. Invalid profiles were again predominantly classified as Inconsistent (n = 26), with fewer Irrelevant classifications (n = 14), and five Suppressed classifications (Table 2). Comparison with published results in a presumed compliant, non-psychiatric sample (Frederick, 2003) revealed significantly lower passing rates in our sample for both Verbal (z = −8.58, p < .001) and Non-verbal subtests (z = −10.35, p < .001). Comparison with published results of presumed compliant, psychotic-disordered samples (Frederick et al., 2000; Pivovarova et al., 2010) revealed significantly lower passing rates in our sample for both Verbal (z = −3.68, p < .001) and Non-verbal subtests (z = −4.24, p < .001) (Table 3). No significant difference in the passing rate was observed between schizophrenia and schizoaffective subgroups for the VIP Verbal subtest (Fisher's Exact Test, p = .337). For the VIP Non-verbal subtest, a trend-significant difference was observed, with the schizophrenia subgroup exhibiting more invalid profiles than the schizoaffective subgroup (Fisher's Exact Test, p = .053).

Table 2.

Test performance

VIP Compliant (n [%]) Inconsistent (n [%]) Irrelevant (n [%]) Suppressed (n [%]) Specificity (n [%]) 
 VIP Verbal 21 (40) 21 (40) 11 (20) 0 (0) 21/53 (40) 
 VIP Non-verbal 9 (17) 26 (48) 14 (26) 5 (9) 9/54 (17) 
Test of Memory Malingering Valid (n [%]) Invalid (n [%]) Specificity (n [%])   
 Trial 2 39 (72) 15 (28) 39/54 (72)   
 Retention 45 (83) 9 (17) 45/54 (83)   
WRAT-4 Reading Valid (n [%]) Invalid (n [%])    
 ≤79 (n = 12)    
  VIP Verbal 0 (0) 11 (100)    
  VIP Non-verbal 1 (8) 11 (92)    
  TOMM Retention 5 (42) 7 (58)    
 ≥80 (n = 42)    
  VIP Verbal 21 (50) 21 (50)    
  VIP Non-verbal 8 (19) 34 (81)    
  TOMM Retention 40 (95) 2 (5)    
Short Test of Mental Status Valid (n [%]) Invalid (n [%])    
 ≤29 (n = 19)    
  VIP Verbal 2 (11) 16 (89)    
  VIP Non-verbal 1 (5%) 18 (95)    
  TOMM Retention 11 (58) 8 (42)    
 ≥30 (n = 35)    
  VIP Verbal 19 (56) 16 (34)    
  VIP Non-verbal 8 (23) 27 (77)    
  TOMM Retention 34 (97) 1 (3)    
VIP Compliant (n [%]) Inconsistent (n [%]) Irrelevant (n [%]) Suppressed (n [%]) Specificity (n [%]) 
 VIP Verbal 21 (40) 21 (40) 11 (20) 0 (0) 21/53 (40) 
 VIP Non-verbal 9 (17) 26 (48) 14 (26) 5 (9) 9/54 (17) 
Test of Memory Malingering Valid (n [%]) Invalid (n [%]) Specificity (n [%])   
 Trial 2 39 (72) 15 (28) 39/54 (72)   
 Retention 45 (83) 9 (17) 45/54 (83)   
WRAT-4 Reading Valid (n [%]) Invalid (n [%])    
 ≤79 (n = 12)    
  VIP Verbal 0 (0) 11 (100)    
  VIP Non-verbal 1 (8) 11 (92)    
  TOMM Retention 5 (42) 7 (58)    
 ≥80 (n = 42)    
  VIP Verbal 21 (50) 21 (50)    
  VIP Non-verbal 8 (19) 34 (81)    
  TOMM Retention 40 (95) 2 (5)    
Short Test of Mental Status Valid (n [%]) Invalid (n [%])    
 ≤29 (n = 19)    
  VIP Verbal 2 (11) 16 (89)    
  VIP Non-verbal 1 (5%) 18 (95)    
  TOMM Retention 11 (58) 8 (42)    
 ≥30 (n = 35)    
  VIP Verbal 19 (56) 16 (34)    
  VIP Non-verbal 8 (23) 27 (77)    
  TOMM Retention 34 (97) 1 (3)    

Notes: VIP = Validity Indicator Profile; TOMM = Test of Memory Malingering.

Table 3.

Comparison of reported sample with previously published results

VIP % Psychotic disorder Current sample
 
VIP Verbal VIP Non-verbal 
Compliant combined 
Frederick (2003) 0% z = −8.58* z = −10.35*- 
Psychosis 
Pivovarova and colleagues (2010) 85% z = −3.68* — 
Frederick and colleagues (2000) 100% — z = −4.24* 
  Current sample 
TOMM % Psychotic Disorder Trial 2 Retention 
Honest Responders Combined 
Tombaugh (1997) 0% z = −4.29* z = −3.78* 
Psychosis 
Pivovarova and colleagues (2010) 85% z = −3.40* — 
Hubbard (2007) 100% z = −1.45 z = −0.09 
Duncan (2005) 72% z = −2.93** z = −1.70*** 
Weinborn and colleagues (2003) 69% z = −1.22 z = −0.60 
 TOMM Psychosis Pooled 84% z = −3.35* z = −1.02 
VIP % Psychotic disorder Current sample
 
VIP Verbal VIP Non-verbal 
Compliant combined 
Frederick (2003) 0% z = −8.58* z = −10.35*- 
Psychosis 
Pivovarova and colleagues (2010) 85% z = −3.68* — 
Frederick and colleagues (2000) 100% — z = −4.24* 
  Current sample 
TOMM % Psychotic Disorder Trial 2 Retention 
Honest Responders Combined 
Tombaugh (1997) 0% z = −4.29* z = −3.78* 
Psychosis 
Pivovarova and colleagues (2010) 85% z = −3.40* — 
Hubbard (2007) 100% z = −1.45 z = −0.09 
Duncan (2005) 72% z = −2.93** z = −1.70*** 
Weinborn and colleagues (2003) 69% z = −1.22 z = −0.60 
 TOMM Psychosis Pooled 84% z = −3.35* z = −1.02 

Notes: VIP = Validity Indicator Profile; TOMM = Test of Memory Malingering.

*p ≤ .001.

**p ≤ .01.

***p ≤ .05.

TOMM Classification Accuracy

The TOMM Retention Trial exhibited a passing rate of 83%. Of protocols classified as invalid, no Retention score was significantly below chance (40; 40; 39; 39; 39; 38; 34; 34; 34) (Table 2). Comparison with published results of a compliant, non-psychiatric sample (Tombaugh, 1997) revealed significantly lower passing rates in our sample (z = −3.78, p < .001). Comparison with published results of compliant, psychotic-disordered samples pooled across three previous studies (Duncan, 2005; Hubbard, 2007; Weinborn et al., 2003) was not significant (z = −1.02; Table 3). For the TOMM Retention Trial, a trend-significant difference was observed between schizophrenia and schizoaffective subgroups, with the schizophrenia subgroup exhibiting more invalid profiles than the schizoaffective subgroup (Fisher's Exact Test, p = .078).

VIP and TOMM Agreement

Disagreement between classifications of the VIP and TOMM Retention Trial were significant for both the VIP Verbal subtest (z-test = −4.65, p < .001) and VIP Non-verbal subtest (z-test = −6.93, p < .001). Of 45 valid TOMM performances, 21 subjects (47%) were classified as Compliant on the VIP Verbal subtest, and nine subjects (2%) were classified as Compliant on the VIP Non-verbal subtest (Table 4).

Table 4.

VIP Non-verbal, Verbal, TOMM Trial 2, and TOMM Retention Agreement

 Compliant (n [%]) Inconsistent (n [%]) Irrelevant (n [%]) Suppressed (n [%]) 
 VIP Non-verbal 
VIP Verbal     
 Compliant 6 (11) 10 (19) 3 (6) 2 (4) 
 Inconsistent 1 (2) 11 (20) 8 (15) 2 (4) 
 Irrelevant 2 (4) 5 (9) 3 (6) 1 (2) 
 Suppressed 0 (0) 0 (0) 0 (0) 0 (0) 
 VIP Non-verbal 
TOMM     
 Retention Valid 9 (17) 22 (41) 10 (19) 4 (7) 
 Retention Invalid 0 (0) 4 (7) 4 (7) 1 (2) 
 VIP Verbal 
TOMM     
 Retention Valid 21 (31) 17 (32) 7 (13) 0 (0) 
 Retention Invalid 0 (0) 5 (9) 4 (7) 0 (0) 
 Compliant (n [%]) Inconsistent (n [%]) Irrelevant (n [%]) Suppressed (n [%]) 
 VIP Non-verbal 
VIP Verbal     
 Compliant 6 (11) 10 (19) 3 (6) 2 (4) 
 Inconsistent 1 (2) 11 (20) 8 (15) 2 (4) 
 Irrelevant 2 (4) 5 (9) 3 (6) 1 (2) 
 Suppressed 0 (0) 0 (0) 0 (0) 0 (0) 
 VIP Non-verbal 
TOMM     
 Retention Valid 9 (17) 22 (41) 10 (19) 4 (7) 
 Retention Invalid 0 (0) 4 (7) 4 (7) 1 (2) 
 VIP Verbal 
TOMM     
 Retention Valid 21 (31) 17 (32) 7 (13) 0 (0) 
 Retention Invalid 0 (0) 5 (9) 4 (7) 0 (0) 

Association with STMS Performance

Performance on the Short Test of Mental Status was significantly associated with classification on the TOMM (Fisher's Exact Test, p < .001), and with classification on the VIP Verbal subtest (Fisher's Exact Test, p < .01), with subjects in the low STMS group being more likely to be classified as invalid on either test (Table 2). No significant relationship was found between STMS performance and VIP Non-verbal classification.

Association with WRAT-4 Reading Score

Performance on the WRAT-4 Reading subtest was significantly associated with classification on the TOMM (Fisher's Exact Test, p < .001), and with classification on the VIP Verbal subtest (Fisher's Exact Test, p < .01), with subjects in the low WRAT-4 Reading group being more likely to be classified as invalid on either test (Table 2). No significant relationship was found between WRAT-4 Reading performance and VIP Non-verbal classification.

Discussion

Standalone and embedded measures of effort and motivation rely to a greater or lesser extent on the assumption that true cognitive impairment does not affect performance on these tests. This assumption is based on prior work documenting the performance of cognitively-impaired clinical groups on such measures which find adequate specificity (≥90) for honest or compliant responders. Although the performance of neurologically affected patient groups has been extensively studied on tests of effort and motivation, less is known about the effects of neurocognitive complications of psychiatric disorders, and specifically of psychosis-spectrum disorders, on effort testing.

Since individuals diagnosed with schizophrenia or schizoaffective disorder may be seen in a forensic context, more extensive normative data are needed to address the influence of cognitive dysfunction on effort testing in affected individuals. We chose to focus on the VIP because of the comparatively few published studies that have evaluated this measure on patient groups and, more specifically, in individuals diagnosed with a schizophrenia or schizoaffective disorder. VIP performance was compared with normative data, with the limited, available data in similar psychiatric samples, and with TOMM classifications to assess agreement between measures in this cohort. We further assessed the influence of mental status and premorbid cognitive ability on effort test performance.

First, our findings indicate that performance on the VIP is significantly decreased in this sample of individuals diagnosed with schizophrenia or schizoaffective disorder. Passing rates on the Verbal (40%) and Non-verbal (17%) subtests in our sample were significantly lower than normative data for honest responders in the original standardization sample (94% and 90%, respectively; Frederick, 2003), and significantly lower than those found in previous research in predominantly psychotic-disordered samples (Verbal 70%; Non-verbal 55%; Frederick et al., 2000; Pivovarova et al., 2010). The majority of invalid classifications for both the Verbal and Non-verbal subtests were classified as Inconsistent, suggesting that performance was uneven for less demanding items in Sector 1. Although five patients were classified as Suppressed on the Non-verbal subtest, the second most common invalid classification was that of Irrelevant for both Verbal and Non-verbal subtests. Of note, the passing rate on the TOMM (83%) was less affected by psychiatric and neurocognitive difficulties in our sample, but still lower than normative data for a mixed sample of cognitively intact and cognitively impaired individuals (99%; Tombaugh, 1997). In contrast, when compared with a pooled sample of previously published passing rates in psychotic-spectrum individuals (Duncan, 2005; Hubbard, 2007; Weinborn et al., 2003), no significant difference was observed (83% vs. 89%). Disagreement between the classifications of the TOMM and the VIP was noted to be higher in the present study than in previous work (Farkas, Rosenfeld, Robbins, & van Gorp, 2006). In that study, using TOMM classifications as the index variable, VIP classifications exhibited higher sensitivity and specificity in a sample of forensic referrals for both Verbal (Sen: 75%; Spec: 93%) and Non-verbal (Sen: 75%; Spec: 50%) subtests than in the current study.

The significantly lower VIP passing rates in our sample than in Pivovarova and colleagues (2010) may be partly due to differences in participant characteristics between studies, specifically in disease severity and intellectual ability. Although disease severity cannot be definitively compared between the two studies, participants in the Pivovarova study were recruited as outpatients through a psychiatric hospital, whereas all of our participants required supervised residential care in a facility for the chronically mentally ill. In regard to intellectual ability, which also cannot be directly compared between studies, we note that in our study, 12 of 54 participants had estimated intellectual ability below a standard score of 80 and that intellectual ability was significantly associated with effort test performance. Although the VIP manual cautions against use of the VIP with individuals diagnosed with mental retardation (Frederick, 2003), lower intellectual ability without a documented history of mental retardation is not considered a contraindication to administration. Results from this study suggest that even in individuals not diagnosed with mental retardation, estimated borderline range of intellectual ability may significantly influence tested performance.

These results serve to underscore the importance of testing the influence of both psychiatric and neurocognitive symptoms on effort test performance in distinct clinical groups. Our results suggest that the VIP, specifically, will be potentially influenced by psychiatric symptomatology, associated neurocognitive deficits, current mental status, as well as estimated intellectual ability. One important issue is the extent to which our sample is representative of schizophrenia or schizoaffective samples more broadly. We note that our sample consisted of chronically affected individuals whose psychiatric symptoms had necessitated primary placement in a residential facility. Certainly, the range of severity in psychotic disorders more generally is broad and the individuals included in our sample may represent the most severely affected. Of note, however, is the finding that the TOMM passing rate in our sample was not significantly lower than the aggregate specificity rate of samples that included psychotic-disordered individuals in previous studies, while significantly lower passing rates were observed only for the VIP Verbal and Non-verbal subtests.

Although our results are suggestive, limitations are identified that should be addressed in future studies. One question that should be further clarified is the issue of whether the invalid VIP classifications are actually in error in our patient sample. As noted above, the majority invalid classification was for an Inconsistent profile, with the second most common classification being for an Irrelevant profile. Both of these classifications would suggest either uneven attention and inconsistent focus or little effort to answer accurately, but at the same time no strong effort to answer incorrectly. Given the effects of schizophrenia and schizoaffective disorder on attention, focus, and disinhibition, it could be argued that the VIP classifications are accurately representing the attitude and approach to testing taken by our patient sample. Furthermore, absent the direct effect of cognitive deficits on performance, there is also the possibility that amotivation, disengagement, or lack of interest in the testing in a research setting may have adversely affected performance. Indeed, the VIP manual (Frederick, 2003) suggests that individuals with lower intellectual ability may “stop trying after they encounter more difficult items mixed among easy ones.” (p. 42). In such cases, the manual indicates that test administration may be altered by guiding evaluees through the administration using prompts and altered instructions for completion.

In summary, results of this study indicate significantly altered performance on standalone tests of effort and motivation in a sample of individuals diagnosed with schizophrenia and schizoaffective disorder. At this point, the independent influence of psychiatric, neurocognitive, intellectual, and motivational factors cannot be further assessed. Findings in our sample suggest that intellectual ability and mental status may both be implicated in higher failure rates, particularly on the VIP Verbal subtest. Future studies might clarify independent effects on effort test performance with the use of a broader battery that would assess psychiatric, neurocognitive, intellectual, and motivational factors. These observations notwithstanding, our results would recommend caution in using effort tests in individuals diagnosed with schizophrenia or schizoaffective disorder, particularly in those individuals in whom low intellectual ability is assessed or suspected.

Funding

This work was supported by the PSC-CUNY Research Award.

Conflict of Interest

None declared.

Acknowledgements

The authors wish to thank Raffi Leicht, who assisted in database entry and management of records.

References

American Psychiatric Association
Diagnostic and Statistical Manual of Mental Disorders
 , 
2000
Washington, DC
Author
Arnold
G.
Boone
K. B.
Lu
P.
Dean
A.
Wen
J.
Nitch
S.
, et al.  . 
Sensitivity and specificity of finger tapping test scores for the detection of suspect effort
The Clinical Neuropsychologist
 , 
2005
, vol. 
19
 
1
(pg. 
105
-
120
)
Back
C. A.
Boone
K. B.
Edwards
C.
Parks
C.
Burgoyne
K.
Silver
B.
The performance of schizophrenics on three Cognitive Tests of Malingering, Rey 15-Item Memory Test, Rey Dot Counting, and Hiscock Forced-Choice Method
Assessment
 , 
1996
, vol. 
3
 
4
(pg. 
449
-
457
)
Dean
A. C.
Victor
T. L.
Boone
K. B.
Arnold
G.
The relationship of IQ to effort test performance
The Clinical Neuropsychologist
 , 
2008
, vol. 
22
 
4
(pg. 
705
-
722
)
Dean
A. C.
Victor
T. L.
Boone
K. B.
Philpott
L. M.
Hess
R. A.
Dementia and effort test performance
The Clinical Neuropsychologist
 , 
2009
, vol. 
23
 
1
(pg. 
133
-
152
)
Duncan
A.
The impact of cognitive and psychiatric impairment of psychotic disorders on the test of memory malingering (TOMM)
Assessment
 , 
2005
, vol. 
12
 
2
(pg. 
123
-
129
)
Egeland
J.
Sundet
K.
Rund
B. R.
Asbjornsen
A.
Hugdahl
K.
Landro
N. I.
, et al.  . 
Sensitivity and specificity of memory dysfunction in schizophrenia: A comparison with major depression
Journal of Clinical and Experimental Neuropsychology
 , 
2003
, vol. 
25
 
1
(pg. 
79
-
93
)
Farkas
M. R.
Rosenfeld
B.
Robbins
R.
van Gorp
W.
Do Tests of Malingering concur? Concordance among malingering measures
Behavioral Sciences and the Law
 , 
2006
, vol. 
24
 
5
(pg. 
659
-
671
)
Frederick
R. I.
Validity Indicator Profile manual
 , 
2003
Texas
Pearson
Frederick
R. I.
Crosby
R. D.
Development and validation of the Validity Indicator Profile
Law and Human Behavior
 , 
2000
, vol. 
24
 
1
(pg. 
59
-
82
)
Frederick
R. I.
Crosby
R. D.
Wynkoop
T. F.
Performance curve classification of invalid responding on the Validity Indicator Profile
Archives of Clinical Neuropsychology
 , 
2000
, vol. 
15
 
4
(pg. 
281
-
300
)
Gierok
S. D.
Dickson
A. L.
Cole
J. A.
Performance of forensic and non-forensic adult psychiatric inpatients on the Test of Memory Malingering
Archives of Clinical Neuropsychology
 , 
2005
, vol. 
20
 
6
(pg. 
755
-
760
)
Goldberg
H. E.
Back-Madruga
C.
Boone
B. B.
Boone
B. B.
The impact of psychiatric disorders on cognitive Symptom Validity Test scores
Assessment of feigned cognitive impairment: A neuropsychological perspective
 , 
2007
New York
The Guilford Press
(pg. 
281
-
309
)
Gorissen
M.
Sanz
J. C.
Schmand
B.
Effort and cognition in schizophrenia patients
Schizophrenia Research
 , 
2005
, vol. 
78
 
2–3
(pg. 
199
-
208
)
Hoff
A. L.
Svetina
C.
Shields
G.
Stewart
J.
DeLisi
L. E.
Ten year longitudinal study of neuropsychological functioning subsequent to a first episode of schizophrenia
Schizophrenia Research
 , 
2005
, vol. 
78
 
1
(pg. 
27
-
34
)
Hubbard
K. L.
Feigning cognitive deficits among psychiatric inpatients: A validation of three measures of cognitive malingering
 , 
2007
Graduate School of The University of Alabama, Proquest
 
PhD
Kokmen
E.
Naessens
J. M.
Offord
K. P.
A short test of mental status: Description and preliminary results
1987
, vol. 
62
 
4
(pg. 
281
-
288
)
Kuperberg
G.
Heckers
S.
Schizophrenia and cognitive function
Current Opinion in Neurobiology
 , 
2000
, vol. 
10
 
2
(pg. 
205
-
210
)
Lee
J.
Park
S.
Working memory impairments in schizophrenia: A meta-analysis
Journal of Abnormal Psychology
 , 
2005
, vol. 
114
 
4
(pg. 
599
-
611
)
Lesser
I. M.
Miller
B. L.
Boone
K. B.
Hill-Gutierrez
E.
Mehringer
C. M.
Wong
K.
, et al.  . 
Brain injury and cognitive function in late-onset psychotic depression
Journal of Neuropsychiatry and Clinical Neurosciences
 , 
1991
, vol. 
3
 
1
(pg. 
33
-
40
)
Miller
B. L.
Lesser
I. M.
Boone
K. B.
Hill
E.
Mehringer
C. M.
Wong
K.
Brain lesions and cognitive function in late-life psychosis
British Journal of Psychiatry
 , 
1991
, vol. 
158
 (pg. 
76
-
82
)
Pivovarova
E.
Rosenfeld
B.
Dole
T.
Green
D.
Zapf
P.
Are measures of cognitive effort and motivation useful in differentiating feigned from genuine psychiatric symptoms?
International Journal of Forensic Mental Health
 , 
2010
, vol. 
9
 
4
(pg. 
271
-
278
)
Sharma
T.
Antonova
L.
Cognitive function in schizophrenia. Deficits, functional consequences, and future treatment
Psychiatric Clinics of North America
 , 
2003
, vol. 
26
 
1
(pg. 
25
-
40
)
Tombaugh
T. N.
Test of Memory Malingering manual
 , 
1996
New York
MHS
Tombaugh
T. N.
The Test of Memory Malingering (TOMM): Normative data from cognitively intact and cognitively impaired individuals
Psychological Assessment
 , 
1997
, vol. 
9
 
3
(pg. 
260
-
268
)
Weinborn
M.
Orr
T.
Woods
S. P.
Conover
E.
Feix
J.
A validation of the test of memory malingering in a forensic psychiatric setting
Journal of Clinical and Experimental Neuropsychology
 , 
2003
, vol. 
25
 
7
(pg. 
979
-
990
)
Wilkinson
G.
Robertson
G.
Wide Range Achievement Test-fourth edition
 , 
2006
Florida
Psychological Assessment Resources