Inadequate effort during neuropsychological examination results in inaccurate representations of an individual's true abilities and difficulties. As such, performance validity tests (PVTs) are strongly recommended as standard practice during adult-based evaluations. One concern with using PVTs with children is that failure reflects immature cognitive ability rather than non-credible effort. The current study examined performance on the Medical Symptom Validity Test (MSVT) in two large pediatric clinical samples with strikingly different neuropsychological profiles: (1) mild traumatic brain injury (mTBI; n = 510) and (2) fetal alcohol spectrum disorder (FASD; n = 120). Despite higher IQ scores and reading ability, the mTBI group performed significantly worse than the FASD group on all effort indices. Sixteen percent of the mTBI group failed the MSVT, whereas only 5% of the FASD group did. Our findings support the idea that the MSVT measures effort, not ability, in most cases and help to justify incorporating PVTs into pediatric neuropsychological batteries.
Neuropsychological test interpretation relies on individuals putting forth their best effort to perform well. Because inadequate effort during neuropsychological examination will result in inaccurate representation and misinterpretation of an individual's true abilities and difficulties, performance validity tests (PVTs) are strongly recommended as standard practice during adult-based evaluations by both the American Academy of Clinical Neuropsychology and the National Academy of Neuropsychology (Bush et al., 2005; Heilbronner et al., 2009). While youth have been thought historically to be reliable reporters of their symptoms and less capable of deception than adults (Salekin, Kubak, & Lee, 2008), several studies have shown that children and adolescents are capable of exaggerating symptoms and feigning cognitive impairment during neuropsychological evaluation (Faust, Hart, & Guilmette, 1988; Faust, Hart, Guilmette, & Arkes, 1988; Flaro & Boone, 2009; Henry, 2005; Kirkwood, Kirk, Blaha, & Wilson, 2010; Lu & Boone, 2002; MacCaffery & Lynch, 2009).
In order to evaluate non-credible effort in children and adolescents, a growing body of research has examined the value of PVTs in pediatric populations. Recent studies indicate that high percentages of school-aged children are capable of passing several stand-alone PVTs using adult-based cutoff scores (Deright & Carone, 2015; Kirkwood, Yeates, Randolph, & Kirk, 2012).
The Medical Symptom Validity Test (MSVT) was developed for use with both adult and pediatric populations (Green, 2004) and has been well validated in both populations (see Green, Flaro, Brockhaus, & Montijo, 2012). It involves computerized presentation of 10 word pairs over two trials and then examination of both forced-choice word recognition (i.e., effort indices or “easy” subtests) and recall (i.e., memory indices or “hard” subtests). The MSVT effort indices can be easily passed with mean scores well above the pass/fail cutoff by English- and non-English-speaking children and adolescents with a reading level greater than 2nd grade who put forth adequate effort. Studies have reported that 96%–98% of healthy children from Germany, Brazil, and Canada passed the MSVT (Blaskewitz, Merten, & Kathmann, 2008; Green, 2004). Furthermore, ∼95% of children with moderate-to-severe brain injury or developmental disability scored above the recommended pass/fail cutoff (Carone, 2008; Green et al., 2012).
Despite evidence that the MSVT effort indices can be passed when effort is adequate, several studies have reported relatively high rates of failure on the MSVT effort indices across certain populations of neurologically impaired patients, such as adults with dementia (Green et al., 2012; Howe & Loring, 2009). Given the impairments associated with dementia, it is not unexpected that adults with such a condition could fail the MSVT. More specifically, genuine memory impairment is identified among individuals who show significantly impaired recall in the context of relatively spared recognition. So as to not misclassify individuals with severe impairment as producing non-credible effort, a “dementia profile” (a.k.a., “severe impairment profile”) was developed for the MSVT to identify genuine versus feigned problems (Howe & Loring, 2009).
Given that even low functioning children have been found to pass the MSVT, the base rate of MSVT failure in some subgroups of children have been surprisingly high. Specifically, in a sample of 96 children whose parents were seeking disability from the Disability Determination Service on their behalf, 20% of the claimants failed the MSVT (Chafetz, 2008). Similarly, in a large mild traumatic brain injury (mTBI) sample of children aged 8–17 years (n = 193), Kirkwood & Kirk (2010) found that 17% of the sample failed the MSVT. The authors attributed most failures to non-credible effort, rather than ability, because those participants who failed the MSVT were first, no different in terms of injury severity or premorbid reading/learning difficulties from those participants who passed the MSVT; second, considerably older and in a higher grade than children from prior studies who were able to pass the MSVT when putting forth good effort; and third, found to have other inconsistencies in their presentation. A follow-up study replicated these findings in a sample of 276 children ages 8–16 years with mTBI and found a 19% failure rate on the MSVT (Kirkwood et al., 2012). The authors also found that MSVT performance was significantly correlated with ability-based neuropsychological tests, accounting for 38% of the variance overall. Those participants failing the MSVT effort indices performed significantly worse on nearly all neuropsychological tests. Likewise, a more recent study by the same group found that those participants failing the MSVT were significantly more likely to endorse postconcussive symptoms than those participants who passed the MSVT, even after controlling for other factors known to influence symptom reporting in this sample (Kirkwood, Peterson, Connery, Baker, & Grubenhoff, 2014).
Some practitioners remain concerned that failure on PVTs like the MSVT may reflect immature or impaired cognitive ability rather than non-credible effort (see Bigler, 2012; Larabee, 2012). To better understand the limits of validity testing and to determine if the MSVT measures effort or ability in children, the current study examined performance on the MSVT in two pediatric clinical samples with strikingly different neuropsychological profiles: first, mTBI and second, fetal alcohol spectrum disorder (FASD).While mTBI causes alterations in neurocognitive functioning, individuals typically return to their baseline level of performance on objective cognitive tests within a few weeks or less (e.g., Babikian et al., 2011). In contrast, children with FASD have an average IQ ∼70 and marked deficits across numerous neurobehavioral domains (Mattson & Vaurio, 2010). If the MSVT is a test of ability, those with more severe impairment (i.e., FASD group) would be expected to exhibit greater failure rates than those with less severe impairment (i.e., mTBI group). In contrast, if the MSVT is a test of effort and not ability, then those with severe impairment would be expected to perform no worse than those with mild impairment. In line with this expectation and based on previous work (e.g., Carone, 2008; Green, Flaro, & Courtney, 2009; Kirkwood & Kirk, 2010), we hypothesized that clinically referred participants with mTBI would have an equal (or greater) failure rate on the MSVT effort indices than those with FASD. In addition, consistent with the idea that the recall conditions of the MSVT are more difficult and are measures of ability (i.e., memory), not effort, we expected that the FASD group would perform more poorly than the mTBI group on these indices.
A total of 630 children aged 8–17 years were included in the study. Of the total sample, 510 sustained mTBI, and 120 participants had FASD. The children in the mTBI sample were drawn from a large clinical database of a 5-year series of clinical cases referred consecutively to an outpatient concussion program at a children's hospital in the Intermountain region of the United States. Earlier subgroups of this same case series have been presented previously (Baker, Connery, Kirk, & Kirkwood, 2014; Green, Kirk, Connery, Baker, & Kirkwood, 2014; Kirk, Hutaff-Lee, Connery, Baker, & Kirkwood, 2014; Kirkwood, Connery, Kirk, & Baker, 2014; Kirkwood, Hargrave, & Kirk, 2011; Kirkwood & Kirk, 2010; Kirkwood et al. 2012; Kirkwood, Peterson, et al., 2014). All patients in the mTBI group sustained blunt head trauma within 12 months of their evaluation and underwent testing no earlier than 1 week post-injury and no later than 52 weeks post-injury (median testing time was 6 weeks post-injury). Patients were generally referred because of concerns about persistent symptoms related to underlying brain injury. With the exception of a few select cases in which the head injury was unwitnessed and reliable acute injury data were unavailable, all patients were reported to have displayed clear evidence of concussion (i.e., alteration in mental status, loss of consciousness, post-traumatic amnesia, or transient neurologic disturbance). The causes of injury in this sample were: sports or recreational (60%), falls (17%), motor-vehicle-related trauma (11%), assault (3%), and other (4%). For those children who had intracranial pathology on neuroimaging (8%), their Glasgow Coma Scale was never less than a score of 13. Exclusionary criteria for the mTBI group were forensic referral, neurosurgical intervention, injury resulting from abuse, and nontraumatic brain injury (e.g., hypoxia). Of note, a small percentage of the families (7.25%) referred for clinical evaluation were engaged in or considering litigation related to the cause of the injury; however, given that these few children were referred clinically and not for an independent forensic evaluation they were deemed eligible for inclusion in the study.
The participants with FASD were drawn from a large clinical database of an 8- to 10-year series of clinical cases referred for neuropsychological assessment to the second author's private practice located in the Western region of Canada. With the exception of two children, both of whom were referrals from lawyers with respect to insurance claims, all children in the FASD group were clinical referrals. Children in the FASD group were considered eligible for this study if they had a primary diagnosis of FASD. This diagnosis was established by a multidisciplinary team (i.e., pediatrician, speech therapist, occupational therapist, occupational therapist, and neuropsychologist) working within a clinic specializing in FASD assessment and diagnosis using a four variable model: first, growth deficiency; second, facial dysmorphology; third, central nervous system functioning (i.e., neuropsychological assessment and the obtained profile of strengths and weaknesses); and fourth, parent admission of prenatal alcohol exposure.
All children in this study received the MSVT as part of a routine neuropsychological battery. Those children in the mTBI group were administered an abbreviated neuropsychological battery that lasted ∼1–2 h. Those children in the FASD group were administered a neuropsychological test battery over a day and a half of testing.
The MSVT (Green, 2004) is a computerized forced-choice verbal memory test designed to evaluate effort and memory. Its normative data is based on over 1,000 children and adults with various clinical diagnoses and healthy controls. The primary effort indices are the Immediate Recognition (IR), Delayed Recognition (DR), and Consistency (CON) scores. The consistency score reflects the consistency of responses between the IR and DR conditions. The test requires ∼5 min of direct administration time (i.e., not including the delay time between IR and DR). Examinees are presented with 10 semantically related word pairs twice on a computer screen. During the IR and DR conditions, they are then asked to choose the correct word from pairs consisting of the target and a foil. The IR condition is completed immediately following the second presentation of the 10 word pairs and the DR condition is completed following a 10-min break. During both conditions, examinees receive auditory and visual feedback about the correctness of each response. Following the DR condition, examinees are asked to recall the words during Paired Associate (PA) and Free Recall (FR) conditions. These latter two conditions are thought to be more reflective of memory as opposed to effort (Green, 2004). The published criteria proposed by Green (2004) were considered indicative of non-credible effort.
Two different measures that are nationally standardized with satisfactory psychometric properties and that are highly correlated (>0.81) were used to estimate overall cognitive ability (Strauss, Sherman, & Spreen, 2006; Flanagan & Kaufman, 2009). The children in the mTBI group completed the two-subtest version of the Wechsler Abbreviated Scale of Intelligence (Wechsler, 1999). The children in the FASD group completed either the Third or Fourth edition of the Wechsler Intelligence Scale for Children (Wechsler, 1991, 2003). Additionally, three different nationally standardized measures of academic achievement were used to assess single-word reading ability across the two groups. The children in the mTBI group completed the Letter-Word Identification subtest from the Woodcock–Johnson 3rd edition Tests of Achievement (Woodcock, McGrew, & Mather, 2001). The FASD group completed the Basic Word Reading subtest from either the Test of Academic Performance (Adams, Erb, & Sheslow, 1998) or the Wide Range Achievement Test-4th edition (Wilkenson & Roberston, 2006). Grade-level reading equivalence was obtained from the reading measures, to serve as a gross measure of reading level for both groups.
Independent sample t-tests revealed no significant differences between groups on age (t = 0.22, df = 628, p = .826; Cohen's d = 0.019). However, significant differences were reported for FSIQ (or estimated FSIQ) (t = 20.18, df = 628, p< .001; Cohen's d = 2.07) and single-word reading grade equivalence (t = 10.82, df = 628, p < .001; Cohen's d = 1.20). As expected, the mTBI group had a significantly higher estimated FSIQ and reading ability (FSIQ mean = 103, reading grade equivalence mean = 9:1) than the FASD group (FSIQ mean = 78, reading grade equivalence mean = 5:8). The sample was predominantly male (61%), and there were no differences in gender distribution across groups (χ2 = 0.312). Demographic variables for the two groups are described in Table 1.
|Entire sample||Passed MSVT||Failed MSVT|
|FASD (n = 120)||mTBI (n = 510)||FASD (n = 114)||mTBI (n = 429)||FASD (n = 6)||mTBI (n = 81)|
|Gender (% male)||65%||60%||65%||60%||67%||59%|
|Entire sample||Passed MSVT||Failed MSVT|
|FASD (n = 120)||mTBI (n = 510)||FASD (n = 114)||mTBI (n = 429)||FASD (n = 6)||mTBI (n = 81)|
|Gender (% male)||65%||60%||65%||60%||67%||59%|
Notes: MSVT= Medical Symptom Validity Test; FSIQ= full-scale IQ; GE = grade equivalence.
*Group difference of p < .05.
**Group difference of p < .01.
Using the recommended published cutoff scores (Green, 2004), 87 (14%) of the 630 participants failed one or more of the primary MSVT effort indices. When comparing the two clinical groups, a significantly greater percentage of children with mTBI failed the MSVT than did children with FASD (χ2 = 0.001; 16% mTBI (n = 81) vs. 5% FASD (n = 6); odds ratio = 3.6, 95% CI = 1.5–8.4; see Fig. 1).
Within the FASD group, MSVT failure was not related to gender. However, failure in the FASD group was associated with age and grade, with those participants failing the MSVT being significantly younger (p = .014) and in a lower grade (p = .004). When comparing mTBI participants who failed the MSVT to those mTBI participants who passed the MSVT, the two groups did not differ in gender, age, grade, maternal education, history of premorbid LD, ADHD, litigation status, time since injury, or whether the injury was associated with loss of consciousness. In all cases p > .05. Interestingly, however, when examining neuroimaging pathology a greater percentage of children with negative neuroimaging findings failed the MSVT than those with positive neuroimaging findings (i.e., 18% vs. 5%, respectively), suggesting that MSVT failure could not be accounted for by injury complexity. Regarding litigation status, of the 81 mTBI participants who failed the MSVT, only six families (7.4%) reported being engaged in or considering litigation. This percentage is consistent with the percent of families engaged in or considering litigation in the mTBI group that passed the MSVT (i.e., 31 families or 7.2% of participants).
Across both groups, those participants who passed the MSVT performed nearly perfectly on all conditions except for FR. Mean performance across groups for the effort conditions, was as follows, with very little variability: IR (99.4%, SD = 1.8), DR (98.9%, SD = 2.4), and CON (98.7%, SD = 2.6). More variability was seen in the PA condition, but mean performance was also nearly perfect for both the mTBI and FASD pass groups (97.1%, SD = 9.2). Mean performance on FR was 74.6% (SD = 14.7), which is comparable with normative data for adolescents published by the test developer (Green, 2004). In contrast, for those participants who failed the MSVT, mean performance across all of the MSVT conditions was well below the published cutoff scores: IR (78.6%), DR (70.0%), CON (70.3%), PA (65.1%), and FR (49.5%).
As expected, when comparing the clinical groups, the FASD group performed no worse than the mTBI group on the effort indices. Indeed, despite having lower IQ and worse reading skills, the FASD group actually significantly outperformed the mTBI group on all effort indices: IR (t = 2.46, df = 628, p = .016; Cohen's d = 0.24), DR (t = 2.99, df = 628, p = .005; Cohen's d = 0.37), and CON (t = 2.68, df = 628, p = .009; Cohen's d = 0.32) conditions. When examining the MSVT primary memory indices that are thought to reflect ability more than effort, there was no difference in mean performance across groups on the PA condition (t = 0.750, df = 628, p = .453; Cohen's d = 0.08), but the mTBI group performed significantly better on the FR condition (t = 3.37, df = 628, p = .002; Cohen's d = 0.33, see Fig. 2).
Group differences in the PA condition emerged when the mTBI group was separated into those who passed the MSVT effort indices and those who did not. More specifically, performance on the PA condition was significantly lower in the FASD pass group as compared with the mTBI pass group (t = 3.41, df = 541, p = .002; Cohen's d = 0.28), although the FASD pass group was still 94.5% accurate. Performance in the FASD pass group was significantly higher than the mTBI fail group (t = −11.2, df = 193, p < .001; Cohen's d = 1.58). Examination of the FR condition across groups yielded similar results. The FASD pass group performed significantly lower than the mTBI pass group (t = 6.13, df = 541, p < .001; Cohen's d = 0.59), but significantly better than the mTBI fail group (t = −6.96, df = 193, p < .001; Cohen's d = 1.01, see Fig. 3). Notably, the p values reported above for all multiple group comparisons were adjusted for multiplicity using the Benjamini and Hochberg (1995) correction to minimize false discovery rate.
The current study examined whether the MSVT validity indices measure effort or ability by contrasting the performance of two pediatric clinical populations with strikingly different neuropsychological profiles: mTBI and FASD. Consistent with our first hypothesis and with previous literature (e.g., Carone, 2008; Green et al., 2009; Kirkwood & Kirk, 2010), our findings support the idea that the MSVT is more likely to measure effort than ability in most children 8 years and older. This is evidenced by the FASD group significantly outperforming the mTBI group on all effort indices, despite their considerably lower cognitive abilities. In fact, the MSVT was easy enough that 95% of the children with FASD, whose mean IQ fell in the borderline range, passed the effort indices with almost perfect accuracy. In contrast, only 84% of the children with mTBI passed the MSVT effort indices, even though their mean IQ fell solidly in the average range (see Fig. 1). Notably, although failure in the children with FASD was associated with being younger and in a lower grade, this was not the case in children with mTBI. In fact, MSVT failure in the children with mTBI was not associated with any demographic or injury-related factors, premorbid LD, ADHD, or reading problems. Overall, our finding is consistent with several other studies which have found that the MSVT can be easily passed by a wide range of clinical and non-clinical pediatric samples, including a sample of children with moderate-to-severe brain dysfunction (Carone, 2008).
Rates of non-credible effort in the mTBI sample are consistent with previous analyses of a subset of the same case series, which found 15% or more of the children failed the MSVT (Kirkwood & Kirk 2010; Kirkwood et al. 2011, 2012; Kirkwood, Connery, et al., 2014). The reasons why children provided non-credible performance on the MSVT was not the focus of the current study. Suffice it to say, however, the clinicians evaluating the study participants judged the reasons to be quite varied and to include attempts to obtain external gains (e.g., additional support at school) and/or fulfill internal psychological needs (e.g., somatoform disorder). Certain children were also judged to be simply noncompliant. These and many other possible explanatory factors for non-credible effort in children have been discussed elsewhere in a separate case-based analysis (Kirkwood et al., 2010).
While the primary aim of the paper was to examine performance on the MSVT effort indices, the secondary aim was to examine performance on the recall conditions of the MSVT that are considered to be more difficult and reflective of ability (i.e., memory), rather than effort. Our hypothesis that the FASD group would perform more poorly than the mTBI group on the MSVT memory measures was partially supported. Consistent with expectations, performance on the FR condition was significantly lower in the FASD than in the mTBI sample. These findings further support the publisher's intent that the “easy” conditions (i.e., effort conditions) of the MSVT tap effort-related processes, while the FR condition taps ability-related processes (e.g., memory). In contrast, performance on the PA condition was comparable across clinical groups, with both the FASD and mTBI groups performing quite accurately on this condition. Together, the current findings corroborate previous reports of performance on the FR and PA conditions in pediatric samples (Carone, 2008; Kirkwood & Kirk, 2010), in that FR performance was found to be associated with cognitive ability, whereas no differences were found in the PA condition. Notably, when performance on the PA condition is examined by comparing those children who put forth credible effort, as opposed to non-credible effort, differences in PA performance begin to emerge. Specifically, findings indicated that those children with FASD who passed the MSVT, significantly outperformed the children in the mTBI group who failed the MSVT, with performances averaging 30 points higher for children with FASD. As suggested by Kirkwood and Kirk (2010), the PA condition may serve as a better indicator of effort than ability, although future research is required in this regard.
The results of the current study need to be interpreted in the context of several limitations. First, the data were collected by two different groups of practitioners for clinical purposes and thus were constrained by specific patient needs and the protocol being followed at the time of evaluation. Although all patients were administered the MSVT, the number of ability-based measures shared by all participants was limited, restricting the analyses that could be conducted to examine the impact of PVT failure on other ability-based measures. Future studies should work towards collecting a broader array of both effort and ability-based measures to better understand the implications of MSVT failure across different pediatric populations. Furthermore, another limitation to this study was that the MSVT was the only stand-alone PVT administered to all patients so misclassification may have occurred. However, false negatives were thought to be relatively few based on an earlier analysis of a subset of the same case series of children with mTBI that carefully considered potential false positives and false negatives (Kirkwood & Kirk, 2010). Another limitation was that participants in this study were drawn from samples of convenience. Participants were comprised of children and adolescents for whom persistent concerns were apparent following mTBI, or those with FASD who had been identified as having learning or behavioral difficulties significant enough to warrant a neuropsychological evaluation. As such, our participants do not represent all individuals with mTBI or FASD. Future research could beneficially examine PVT performance in non-referred community samples to better understand how frequently PVT failure occurs in non-selected samples. Finally, it is important to acknowledge that within the FASD group, age and lower grade were associated with failure on the MSVT, suggesting that the MSVT should be used cautiously in lower functioning children in the early school years.
To our knowledge, this is the first study to directly examine validity testing in two large pediatric clinical samples. By comparing two samples with presumably such strikingly different neuropsychological profiles, we can be fairly certain that failure on the MSVT effort indices reflects non-credible effort and not genuine ability-based impairment for most school-age children. Our findings, together with previous studies, provide compelling evidence that PVTs could be confidently added to most pediatric batteries. This is of paramount importance for several reasons including that some children undoubtedly put forth non-credible effort at least some of the time and failure to identify non-credible effort can deleteriously impact diagnostic and clinical management decisions.
Conflict of Interest