Abstract

Suppressed Digit Span performance has been proposed as an embedded indicator for suboptimal effort detection in neuropsychological evaluations in Western societies, particularly in the USA. However, its effectiveness in Chinese countries remains unexplored. The purposes of this study were first to explore normative Digit Span performance patterns between the Taiwan and American standardization samples, then to examine performances of patients with traumatic brain injury and with psychiatric diseases on the embedded measures (the Digit Span Scaled Score, Vocabulary minus Digit Span difference score, Reliable Digit Span, and the longest string of digits forward and backward) through retrospective data analysis. The normative Digit Span performance differs between the two cultural populations. Although litigating and nonlitigating participants perform differently on these measures, further prospective studies are needed to explore this issue with comprehensive external corroborating validity data.

Introduction

Neuropsychological assessments can contribute important information on diagnosis, prognosis, and treatment to individuals with brain injury or disease (Lezak, Howieson, & Loring, 2004). However, test results may be useless because of patients' feigning or suboptimal effort with various grounds, especially when incentives are present (Mathias, Greve, Bianchini, Houston, & Crouch, 2002; Tan, Slick, Strauss, & Hultsch, 2002). Recently, detection of suboptimal effort as an essential component of neuropsychological assessments has become a consensus (Heilbronner, Sweet, Morgan, Larrabee, & Millis, 2009).

Techniques of Suboptimal Effort Detection

In the Western societies, detection of poor effort or response bias in neuropsychological testing has followed two main approaches (Iverson, 2010). One has been the development of specific measures of effort, such as the Test of Memory Malingering (Tombaugh, 1996) which generally follows a forced-choice format. Similarly, the Portland Digit Recognition Test (Binder, 2003) is another early application of forced-choice recognition testing which takes 40–60 min to administer. Although these dedicated tests are widely used in clinical practice and supported by empirical data, their vulnerability to coaching and time-consuming limitation have been strongly criticized (Grote & Hook, 2007).

Recently, investigators have recommended to employ the standard cognitive tasks to detect noncredible test performance as adjunct indicators of these free-standing symptom validity tests and to circumvent coaching (Babikian, Boone, Lu, & Arnold, 2006). The principle of detection method is identifying cutoff scores or response patterns associated with exaggeration on standard assessment instruments (Larrabee, 2007). Several neuropsychological tests have been examined for their ability to detect malingering, including the Wechsler Adult Intelligence Scale-Third Edition (WAIS-III; Mittenberg et al., 2001; Wechsler, 1997a), Wechsler Memory Scale-Third Edition (WMS-III; Killgore & DellaPietra, 2000; Wechsler, 1997b), Trail Making Test (O'Bryant, Hilsabeck, Fisher, & McCaffrey, 2003), Rey-Osterrieth Complex Figure Test (Lu, Boone, Cozolino, & Mitchell, 2003), Wisconsin Card Sorting Test (Bernard, McGrath, & Houston, 1996), Auditory Verbal Learning Test (Binder, Villanueva, Howieson, & Moore, 1993), California Verbal Learning Test (Millis, Putnam, Adams, & Ricker, 1995), and Finger Tapping Test (Arnold et al., 2005), among others. Since the WAIS-III is the most frequently used instrument in clinical settings (Rabin, Barr, & Burton, 2005) and has been developed within a large-scale standardization sample, interest in indices embedded within the WAIS-III has grown in recent years. For example, very poor performance on the Digit Span subtest is relatively uncommon in patients with numerous different types of brain damage (Iverson & Tulsky, 2003). The Digit Span subtest is easily and quickly administered either individually or as part of a battery, so is particularly user friendly in a clinical setting (Jasinski et al., 2011). Therefore, indicators derived from the Digit Span subtest have been recommended as effective measures for identifying suspect effort, such as the Digit Span age-corrected scaled score (DS-SS; Axelrod, Fichtenberg, Millis, & Wertheimer, 2006; Harrison, Rosenblum, & Currie, 2010; Heinly, Greve, Bianchini, Love, & Brennan, 2005; Iverson & Tulsky, 2003), the longest forward/backward span (Heinly et al., 2005; Iverson & Franzen, 1996; Iverson & Tulsky, 2003), Reliable Digit Span (RDS; Babikian et al., 2006; Greiffenstein, Baker, & Gola, 1994; Harrison et al., 2010), and Vocabulary minus Digit Span difference score (VDS; Curtis, Greve, & Bianchini, 2009; Mittenberg et al., 2001; Schwarz, Gfeller, & Oliveri, 2006). Because borderline or impaired DS-SS (below 5 or 4) occur in less than 5% of both normal and clinical groups (Iverson & Tulsky, 2003), the Digit Span has been viewed as a promising effort indicator. In addition, Iverson and Tulsky (2003) offered that the longest digits forward and backward should rarely be less than five and three, respectively. Otherwise, RDS is computed by adding the longest string of digits forward and backward in which both trials at that level were passed (Greiffenstein et al., 1994). Most investigators (e.g., Greiffenstein et al., 1994; Larrabee, 2003) suggested a raw cutoff score of ≤7 in differentiating probable malingers from patients with traumatic brain injury (TBI). Some researchers (Mittenberg et al., 2001) found that the suppressed Digit Span performance relative to Vocabulary subtest performance could be used as a suspicion index for poor effort because large VDS (≥5) were rare in the patients with TBI but yet relatively common in the analog simulators. In conclusion, increasing evidence reveals that the Digit Span subtest may have utility as a measure of suboptimal effort.

Cross-Cultural Issues

Although cultural, ethnic, and socio-economic factors should be taken into assessment and diagnostic considerations (Heilbronner et al., 2009; Judd & Beggs, 2008), clinicians assessing patients of a specific cultural group or minority populations are often forced to infer the validity, and justify the use, of common malingering instruments, based largely on their own unique experience and expertise (Salazar, Lu, Wen, & Boone, 2007). Specifically, the Chinese have been the largest Asian subgroup in America and accounted for about 1% of the total U.S. population (U.S. Census Bureau, 2011a). Considering approximately 70% Chinese Americans are foreign born and about 1.5 m of them speak English less than “very well” (U.S. Census Bureau, 2011b), language and cultural factors may play a large role in their performance on neuropsychological tests developed on individuals of Euro-American background (Wong, 2011). In contrast to the robust data of poor effort detection in the Northern American and European countries, the literature addressing the utility of effort tests with non-Caucasians and individuals who speak other languages is sparse to nonexistent. In addition, few studies focused on patients in other societies, particularly native Chinese individuals, and their performance on measures designed to detect suboptimal effort (Salazar et al., 2007). Some researchers (e.g., Chiu & Lee, 2002; Gao, Liu, Ding, & Lu, 2002) in Hong Kong and China examined the effectiveness of various detection techniques commonly used in the Western countries, particularly the forced-choice paradigm, in identifying noncredible effort. Nevertheless, major limitations of these studies were the use of the simulation design and restricted to the technique of forced-choice paradigm, which markedly reduced the generalizability and increased vulnerability to coaching. Otherwise, in the past two decades in Taiwan, only a few studies (e.g., Chiang et al., 2006) paid attention to feigning and most of them did not focus on the issue of suboptimal effort detection. Therefore, the need to develop measures to objectively document suspect effort and noncredible cognitive impairment for individuals in Chinese societies was highlighted. Since the normative data of the WAIS-III Chinese version (WAIS-IIIC; Chen & Chen, 2002) was derived from one representative, large-scale population in Taiwan, utilizing embedded indicators of suboptimal effort detection, especially the Digit Span from the WAIS-IIIC, might be beneficial to local clinical practitioners. However, although the atypical performance patterns on the Digit Span subtest were drawn from the Western societies, the Taiwanese normative performance should be examined. On the other hand, participants in most existing studies have been restricted to patients with TBI and few studies have investigated the extent to whether these indices might misclassify individuals with genuine psychiatric disorders as patients with suboptimal effort.

Goal of the Present Study

The aim of the present study was two-fold. The first purpose was to examine the universality of the Digit Span performance patterns through the contrast between Taiwan and American standardization samples. The second aim was to examine the rates of Taiwanese clinical samples identified by the embedded Digit Span-related symptom validity variables, specifically the DS-SS, longest digits forward, longest digits backward, RDS, and VDS.

Study 1: Cultural Difference in Normative Performance on the Digit Span

Methods

Participants

Taiwanese standardization sample

The WAIS-IIIC and the WMS-IIIC (Hua et al., 2005) standardization samples were included in this study. The standardization sample of the WAIS-IIIC was initially collected in 1999 on a stratified random sample of 888 participants in Taiwan. The sampling followed the major demographic distribution of the national population in Taiwan: gender (50.97% men, 49.03% women), age (16–84 years collapsed into 11 age groups), education level (less than grade 6, grades 7–10, high-school diploma, some post-secondary training, or university degree). Each age group consisted of 80 participants, with the exception of the oldest age group (75–84) that enrolled 88 participants. The WMS-IIIC published in 2005 was derived on another standardization sample in Taiwan. The WMS-IIIC sampling followed the WAIS-IIIC in gender, age, and education level, but the stratified random sample included 770 participants equally divided into 11 age groups. The sampling exclusion and inclusion criteria of the WAIS-IIIC and WMS-IIIC were patterned after those of the American versions.

American standardization sample

The WAIS-III standardization sample was presented in the Technical Manual (Psychological Corporation, 1997). The nationally stratified standardization sample consisted of 2,450 subjects divided into 13 age groups. Each age group contained 200 participants with the exception of the two oldest age groups, 80–84 (n = 150) and 85–89 (n = 100). The gender ratio was equal between the eight younger age groups, whereas the other age groups (older than 65) consisted of more women.

Procedures and statistics

Since the WAIS-IIIC and the WMS-IIIC did not provide available information for computing the VDS of the normal population, we only compared the longest string of digits forward and backward in the two Taiwan standardization samples with those in the American normative population.

The WAIS-IIIC and the WMS-IIIC standardization samples were compared by the independent t-test for the longest string of digits forward and backward for examining whether the Digit Span performance was stable across the two groups. These two samples would be combined into one Taiwan standardization sample if they performed similarly on the Digit Span. Then, further comparison of the two standardization samples (Taiwan and American) was performed by the independent t-test. Otherwise, the χ2 test for homogeneity was made to determine whether proportions of the two populations performed on the longest string of digits forward and backward below the cutoffs (5 and 3, respectively) proposed by Iverson and Tulsky (2003) was similar or homogenous. The statistical significance was set at p < .05. Effect sizes were reported as Cohen's d (Cohen, 1988). All statistical procedures were performed by using PASW 18.0 for Windows.

Results

The means and the standard deviations of the longest string of Digit Span forward and backward for the three groups are presented in Table 1. Since the two Taiwan groups did not differ in the longest digit forward, t(1656) = 0.97, p = .33, d = 0.05 (95% CI: −0.07, 0.21), and digit backward, t(1656) = 0.45, p = .65, d = 0.02 (95% CI: −0.13, 0.21), we merged the two Taiwan standardization samples into one combined group for the following comparison. The mixed Taiwan standardization sample performed significantly more digits than the American standardization sample in the longest forward span, t(4106) = 27.09, p < .001, d = 0.86 (95% CI: 1.11, 1.29), and backward span, t(4106) = 10.09, p < .001, d = 0.31 (95% CI: 0.41, 0.61). In addition, the χ2 analysis indicated that the proportions of subjects performed below the cutoffs significantly differed in the longest digit forward, χ2(2, n = 4,108) = 76.57, p < .001, and digit backward, χ2(2, n = 4,108) = 35.25, p < .001, across the two cultural groups.

Table 1.

Performance of longest spans forward and backward in the standardization samples

  Taiwan standardization sample
 
Mixed Taiwan sample (n = 1,658) WAIS-III standardization sample (n = 2,450) p-valuea 
WAIS-IIIC (n = 888) WMS-IIIC (n = 770) 
FDS 7.60 (1.43) 7.67 (1.45) 7.63 (1.44) 6.43 (1.36) <.001 
BDS 5.19 (1.77) 5.23 (1.83) 5.21 (1.80) 4.70 (1.43) <.001 
FDS < 5 (%) 2.10 3.10 2.60 6.00 <.001b 
BDS < 3 (%) 3.70 5.10 4.30 3.30 <.001b 
  Taiwan standardization sample
 
Mixed Taiwan sample (n = 1,658) WAIS-III standardization sample (n = 2,450) p-valuea 
WAIS-IIIC (n = 888) WMS-IIIC (n = 770) 
FDS 7.60 (1.43) 7.67 (1.45) 7.63 (1.44) 6.43 (1.36) <.001 
BDS 5.19 (1.77) 5.23 (1.83) 5.21 (1.80) 4.70 (1.43) <.001 
FDS < 5 (%) 2.10 3.10 2.60 6.00 <.001b 
BDS < 3 (%) 3.70 5.10 4.30 3.30 <.001b 

Notes: WAIS-IIIC = Wechsler Adult Intelligence Scale-Third Edition Chinese version; WMS-IIIC = Wechsler Memory Scale-Third Edition Chinese version; FDS = longest forward digits; BDS = longest backward digits; Mean (SD) was presented if not specified.

aFrom the statistic difference between the mixed Taiwan and American WAIS-III samples.

bChi-square test.

Study 2: Digit Span Performance in Taiwan Clinical Population

Methods

Participants

Traumatic brain injury

One hundred and thirty-two patients with TBI were culled from the archival neuropsychological evaluation records consecutively referred by neurosurgeons or neurologists at National Taiwan University Hospital spanning between May 2005 and August 2011. To avoid confounding educational factors and ensure clinically useful comparisons, patients were excluded from the study if they had <6 years of education, previous history of developmental problems (e.g., mental retardation, autistic spectrum disorders, learning disabilities), or incomplete WAIS-IIIC scores. Finally, 96 TBI patients were included in the study. Thirty-three of these were classified as mild TBI while the remaining 63 were moderate or severe in severity according to the criteria set by the Mild Traumatic Brain Injury Committee of the Head Injury Interdisciplinary Special Interest Group of the American Congress of Rehabilitation Medicine (1993). Seventy-four (77.1%) TBI subjects revealed no presence of external incentive, whereas the other 22 participants (including 11 patients with mild TBI) were involved in litigation or pursuing insurance compensation claims.

Psychiatric patients

Participants were culled from the archival records of 253 inpatients or outpatients referred for intellectual evaluation at Taipei City Hospital, Songde Branch from January 2007 through August 2011. These psychiatric patients were diagnosed by psychiatrists according to the Diagnostic and Statistical Manual of Mental Disorders–Fourth Edition, Text Revision (American Psychiatric Association, 2000). Per the aforementioned exclusion criteria, patients who had <6 years of education, with previous history of developmental problems, with undetermined diagnosis, or without complete WAIS-IIIC scores were excluded. Finally, 110 psychiatric patients were included in the study. The breakdown for primary diagnosis was as follows: schizophrenia spectrum disorders (i.e., schizophrenia, schizoaffective, and psychotic disorder not otherwise specified) were the most common diagnoses (n = 72), followed by depressive disorders (n = 22), and bipolar disorders (n = 16). Ninety-two (83.6%) psychiatric patients had no presence of external incentive, while the remaining 18 (16.4%) patients were involved in litigation or pursuing insurance compensation claims.

The WAIS-IIIC was administered by trained intern clinical psychologists or licensed clinical psychologists. The study was approved by our research ethics committees.

Procedures and statistics

Five Digit Span variables (the DS-SS, VDS, RDS, longest string of digits forward, and longest string of digits backward) collected for following analyses were reproduced in Table 2.

Table 2.

Description and cutoffs of the five Digit Span variables

Digit Span variables Description cutoff 
Digit Span Scaled Score (DS-SS) Derived from the WAIS-IIIC manual ≤5 (Iverson & Tulsky, 2003
Vocabulary minus Digit Span difference score (VDS) The discrepancy between the scaled score of the Vocabulary and Digit Span subtest >5 (Iverson & Tulsky, 2003
Reliable Digit Span (RDS) Sum of the longest string of digits repeated without error over both trials for forward and backward conditions ≤7 (Greiffenstein et al., 1994
Longest string of forward digits Longest string of forward digits where at least one of a given pair of trials was correctly recited <5 (Iverson & Tulsky, 2003
Longest string of backward digits Longest string of backward digits where at least one of a given pair of trials was correctly recited <3 (Iverson & Tulsky, 2003
Digit Span variables Description cutoff 
Digit Span Scaled Score (DS-SS) Derived from the WAIS-IIIC manual ≤5 (Iverson & Tulsky, 2003
Vocabulary minus Digit Span difference score (VDS) The discrepancy between the scaled score of the Vocabulary and Digit Span subtest >5 (Iverson & Tulsky, 2003
Reliable Digit Span (RDS) Sum of the longest string of digits repeated without error over both trials for forward and backward conditions ≤7 (Greiffenstein et al., 1994
Longest string of forward digits Longest string of forward digits where at least one of a given pair of trials was correctly recited <5 (Iverson & Tulsky, 2003
Longest string of backward digits Longest string of backward digits where at least one of a given pair of trials was correctly recited <3 (Iverson & Tulsky, 2003

Note: WAIS-IIIC = Wechsler Adult Intelligence Scale-Third Edition Chinese version.

Descriptive analyses were conducted. For comparison of the four subgroups (TBI not in litigation, TBI in litigation, psychiatric disorders not in litigation, and psychiatric disorders in litigation), the analysis of variance with the Scheffe post hoc comparison (p < .05) was performed to examine the group difference. Otherwise, if the data violated the assumption of homogeneity of variance, the nonparametric Kruskal–Wallis test with the Mann–Whitney pairwise comparison was conducted. The statistical significance was set at p < .05. All statistical procedures were performed by using PASW 18.0 for Windows.

Results

Information regarding demographic and performance data for the four groups appear in Table 3. As can be seen, there were significant differences in age, χ2(3, n = 206) = 25.48, p < .001, education, F(3, 206) = 3.83, p = .011, Verbal IQ, χ2(3, n = 206) = 16.75, p = .001, Performance IQ, F(3, 206) = 12.72, p < .001, Full-Scale IQ (FSIQ), F(3, 206) = 10.29, p < .001, DS-SS, χ2(3, n = 206) = 12.31, p = .006, longest forward digits, χ2(3, n = 206) = 16.22, p = .001, and RDS, χ2(3, n = 199) = 16.79, p = .001, whereas there were no significant group differences in longest backward digits, F(3, 206) = 2.05, p = .109, and VDS, χ2(3, n = 198) = 1.21, p = .752.

Table 3.

Characteristics of the demographic variables and WAIS-III-related scores

  TBI (n = 96)
 
Psychiatric group (n = 110)
 
p-value 
No litigating Litigating No litigating Litigating 
Number (n74 22 92 18  
Gender (men, %) 67.6 68.2 44.6 50.0  
Handedness (right, %) 93.2 90.9 100.0 100.0  
Age (years) 36.61 (14.17)a 27.55 (7.73)b 35.10 (11.71)a 48.33 (10.23)c <.0011 
Education (years) 13.01 (2.90)a 12.50 (2.86)a 13.34 (3.00)a 10.83 (2.83)b .0112 
Verbal IQ 99.84 (17.01)a 87.68 (19.65)b,c 94.88 (12.89)b 83.17 (19.79)c .0011 
Performance IQ 97.43 (17.21)a 85.68 (19.19)b,c 86.71 (14.97)b 74.11 (13.15)c <.0012 
Full-Scale IQ 98.64 (16.62)a 86.50 (18.88)b 90.71 (13.71)c 78.28 (15.54)b <.0012 
DS-SS 10.34 (2.99)a 8.50 (3.89)a,b 9.39 (3.20)b 7.94 (4.19)c .0061 
FDS 8.01 (1.13)a 7.05 (1.79)b 7.87 (1.25)a 6.17 (2.12)b .0011 
BDS 5.36 (1.55)a 5.23 (1.85)a 5.03 (1.62)a 4.33 (2.03)a .1092 
VDS −0.35 (2.84)a −1.00 (2.60)a −0.29 (3.89)a −0.41 (3.64)a .7521 
RDS 11.91 (2.27)a 11.09 (3.22)a,b 11.93 (2.36)b 8.83 (3.05)c .0011 
  TBI (n = 96)
 
Psychiatric group (n = 110)
 
p-value 
No litigating Litigating No litigating Litigating 
Number (n74 22 92 18  
Gender (men, %) 67.6 68.2 44.6 50.0  
Handedness (right, %) 93.2 90.9 100.0 100.0  
Age (years) 36.61 (14.17)a 27.55 (7.73)b 35.10 (11.71)a 48.33 (10.23)c <.0011 
Education (years) 13.01 (2.90)a 12.50 (2.86)a 13.34 (3.00)a 10.83 (2.83)b .0112 
Verbal IQ 99.84 (17.01)a 87.68 (19.65)b,c 94.88 (12.89)b 83.17 (19.79)c .0011 
Performance IQ 97.43 (17.21)a 85.68 (19.19)b,c 86.71 (14.97)b 74.11 (13.15)c <.0012 
Full-Scale IQ 98.64 (16.62)a 86.50 (18.88)b 90.71 (13.71)c 78.28 (15.54)b <.0012 
DS-SS 10.34 (2.99)a 8.50 (3.89)a,b 9.39 (3.20)b 7.94 (4.19)c .0061 
FDS 8.01 (1.13)a 7.05 (1.79)b 7.87 (1.25)a 6.17 (2.12)b .0011 
BDS 5.36 (1.55)a 5.23 (1.85)a 5.03 (1.62)a 4.33 (2.03)a .1092 
VDS −0.35 (2.84)a −1.00 (2.60)a −0.29 (3.89)a −0.41 (3.64)a .7521 
RDS 11.91 (2.27)a 11.09 (3.22)a,b 11.93 (2.36)b 8.83 (3.05)c .0011 

Notes: TBI = traumatic brain injury; DS-SS = Digit Span scaled score; FDS = longest forward digits; BDS = longest backward digits; VDS = Vocabulary minus Digit Span difference score; RDS = Reliable Digit Span.

Mean (SD) was presented if not specified. Row means with the same letter are not significantly different from each other.

1The Kruskal–Wallis Test with the Mann–Whitney pairwise comparison.

2The one-way analysis of variance with the Scheffe post hoc comparison.

Since reduced effort may been found in patients outside medical-legal circumstances (Locke, Smigielski, Powell, & Stevens, 2008), it is impossible to present the sensitivities, specificities, and accurate diagnostic rates in our study. It should also be acknowledged that litigating and nonlitigating patients falling below the cut-offs do not necessarily represent true-positive and false-positive errors, respectively. Cumulative percentages of the DS-SS, VDS, and RDS by groups are presented in Table 4. DS-SS of ≤5 in nonlitigating samples were compatible with the WAIS-III norms, occurring in 12 (7.2%) participants. Of those participants with external incentive, 12 (30.0%) were below the cutoff proposed by Iverson and Tulsky (2003). Although the VDS of ≥6 were uncommon, occurring in three (1.9%) nonlitigating participants, none of the litigating participants scored above the cutoff. The incidence of the RDS scores of ≤7 were very rare in patients with no external incentives, occurring in only 2 (1.3%) of the 159 participants. In contrast, eight (20.0%) of the litigating participants performed below the cutoff proposed by Greiffenstein and colleagues (1994).

Table 4.

Cumulative percentages of the DS-SS, VDS, and RDS scores in the participants

Variable TBI
 
Psychiatric group
 
No litigating patients (n = 159bLitigating patients (n = 40) 
No litigating (n = 74) Litigating (n = 22) No litigating (n = 85aLitigating (n = 18) 
DS-SS 
 ≤4 1.4 13.6 1.1 22.2 1.2 17.5 
 5 8.1 27.3 6.5 33.3 7.2 30.0 
 6 12.2 31.8 20.7 50.0 16.9 40.0 
 ≥7 100.0 100.0 100.0 100.0 100.0 100.0 
VDS 
 ≥6 1.4 – 3.5 – 2.5 – 
 5 6.8 – 9.4 5.9 8.2 2.6 
 4 12.2 – 20.0 11.8 16.4 5.1 
 ≤3 100.0 100.0 100.0 100.0 100.0 100.0 
RDS 
 ≤6 1.5 13.6 – 22.2 0.6 17.5 
 7 1.5 13.6 1.1 27.8 1.3 20.0 
 8 6.0 18.2 12.0 61.1 9.4 37.5 
 9 16.4 31.8 26.1 72.2 22.0 50.0 
 ≥10 100.0 100.0 100.0 100.0 100.0 100.0 
Variable TBI
 
Psychiatric group
 
No litigating patients (n = 159bLitigating patients (n = 40) 
No litigating (n = 74) Litigating (n = 22) No litigating (n = 85aLitigating (n = 18) 
DS-SS 
 ≤4 1.4 13.6 1.1 22.2 1.2 17.5 
 5 8.1 27.3 6.5 33.3 7.2 30.0 
 6 12.2 31.8 20.7 50.0 16.9 40.0 
 ≥7 100.0 100.0 100.0 100.0 100.0 100.0 
VDS 
 ≥6 1.4 – 3.5 – 2.5 – 
 5 6.8 – 9.4 5.9 8.2 2.6 
 4 12.2 – 20.0 11.8 16.4 5.1 
 ≤3 100.0 100.0 100.0 100.0 100.0 100.0 
RDS 
 ≤6 1.5 13.6 – 22.2 0.6 17.5 
 7 1.5 13.6 1.1 27.8 1.3 20.0 
 8 6.0 18.2 12.0 61.1 9.4 37.5 
 9 16.4 31.8 26.1 72.2 22.0 50.0 
 ≥10 100.0 100.0 100.0 100.0 100.0 100.0 

Notes: TBI = traumatic brain injury; DS-SS = Digit Span Scaled Score; VDS = Vocabulary minus Digit Span difference score; RDS = Reliable Digit Span.

an = 92 in the analysis of the DS-SS.

bn = 166 in the analysis of the DS-SS.

Cumulative percentages for the longest string of digits forward and backward are presented in Table 5. The longest string of digits forward of ≤4 was very unusual (0.6%) in nonlitigating patients, occurring in only one (1.4%) nonlitigating patient with TBI and none nonlitigating psychiatric patients. In contrast, the longest string of digits forward of ≤4 for those in the litigating group occurred in six (15%) participants. Similarly, only 9 (5.4%) nonlitigating patients performed ≤5 on the longest string of digits forward compared with 12 (30.0%) of litigating patients performing such scores. None of the nonlitigating patients performed the longest string of digits backward of ≤2. However, two (5.0%) of the litigating patients performed the longest string of digits backward of ≤2, and the proportion was compatible with the Taiwan standardization sample.

Table 5.

Cumulative percentages of longest forward and backward digits in the clinical groups and the Taiwan standardization sample

  TBI
 
Psychiatric group
 
No litigating patients (n = 166) Litigating patients (n = 40) Mixed Taiwan standardization sample (n = 1,658) 
No litigating (n = 74) Litigating (n = 22) No litigating (n = 92) Litigating (n = 18) 
Longest digits forward 
 3 — 9.1 — 11.1 — 10.0 0.5 
 4 1.4 9.1 — 22.2 0.6 15.0 2.6 
 5 4.1 18.2 6.5 44.4 5.4 30.0 12.0 
 6 10.8 31.8 18.5 61.1 15.1 45.0 22.6 
 7 23.0 45.5 28.3 72.2 25.9 57.5 35.2 
 8 59.5 81.8 59.8 72.2 59.6 77.5 64.1 
 9 100.0 100.0 100.0 100.0 100.0 100.0 100.0 
Longest digits backward 
 2 — — — 11.1 — 5.0 4.3 
 3 13.5 22.7 21.7 44.4 18.1 32.5 18.7 
 4 28.4 45.5 39.1 72.2 34.3 57.5 37.5 
 5 58.1 54.5 67.4 77.8 63.3 65.0 59.7 
 6 78.4 72.7 81.5 77.8 80.1 75.0 73.9 
 7 85.1 81.8 88.0 83.3 86.7 82.5 83.6 
 8 100.0 100.0 100.0 100.0 100.0 100.0 100.0 
  TBI
 
Psychiatric group
 
No litigating patients (n = 166) Litigating patients (n = 40) Mixed Taiwan standardization sample (n = 1,658) 
No litigating (n = 74) Litigating (n = 22) No litigating (n = 92) Litigating (n = 18) 
Longest digits forward 
 3 — 9.1 — 11.1 — 10.0 0.5 
 4 1.4 9.1 — 22.2 0.6 15.0 2.6 
 5 4.1 18.2 6.5 44.4 5.4 30.0 12.0 
 6 10.8 31.8 18.5 61.1 15.1 45.0 22.6 
 7 23.0 45.5 28.3 72.2 25.9 57.5 35.2 
 8 59.5 81.8 59.8 72.2 59.6 77.5 64.1 
 9 100.0 100.0 100.0 100.0 100.0 100.0 100.0 
Longest digits backward 
 2 — — — 11.1 — 5.0 4.3 
 3 13.5 22.7 21.7 44.4 18.1 32.5 18.7 
 4 28.4 45.5 39.1 72.2 34.3 57.5 37.5 
 5 58.1 54.5 67.4 77.8 63.3 65.0 59.7 
 6 78.4 72.7 81.5 77.8 80.1 75.0 73.9 
 7 85.1 81.8 88.0 83.3 86.7 82.5 83.6 
 8 100.0 100.0 100.0 100.0 100.0 100.0 100.0 

Note: TBI = traumatic brain injury.

Discussion

The two studies were conducted to examine the normative performance measured by the Digit Span subtest of the WAIS-III across different cultural groups and to examine how many patients with TBI and psychiatric diseases in Taiwan classified by the embedded Digit Span-related symptom validity measures. Previous studies have less reported results regarding the impact of culture or ethnicity on measures of effort detection. Some findings of these effort detection indicators may be successfully utilized in patients from ethnic minority in the USA (Salazar et al., 2007) and others reporting instruments of the forced-choice format are as sensitive to detect poor effort in the native Chinese population as North American and Europeans (Chiu & Lee, 2002; Gao et al., 2002). Because relatively high sensitivities and specificities of the Digit Span subtest of the WAIS-III for effort detection have been reported in Western societies (Jasinski et al., 2011), it is important to understand the Digit Span-related performance patterns in Chinese societies.

In the first study, neither the longest string of digit forward nor the longest string of digit backward performed by the two standardization samples showed between-group equivalence on average. In addition, since the two Taiwanese normative data were compatible in the longest string of digit forward and backward, the performance pattern of the Digit Span in native Taiwanese population was reliable and consistent. In another study regarding normative data on cross-cultural neuropsychological tests in Mainland China (Hsieh & Tori, 2007), results of the Digit Span subtest of the WAIS-R were comparable with those of the WAIS-IIIC in Taiwan. Furthermore, our results are consistent with cross-language studies (Chan & Elliott, 2011; Chen & Stevenson, 1988; Chen, Cowell, Varley, & Wang, 2009; Stigler, Lee, & Stevenson, 1986) which found that the Chinese speakers outperformed their English counterparts in Digit Span tasks. Given that digit articulation times in Chinese are shorter than those in English (Chen & Stevenson, 1988; Stigler et al., 1986), our findings further support the “word-length effect” (Baddeley, Thomson, & Buchanan, 1975) in large-scale populations. Considering bilingual studies (Chincotta & Hoosain, 1995; Hoosain, 1979, 1984) demonstrated the Digit Span superiority in Chinese-English bilinguals' mother tongue, our study highlighted the importance of spoken language in administering the Digit Span to Chinese Americans and interpretation of their performances. Generally speaking, the Digit Span performance difference might be based on cultural factors, particularly the linguistic feature, rather than on fundamental difference in cognitive resource between the two groups. Moreover, one recent retrospective study (Young, Sawyer, Roper, & Baughman, 2012) found effort indices from the recently revised Digit Span contribute to suboptimal effort detection with limited sensitivity. Since the “word-length effect” may play an important role in the revised portion of the Digit Span, namely Digit Span sequencing, utilizing these effort detection indicators derived from the new Digit Span version to Chinese Americans with caution is also suggested.

The purpose of the second study was to investigate the rates of Taiwanese clinical samples identified by the embedded Digit Span-related symptom validity variables proposed in the Western societies (low DS-SS, reduced number of the longest digit forward and backward, high VDS, and low RDS). Consistent with the previous studies, the second study indicates that litigating patients performed poorer on the intellectual assessment than nonlitigating ones. The DS-SS of ≤5 was relatively rare in litigating patients, in contrast to nearly one third of the patients in litigation performing scores below this cutoff. Otherwise, since the native Taiwanese produced more digits forward and backward than the U.S. sample did in the first study, applying the cutoffs from forward and backward digits derived from the U.S. standardization sample may take into consideration for the clinical populations in Taiwan. Our results suggest that a cutoff for the longest forward digits may partially differentiate the standardization sample and nonlitigating patients from patients in litigation in comparison with poor discriminability for the backward digits due to compatible proportions below the cutoff among groups. Similarly, since the RDS is associated with the numbers of longest forward and backward digits, a cutoff of ≤7 as proposed by Greiffenstein and colleagues (1994) may also be questioned based on the results of our first study. Our results indicate that RDS of ≤8 for patients with TBI and ≤7 for patients with psychiatric diseases may identify the usual rates in nonlitigating groups. However, contrary to Iverson and Tulsky's (2003) recommendation, a cutoff for the VDS showed low discriminability for our samples. Interestingly, our results reveal that nonlitigating patients show more discrepancy between the Vocabulary and DS-SS. In fact, recent evidence (e.g., Axelrod et al., 2006; Harrison et al., 2010) suggested that the VDS might not be an effective indicator for effort detection but further studies are needed to explore possible explanations. It is especially noteworthy that our litigating samples performed lower on the WAIS-IIIC compared with the nonlitigating groups, suggesting that the relatively low performance on the Digit Span indices in our litigating sample may be secondary to low ability. However, the discrepant patterns persisted after controlling for their premorbid IQ (data not shown). Given that the premorbid IQ estimates derived from an unpublished work (Chen, Hua, Zhu, & Chen, 2006) accounted for about only one third of the variance in FSIQ, the possibility of different premorbid function levels between groups contributing to the lower scores on Digit Span performance indices may not be ruled out.

Limitations of the retrospective study must also be noted. Patients with or without incentive recorded from the documents may not represent their real motive and effort level. The sample sizes of the litigating groups are apparently too small to conduct further analysis. In addition, the high exclusion rate raises the question of selection bias and reduces generalizability. In fact, the high exclusion rate resulted from patients with incomplete WAIS-III scores because they were assessed with abbreviated versions of the WAIS-IIIC (Chen, Hua, Zhu, & Chen, 2008). These tetrad short-forms did not include the Digit Span subtest. Moreover, subjects with <6 years of education and with history of developmental problems were excluded as well, thus limiting the sampling representativeness. Furthermore, clinical information (e.g., medication, duration of illness, age of onset, and severity of psychiatric symptom) of our clinical populations was incomplete or missing.

In summary, findings of the current study reveal that the normative performance pattern of the Digit Span is not compatible between the populations in Taiwan and America. Except for the VDS, most of the embedded Digit Span-related symptom validity measures proposed in the Western societies classify relative few nonlitigating participants below cutoffs in Taiwan. We thus suggest that practitioners in Euro-American regions may pay attention to the cultural-linguistic effect while administering the Digit Span to their bilingual Chinese clients, irrespective of external incentive. In Chinese societies, the Digit Span may be an essential effort detection instrument as an adjunct indicator. Future studies in cross-validation may conduct known-group designs and examine its utility for other populations, such as patients with chronic pain or criminals who claimed mental disability. However, it is important to note that the Digit Span may not be used as the sole instrument for effort detection as Axelrod and colleagues (2006) suggestion. Thus, it would be quite useful in combination with other effort indices for detecting incomplete effort if possible. Since there is no other validated instrument for effort detection in Taiwan, it is urgent to examine the effectiveness of other tests utilized in forensic neuropsychological evaluations in the Western societies.

Funding

This work was partially supported by grants from the National Science Council (NSC-98-2410-H-002-026-MY3) and Taipei City Hospital (10101-62-010), Taipei, Taiwan.

Conflict of Interest

None declared.

References

American Psychiatric Association, & American Psychiatric Association Task Force on DSM-IV
Diagnostic and Statistical Manual of Mental Disorders: DSM-IV-TR
 , 
2000
4th ed.
Washington, DC
Author
Arnold
G.
Boone
K. B.
Lu
P.
Dean
A.
Wen
J.
Nitch
S.
, et al.  . 
Sensitivity and specificity of finger tapping test scores for the detection of suspect effort
The Clinical Neuropsychologist
 , 
2005
, vol. 
19
 
1
(pg. 
105
-
120
)
Axelrod
B. N.
Fichtenberg
N. L.
Millis
S. R.
Wertheimer
J. C.
Detecting incomplete effort with Digit Span from the Wechsler Adult Intelligence Scale-Third Edition
The Clinical Neuropsychologist
 , 
2006
, vol. 
20
 
3
(pg. 
513
-
523
)
Babikian
T.
Boone
K. B.
Lu
P.
Arnold
G.
Sensitivity and specificity of various digit span scores in the detection of suspect effort
The Clinical Neuropsychologist
 , 
2006
, vol. 
20
 
1
(pg. 
145
-
159
)
Baddeley
A. D.
Thomson
N.
Buchanan
M.
Word length and the structure of short-term memory
Journal of Verbal Learning and Verbal Behavior
 , 
1975
, vol. 
14
 
6
(pg. 
575
-
589
)
Bernard
L. C.
McGrath
M. J.
Houston
W.
The differential effects of simulating malingering, closed head injury, and other CNS pathology on the Wisconsin Card Sorting Test: support for the “pattern of performance” hypothesis
Archives of Clinical Neuropsychology
 , 
1996
, vol. 
11
 
3
(pg. 
231
-
245
)
Binder
L. M.
Villanueva
M. R.
Howieson
D.
Moore
R. T.
The Rey AVLT recognition memory task measures motivational impairment after mild head trauma
Archives of Clinical Neuropsychology
 , 
1993
, vol. 
8
 
2
(pg. 
137
-
147
)
Binder
L. M.
The Portland Digit Recognition Test: A review of validation data and clinical use
Journal of Forensic Neuropsychology
 , 
2003
, vol. 
2
 
3
(pg. 
27
-
41
)
Chan
M. E.
Elliott
J. M.
Cross-linguistic differences in digit memory span
The Australian Psychologist
 , 
2011
, vol. 
46
 
1
(pg. 
25
-
30
)
Chen
C.
Stevenson
H. W.
Cross-linguistic differences in digit span of preschool children
Journal of Experimental Child Psychology
 , 
1988
, vol. 
46
 
1
(pg. 
150
-
158
)
Chen
H. J.
Chen
Y. H.
Wechsler Adult Intelligence Scale-III (Chinese version): Administration and scoring manual
 , 
2002
Taipei
Chinese Behavioral Science Corporation
Chen
H. Y.
Hua
M. S.
Zhu
J.
Chen
Y. H.
Estimating WAIS-III IQ premorbid ability: Regression models combine demographic variables and subtest performance predictors derived from the Taiwan WAIS-III standardization sample
2006
 
Unpublished manuscript
Chen
H. Y.
Hua
M. S.
Zhu
J.
Chen
Y. H.
Selection of factor-based WAIS-III tetrads in the Taiwan standardization sample: A guide to clinical practice
Chinese Journal of Psychology
 , 
2008
, vol. 
50
 
1
(pg. 
91
-
109
)
Chen
Z. Y.
Cowell
P. E.
Varley
R.
Wang
Y. C.
A cross-language study of verbal and visuospatial working memory span
Journal of Clinical and Experimental Neuropsychology
 , 
2009
, vol. 
31
 
4
(pg. 
385
-
391
)
Chiang
Y.-C.
Tzeng
D.-S.
Chow
P. C. K.
Chang
C.
Pan
C.-C.
Lu
T.
, et al.  . 
Characteristics of the draftees with suspected malingering
Taiwanese Journal of Psychiatry
 , 
2006
, vol. 
20
 
2
(pg. 
145
-
154
)
Chincotta
D.
Hoosain
R.
Reading rate, articulatory suppression and bilingual digit span
European Journal of Cognitive Psychology
 , 
1995
, vol. 
7
 
2
(pg. 
201
-
211
)
Chiu
V. W. Y.
Lee
T. M. C.
Detection of malingering behavior at different levels of task difficulty in Hong Kong Chinese
Rehabilitation Psychology
 , 
2002
, vol. 
47
 
2
(pg. 
194
-
203
)
Cohen
J.
Statistical power analysis for the behavioral sciences
 , 
1988
2nd ed.
Hillsdale, NJ
L. Erlbaum Associates
Curtis
K. L.
Greve
K. W.
Bianchini
K. J.
The Wechsler Adult Intelligence Scale-III and malingering in traumatic brain injury: Classification accuracy in known groups
Assessment
 , 
2009
, vol. 
16
 
4
(pg. 
401
-
414
)
Gao
B.
Liu
R.
Ding
S.
Lu
S.
Application of the Binomial Forced-Choice Digit Memory Test in patients with compensable head trauma
Chinese Journal of Clinical Psychology
 , 
2002
, vol. 
10
 
4
(pg. 
256
-
259
)
Greiffenstein
M. F.
Baker
W. J.
Gola
T.
Validation of malingered amnesia measures with a large clinical sample
Psychological Assessment
 , 
1994
, vol. 
6
 (pg. 
218
-
224
)
Grote
C. L.
Hook
J. N.
Larrabee
G. J.
Forced choice recognition tests of malingering
Assessment of malingered neuropsychological deficits
 , 
2007
Oxford, New York
Oxford University Press
(pg. 
44
-
79
)
Harrison
A. G.
Rosenblum
Y.
Currie
S.
Examining unusual digit span performance in a population of postsecondary students assessed for academic difficulties
Assessment
 , 
2010
, vol. 
17
 
3
(pg. 
283
-
293
)
Heilbronner
R. L.
Sweet
J. J.
Morgan
J. E.
Larrabee
G. J.
Millis
S. R.
American Academy of Clinical Neuropsychology Consensus Conference Statement on the neuropsychological assessment of effort, response bias, and malingering
The Clinical Neuropsychologist
 , 
2009
, vol. 
23
 
7
(pg. 
1093
-
1129
)
Heinly
M. T.
Greve
K. W.
Bianchini
K. J.
Love
J. M.
Brennan
A.
WAIS Digit Span-based indicators of malingered neurocognitive dysfunction: Classification accuracy in traumatic brain injury
Assessment
 , 
2005
, vol. 
12
 
4
(pg. 
429
-
444
)
Hoosain
R.
Forward and backward digit span in the languages of the bilingual
Journal of Genetic Psychology
 , 
1979
, vol. 
135
 
2
(pg. 
263
-
268
)
Hoosain
R.
Lateralization of bilingual digit span functions
Perceptual and Motor Skills
 , 
1984
, vol. 
58
 
1
(pg. 
21
-
22
)
Hsieh
S. L.
Tori
C. D.
Normative data on cross-cultural neuropsychological tests obtained from Mandarin-speaking adults across the life span
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22
 
3
(pg. 
283
-
296
)
Hua
M.-S.
Chang
B.-S.
Lin
K.-N.
Yang
C.-M.
Lu
L. H.-J.
Chen
H.-Y.
Wechsler Memory Scale- III (Chinese version): Administration and scoring manual
 , 
2005
Taipei
Chinese Behavioral Science Corporation
Iverson
G. L.
Horton
A. M.
Hartlage
L. C.
Detecting exaggeration, poor effort, and malingering in neuropsychology
Handbook of forensic neuropsychology
 , 
2010
2nd ed.
New York
Springer Pub
(pg. 
91
-
135
)
Iverson
G. L.
Franzen
M. D.
Using multiple objective memory procedures to detect simulated malingering
Journal of Clinical and Experimental Neuropsychology
 , 
1996
, vol. 
18
 
1
(pg. 
38
-
51
)
Iverson
G. L.
Tulsky
D. S.
Detecting malingering on the WAIS-III. Unusual Digit Span performance patterns in the normal population and in clinical groups
Archives of Clinical Neuropsychology
 , 
2003
, vol. 
18
 
1
(pg. 
1
-
9
)
Jasinski
L. J.
Berry
D. T. R.
Shandera
A. L.
Clark
J. A.
Use of the Wechsler Adult Intelligence Scale Digit Span subtest for malingering detection: A meta-analytic review
Journal of Clinical and Experimental Neuropsychology
 , 
2011
, vol. 
33
 
3
(pg. 
300
-
314
)
Judd
T.
Beggs
B.
Bartol
C. R.
Bartol
A. M.
Cross-cultural forensic neuropsychological assessment
Current perspectives in forensic psychology and criminal behavior
 , 
2008
2nd ed.
Los Angeles
Sage Publications
(pg. 
193
-
206
)
Killgore
W. D.
DellaPietra
L.
Using the WMS-III to detect malingering: Empirical validation of the rarely missed index (RMI)
Journal of Clinical and Experimental Neuropsychology
 , 
2000
, vol. 
22
 
6
(pg. 
761
-
771
)
Larrabee
G. J.
Detection of malingering using atypical performance patterns on standard neuropsychological tests
The Clinical Neuropsychologist
 , 
2003
, vol. 
17
 
3
(pg. 
410
-
425
)
Larrabee
G. J.
Larrabee
G. J.
Identification of malingering by pattern analysis on neuropsychological tests
Assessment of malingered neuropsychological deficits
 , 
2007
Oxford, New York
Oxford University Press
(pg. 
80
-
99
)
Lezak
M. D.
Howieson
D. B.
Loring
D. W.
Neuropsychological assessment
 , 
2004
4th ed.
Oxford, New York
Oxford University Press
Locke
D. E.
Smigielski
J. S.
Powell
M. R.
Stevens
S. R.
Effort issues in post-acute outpatient acquired brain injury rehabilitation seekers
NeuroRehabilitation
 , 
2008
, vol. 
23
 
3
(pg. 
273
-
281
)
Lu
P. H.
Boone
K. B.
Cozolino
L.
Mitchell
C.
Effectiveness of the Rey-Osterrieth Complex Figure Test and the Meyers and Meyers recognition trial in the detection of suspect effort
The Clinical Neuropsychologist
 , 
2003
, vol. 
17
 
3
(pg. 
426
-
440
)
Mathias
C. W.
Greve
K. W.
Bianchini
K. J.
Houston
R. J.
Crouch
J. A.
Detecting malingered neurocognitive dysfunction using the reliable digit span in traumatic brain injury
Assessment
 , 
2002
, vol. 
9
 
3
(pg. 
301
-
308
)
Mild Traumatic Brain Injury Committee of the Head Injury Interdisciplinary Special Interest Group of the American Congress of Rehabilitation Medicine
Definition of mild traumatic brain injury
Journal of Head Trauma Rehabilitation
 , 
1993
, vol. 
8
 
3
(pg. 
86
-
87
)
Millis
S. R.
Putnam
S. H.
Adams
K. M.
Ricker
J. H.
The California Verbal Learning Test in the detection of incomplete effort in neuropsychological evaluation
Psychological Assessment
 , 
1995
, vol. 
7
 
4
(pg. 
463
-
471
)
Mittenberg
W.
Theroux
S.
Aguila-Puentes
G.
Bianchini
K.
Greve
K.
Rayls
K.
Identification of malingered head injury on the Wechsler Adult Intelligence Scale—3rd edition
The Clinical Neuropsychologist
 , 
2001
, vol. 
15
 
4
(pg. 
440
-
445
)
O'Bryant
S. E.
Hilsabeck
R. C.
Fisher
J. M.
McCaffrey
R. J.
Utility of the trail making test in the assessment of malingering in a sample of mild traumatic brain injury litigants
The Clinical Neuropsychologist
 , 
2003
, vol. 
17
 
1
(pg. 
69
-
74
)
Psychological Corporation
WAIS-III/WMS-III technical manual
 , 
1997
San Antonio
Harcourt Brace & Company
Rabin
L. A.
Barr
W. B.
Burton
L. A.
Assessment practices of clinical neuropsychologists in the United States and Canada: a survey of INS, NAN, and APA Division 40 members
Archives of Clinical Neuropsychology
 , 
2005
, vol. 
20
 
1
(pg. 
33
-
65
)
Salazar
X. F.
Lu
P. H.
Wen
J.
Boone
K. B.
Boone
K. B.
The use of effort tests in ethnic minorities and in non-English-speaking and English as a second language populations
Assessment of feigned cognitive impairment a neuropsychological perspective
 , 
2007
New York
Guilford Press
(pg. 
405
-
427
)
Schwarz
L. R.
Gfeller
J. D.
Oliveri
M. V.
Detecting feigned impairment with the digit span and vocabulary subtests of the Wechsler Adult Intelligence Scale-third edition
The Clinical Neuropsychologist
 , 
2006
, vol. 
20
 
4
(pg. 
741
-
753
)
Stigler
J. W.
Lee
S. Y.
Stevenson
H. W.
Digit memory in Chinese and English—evidence for a temporally limited store
Cognition
 , 
1986
, vol. 
23
 
1
(pg. 
1
-
20
)
Tan
J. E.
Slick
D. J.
Strauss
E.
Hultsch
D. F.
How'd they do it? Malingering strategies on symptom validity tests
The Clinical Neuropsychologist
 , 
2002
, vol. 
16
 
4
(pg. 
495
-
505
)
Tombaugh
T. N.
Test of Memory Malingering (TOMM)
 , 
1996
New York
Multi-Health Systems
U.S. Census Bureau
Profile of General Population and Housing Characteristics: 2010
2011
 
U.S. Census Bureau
Selected population profile in the United States
2011
 
Wechsler
D. A.
Wechsler Adult Intelligence Scale- III: Administration and scoring manual
 , 
1997
San Antonio, TX
The Psychological Corporation
Wechsler
D. A.
Wechsler Memory Scale- III: Administration and scoring manual
 , 
1997
San Antonio, TX
The Psychological Corporation
Wong
T. M.
Fujii
D. E. M.
Neuropsychology of Chinese Americans
The neuropsychology of Asian-Americans
 , 
2011
New York
Taylor & Francis
(pg. 
29
-
46
)
Young
J. C.
Sawyer
R. J.
Roper
B. L.
Baughman
B. C.
Expansion and re-examination of Digit Span effort indices on the WAIS-IV
The Clinical Neuropsychologist
 , 
2012
, vol. 
26
 
1
(pg. 
147
-
159
)