Abstract

There is a dearth of non-Western normative data for neuropsychological batteries designed to measure cognitive deficits in schizophrenia. Here, we provide normative data for English-speaking ethnic Chinese on the widely used Brief Assessment of Cognition in Schizophrenia acquired from 595 healthy community participants between ages 14 and 55. Means and standard deviations of subtests and composite scores were stratified by age group and sex. We also explored linear regression approaches to generate continuous norms adjusted for age, sex, and education. Notable differences in subtest performances were found against a Western comparison sample. Normative data established in the current sample are essential for clinical and research purposes as it serves as a reference source of cognition for ethnic Chinese.

Introduction

The accurate measurement of cognitive deficits in schizophrenia plays an important role in assessing the efficacy of remediation therapy and other interventions and will be a part of the assessment of domains for the next generation of psychiatric diagnostic manuals. There is accumulating evidence that Western patients with schizophrenia from English speaking and other European countries perform at 1.5–2 SD below the population mean (Addington, Brooks, & Addington, 2003; Bilder et al., 2000; Harvey & Keefe, 1997; Hoff et al., 1999; Keefe et al., 2004; Wilk et al., 2004). However, a lack of neuropsychological data derived from non-Western clinical samples hinders efforts in assessment, intervention planning, and clinical drug trials that aim to ameliorate cognitive deficits in schizophrenia.

There is a need to deliver an accurate neuropsychological assessment in Asian populations using psychometrically sound and informative measures for practitioners and researchers (Collinson, Lam, & Hayes, 2010). Most neuropsychological tests are developed based on Western samples. In earlier years, neuropsychological testing in Asian samples was typically performed using test batteries that were directly translated in their entirety. Today, the changing economic and education landscapes in Asia has increased the ubiquity of English as the main medium of communication. Similarly, in Singapore, a large and increasing proportion of the adult population is schooled in English-medium institutions. Although English is not a first language in some Singaporeans, many are bilinguals or trilinguals who are able to converse in dialects, native languages, and English. Neuropsychological testing is predominantly conducted in English within clinical settings in Singapore. There have been other standardized neuropsychological measures that have been translated into Mandarin, for example, the WAIS-RC (Gong, 1983). However, these were designed to measure general intelligence and cognition and were not specific to the impairments often found impaired in schizophrenia.

The Brief Assessment of Cognition in Schizophrenia (BACS) was designed to measure cognition in schizophrenia. The domains of cognitive function measured by the BACS including verbal memory, working memory, motor speed, attention, executive functions, and verbal fluency found to be impaired in schizophrenia (Addington, Brooks, & Addington, 2003; Bilder et al., 2002; Censits, Ragland, Gur, & Gur, 1997; Hobart, Goldberg, Bartko, & Gold, 1999; Keefe et al., 2004; Mohamed, Paulsen, O'Leary, Arndt, & Andreasen, 1999; Nuechterlein et al., 2004; Saykin et al., 1991). The BACS has demonstrated high test–retest reliability, concurrent validity with standard neuropsychological batteries, and is as sensitive as longer test batteries in measuring cognitive deficits in schizophrenia (Chianetta, Lefebvre, LeBlanc, & Grignon, 2008; Keefe et al., 2004). The battery has good tolerability, allowing higher percentages of patients to complete BACS subtests compared with standard measures (Chianetta, Lefebvre, LeBlanc, & Grignon, 2008; Keefe et al., 2004).

The BACS has been used frequently as a neuropsychological assessment tool for clinical trials in schizophrenia (Cavallaro et al., 2009; Friedman et al., 2008; Geffen, Keefe, Rabinowitz, Anand, & Davidson, 2013; Goff et al., 2007; Hill et al., 2008; Keefe et al., 2007; Müller, Werheid, Hammerstein, Jungmann, & Becker, 2005; Ogino et al., 2011), is found to be a valid outcome measure for the effects of rehabilitation, medication, and therapy on cognitive function, and is an indicator of functional relevance with respect to independent living skills, performance-based assessments of everyday living skills, and interview-based assessments of cognition in patients with schizophrenia (Keefe et al., 2004; Keefe, Poe, Walker, & Harvey, 2006).

Originally developed and normed in the USA, BACS normative studies have since been reported in Italy and Russia (Anselmetti et al., 2008; Keefe et al., 2008; Sarkisyan, Gurovich, & Keefe, 2011) and the battery has been validated in Japanese, Spanish, French, Brazilian, and German samples (Bralet et al., 2007; Kaneda et al., 2007; Sachs, Winklbaur, Jagsch, & Keefe, 2011; Segarra et al., 2011). Backed by several validation data sets, the BACS is among the few neuropsychological instruments that permit the cross-cultural assessment of cognition in schizophrenia. However the appropriateness of applying norms derived from English-speaking Western individuals in an English-speaking ethnic Chinese sample is questionable. Crucial considerations include comparability, cultural, socio-economic, and education factors in neuropsychological testing (Ardila, 1995; Ardila, Rosselli, & Ostrosky-Solis, 1992; O'Bryant, O'Jile, & McCaffrey, 2004; Ostrosky-Solis et al., 1985).

The aim of this study was to establish normative data in a sample of English-speaking Chinese individuals. Methodologies from previous neuropsychological normative studies were evaluated, and two approaches were considered suitable (Crawford & Howell, 1998). The first involved discrete sample stratification where age groups and sex were used to generate normative information of BACS performance in each stratum. The second methodology established continuous norms using regression approaches to adjust for sex, age, and education effects (Crawford & Howell, 1998). We then evaluated the effects of culture on test performances by comparing test scores of the current sample of ethnic Chinese to age-matched data from previously published U.S. norms (Keefe et al., 2008).

Methods

Subjects

The normative sample consists of a community sample of 595 subjects (295 men and 300 women) in Singapore. Recruitment was conducted as part of the Singapore Translational and Clinical Research in Neuroscience. Only 595 subjects out of a total pool of 1,110 subjects were included in this normative study. Subjects were excluded from this normative sample due to stratification procedures to fit the population profile in Singapore and missing data. The study was conducted on a predominantly urban sample, as Singapore is a city-state. All subjects were English-speaking ethnic Chinese aged between 14 and 55 (mean = 35.52, SD = 11.85). Subjects were excluded if there was a history of diagnostic and statistical manual, fourth edition (DSM-IV) Axis 1 Disorders (as determined by the Structured Clinical Interview for DSM-IV-text-revision Axis 1 Disorders, structured clinical interview for DSM-IV-TR axis I disorders Non-Patient version; First et al., 2002), clinically significant neurological disease or head injury, alcohol or drug abuse over the past 6 months, or had less than 9 years of formal education. The study was approved by the relevant institutional and ethics committees. Prior to data collection, study procedures were explained to all research participants and only those who gave written informed consent were included in the study. Parental consent was also taken for subjects who were younger than 21 years in adherence with local ethical regulations. All subjects had a minimum of 6 years of primary school education and had completed the Primary Six Leaving Examination. All subjects were able to comprehend informed consent and study procedures. Neuropsychological testing and clinical assessments were conducted in English.

Assessment Procedures

All subjects were assessed by experienced psychometricians certified on the BACS. The full BACS battery was administered: Verbal Memory, Digit Sequencing, Token Motor Task, Semantic Fluency (Animals, Fruits, and Vegetables), Symbol Coding, and Tower of London. Demographic and education information were also collected. A recent study we conducted demonstrated that years of education was not the optimum index of educational attainment (Lam et al., 2013). Therefore, for the purpose of establishing normative data, an index of education attainment (Adjusted Years of Education or AYE; (Lam et al., 2013) was measured as a proxy of academic attainment. The AYE takes into consideration the presence of alternative education choices to the mainstream education and is calculated by summing the adjusted years of education at each stage, regardless of the institution and candidature (Lam et al., 2013). The mean number of years of education in this sample was 13.14 (SD = 2.77) and mean AYE was 12.38 (SD = 2.45).

Statistical Analyses

Discrete stratification and linear regression approaches (Cohen, Cohen, West, & Aiken, 2002; Crawford & Howell, 1998; Keefe et al., 2008) were utilized to describe normative information. The Bland–Altman approach was applied on z-scores to examine the agreement of the two methodologies. Limits of agreement were calculated to examine the comparability of both norming methods. Paired t-tests were also conducted to explore effect sizes of the difference between both methods. Neuropsychological performance in the current sample was also evaluated against a U.S. normed sample by computing Cohen's d (Keefe et al., 2008). All data analyses were carried out using PASW (Version 19.0).

Calculation of Scaled Subtest Scores and Composite Scores

Scaled scores of subtests and composite scores were computed as follows (first discussed in Keefe et al., 2008): 

formula

Where Zij is the scaled score of the ith subject for subtest j, Yij the raw score of the ith subject on the subtest j, Mj and SDj the mean and the standard deviation for test j, respectively, of the sample. In the discrete stratification method, Mj and SDj are the age–sex stratum-specific means and standard deviations of subtest j, respectively. In the linear regression computation, which will be described later, Mj is the predicted performance on subtest j of the ith subject given his/her age, sex, and AYE and SDj the standard error associated with this individual prediction. Subject T-scores, Tij, will be derived by multiplying Zij by 10 and adding 50: 

formula

This has the effect of producing a distribution of T-scores with a mean of 50 and standard deviation of 10 for each subtest. BACS Composite score for each subject is computed as follows: 

formula

Where Ccomposite,i is the BACS composite score for the ith subject, forumla the sum of the six scaled test scores of the ith subject, and SD the standard deviation of the sum of the scaled scores of the sample.

Analytical Approach: Discrete Stratification by Age Group and Sex

The study sample was matched according to distributions of age and sex reported in the population demographics of the ethnic Chinese population in Singapore (Government of Singapore, 2010). A total of 595 participants were stratified according to their age groups and sex. The normative sample was stratified into eight age groups. Table 1 displays the expected and actual number of subjects in each age group based on the stratification.

Table 1.

Demographics of normative sample

 Age groups
 
14–20
 
21–24
 
25–29
 
30–34
 
35–39
 
40–44
 
45–49
 
50–55
 
Projected Actual Projected Actual Projected Actual Projected Actual Projected Actual Projected Actual Projected Actual Projected Actual 
Sex 
 Men 44 45 23 24 32 32 34 35 37 39 37 39 40 42 46 39 
 Women 42 43 23 29 33 20 38 39 41 40 39 40 40 42 46 47 
Total 86 88 46 53 65 52 72 74 78 79 76 79 80 84 92 86 
 Age groups
 
14–20
 
21–24
 
25–29
 
30–34
 
35–39
 
40–44
 
45–49
 
50–55
 
Projected Actual Projected Actual Projected Actual Projected Actual Projected Actual Projected Actual Projected Actual Projected Actual 
Sex 
 Men 44 45 23 24 32 32 34 35 37 39 37 39 40 42 46 39 
 Women 42 43 23 29 33 20 38 39 41 40 39 40 40 42 46 47 
Total 86 88 46 53 65 52 72 74 78 79 76 79 80 84 92 86 

Results

Mean performances on the BACS subtests stratified by age groups and sex are presented in Table 2. Consistent with the directionality of the findings reported in Keefe and colleagues (2004), a general decline in overall performance was observed with increase in age. Women had also showed better performances in language-related tasks, such as verbal memory, whereas men had better performances than women in Token Motor Task and Tower of London.

Table 2.

Raw scores by age group and gender

 Age group: 14–20
 
Age group: 21–24
 
Age group: 25–29
 
Age group: 30–34
 
Age group: 35–39
 
Age group: 40–44
 
Age group: 45–49
 
Age group: 50–55
 
Overall n = 595
 
Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD 
Verbal Memory 
 Men 46.80 9.32 0.36 46.08 8.73 0.29 39.63 8.34 −0.34 41.46 10.69 −0.16 42.46 9.35 −0.06 41.72 8.93 −0.13 36.17 8.36 −0.67 38.56 8.72 −0.44 43.09 10.25 
 Women 49.79 8.22 0.65 49.90 8.20 0.66 46.85 7.55 0.37 46.18 9.65 0.30 43.10 11.19 0.00 42.90 11.78 −0.02 40.98 9.03 −0.21 40.70 12.48 −0.23 
Digit Sequencing 
 Men 21.93 3.47 0.32 22.50 3.61 0.48 20.47 3.73 −0.10 20.83 3.56 0.01 22.00 3.31 0.34 21.18 3.63 0.11 19.69 3.61 −0.32 21.13 3.47 0.09 20.81 3.50 
 Women 20.67 2.93 −0.04 22.31 2.62 0.43 20.95 3.44 0.04 20.67 3.34 −0.04 19.33 4.24 −0.42 20.00 3.06 −0.23 19.79 3.79 −0.29 20.62 2.80 −0.05 
Token Motor 
 Men 77.40 13.36 −0.03 85.58 9.29 0.63 82.13 9.69 0.35 80.97 11.54 0.26 80.62 10.78 0.23 79.33 13.07 0.13 72.19 13.18 −0.45 71.64 13.68 −0.49 77.74 12.44 
 Women 81.35 11.50 0.29 81.31 9.58 0.29 78.80 13.37 0.09 79.05 13.48 0.11 75.40 11.77 −0.19 79.85 11.46 0.17 73.10 10.60 −0.37 72.26 12.03 −0.44 
Animal Fluency 
 Men 21.04 6.01 0.20 21.54 5.51 0.30 18.44 3.95 −0.31 20.49 4.46 0.09 20.21 4.47 0.04 21.00 5.37 0.19 20.60 5.61 0.11 17.87 4.82 −0.42 20.02 5.13 
 Women 22.12 4.73 0.41 21.28 4.83 0.25 20.70 4.49 0.13 20.51 5.00 0.10 19.45 4.61 −0.11 18.78 6.19 −0.24 19.50 5.05 −0.10 17.87 4.42 −0.42 
Fruits Fluency 
 Men 15.16 3.20 −0.23 14.79 3.55 −0.34 14.53 3.60 −0.41 15.37 3.35 −0.17 15.49 2.85 −0.13 15.31 3.20 −0.18 15.24 3.53 −0.20 14.31 3.31 −0.48 15.93 3.39 
 Women 16.42 2.82 0.14 17.34 3.09 0.42 16.85 3.36 0.27 16.90 3.55 0.28 16.95 3.30 0.30 17.60 3.42 0.49 17.07 3.38 0.34 15.68 3.08 −0.07 
Vegetables Fluency 
 Men 9.22 3.47 −0.59 9.83 3.06 −0.44 9.53 4.08 −0.51 10.46 3.52 −0.28 9.56 3.32 −0.50 9.97 3.17 −0.40 11.10 4.37 −0.12 11.33 3.79 −0.06 11.59 4.03 
 Women 10.60 3.28 −0.24 12.45 3.24 0.21 12.90 3.67 0.33 13.36 4.02 0.44 13.13 4.36 0.38 14.08 3.71 0.62 13.88 3.40 0.57 13.53 3.93 0.48 
Total Verbal Fluency 
 Men 45.42 10.33 −0.22 46.17 9.67 −0.15 42.50 9.25 −0.53 46.31 8.93 −0.13 45.26 6.73 −0.24 46.28 8.18 −0.13 46.93 11.25 −0.06 43.51 8.43 −0.43 47.54 9.45 
 Women 49.14 7.94 0.17 51.07 8.17 0.37 50.45 9.43 0.31 50.77 9.55 0.34 49.53 9.72 0.21 50.45 10.91 0.31 50.45 9.02 0.31 47.09 8.63 −0.05 
Symbol Coding 
 Men 63.80 10.22 0.19 63.67 8.08 0.18 63.13 8.46 0.12 61.40 10.77 −0.05 63.59 8.62 0.17 59.74 9.14 −0.22 53.17 8.94 −0.89 55.00 8.59 −0.71 61.92 9.79 
 Women 68.05 11.53 0.63 69.83 6.40 0.81 67.30 9.59 0.55 65.54 8.65 0.37 64.73 7.34 0.29 59.48 6.55 −0.25 59.79 9.10 −0.22 58.55 7.63 −0.34 
Tower of London 
 Men 18.07 2.29 0.29 18.96 2.10 0.57 18.81 2.29 0.52 17.40 3.46 0.07 17.62 2.66 0.14 16.72 3.75 −0.15 15.62 3.89 −0.50 17.18 2.55 0.00 17.17 3.12 
 Women 17.91 3.45 0.23 18.10 1.35 0.30 18.35 2.50 0.38 17.41 2.46 0.08 16.65 3.24 −0.17 16.53 2.72 −0.21 16.33 2.31 −0.27 15.34 4.14 −0.59 
Composite 
 Men   0.26   0.56   0.01   0.00   0.16   −0.11   −0.81   −0.55   
 Women   0.54   0.80   0.48   0.32   −0.08   −0.06   −0.29   −0.48   
 Age group: 14–20
 
Age group: 21–24
 
Age group: 25–29
 
Age group: 30–34
 
Age group: 35–39
 
Age group: 40–44
 
Age group: 45–49
 
Age group: 50–55
 
Overall n = 595
 
Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD Z Mean SD 
Verbal Memory 
 Men 46.80 9.32 0.36 46.08 8.73 0.29 39.63 8.34 −0.34 41.46 10.69 −0.16 42.46 9.35 −0.06 41.72 8.93 −0.13 36.17 8.36 −0.67 38.56 8.72 −0.44 43.09 10.25 
 Women 49.79 8.22 0.65 49.90 8.20 0.66 46.85 7.55 0.37 46.18 9.65 0.30 43.10 11.19 0.00 42.90 11.78 −0.02 40.98 9.03 −0.21 40.70 12.48 −0.23 
Digit Sequencing 
 Men 21.93 3.47 0.32 22.50 3.61 0.48 20.47 3.73 −0.10 20.83 3.56 0.01 22.00 3.31 0.34 21.18 3.63 0.11 19.69 3.61 −0.32 21.13 3.47 0.09 20.81 3.50 
 Women 20.67 2.93 −0.04 22.31 2.62 0.43 20.95 3.44 0.04 20.67 3.34 −0.04 19.33 4.24 −0.42 20.00 3.06 −0.23 19.79 3.79 −0.29 20.62 2.80 −0.05 
Token Motor 
 Men 77.40 13.36 −0.03 85.58 9.29 0.63 82.13 9.69 0.35 80.97 11.54 0.26 80.62 10.78 0.23 79.33 13.07 0.13 72.19 13.18 −0.45 71.64 13.68 −0.49 77.74 12.44 
 Women 81.35 11.50 0.29 81.31 9.58 0.29 78.80 13.37 0.09 79.05 13.48 0.11 75.40 11.77 −0.19 79.85 11.46 0.17 73.10 10.60 −0.37 72.26 12.03 −0.44 
Animal Fluency 
 Men 21.04 6.01 0.20 21.54 5.51 0.30 18.44 3.95 −0.31 20.49 4.46 0.09 20.21 4.47 0.04 21.00 5.37 0.19 20.60 5.61 0.11 17.87 4.82 −0.42 20.02 5.13 
 Women 22.12 4.73 0.41 21.28 4.83 0.25 20.70 4.49 0.13 20.51 5.00 0.10 19.45 4.61 −0.11 18.78 6.19 −0.24 19.50 5.05 −0.10 17.87 4.42 −0.42 
Fruits Fluency 
 Men 15.16 3.20 −0.23 14.79 3.55 −0.34 14.53 3.60 −0.41 15.37 3.35 −0.17 15.49 2.85 −0.13 15.31 3.20 −0.18 15.24 3.53 −0.20 14.31 3.31 −0.48 15.93 3.39 
 Women 16.42 2.82 0.14 17.34 3.09 0.42 16.85 3.36 0.27 16.90 3.55 0.28 16.95 3.30 0.30 17.60 3.42 0.49 17.07 3.38 0.34 15.68 3.08 −0.07 
Vegetables Fluency 
 Men 9.22 3.47 −0.59 9.83 3.06 −0.44 9.53 4.08 −0.51 10.46 3.52 −0.28 9.56 3.32 −0.50 9.97 3.17 −0.40 11.10 4.37 −0.12 11.33 3.79 −0.06 11.59 4.03 
 Women 10.60 3.28 −0.24 12.45 3.24 0.21 12.90 3.67 0.33 13.36 4.02 0.44 13.13 4.36 0.38 14.08 3.71 0.62 13.88 3.40 0.57 13.53 3.93 0.48 
Total Verbal Fluency 
 Men 45.42 10.33 −0.22 46.17 9.67 −0.15 42.50 9.25 −0.53 46.31 8.93 −0.13 45.26 6.73 −0.24 46.28 8.18 −0.13 46.93 11.25 −0.06 43.51 8.43 −0.43 47.54 9.45 
 Women 49.14 7.94 0.17 51.07 8.17 0.37 50.45 9.43 0.31 50.77 9.55 0.34 49.53 9.72 0.21 50.45 10.91 0.31 50.45 9.02 0.31 47.09 8.63 −0.05 
Symbol Coding 
 Men 63.80 10.22 0.19 63.67 8.08 0.18 63.13 8.46 0.12 61.40 10.77 −0.05 63.59 8.62 0.17 59.74 9.14 −0.22 53.17 8.94 −0.89 55.00 8.59 −0.71 61.92 9.79 
 Women 68.05 11.53 0.63 69.83 6.40 0.81 67.30 9.59 0.55 65.54 8.65 0.37 64.73 7.34 0.29 59.48 6.55 −0.25 59.79 9.10 −0.22 58.55 7.63 −0.34 
Tower of London 
 Men 18.07 2.29 0.29 18.96 2.10 0.57 18.81 2.29 0.52 17.40 3.46 0.07 17.62 2.66 0.14 16.72 3.75 −0.15 15.62 3.89 −0.50 17.18 2.55 0.00 17.17 3.12 
 Women 17.91 3.45 0.23 18.10 1.35 0.30 18.35 2.50 0.38 17.41 2.46 0.08 16.65 3.24 −0.17 16.53 2.72 −0.21 16.33 2.31 −0.27 15.34 4.14 −0.59 
Composite 
 Men   0.26   0.56   0.01   0.00   0.16   −0.11   −0.81   −0.55   
 Women   0.54   0.80   0.48   0.32   −0.08   −0.06   −0.29   −0.48   

Information on the distribution of age group and sex standardized scores that are computed using the stratification methodology is presented in Table 3. Although education is closely associated with neuropsychological task performance, further stratification by education was not conducted as this would have reduced cell sizes to an extent where robust comparisons could not be made.

Table 3.

Distributions of performances after adjusting for age and gender, separated by methodology

 Stratification method
 
Regression method
 
n Percentiles
 
Percentiles
 
25th 50th 75th 25th 50th 75th 
Verbal Memory 595 −0.60 0.03 0.71 −0.61 0.04 0.69 
Digit Sequencing 595 −0.75 0.06 0.70 −0.71 0.06 0.74 
Token Motor Task 595 −0.71 0.05 0.74 −0.63 0.05 0.74 
Semantic Fluency 595 −0.71 −0.04 0.61 −0.71 −0.02 0.60 
Symbol Coding 595 −0.68 0.03 0.58 −0.63 −0.01 0.55 
Tower of London 595 −0.51 0.11 0.64 −0.47 0.15 0.60 
Composite 595 −0.66 −0.02 0.68 −0.68 0.02 0.71 
 Stratification method
 
Regression method
 
n Percentiles
 
Percentiles
 
25th 50th 75th 25th 50th 75th 
Verbal Memory 595 −0.60 0.03 0.71 −0.61 0.04 0.69 
Digit Sequencing 595 −0.75 0.06 0.70 −0.71 0.06 0.74 
Token Motor Task 595 −0.71 0.05 0.74 −0.63 0.05 0.74 
Semantic Fluency 595 −0.71 −0.04 0.61 −0.71 −0.02 0.60 
Symbol Coding 595 −0.68 0.03 0.58 −0.63 −0.01 0.55 
Tower of London 595 −0.51 0.11 0.64 −0.47 0.15 0.60 
Composite 595 −0.66 −0.02 0.68 −0.68 0.02 0.71 

Analytical Approach: Computation of Continuous Norms

Linear regression methods were employed on the sample to evaluate the effects of age, sex, and education on subtest performances. The predicted score on subtest j for the ith subject, forumla, is calculated by using the beta-weights provided in Table 4, for instance (Cohen, Cohen, West, & Aiken, 2002)  

(1)
formula
where Sexi = 0 if the ith subject is a man and Sexi = 1 if the ith subject is a woman, Agei the age in years of the ith subject, and Boj, B1j, and B2j the estimated intercept and slopes, respectively, for subtest j.

Table 4.

Effect sizes of age, gender, and education index on neurocognitive subtests and composite score

    Intercept Age
 
Gender
 
AYE
 
 F R2 forumla Bo B1 t (594) B2 t (594) B3 t (594) 
Verbal Memory 
 VMPredicted = Bo + B1 × Age 61.20** .09** .09 52.49 −0.27** −7.82     
 VMPredicted = Bo + B2 × Gender 14.68** .02** .02 41.48   3.19** 3.83   
 VMPredicted = Bo + B3 × AYE 42.04** .07** .07 29.74     1.08** 6.48 
 VMPredicted = Bo + B1 × Age + B2 × Gender 40.86** .12** .12 50.95 −0.27** −8.09 3.42** 4.32   
 VMPredicted = Bo + B1 × Age + B3 × AYE 56.97** .16** .16 39.06 −0.27** −8.20   1.09** 6.92 
 VMPredicted = Bo + B2 × Gender + B3 × AYE 29.04** .09** .09 28.27   3.12** 3.88 1.07** 6.51 
 VMPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 45.63** .19** .18 37.66 −0.27** −8.48 3.35** 4.41 1.08** 6.97 
Digit Sequencing 
 DSPredicted = Bo + B1 × Age 10.17** .02** .02 22.17 −0.04** −3.19     
 DSPredicted = Bo + B2 × Gender 6.45* .01* .01 21.17   −0.73* −2.54   
 DSPredicted = Bo + B3 × AYE 1.85 — — 19.82     0.08 1.36 
 DSPredicted = Bo + B1 × Age + B2 × Gender 8.12** .03** .02 22.48 −0.04** −3.11 −0.69* −2.44   
 DSPredicted = Bo + B1 × Age + B3 × AYE 6.08** .02** .02 21.17 −0.04** −3.21   0.08 1.40 
 DSPredicted = Bo + B2 × Gender + B3 × AYE 4.21* .01* .01 20.16   −0.73* −2.56 0.08 1.40 
 DSPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 6.11** .03** .03 21.46 −0.04** −3.13 −0.70* −2.47 0.08 1.44 
Token Motor Task 
 TMTPredicted = Bo + B1 × Age 36.24** .06** .06 86.69 −0.25** −6.02     
 TMTPredicted = Bo + B2 × Gender .74 — — 78.18   −0.88 −0.86   
 TMTPredicted = Bo + B3 × AYE 5.53* .01* .01* 71.69     0.49* 2.35 
 TMTPredicted = Bo + B1 × Age + B2 × Gender 18.32** .06** .06** 86.99 −0.25** −5.99 −0.66 −0.67   
 TMTPredicted = Bo + B1 × Age + B3 × AYE 21.37** .07** .06 80.52 −0.25** −6.07   0.50* 2.49 
 TMTPredicted = Bo + B2 × Gender + B3 × AYE 3.17* .01* .01 72.11   −0.91 −0.89 0.49* 2.36 
 TMTPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 14.40** .07** .07** 80.81 −0.25** −6.04 −0.69 −0.70 0.50* 2.49 
Semantic Fluency: Animals 
 AniFPredicted = Bo + B1 × Age 19.75** .03** .03 22.78 −0.08** −4.44     
 AniFPredicted = Bo + B2 × Gender .31 — — 20.14   −0.24 −0.56   
 AniFPredicted = Bo + B3 × AYE 20.87** .03** .03 15.23     0.39** 4.57 
 AniFPredicted = Bo + B1 × Age + B2 × Gender 9.94** .03** .03 22.86 −0.08** −4.42 −0.17 −0.41   
 AniFPredicted = Bo + B1 × Age + B3 × AYE 21.23** .07** .06 17.97 −0.08** −4.57   0.39** 4.69 
 AniFPredicted = Bo + B2 × Gender + B3 × AYE 10.62** .04** .03 15.35   −0.26 −0.63 0.39** 4.57 
 AniFPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 14.21** .07** .06 18.05 −0.08** −4.55 −0.19 −0.48 0.39** 4.69 
Semantic Fluency: Fruits 
 FruFPredicted = Bo + B1 × Age .09 — — 16.06 −0.00 −0.30     
 FruFPredicted = Bo + B2 × Gender 42.63** .07** .07 15.05   1.76** 6.53   
 FruFPredicted = Bo + B3 × AYE 19.46** .03** .03 12.87     0.25** 4.41 
 FruFPredicted = Bo + B1 × Age + B2 × Gender 21.44** .07** .06 15.26 −0.00 −0.54 1.76** 6.54   
 FruFPredicted = Bo + B1 × Age + B3 × AYE 9.77** .03** .03 13.01 −0.00 −0.35   0.25** 4.41 
 FruFPredicted = Bo + B2 × Gender + B3 × AYE 32.03** .10** .10 12.06   1.74** 6.57 0.24** 4.48 
 FruFPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 21.45** .10** .09 12.29 −0.01 −0.60 1.75** 6.59 0.24** 4.48 
Semantic Fluency: Vegetables 
 VegFPredicted = Bo + B1 × Age 30.13** .05** .05 8.93 0.08** 5.49     
 VegFPredicted = Bo + B2 × Gender 86.30** .13** .13 10.14   2.87** 9.29   
 VegFPredicted = Bo + B3 × AYE 16.21** .03** .03 8.27     0.27** 4.03 
 VegFPredicted = Bo + B1 × Age + B2 × Gender 60.60** .17** .17 7.67 0.07** 5.53 2.81** 9.31   
 VegFPredicted = Bo + B1 × Age + B3 × AYE 23.73** .07** .07 5.68 0.07** 5.52   0.27** 4.07 
 VegFPredicted = Bo + B2 × Gender + B3 × AYE 53.12** .15** .15 6.92   2.85** 9.36 .26** 4.19 
 VegFPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 47.54** .19** .19 4.51 0.07** 5.57 2.79** 9.39 0.26** 4.24 
Semantic Fluency: Total 
 SFPredicted = Bo + B1 × Age .04 — — 47.77 −0.01 −0.2     
 SFPredicted = Bo + B2 × Gender 33.84** .05** .05 45.33   4.39** 5.82   
 SFPredicted = Bo + B3 × AYE 34.24** .06** .05 36.37     0.90** 5.85 
 SFPredicted = Bo + B1 × Age + B2 × Gender 16.98** .05** .05 45.79 −0.01 −0.42 4.40** 5.82   
 SFPredicted = Bo + B1 × Age + B3 × AYE 17.13** .06** .05 36.67 −0.01 −0.27   0.90** 5.85 
 SFPredicted = Bo + B2 × Gender + B3 × AYE 35.53** .11** .10 34.33   4.33** 5.90 0.89** 5.94 
 SFPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 23.73** .11** .10 34.85 −0.02 −0.49 4.34** 5.91 0.89** 5.94 
Symbol Coding 
 SCPredicted = Bo + B1 × Age 92.67** .14** .13 72.71 −0.30** −9.63     
 SCPredicted = Bo + B2 × Gender 18.72** .03** .03 60.19   3.42** 4.33   
 SCPredicted = Bo + B3 × AYE 20.98** .03** .03 52.76     0.74** 4.58 
 SCPredicted = Bo + B1 × Age + B2 × Gender 60.89** .17** .17 71.05 −0.31** −10.00 3.69** 5.03   
 SCPredicted = Bo + B1 × Age + B3 × AYE 60.95** .17** .17 63.42 −0.31** −9.88   0.76** 5.04 
 SCPredicted = Bo + B2 × Gender + B3 × AYE 20.22** .06** .06 51.17   3.38** 4.34 0.73** 4.59 
 SCPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 50.90** .21** .20 61.90 −0.31** −10.26 3.64** 5.07 0.75** 5.08 
Tower of London 
 TOLPredicted = Bo + B1 × Age 44.72** .07** .07 19.65 −0.07** −6.69     
 TOLPredicted = Bo + B2 × Gender 4.16* .01* .01 17.44   −0.52* −2.04   
 TOLPredicted = Bo + B3 × AYE 19.10** .03** .03 14.39     0.23** 4.37 
 TOLPredicted = Bo + B1 × Age + B2 × Gender 24.21** .08** .07 19.86 −0.07** −6.63 −0.46 −1.87   
 TOLPredicted = Bo + B1 × Age + B3 × AYE 33.74** .10** .10 16.84 −0.07** −6.85   0.23** 4.61 
 TOLPredicted = Bo + B2 × Gender + B3 × AYE 11.87** .04** .04 14.64   −0.54* −2.13 0.23** 4.41 
 TOLPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 23.89** .11** .10 17.04 −0.07** −6.79 −0.48 −1.96 0.23** 4.64 
BACS Composite Score 
 BACS_CompositePredicted = Bo + B1 × Age 93.48** .14** .14 1.11 −0.03** −9.67     
 BACS_CompositePredicted = Bo + B2 × Gender 5.45*   −0.10   0.19* 2.33   
 BACS_CompositePredicted = Bo + B3 × AYE 51.06** .08** .08 −1.42     0.12** 7.15 
 BACS_CompositePredicted = Bo + B1 × Age + B2 × Gender 51.42** .15** .15 1.01 −0.03** −9.83 0.22** 2.87   
 BACS_CompositePredicted = Bo + B1 × Age + B3 × AYE 82.35** .22** .22 −0.33 −0.03** −10.23   0.12** 7.85 
 BACS_CompositePredicted = Bo + B2 × Gender + B3 × AYE 28.45** .09** .09 −1.51   0.18* 2.34 0.12** 7.14 
 BACS_CompositePredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 58.42** .23* .23 −0.42 −0.03** −10.40 0.21** 2.91 0.12** 7.86 
    Intercept Age
 
Gender
 
AYE
 
 F R2 forumla Bo B1 t (594) B2 t (594) B3 t (594) 
Verbal Memory 
 VMPredicted = Bo + B1 × Age 61.20** .09** .09 52.49 −0.27** −7.82     
 VMPredicted = Bo + B2 × Gender 14.68** .02** .02 41.48   3.19** 3.83   
 VMPredicted = Bo + B3 × AYE 42.04** .07** .07 29.74     1.08** 6.48 
 VMPredicted = Bo + B1 × Age + B2 × Gender 40.86** .12** .12 50.95 −0.27** −8.09 3.42** 4.32   
 VMPredicted = Bo + B1 × Age + B3 × AYE 56.97** .16** .16 39.06 −0.27** −8.20   1.09** 6.92 
 VMPredicted = Bo + B2 × Gender + B3 × AYE 29.04** .09** .09 28.27   3.12** 3.88 1.07** 6.51 
 VMPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 45.63** .19** .18 37.66 −0.27** −8.48 3.35** 4.41 1.08** 6.97 
Digit Sequencing 
 DSPredicted = Bo + B1 × Age 10.17** .02** .02 22.17 −0.04** −3.19     
 DSPredicted = Bo + B2 × Gender 6.45* .01* .01 21.17   −0.73* −2.54   
 DSPredicted = Bo + B3 × AYE 1.85 — — 19.82     0.08 1.36 
 DSPredicted = Bo + B1 × Age + B2 × Gender 8.12** .03** .02 22.48 −0.04** −3.11 −0.69* −2.44   
 DSPredicted = Bo + B1 × Age + B3 × AYE 6.08** .02** .02 21.17 −0.04** −3.21   0.08 1.40 
 DSPredicted = Bo + B2 × Gender + B3 × AYE 4.21* .01* .01 20.16   −0.73* −2.56 0.08 1.40 
 DSPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 6.11** .03** .03 21.46 −0.04** −3.13 −0.70* −2.47 0.08 1.44 
Token Motor Task 
 TMTPredicted = Bo + B1 × Age 36.24** .06** .06 86.69 −0.25** −6.02     
 TMTPredicted = Bo + B2 × Gender .74 — — 78.18   −0.88 −0.86   
 TMTPredicted = Bo + B3 × AYE 5.53* .01* .01* 71.69     0.49* 2.35 
 TMTPredicted = Bo + B1 × Age + B2 × Gender 18.32** .06** .06** 86.99 −0.25** −5.99 −0.66 −0.67   
 TMTPredicted = Bo + B1 × Age + B3 × AYE 21.37** .07** .06 80.52 −0.25** −6.07   0.50* 2.49 
 TMTPredicted = Bo + B2 × Gender + B3 × AYE 3.17* .01* .01 72.11   −0.91 −0.89 0.49* 2.36 
 TMTPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 14.40** .07** .07** 80.81 −0.25** −6.04 −0.69 −0.70 0.50* 2.49 
Semantic Fluency: Animals 
 AniFPredicted = Bo + B1 × Age 19.75** .03** .03 22.78 −0.08** −4.44     
 AniFPredicted = Bo + B2 × Gender .31 — — 20.14   −0.24 −0.56   
 AniFPredicted = Bo + B3 × AYE 20.87** .03** .03 15.23     0.39** 4.57 
 AniFPredicted = Bo + B1 × Age + B2 × Gender 9.94** .03** .03 22.86 −0.08** −4.42 −0.17 −0.41   
 AniFPredicted = Bo + B1 × Age + B3 × AYE 21.23** .07** .06 17.97 −0.08** −4.57   0.39** 4.69 
 AniFPredicted = Bo + B2 × Gender + B3 × AYE 10.62** .04** .03 15.35   −0.26 −0.63 0.39** 4.57 
 AniFPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 14.21** .07** .06 18.05 −0.08** −4.55 −0.19 −0.48 0.39** 4.69 
Semantic Fluency: Fruits 
 FruFPredicted = Bo + B1 × Age .09 — — 16.06 −0.00 −0.30     
 FruFPredicted = Bo + B2 × Gender 42.63** .07** .07 15.05   1.76** 6.53   
 FruFPredicted = Bo + B3 × AYE 19.46** .03** .03 12.87     0.25** 4.41 
 FruFPredicted = Bo + B1 × Age + B2 × Gender 21.44** .07** .06 15.26 −0.00 −0.54 1.76** 6.54   
 FruFPredicted = Bo + B1 × Age + B3 × AYE 9.77** .03** .03 13.01 −0.00 −0.35   0.25** 4.41 
 FruFPredicted = Bo + B2 × Gender + B3 × AYE 32.03** .10** .10 12.06   1.74** 6.57 0.24** 4.48 
 FruFPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 21.45** .10** .09 12.29 −0.01 −0.60 1.75** 6.59 0.24** 4.48 
Semantic Fluency: Vegetables 
 VegFPredicted = Bo + B1 × Age 30.13** .05** .05 8.93 0.08** 5.49     
 VegFPredicted = Bo + B2 × Gender 86.30** .13** .13 10.14   2.87** 9.29   
 VegFPredicted = Bo + B3 × AYE 16.21** .03** .03 8.27     0.27** 4.03 
 VegFPredicted = Bo + B1 × Age + B2 × Gender 60.60** .17** .17 7.67 0.07** 5.53 2.81** 9.31   
 VegFPredicted = Bo + B1 × Age + B3 × AYE 23.73** .07** .07 5.68 0.07** 5.52   0.27** 4.07 
 VegFPredicted = Bo + B2 × Gender + B3 × AYE 53.12** .15** .15 6.92   2.85** 9.36 .26** 4.19 
 VegFPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 47.54** .19** .19 4.51 0.07** 5.57 2.79** 9.39 0.26** 4.24 
Semantic Fluency: Total 
 SFPredicted = Bo + B1 × Age .04 — — 47.77 −0.01 −0.2     
 SFPredicted = Bo + B2 × Gender 33.84** .05** .05 45.33   4.39** 5.82   
 SFPredicted = Bo + B3 × AYE 34.24** .06** .05 36.37     0.90** 5.85 
 SFPredicted = Bo + B1 × Age + B2 × Gender 16.98** .05** .05 45.79 −0.01 −0.42 4.40** 5.82   
 SFPredicted = Bo + B1 × Age + B3 × AYE 17.13** .06** .05 36.67 −0.01 −0.27   0.90** 5.85 
 SFPredicted = Bo + B2 × Gender + B3 × AYE 35.53** .11** .10 34.33   4.33** 5.90 0.89** 5.94 
 SFPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 23.73** .11** .10 34.85 −0.02 −0.49 4.34** 5.91 0.89** 5.94 
Symbol Coding 
 SCPredicted = Bo + B1 × Age 92.67** .14** .13 72.71 −0.30** −9.63     
 SCPredicted = Bo + B2 × Gender 18.72** .03** .03 60.19   3.42** 4.33   
 SCPredicted = Bo + B3 × AYE 20.98** .03** .03 52.76     0.74** 4.58 
 SCPredicted = Bo + B1 × Age + B2 × Gender 60.89** .17** .17 71.05 −0.31** −10.00 3.69** 5.03   
 SCPredicted = Bo + B1 × Age + B3 × AYE 60.95** .17** .17 63.42 −0.31** −9.88   0.76** 5.04 
 SCPredicted = Bo + B2 × Gender + B3 × AYE 20.22** .06** .06 51.17   3.38** 4.34 0.73** 4.59 
 SCPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 50.90** .21** .20 61.90 −0.31** −10.26 3.64** 5.07 0.75** 5.08 
Tower of London 
 TOLPredicted = Bo + B1 × Age 44.72** .07** .07 19.65 −0.07** −6.69     
 TOLPredicted = Bo + B2 × Gender 4.16* .01* .01 17.44   −0.52* −2.04   
 TOLPredicted = Bo + B3 × AYE 19.10** .03** .03 14.39     0.23** 4.37 
 TOLPredicted = Bo + B1 × Age + B2 × Gender 24.21** .08** .07 19.86 −0.07** −6.63 −0.46 −1.87   
 TOLPredicted = Bo + B1 × Age + B3 × AYE 33.74** .10** .10 16.84 −0.07** −6.85   0.23** 4.61 
 TOLPredicted = Bo + B2 × Gender + B3 × AYE 11.87** .04** .04 14.64   −0.54* −2.13 0.23** 4.41 
 TOLPredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 23.89** .11** .10 17.04 −0.07** −6.79 −0.48 −1.96 0.23** 4.64 
BACS Composite Score 
 BACS_CompositePredicted = Bo + B1 × Age 93.48** .14** .14 1.11 −0.03** −9.67     
 BACS_CompositePredicted = Bo + B2 × Gender 5.45*   −0.10   0.19* 2.33   
 BACS_CompositePredicted = Bo + B3 × AYE 51.06** .08** .08 −1.42     0.12** 7.15 
 BACS_CompositePredicted = Bo + B1 × Age + B2 × Gender 51.42** .15** .15 1.01 −0.03** −9.83 0.22** 2.87   
 BACS_CompositePredicted = Bo + B1 × Age + B3 × AYE 82.35** .22** .22 −0.33 −0.03** −10.23   0.12** 7.85 
 BACS_CompositePredicted = Bo + B2 × Gender + B3 × AYE 28.45** .09** .09 −1.51   0.18* 2.34 0.12** 7.14 
 BACS_CompositePredicted = Bo + B1 × Age + B2 × Gender + B3 × AYE 58.42** .23* .23 −0.42 −0.03** −10.40 0.21** 2.91 0.12** 7.86 

Note: AYE = Adjusted Years of Education.

*p < .05.

**p < .01.

If a single predictor is entered in the model, the standard error is calculated as follows,  

(2)
formula

Where forumla is the standard error of forumla, where i is a new observation, Sy.x the root of the mean square error (MSE), N the sample size, Xo the individual's score on the predictor, X the mean score on the predictor variable, and SDx the standard deviation of the predictor variable.

For a multiple predictor model with k predictors, the standard error, forumla, associated with this predicted score is  

(3)
formula

Where rmm is the main diagonal elements, rmm the off-diagonal elements of the (k x k) inverted correlation matrix between pairs of predictors, Xmi the ith subject's value on the mth predictor in standardized form. Inside the square root expression, the first summation is over the k diagonal elements, whereas the second summation is over the k(k−1)/2 off-diagonal elements.

The z-score of an individual is then calculated by:  

(4)
formula

Results

Means and Standard Deviations

Information on the distribution of age- and sex-standardized scores computed using the regression methodology is presented in Table 3.

Regression Results

Age, Sex, and AYE (as a measure of education) were regressed on each of the BACS subtests first as single predictors to adjust neurocognitive performances for each predictor. Subsequently, Sex and Age, Sex and AYE, Age and AYE, and Age, Sex, and AYE were entered in separate regression models. Beta-coefficients of these predictors are reported in Table 4. These coefficients are utilized to establish the expected performance of an individual of a particular demographic profile.

Table 4 revealed that women had better performances in language-related tasks such as Verbal Memory and Fluency tasks, as evident by the positive directionality of the beta-coefficients. On the other hand, men performed better in visuospatial tasks, such as the Tower of London. These are consistent with the findings derived from the stratification methodology.

In terms of the relative importance of the predictors to subtest performances, standardized coefficients for age were highest compared with sex and AYE, except for the Semantic Fluency subtest. The standardized coefficients for age have a mean of −0.22 and varied between −0.13 in Digit Sequencing and −0.38 in Symbol Coding. The relative contribution of Sex ranges from −0.03 in Token Motor Task to 0.23 in Semantic Fluency, whereas AYE varied from 0.06 in Digit Sequencing to 0.26 in Verbal Memory. Sex and AYE have relative similar contributions to the Semantic Fluency with a standardized beta of 0.23. For composite scores, age has the highest standardized beta of −0.38, followed by AYE of 0.28 and Sex of 0.11.

Residual plots of subtests and composite scores also suggested that data fitted a regression model reasonably well, as measured by residuals that fall between ±3 SD of the standardized residuals. Residual plots of the composite scores are presented in Fig. 1. We discuss these implications in the “Discussion” section.

Fig. 1.

Regression residual plots for composite scores.

Fig. 1.

Regression residual plots for composite scores.

Agreement of the Two Analytical Methods

Next, Bland–Altman analyses were performed to establish the comparability of the stratification via the age group and sex method and the linear regression approach using age and sex as predictors. We conclude that the two methods agree if the 95% lower and upper limits of agreement are within ±0.25.

Results from the Bland–Altman analyses indicate that the two methods may not be comparable. Discrepancies were observed at the upper and lower limits of nearly all scores (Table 5). A graphical presentation of the Bland–Altman plot of the composite scores is presented in Fig. 2. We performed paired sample t-tests to compare the means (MStrata vs. Predicted ŶSubtest) and standard deviations (SDStrata vs. forumla) of each methodology that were utilized to calculate z-scores to investigate if these may contribute to the lack of agreement. We set a threshold of 0.25 SD so that comparisons that fall below this threshold are considered acceptable. Results showed that all subtest means and most standard deviation comparisons fall under 0.2 SD. Next, we also performed paired sample t-tests on all z-scores of subtest and composite scores that were derived from both methodologies. Results showed that all comparisons fell under 0.0005 SD.

Table 5.

Bland–Altman (BA) limits and the total range of agreement for the comparison of normative methods for BACS subtests and composite scores

 Composite VM DS TMT Ani Flu Fru Flu Veg Flu SF SC TOL 
BA limits (95% CI) 
 Upper limit 0.51 (0.48, 0.55) 0.43 (0.40, 0.46) 0.47 (0.43, 0.50) 0.49 (0.46, 0.53) 0.42 (0.40, 0.45) 0.33 (0.31, 0.36) 0.37 (0.34, 0.39) 0.38 (0.35, 0.41) 0.51 (0.48, 0.55) 0.60 (0.56, 0.64) 
 Lower limit −0.51 (−0.55, −0.48) −0.43 (−0.46, −0.40) −0.47 (−0.50, −0.43) −0.49 (−0.53, −0.46) −0.42 (−0.45, −0.40) −0.33 (−0.36, −0.31) −0.37 (−0.39, −0.34) −0.38 (−0.41, −0.35) −0.51 (−0.55, −0.48) −0.60 (−0.64, −0.56) 
 Composite VM DS TMT Ani Flu Fru Flu Veg Flu SF SC TOL 
BA limits (95% CI) 
 Upper limit 0.51 (0.48, 0.55) 0.43 (0.40, 0.46) 0.47 (0.43, 0.50) 0.49 (0.46, 0.53) 0.42 (0.40, 0.45) 0.33 (0.31, 0.36) 0.37 (0.34, 0.39) 0.38 (0.35, 0.41) 0.51 (0.48, 0.55) 0.60 (0.56, 0.64) 
 Lower limit −0.51 (−0.55, −0.48) −0.43 (−0.46, −0.40) −0.47 (−0.50, −0.43) −0.49 (−0.53, −0.46) −0.42 (−0.45, −0.40) −0.33 (−0.36, −0.31) −0.37 (−0.39, −0.34) −0.38 (−0.41, −0.35) −0.51 (−0.55, −0.48) −0.60 (−0.64, −0.56) 

Notes: VM = Verbal Memory; DS = Digit Sequencing; TMT = Token Motor Task; Ani Flu = Animal Fluency; Fru Flu = Fruits Fluency; SF = Total Semantic Fluency; SC = Symbol Coding; TOL = Tower of London.

Fig. 2.

Bland–Altman plot for composite scores.

Fig. 2.

Bland–Altman plot for composite scores.

Comparison with U.S. Normative Sample

We compared the U.S. normative sample (Keefe et al., 2008) and those established in the two analytical approaches to demonstrate the overall normative difference in both samples. As the overall age ranges in the two normative samples were discrepant, subjects within the overlapping age range of 20–55 were selected for this analysis. A total of 267 subjects from the U.S. normative sample (mean age = 41.22, SD = 10.29) were compared with 595 subjects from this study (mean age = 35.96, SD = 10.88). The two samples were compared using Cohen's d. Cohen's d was calculated using a pooled standard deviation (Zakzanis, 2001).

Results

Comparisons with U.S. Normative Samples

Table 6 presents the results derived from using Cohen's d. Performances between the two samples were compared using raw scores and standardized scores using both stratification and regression methodologies. The Western subjects consistently performed approximately half a standard deviation higher than the English-speaking ethnic Chinese in language-related tasks, such as Verbal Memory and Semantic Fluency. However, the current normative sample had better performance on non-verbal tasks such as Token Motor Task. Overall, composite scores revealed that the Western sample was approximately one third of a standard deviation higher than the current sample.

Table 6.

Comparison of neurocognitive subtests with the current normative sample and the Keefe and colleagues (2004) normative sample

 Raw scores
 
Z-scores standardized using current norms
 
  Stratification methoda
 
Regression methodb
 
 Current sample (n = 516)
 
U.S. sample (n = 267)
 
SDPooled d U.S. sample (n = 267)
 
SDPooled d U.S. sample (n = 267)
 
SDPooled d 
 n Mean SD n Mean SD Mean SD Mean SD 
Verbal Memory 516 42.19 10.22 266 47.88 8.93 10.16 0.56 0.67 0.92 1.01 0.66 0.64 0.85 1.00 0.65 
Digit Sequencing 516 20.69 3.55 267 21.71 3.80 3.67 0.28 0.31 1.13 1.05 0.30 0.32 1.07 1.03 0.31 
Token Motor 516 77.47 12.41 259 71.42 14.64 13.49 −0.45 −0.38 1.17 1.07 −0.36 −0.42 1.19 1.08 −0.39 
Animals Fluency 516 19.76 5.10 267 22.56 5.48 5.40 0.52 0.62 1.10 1.07 0.58 0.59 1.08 1.06 0.56 
Total Fluency 516 47.58 9.60 266 52.95 11.64 10.64 0.50 0.62 1.32 1.15 0.54 0.57 1.26 1.12 0.51 
Symbol Coding 516 61.29 9.37 266 59.50 12.17 10.43 −0.17 −0.10 1.33 1.11 −0.09 −0.10 1.29 1.11 −0.09 
Tower of London 516 17.07 3.14 267 17.13 3.51 3.27 0.02 0.04 1.27 1.09 0.04 0.11 1.16 1.06 0.11 
Composite         0.38 1.33 1.12 0.34 0.36 1.25 1.10 0.33 
 Raw scores
 
Z-scores standardized using current norms
 
  Stratification methoda
 
Regression methodb
 
 Current sample (n = 516)
 
U.S. sample (n = 267)
 
SDPooled d U.S. sample (n = 267)
 
SDPooled d U.S. sample (n = 267)
 
SDPooled d 
 n Mean SD n Mean SD Mean SD Mean SD 
Verbal Memory 516 42.19 10.22 266 47.88 8.93 10.16 0.56 0.67 0.92 1.01 0.66 0.64 0.85 1.00 0.65 
Digit Sequencing 516 20.69 3.55 267 21.71 3.80 3.67 0.28 0.31 1.13 1.05 0.30 0.32 1.07 1.03 0.31 
Token Motor 516 77.47 12.41 259 71.42 14.64 13.49 −0.45 −0.38 1.17 1.07 −0.36 −0.42 1.19 1.08 −0.39 
Animals Fluency 516 19.76 5.10 267 22.56 5.48 5.40 0.52 0.62 1.10 1.07 0.58 0.59 1.08 1.06 0.56 
Total Fluency 516 47.58 9.60 266 52.95 11.64 10.64 0.50 0.62 1.32 1.15 0.54 0.57 1.26 1.12 0.51 
Symbol Coding 516 61.29 9.37 266 59.50 12.17 10.43 −0.17 −0.10 1.33 1.11 −0.09 −0.10 1.29 1.11 −0.09 
Tower of London 516 17.07 3.14 267 17.13 3.51 3.27 0.02 0.04 1.27 1.09 0.04 0.11 1.16 1.06 0.11 
Composite         0.38 1.33 1.12 0.34 0.36 1.25 1.10 0.33 

aStandardization by age group and sex.

bStandardization by age and sex.

Discussion

This study provides normative data for the BACS in a sample of English-speaking ethnic Chinese aged 14–55 in Singapore. Data were stratified by age group and sex. Consistent with previous studies (Bopp & Verhaeghen, 2005; Charlton et al., 2006; Hsieh & Tori, 2007; Lee, Yuen, & Chan, 2002), declining cognitive performance was observed with increasing age. We also found that women were superior in Language and Symbol Coding tasks performances, whereas men performed better in tasks that require visuospatial abilities, such as Tower of London. Our findings are comparable with previous studies that reported sex-related cognitive differences (Lynn & Irwing, 2008; Majeres, 1999; Mann, Sasanuma, Sakuma, & Masaki, 1990; Rönnlund, Lövdén, & Nilsson, 2001).

Although English was used for test administration, a comparison between the current normative sample and a Western sample revealed some notable differences. The Western sample performed better on language-related tasks such as Verbal Memory and Semantic Fluency, whereas ethnic Chinese participants scored higher on non-language tasks such as Symbol Coding. Findings could be attributed to the familiarity with the use of iconographic symbols in the Chinese language (Perfetti, Liu, & Tan, 2005; Tan, Laird, Li, & Fox, 2005). We found cross cultural differences in performance even when a common language medium is employed. This highlights the importance of establishing a set of culturally appropriate norms where influences of culture, language, and education should be taken into consideration during cognitive evaluation (Boone, Victor, Wen, Razani, & Pontón, 2007; Patton et al., 2003; Shan, Chen, Lee, & Su, 2008).

Sampling methodologies play a crucial role in establishing normative data for neuropsychological performances. The U.S. normative sample was matched to the 2005 United States Census (Keefe et al., 2008) and was of a higher mean age when compared with the current sample. Education was also adjusted differently in the U.S. normative sample. Sample-specific differences in culture, language, and education may lead to inaccurate estimates of task performance in the Singaporean sample if U.S. norms were applied. Direct application of Western norms on a sample of ethnic Chinese with Schizophrenia may result in spurious interpretations. In particular, patients may appear less impaired than actual in tasks such as the token motor task, but may be more susceptible to show deficits in the verbal memory task. This is problematic as such a cognitive profile is not an accurate representation of the impairments in these patients, and the utilization of such information could confound and confuse remediation efforts. In view of the findings, the use of Western norms in an ethnic Chinese sample increases the likelihood of spurious impairments in language and executive functioning which could have serious implications for the interpretation of cognitive performance in intervention trials, or program outcome evaluation. Future studies should use appropriate norms to evaluate the cognitive profile of ethnic Chinese patients with Schizophrenia and compare the observed performances against the sample of healthy controls.

To our knowledge, this is the first neuropsychological normative study on an Asian sample that allows continuous adjustments for age, sex, and education. Linear regression approaches were utilized to allow adjustments for demographic variables of age, sex, and education (as measured by AYE). We developed program files to compute scores on subtests and composite performances using the stratification and linear regression methodologies. Users can make use of the normative data by imputing the observed scores into the program files to calculate T- or z-scores adjusted for the effects of demographic variables. Stratification program files only allow users to adjust for the combined effects of age group and sex. On the other hand, linear regression program files allow studies to specify any combinations of age, sex, and education. The use of linear regression approaches to adjust demographic influences is consistent with the contemporary computation of neuropsychological composites (e.g., MATRICS Cognitive Consensus Battery, Kern et al., 2008). The linear regression methodology allows for the adjustment of individual level demographic variables, compared with conventional group sample adjustment techniques reported in neuropsychological normative studies. Our program files will be made available for clinicians and researchers upon request.

Bland–Altman analysis was conducted to assess the agreement between the two methodologies. Results indicated that both methodologies showed discrepancies. We conducted regression residual plots of all subtests and composite scores and demonstrated that our data fit a regression model reasonably well. Also, we explored the means and standard errors that were used in the calculation of z-scores in both methodologies. Specifically, the stratification approach calculates its z-scores by using the standard deviation derived from the sample of a particular age group and sex strata. This standard deviation reflects the variability in the distribution of subtest performances within each stratum and z-scores indicate performance relative to the sample within the strata. The linear regression approach serves as an approximation of subtest performances relative to the full sample, after controlling for the contributions of demographic variables to subtest performances. The standard error of prediction that is used in the linear regression approach represents the uncertainty in the prediction of an individual score (Hayashi, 2000), based on the contributions of the demographic variables. Paired sample t-tests comparisons of the means and standard deviations of both methodologies suggest that both methodologies seem comparable. Further investigations can also employ data simulation methodologies to refine the age group stratum and compare its agreeability with the regression approach.

Despite the comparability of both methodologies, researchers should still be aware of the merits and limitations of both analytical methodologies. The stratification approach allows the comparison of the performance of an individual to his or her respective demographic group. This analytical approach would be suitable for studies or clinical trials that assume equal neuropsychological performances across subjects within each age group and sex strata. As z-scores indicate performances that are particular to an age group and sex strata, researchers and clinicians should be cautious when making comparisons of performances across the stratum. In addition, z-scores are computed based on the sample in a particular group, for example, number of men who are between the age group of 25–29, which limits the generalizability of test scores. On the other hand, the linear regression approach calculates an individuals' neuropsychological performance against the full sample and controls for the effects of age, sex, and education. An advantage over the stratification methodology is that the linear regression approach allows the user to take into account pertinent influences of education on neuropsychological performance. Moreover, the linear regression approach also provides the flexibility for studies to adjust for combinations of demographic variables. Considering that residual plots did not show patterns that suggest contradictions of an adequate fit, the linear regression approach appears better than the discrete stratification methodology in this study.

Even though the linear regression approach serves as a reasonable approximation of relative performance, it is important to note that this approach assumes linear effects of demographic variables to test performances. Although it may be true that a polynomial regression model may yield a better model fit, this approach would involve including more predictors into the model, which may result in overfitting. Therefore, a conservative approach of linear regression was used in attempting to establish a model that is parsimonious (Hawkins, 2004).

Considering that decreasing frequencies are typically present at the extreme ends of a normal distribution, future studies could also explore using asymmetrical confidence intervals and percentile ranks to express neuropsychological test scores (Crawford & Garthwaite, 2009). This methodology may be more accurate and potentially useful in calibrating test measures of cognitive domains.

To summarize, the BACS has been widely used to measure cognitive deficits in schizophrenia (Chianetta, Lefebvre, LeBlanc, & Grignon, 2008; Keefe et al., 2004). This study provides normative data in an English-speaking ethnic Chinese sample using two different normative approaches. Future studies that intend to utilize normative information for the use of research or clinical applications should consider which analytic approach to adopt, given that each has its own advantages. Nevertheless, normative information established in this study can potentially bolster clinical and research efforts in English-speaking ethnic Chinese samples.

Strengths and Limitations

This normative sample for the BACS is among the largest reported. We presented an alternative methodology using linear regression that enables users to calculate t- and z-scores that can be adjusted for any combinations of age, sex, and education. We established norms for subcategories of the semantic fluency task allowing greater flexibility in using test batteries for subsequent research endeavors. The age range for the current normative sample included individuals as young as 14 years of age, strengthening subsequent utility of the BACS to studies investigating at-risk mental states. Finally, the current study presents much needed information in the literature for neuropsychological normative data established in an ethnic Chinese sample that will enable further cognitive research in ethnic Chinese schizophrenia samples.

Several limitations of this study should be noted. Although steps have been taken to ensure that the current sample is similar to the demographics of Chinese in Singapore, we recruited participants with adequate English language abilities for neuropsychological testing. Therefore, caution has to be exercised if norms are to be generalized to samples that are not tested in English. Similarly, interpretations should also be treated with discretion if norms are extended to samples of other ethnic groups or samples with participants' ages that fall out of the limits of our normative sample of 14–55. Lastly, users who decide to utilize the regression approach of the normative data should check graphically if the relationship of subtest scores and demographic variables are linear, as the regression approach would be inappropriate if the linearity assumption is not fulfilled.

Funding

This study was supported by the Institute of Mental Health – Institutional Block Grant (Clincal Research Committee Reference No.: 300/2010). The Singapore Translational and Clinical Research in Psychosis is supported by the National Research Foundation Singapore under the National Medical Research Council Translational and Clinical Research Flagship Programme (Grant No.: NMRC/TCR/003/2008).

Conflict of Interest

R.K., Ph.D. reports that he currently or in the past 3 years has received investigator-initiated research funding support from the Allon, AstraZeneca, Novartis, National Institute of Mental Health, Allon, GlaxoSmithKline, Department of Veteran's Affairs, GlaxoSmithKline, National Institute of Mental Health, Novartis, PsychoGenics, and the Singapore National Medical Research Council. He currently or in the past 3 years has received honoraria, served as a consultant, or advisory board member for Abbott, Amgen, Astellas, Asubio, BiolineRx, Boehringer-Ingelheim, BrainCells, Bristol-Myers Squibb, CHDI, Eli Lilly, EnVivo, Helicon, Lundbeck, Memory Pharmaceuticals, Mitsubishi, NeuroSearch, Novartis, Orexigen, Organon Pharmaceuticals, Orion, Otsuka, Pfizer, Roche, Sanofi-Aventis, Shering-Plough, Shire, Solvay, Sunovion, Takeda, Targacept, Wyeth, and Xenoport. Dr. R.K. receives royalties from the Brief Assessment of Cognition in Schizophrenia (BACS) testing battery and the MATRICS Battery (BACS Symbol Coding). He is also a shareholder in NeuroCog Trials, Inc. The other authors have no conflict of interests to declare.

References

Addington
J.
Brooks
B. L.
Addington
D.
Cognitive functioning in first episode psychosis: Initial presentation
Schizophrenia Research
 , 
2003
, vol. 
62
 
1–2
(pg. 
59
-
64
)
Anselmetti
S.
Poletti
S.
Ermoli
E.
Bechi
M.
Cappa
S.
Venneri
A.
, et al.  . 
The Brief Assessment of Cognition in Schizophrenia: Normative data for the Italian population
Neurological Sciences
 , 
2008
, vol. 
29
 
2
(pg. 
85
-
92
)
Ardila
A.
Directions of research in cross-cultural neuropsychology
Journal of Clinical and Experimental Neuropsychology
 , 
1995
, vol. 
17
 
1
(pg. 
143
-
150
)
Ardila
A.
Rosselli
M.
Ostrosky-Solis
F.
Puente
A. E.
McCaffrey
R. J.
Sociocultural factors in neuropsychological assessment
Handbook of neuropsychological assessment: A biopsychosocial perspective
 , 
1992
New York
Plenum Press
(pg. 
181
-
192
)
Bilder
R. M.
Goldman
R. S.
Robinson
D.
Reiter
G.
Bell
L.
Bates
J. A.
, et al.  . 
Neuropsychology of first-episode schizophrenia: Initial characterization and clinical correlates
American Journal of Psychiatry
 , 
2000
, vol. 
157
 
4
(pg. 
549
-
559
)
Bilder
R. M.
Goldman
R. S.
Volavka
J.
Czobor
P.
Hoptman
M.
Sheitman
B.
, et al.  . 
Neurocognitive effects of clozapine, olanzapine, risperidone, and haloperidol in patients with chronic schizophrenia or schizoaffective disorder
American Journal of Psychiatry
 , 
2002
, vol. 
159
 
6
(pg. 
1018
-
1028
)
Boone
K. B.
Victor
T. L.
Wen
J.
Razani
J.
Pontón
M.
The association between neuropsychological scores and ethnicity, language, and acculturation variables in a large patient population
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22
 
3
(pg. 
355
-
365
)
Bopp
K. L.
Verhaeghen
P.
Aging and verbal memory span: A meta-analysis
Journals of Gerontology Series B: Psychological Sciences and Social Sciences
 , 
2005
, vol. 
60
 
5
(pg. 
P223
-
P233
)
Bralet
M. C.
Falissard
B.
Neveu
X.
Lucas-Ross
M.
Eskenazi
A. M.
Keefe
R. S. E.
Validation of the French version of the BACS (the brief assessment of cognition in schizophrenia) among 50 French schizophrenic patients
European Psychiatry
 , 
2007
, vol. 
22
 
6
(pg. 
365
-
370
)
Cavallaro
R.
Anselmetti
S.
Poletti
S.
Bechi
M.
Ermoli
E.
Cocchi
F.
, et al.  . 
Computer-aided neurocognitive remediation as an enhancing strategy for schizophrenia rehabilitation
Psychiatry Research
 , 
2009
, vol. 
169
 
3
(pg. 
191
-
196
)
Censits
D. M.
Ragland
J. D.
Gur
R. C.
Gur
R. E.
Neuropsychological evidence supporting a neurodevelopmental model of schizophrenia: A longitudinal study
Schizophrenia Research
 , 
1997
, vol. 
24
 
3
(pg. 
289
-
298
)
Charlton
R. A.
Barrick
T. R.
McIntyre
D. J.
Shen
Y.
O'Sullivan
M.
, et al.  . 
White matter damage on diffusion tensor imaging correlates with age-related cognitive decline
Neurology
 , 
2006
, vol. 
66
 
2
(pg. 
217
-
222
)
Chianetta
J. M.
Lefebvre
M.
LeBlanc
R.
Grignon
S.
Comparative psychometric properties of the BACS and RBANS in patients with schizophrenia and schizoaffective disorder
Schizophrenia Research
 , 
2008
, vol. 
105
 
1–3
(pg. 
86
-
94
)
Cohen
J.
Cohen
P.
West
S. G.
Aiken
L. S.
Applied multiple regression/correlation analysis for the behavioral sciences
 , 
2002
2nd ed.
USA
Psychology Press
Collinson
S. L.
Lam
M.
Hayes
C. J.
The utility and benefits of clinical neuropsychology in Asia
Asian Journal of Psychiatry
 , 
2010
, vol. 
3
 
2
(pg. 
50
-
54
)
Crawford
J. R.
Garthwaite
P. H.
Percentiles please: The case for expressing neuropsychological test scores and accompanying confidence limits as percentile ranks
The Clinical Neuropsychologist
 , 
2009
, vol. 
32
 
2
(pg. 
193
-
204
)
Crawford
J. R.
Howell
D. C.
Regression equations in clinical neuropsychology: An evaluation of statistical methods for comparing predicted and obtained scores
Journal of Clinical and Experimental Neuropsychology
 , 
1998
, vol. 
20
 
5
(pg. 
755
-
762
)
First
M. B.
Spitzer
R. L.
Gibbon
M.
Williams
J. B. W.
Structured clinical interview for DSM-IV-TR axis I disorders, research version, non-patient edition (SCID-I/NP)
 , 
2002
New York
Biometrics Research, New York State Psychiatric Institute
Friedman
J. I.
Carpenter
D.
Lu
J.
Fan
J.
Tang
C. Y.
White
L.
, et al.  . 
A pilot study of adjunctive atomoxetine treatment to second-generation antipsychotics for cognitive impairment in schizophrenia
Journal of Clinical Psychopharmacology
 , 
2008
, vol. 
28
 
1
(pg. 
59
-
63
)
Geffen
Y.
Keefe
R.
Rabinowitz
J.
Anand
R.
Davidson
M.
BL-1020 a new GABA-enhanced antipsychotic: Results of six-week, randomized, double-blind, controlled, efficacy and safety study
Journal of Clinical Psychiatry
 , 
2012
, vol. 
73
 
9
(pg. 
e1168
-
e1174
)
Goff
D. C.
Keefe
R.
Citrome
L.
Davy
K.
Krystal
J. H.
Large
C.
, et al.  . 
Lamotrigine as add-on therapy in schizophrenia: Results of 2 placebo-controlled trials
Journal of Clinical Psychopharmacology
 , 
2007
, vol. 
27
 
6
(pg. 
582
-
589
)
Gong
Y.
Revision of Wechsler's Adult Intelligence Scale in China
Acta Psychologica Sinica
 , 
1983
, vol. 
15
 
3
(pg. 
362
-
370
)
Government of Singapore
Statistics: Demography (themes)
 , 
2010
 
Harvey
P. D.
Keefe
R. S. E.
Cognitive impairment in schizophrenia and implications of atypical neuroleptic treatment
CNS Spectrums
 , 
1997
, vol. 
2
 (pg. 
1
-
11
)
Hawkins
D. M.
The problem of overfitting
Journal of Chemical Information and Modeling
 , 
2004
, vol. 
44
 (pg. 
1
-
12
)
Hayashi
F.
Econometrics.
 , 
2000
USA
Princeton University Press
Hill
S. K.
Sweeney
J. A.
Hamer
R. M.
Keefe
R. S. E.
Perkins
D. O.
Gu
H.
, et al.  . 
Efficiency of the CATIE and BACS neuropsychological batteries in assessing cognitive effects of antipsychotic treatments in schizophrenia
Journal of the International Neuropsychological Society
 , 
2008
, vol. 
14
 
2
(pg. 
209
-
221
)
Hobart
M. P.
Goldberg
R.
Bartko
J. J.
Gold
J. M.
Repeatable battery for the assessment of neuropsychological status as a screening test in schizophrenia, II: convergent/discriminant validity and diagnostic group comparisons
American Journal of Psychiatry
 , 
1999
, vol. 
156
 
12
(pg. 
1951
-
1957
)
Hoff
A. L.
Sakuma
M.
Wieneke
M.
Horon
R.
Kushner
M.
DeLisi
L. E.
Longitudinal neuropsychological follow-up study of patients with first-episode schizophrenia
American Journal of Psychiatry
 , 
1999
, vol. 
156
 
9
(pg. 
1336
-
1341
)
Hsieh
S. L. J.
Tori
C. D.
Normative data on cross-cultural neuropsychological tests obtained from Mandarin-speaking adults across the life span
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22
 
3
(pg. 
283
-
296
)
Kaneda
Y.
Sumiyoshi
T.
Keefe
R.
Ishimoto
Y.
Numata
S.
Ohmori
T.
Brief assessment of cognition in schizophrenia: Validation of the Japanese version
Psychiatry and Clinical Neurosciences
 , 
2007
, vol. 
61
 
6
(pg. 
602
-
609
)
Keefe
R. S. E.
Goldberg
T. E.
Harvey
P. D.
Gold
J. M.
Poe
M. P.
Coughenour
L.
The Brief Assessment of Cognition in Schizophrenia: Reliability, sensitivity, and comparison with a standard neurocognitive battery
Schizophrenia Research
 , 
2004
, vol. 
68
 
2–3
(pg. 
283
-
297
)
Keefe
R. S. E.
Harvey
P. D.
Goldberg
T. E.
Gold
J. M.
Walker
T. M.
Kennel
C.
, et al.  . 
Norms and standardization of the Brief Assessment of Cognition in Schizophrenia (BACS)
Schizophrenia Research
 , 
2008
, vol. 
102
 
1–3
(pg. 
108
-
115
)
Keefe
R. S. E.
Malhotra
A. K.
Meltzer
H. Y.
Kane
J. M.
Buchanan
R. W.
Murthy
A.
, et al.  . 
Efficacy and safety of Donepezil in patients with schizophrenia or schizoaffective disorder: Significant placebo/practice effects in a 12-week, randomized, double-blind, placebo-controlled trial
Neuropsychopharmacology
 , 
2007
, vol. 
33
 
6
(pg. 
1217
-
1228
)
Keefe
R. S. E.
Poe
M.
Walker
T. M.
Harvey
P. D.
The relationship of the Brief Assessment of Cognition in Schizophrenia (BACS) to functional capacity and real-world functional outcome
Journal of Clinical and Experimental Neuropsychology
 , 
2006
, vol. 
28
 
2
(pg. 
260
-
269
)
Kern
R. S.
Nuechterlein
K. H.
Green
M. F.
Baade
L. E.
Fenton
W. S.
Gold
J. M.
, et al.  . 
The MATRICS Consensus Cognitive Battery, part 2: Co-norming and standardization
American Journal of Psychiatry
 , 
2008
, vol. 
165
 
2
(pg. 
214
-
220
)
Lam
M.
Eng
G. K.
Rapisarda
A.
Subramaniam
M.
Kraus
M.
Keefe
R. S.
, et al.  . 
Formulation of the age-education index: measuring age and education effects in neuropsychological performance
Psychological Assessment
 , 
2013
, vol. 
25
 
1
(pg. 
61
-
70
)
Lee
T.
Yuen
K.
Chan
C.
Normative data for neuropsychological measures of fluency, attention, and memory measures for Hong Kong Chinese
Journal of Clinical and Experimental Neuropsychology
 , 
2002
, vol. 
24
 
5
(pg. 
615
-
632
)
Lynn
R.
Irwing
P.
Sex differences in mental arithmetic, digit span, and g defined as working memory capacity
Intelligence
 , 
2008
, vol. 
36
 
3
(pg. 
226
-
235
)
Majeres
R. L.
Sex differences in phonological processes: Speeded matching and word reading
Memory and Cognition
 , 
1999
, vol. 
27
 
2
(pg. 
246
-
253
)
Mann
V. A.
Sasanuma
S.
Sakuma
N.
Masaki
S.
Sex differences in cognitive abilities: A cross-cultural perspective
Neuropsychologia
 , 
1990
, vol. 
28
 
10
(pg. 
1063
-
1077
)
Mohamed
S.
Paulsen
J. S.
O'Leary
D.
Arndt
S.
Andreasen
N.
Generalized cognitive deficits in schizophrenia: A study of first-episode patients
Archives of General Psychiatry
 , 
1999
, vol. 
56
 
8
(pg. 
749
-
754
)
Müller
U.
Werheid
K.
Hammerstein
E.
Jungmann
S.
Becker
T.
Prefrontal cognitive deficits in patients with schizophrenia treated with atypical or conventional antipsychotics
European Psychiatry
 , 
2005
, vol. 
20
 
1
(pg. 
70
-
73
)
Nuechterlein
K. H.
Barch
D. M.
Gold
J. M.
Goldberg
T. E.
Green
M. F.
Heaton
R. K.
Identification of separable cognitive factors in schizophrenia
Schizophrenia Research
 , 
2004
, vol. 
72
 
1
(pg. 
29
-
39
)
O'Bryant
S. E.
O'Jile
J. R.
McCaffrey
R. J.
Reporting of demographic variables in neuropsychological research: Trends in the current literature
The Clinical Neuropsychologist
 , 
2004
, vol. 
18
 
2
(pg. 
229
-
233
)
Ogino
S.
Miyamoto
S.
Tenjin
T.
Kitajima
R.
Ojima
K.
Miyake
N.
, et al.  . 
Effects of discontinuation of long-term Biperiden use on cognitive function and quality of life in schizophrenia
Progress in Neuro-psychopharmacology and Biological Psychiatry
 , 
2011
, vol. 
35
 
1
(pg. 
78
-
83
)
Ostrosky-Solis
F.
Canseco
E.
Quintanar
L.
Navarro
E.
Meneses
S.
Ardila
A.
Sociocultural effects in neuropsychological assessment
International Journal of Neuroscience
 , 
1985
, vol. 
27
 
1–2
(pg. 
53
-
66
)
Patton
D. E.
Duff
K.
Schoenberg
M. R.
Mold
J.
Scott
J. G.
Adams
R. L.
Performance of cognitively normal African Americans on the RBANS in community dwelling older adults
The Clinical Neuropsychologist
 , 
2003
, vol. 
17
 
4
(pg. 
515
-
530
)
Perfetti
C. A.
Liu
Y.
Tan
L. H.
The lexical constituency model: Some implications of research on Chinese for general theories of reading
Psychological Review
 , 
2005
, vol. 
112
 
1
(pg. 
43
-
59
)
Rönnlund
M.
Lövdén
M.
Nilsson
L.-G.
Adult age differences in Tower of Hanoi performance: Influence from demographic and cognitive variables
Aging, Neuropsychology, and Cognition
 , 
2001
, vol. 
8
 
4
(pg. 
269
-
283
)
Sachs
G.
Winklbaur
B.
Jagsch
R.
Keefe
R. S. E.
Validation of the German version of the brief assessment of cognition in Schizophrenia (BACS)—preliminary results
European Psychiatry
 , 
2011
, vol. 
26
 
2
(pg. 
74
-
77
)
Sarkisyan
G.
Gurovich
I.
Keefe
R. S. E.
P01-430—Normative data for Russian population and standardization of the scale ‘Brief assessment of cognition in schizophrenia
European Psychiatry
 , 
2011
, vol. 
26
 
Suppl. 1
pg. 
434
 
Saykin
A. J.
Gur
R. C.
Gur
R. E.
Mozley
P. D.
Mozley
L. H.
Resnick
, et al.  . 
Neuropsychological function in schizophrenia. Selective impairment in memory and learning
Archives of General Psychiatry
 , 
1991
, vol. 
48
 
7
(pg. 
618
-
624
)
Segarra
N.
Bernardo
M.
Gutierrez
F.
Justicia
A.
Fernadez-Egea
E.
Allas
M.
, et al.  . 
Spanish validation of the Brief Assessment in Cognition in Schizophrenia (BACS) in patients with schizophrenia and healthy controls
European Psychiatry
 , 
2011
, vol. 
26
 
2
(pg. 
69
-
73
)
Shan
I. K.
Chen
Y. S.
Lee
Y. C.
Su
T. P.
Adult normative data of the Wisconsin Card Sorting Test in Taiwan
Journal of the Chinese Medical Association
 , 
2008
, vol. 
71
 
10
(pg. 
517
-
522
)
Tan
L. H.
Laird
A. R.
Li
K.
Fox
P. T.
Neuroanatomical correlates of phonological processing of Chinese characters and alphabetic words: A meta-analysis
Human Brain Mapping
 , 
2005
, vol. 
25
 
1
(pg. 
83
-
91
)
Wilk
C. M.
Gold
J. M.
Humber
K.
Dickerson
F.
Fenton
W. S.
Buchanan
R. W.
Brief cognitive assessment in schizophrenia: Normative data for the Repeatable Battery for the Assessment of Neuropsychological Status
Schizophrenia Research
 , 
2004
, vol. 
70
 
2–3
(pg. 
175
-
186
)
Zakzanis
K. K.
Statistics to tell the truth, the whole truth, and nothing but the truth: Formulae, illustrative numerical examples, and heuristic interpretation of effect size analyses for neuropsychological researchers
Archives of Clinical Neuropsychology
 , 
2001
, vol. 
16
 
7
(pg. 
653
-
667
)

Author notes

Equally contributed.