Abstract

Cognitive impairment may be detected largely by examining the performance on a single neuropsychological measure. The purpose of the present study was to evaluate the validity and diagnostic accuracy of a coding task in comparison with other related tasks. One hundred thirty-one first-episode psychosis patients were administered five cognitive tasks related to a “speed of processing and executive functioning” dimension (Digit Symbol, Trail Making Test [TMT] parts A and B, Cancellation Test, and Digit Span-backward) and an additional measure of functional outcome. Digit Symbol provided good indices of accuracy and correlations with the global composite score of a comprehensive neuropsychological assessment represented large effect sizes. Correlations with a functional outcome were modest. Similar results were observed with the TMT. The processing speed, as measured by Digit Symbol, may be particularly good in capturing the generalized dysfunction which may be causing the widespread cognitive failures in schizophrenia spectrum disorders.

Introduction

Schizophrenia is a clinical syndrome marked by the presence of symptoms such as auditory hallucinations and delusions, as well as social withdrawal, restricted affect, and poor motivation. Patients with schizophrenia show cognitive deficits that range between 1 and 3 SD below healthy controls, the most prominent of which are related to memory, attention, and executive functions (Heinrichs & Zakzanis, 1998). These deficits are clearly present at the time of treatment initiation (González-Blanch et al., 2006) and are largely independent of clinical symptoms (González-Blanch et al., 2008). As a result, these deficits are becoming increasingly recognized as a critical treatment target (Buchanan et al., 2005; Gold, 2004; Hyman & Fenton, 2003). However, formal cognitive assessments are not often conducted in routine clinical practice. Neuropsychological test batteries take a substantial length of time and the interpretation of the results for individual assessment is complex.

To overcome these limitations, several approaches have been investigated to reduce the time and complexity of comprehensive neuropsychological assessments. Some authors have developed a new set of neuropsychological tests with shorter administration times: The Brief Assessment of Cognition in Schizophrenia (BACS; Keefe Goldberg et al., 2004) and the Screen for Cognitive Impairment in Psychiatry (SCIP; Purdon, 2005) are some examples of validated cognitive screening tools in schizophrenia. Another approach consists of selecting only a small number of standardized tests that examine some key cognitive domains; for example, the Brief Cognitive Assessment (BCA; Velligan et al., 2004) and the Brief Cognitive Assessment Tool for Schizophrenia (B-CATS; Hurford, Marder, Keefe, Reise, & Bilder, in press). Alternatively, other authors have created brief interview-based scales with specific questions referring to cognitive deficits and the degree to which they affect day-to-day functioning, such as the Schizophrenia Cognition Rating Scale (Keefe, Poe, Walker, Kang, & Harvey, 2006) and the Clinical Global Impression of Cognition in Schizophrenia (Ventura, Cienfuegos, Boxer, & Bilder, 2008). This strategy seems particularly useful given the clinicians’ difficulties in differentiating between cognitive and other clinical symptoms (Bromley, 2007), and therefore, the questionable validity of evaluations through the cognitive items of psychopathological scales (Harvey et al., 2001; Hofer et al., 2007). All these types of screens are easy to administer, and the administration times vary from 15 to 30 min approximately, but are still scarcely available in clinical settings.

In a previous research, we found that the speed of processing, as measured by a Digit Symbol coding task, was severely impaired in first-episode schizophrenia spectrum disorders (González-Blanch et al., 2007) and that this might mediate a broader diversity of cognitive deficits (Rodríguez-Sánchez, Crespo-Facorro, González-Blanch, Pérez-Iglesias, & Vázquez-Barquero, 2007). The remarkable importance of speed of processing, as measured by Digit Symbol coding tasks, has been borne out by recent meta-analyses in schizophrenia (Dickinson, Ramsey, & Gold, 2007) and first-episode schizophrenia (Mesholam-Gately, Giuliano, Goff, Faraone, & Seidman, 2009). Likewise, in the Clinical Antipsychotic Trials of Intervention Effectiveness (CATIE) study, the test that explained the greatest part of the global cognitive score at baseline assessment was the Wechsler Adult Intelligence Scale, Third Edition (WAIS-III) Digit Symbol subtest, accounting for 61% of the variance of the global score (Keefe et al., 2006).

Digit Symbol coding tasks are multifactorial. Consistent with this, Digit Symbol tasks have not been linked to specific brain structure or function, and poor performance in tasks of this kind has have been associated with diverse syndromes that are either organic or functional (Groth-Marnat, 2003). However, it has been argued that much of what is measured by traditional neuropsychological assessments is general cognitive ability, not independent, domain-specific performance. Thus, much of the severity of cognitive impairments may arise from generalized factors, which might be the fundamental manifestation of schizophrenia (Dickinson, 2008). In pursuit of very brief and simple screening tools, the fact that generalized cognitive impairment could be the core deficit, rather than specific cognitive processes, has an obvious implication: A great part of cognitive impairment in schizophrenia may be detected by examining the performance on a single neuropsychological measure.

The purpose of the present study was to examine the validity and diagnostic accuracy in the assessment of global cognitive functioning in patients with first-episode psychosis of a Digit Symbol coding task in comparison with other tasks that loaded on the same cognitive dimension (i.e., speed of processing and executive functioning) in a previous factor analytic study (González-Blanch et al., 2007).

Materials and Methods

Setting and Subjects

The study participants were part of a cohort of consecutive admissions to the Cantabria Intervention Programme of First-Episode Psychosis (PAFIP, Spanish abbreviation) from February 2001 to February 2005. The PAFIP is located at the University Hospital “Marqués de Valdecilla” (Santander, Spain). Inclusion criteria for the PAFIP were as follows: age between 15 and 60; Diagnostic and Statistical Manual of Mental Disorder, Fourth edition (DSM-IV) criteria for diagnosis of schizophrenia, schizophreniform disorder, schizoaffective disorder, brief psychotic disorder, or psychosis not otherwise specified; never treated with antipsychotic medication; and living in the catchment area. Exclusion criteria were a history of neurological disease, head injury, mental retardation, or current drug dependence (DSM-IV criteria).

Of the 174 consecutive admissions who met criteria for enrolment, 131 (75.3%) completed baseline neuropsychological assessment. All diagnoses were confirmed 6 months after study entry using the Structured Clinical Interview for DSM-IV (SCID-I; First et al., 1995) by a trained clinician. The majority of participants were male (65%). The mean age of the sample was 27 years and the mean number of years of education was 10 (Table 1). All patients were randomly assigned to haloperidol, olanzapine, or risperidone (Crespo-Facorro et al., 2006). Eighty-eight participants (67%) were prescribed a second-generation antipsychotic medication. This study was approved by the Ethics Committee of the University Hospital “Marqués de Valdecilla,” and written informed consent was obtained from all participants or their legal representative.

Table 1.

Demographic, clinical, and treatment characteristics

 Patients (n = 131) Controls (n = 28) Statistic p-value 
Male sex (n [%]) 85 (64.9) 13 (46.4) χ2 = 3.32 .07 
Age (mean [SD]) 26.8 (7.3) 25.89 (6.29) t = 0.68 .46 
Education (years; mean [SD]) 10.3 (3.2) 11.74 (2.92) t = −1.17 .03 
Premorbid IQ estimationa (mean [SD]) 91.9 (21.0) 101.0 (17.1) t = −2.11 .04 
DUI (months; mean [SD]; median) 24.8 (32.9); 12.0    
DUP (months; mean [SD]; median) 10.8 (23.9); 4.0    
Age of psychosis onset (years; mean [SD]) 25.9 (6.9)    
Diagnoses (n [%]) 
 Schizophrenia 77 (58.8)    
 Schizophreniform disorder 34 (26.0)    
 Brief psychotic disorder 8 (6.1)    
 Psychosis NOS 7 (5.3)    
 Schizoaffective Disorder 4 (3.1)    
 Schizotypal Personality Disorder 1 (.8)    
SANS at week 6 (mean [SD]) 5.1 (5.0)    
SAPS at week 6 (mean [SD]) 3.0 (3.3)    
BPRS at week 6 (mean [SD]) 34.2 (8.8)    
Initial hospitalization (n [%]) 85 (64.9)    
Days of hospitalization (mean [SD]) 20.2 (11.5)    
Antipsychotics (n [%]) 
 Haloperidol 43 (32.8)    
 Risperidone 45 (34.4)    
 Olanzapine 43 (32.8)    
Dose at week 6 (mg/day; mean [SD]) 
 Haloperidol 5.1 (2.13)    
 Risperidone 4.37 (1.49)    
 Olanzapine 15.43 (4.93)    
Concomitant drugs at week 6 (n [%]) 
 Anxiolytics/Hypnotics 44 (33.6)    
 Anticholinergic agents 50 (38.2)    
 Antidepressants 2 (1.5)    
 Mood stabilizers 2 (1.5)    
 Patients (n = 131) Controls (n = 28) Statistic p-value 
Male sex (n [%]) 85 (64.9) 13 (46.4) χ2 = 3.32 .07 
Age (mean [SD]) 26.8 (7.3) 25.89 (6.29) t = 0.68 .46 
Education (years; mean [SD]) 10.3 (3.2) 11.74 (2.92) t = −1.17 .03 
Premorbid IQ estimationa (mean [SD]) 91.9 (21.0) 101.0 (17.1) t = −2.11 .04 
DUI (months; mean [SD]; median) 24.8 (32.9); 12.0    
DUP (months; mean [SD]; median) 10.8 (23.9); 4.0    
Age of psychosis onset (years; mean [SD]) 25.9 (6.9)    
Diagnoses (n [%]) 
 Schizophrenia 77 (58.8)    
 Schizophreniform disorder 34 (26.0)    
 Brief psychotic disorder 8 (6.1)    
 Psychosis NOS 7 (5.3)    
 Schizoaffective Disorder 4 (3.1)    
 Schizotypal Personality Disorder 1 (.8)    
SANS at week 6 (mean [SD]) 5.1 (5.0)    
SAPS at week 6 (mean [SD]) 3.0 (3.3)    
BPRS at week 6 (mean [SD]) 34.2 (8.8)    
Initial hospitalization (n [%]) 85 (64.9)    
Days of hospitalization (mean [SD]) 20.2 (11.5)    
Antipsychotics (n [%]) 
 Haloperidol 43 (32.8)    
 Risperidone 45 (34.4)    
 Olanzapine 43 (32.8)    
Dose at week 6 (mg/day; mean [SD]) 
 Haloperidol 5.1 (2.13)    
 Risperidone 4.37 (1.49)    
 Olanzapine 15.43 (4.93)    
Concomitant drugs at week 6 (n [%]) 
 Anxiolytics/Hypnotics 44 (33.6)    
 Anticholinergic agents 50 (38.2)    
 Antidepressants 2 (1.5)    
 Mood stabilizers 2 (1.5)    

Notes: DUI = duration of untreated illness; DUP = duration of untreated psychosis; NOS = not otherwise specified; BPRS = Brief Psychiatric Rating Scale; SANS = Scale for the Assessment of Negative Symptoms; SAPS = Scale for the Assessment of Positive Symptoms.

aPremorbid IQ (intelligence quotient) scores estimated from the Wechsler Adult Intelligence Scale-III Vocabulary subtest.

For comparison purposes, 28 healthy volunteers (13 male, 15 female) were recruited from the local area through advertisements and underwent the same cognitive battery as patients. Healthy control participants met the following exclusion criteria: Past or current mental disorders screened by the Comprehensive Assessment of Symptoms and History (Andreasen, Flaum, & Arndt, 1992); regular medical prescriptions; neurological or general medical illnesses; or presence of psychosis in first-degree relatives. Patients and controls were similar in age and gender distribution (Table 1). They did, though, differ in years of education and premorbid intelligence quotient (IQ). Given the difference between controls and patients in premorbid IQ, the possibility of having a control sample not representative of the general population was indeed a concern. However, the estimation of current IQ (with Verbal Comprehension Index, a composite measure of verbal intellectual functioning highly correlated with full IQ scale) was similar to the average expected in a general population (mean, 101.2; SD, 14.5). When available, data from controls were compared with Spanish standardized norms for age in individual tests (Peña-Casanova, Gramunt, & Gich, 2004; Wechsler, 1999); the controls' performance was always within the range of average scores of population norms. Therefore, controls recruited in this study showed a cognitive performance that can be considered representative of the age-matched general population.

Procedures

On the basis of a previous factor analytic study, 30 cognitive subtest scores were grouped into eight cognitive dimensions (González-Blanch et al., 2007). To maximize collaboration and avoid acute psychosis state effects of acute psychosis, the cognitive battery was applied following the clinical stabilization of acute psychotic symptoms, with a mean of 10.58 (SD = 3.98) weeks after treatment initiation. For the purpose of test–retest reliability analysis, we used the data of a second application of the same tests 3 months later. Data were standardized to z-scores (with an average of 0 and SD = 1) using baseline healthy control group data. The mean of subtests z-scores was used to compute factor scores. For the purpose of the present study, we used the five variables out of four neuropsychological tests included in the factor called “Speed of processing and executive functioning” and a composite score of the whole battery.

Neuropsychological Measures

Digit Symbol coding test (Wechsler, 1999). This task consists of rows containing small blank squares, each paired with a randomly assigned number from one to nine. Above these rows is a printed key that pairs each number with a different symbol. Using the reference key, the examinee has 120 s to pair specific numbers with given geometric figures. Scaled scores were obtained.

Trail making test (TMT; Spreen & Strauss, 1998). The TMT consists of two parts (A and B). Part A requires the participant to connect series of numbered circles arrayed randomly on a sheet of paper using a pencil. In part B, the array consists of both numbers and letters, and the participant must connect them in alternating order. Part B demands simultaneous processing capacity for two sets of mental operations (number and letter sequencing) as well as a rule-following instruction to alternate between the sets. For parts A and B, scoring is expressed in terms of the time to completion.

Cancellation test (Lezak, Howieson, & Loring, 2004). The participant is asked to scan a large field of either letters or symbols for a target letter (A). The participant is asked to cancel all the targets as quickly as possible for a minute. The score is the number of correct cancellations.

Digit Span-backward subtest (Wechsler, 1999). This test is considered to be related to working memory. In this task, an increasing number of orally presented digits, starting with two, at a rate of one per second have to be repeated in an exactly reversed order. Every other trial the number of digits is increased by one. The test is ended when errors in two consecutive trials are made. Raw score is used as the key-dependent measure.

Global deficit score (GDS). In order to summarize the neuropsychological data of the whole battery into a single score, we used the GDS method. The GDS is computed by converting standard scores (T-scores; with a mean of 50 and an SD of 10) on individual neuropsychological test variables into deficit scores ranging from 0 (no impairment) to 5 (severe impairment). A sum of all deficit scores divided by the total number of tests administered is calculated for each subject in order to obtain the final GDS (for further details on the calculation of this variable, see Carey et al., 2004). A higher value reflects a greater degree of overall impairment. In previous studies, a GDS of ≥0.50 has accurately predicted expert clinical ratings of overall impairment (Carey et al., 2004). The GDS method appears to be relatively resistant to modifications in test batteries or missing data. This method has shown convergent validity with other cognitive composite scores used in schizophrenia studies, such as the Clinically Significant Cognitive Impairment method (Reichenberg et al., 2009). The GDS was based on the eight cognitive dimensions (González-Blanch et al., 2007).

  1. Verbal learning and memory: Rey Auditory Verbal Learning Test (Spreen & Strauss, 1998): immediate memory span, new learning, recall following short and long delay periods, and recognition memory.

  2. Verbal comprehension abilities: WAIS-III subtests (Wechsler, 1999): Vocabulary, Similarities, Information, and Comprehension. Additionally, the Vocabulary subtest was used to estimate premorbid IQ.

  3. Speed of processing and executive functioning: TMT parts A and B (Spreen & Strauss, 1998); Cancellation test (Lezak et al., 2004); Digit Symbol coding; and Digit Span-backward (Wechsler, 1999).

  4. Visual memory: Rey Complex Figure Test immediate and Delayed Recall (Rey, 1987).

  5. Motor dexterity: Grooved Pegboard test (Spreen & Strauss, 1998).

  6. Motor speed: Finger Tapping Test, both hands (Spreen and Strauss, 1998).

  7. Sustained attention/vigilance: Continuous Performance Test Degraded-Stimulus (CPT-DS) hits and reaction time (Cegalis & Bowlin, 1991); and Brief Test of Attention (Schretlen, Bobholz, & Brandt, 1996).

  8. Impulsivity: CPT-DS false alarms score (Cegalis & Bowlin, 1991).

Functional Measures

To rate functional status, we used the global disability item from the Spanish version of Disability Assessment Schedule (DAS; Mañá, Ivorra, & Girón, 1998). The DAS evaluates the ability of the subjects to carry out particular social roles normally expected of them in their environment. Ratings were based on the clinician's judgement of the information obtained from the patient, relatives, case notes, and observation of the patient during the month prior to 1-year follow-up. The global disability item has a score range of 0 (no disability) to 5 (gross disability). The patient and their relatives were assessed by the treating psychiatrist and the social worker independently, with a consensus being reached between both.

Clinical Measures

For characterization purposes, Table 1 shows the values for well-known clinical scales.

The presence of positive and negative psychotic symptoms was assessed by the Scale for the Assessment of Positive Symptoms (SAPS; Andreasen, 1983) and the Scale for the Assessment of Negative Symptoms (SANS; Andreasen, 1981). The Brief Psychiatric Rating Scale was used to assess the severity of general psychopathological symptoms (Overall & Gorham, 1962).

Duration of untreated psychosis (DUP) was defined as the time from the first continuous psychotic symptom corresponding to a score of 4 or more on one of the SAPS items to initiation of an adequate antipsychotic drug treatment. DUP was determined after interviewing the patient and a close relative. Duration of untreated illness was defined as the time from the first nonspecific symptom related to psychosis (for such a symptom to be considered, there should be no return to a previous stable level of functioning) to the initiation of adequate antipsychotic drug treatment.

Data Analyses

We examined the relationship between test variables and full cognitive assessment (i.e., GDS) by the Pearson correlation coefficients. To avoid spuriously inflated correlations, we computed the corrected item-total correlation (CITC) for the five test scores relative to the composite score (the GDS) using the reliability procedure. The CITC is the correlation between a test variable and the rest of the battery, without that item considered part of the test battery, thereby controlling for part-whole correlation.

For the five test variables, test–retest reliability was assessed by using intraclass correlation coefficients (ICC) and the Pearson correlation coefficients in the patient and control groups separately. Test–retest reliability data were available for 111 patients and 20 controls that were tested 3 months apart. Practice effects were measured by comparing data collected at test session 1 to those collected at test session 2 with within-group t-tests. These were determined in the patient and control groups separately.

Receiver operating characteristic (ROC) curves were calculated for the five tests to evaluate their screening performance. The ROC curve is a graphical representation of the sensitivity and specificity values of all possible cut-off values of a continuous diagnostic variable. The presence of a cognitive impairment assigned according to the 0.5 cut-off score in the GDS served as an external criterion. The area under the curve (AUC) directly represents the accuracy of the instrument in screening for cognitive impairment. Differences between the AUC values were tested using the Hanley and McNeil method (Hanley & McNeil, 1983). Determination of optimal cut-off scores was done by maximizing the Youden index (Y = sensitivity + specificity − 100; Youden, 1950).

We calculated sensitivity, specificity, positive predictive values (PPV), and negative predictive values (NPV) for the optimal cut-off score for each test. We then used Mallow's Cp (a measure of goodness of fit) to rank these single variable models and to determine the best subset of test variables.

Finally, we examined Spearman's correlations between global functional outcome and each test variable. All statistical analyses were two-tailed and performed with Statistical Package for Social Sciences (SPSS) version 17.0 and MedCalc version 11.1 for Windows. The latter was mainly used for calculating the differences in AUC values.

Results

Correlation of Individual Test Variables to Global Score

We examined the relationship between the individual tests and GDS for full cognitive assessment. The Pearson correlation coefficients range from 0.39 to 0.71 indicating medium to large effect sizes. The CITC provided large effect sizes for Digit Symbol, TMT-A, and TMT-B (Table 2).

Table 2.

Correlation of individual test variables to global cognitive score

 Pearson r CITC 
Digit Symbol −.657* 0.561 
TMT-A −.591* 0.492 
TMT-B −.714* 0.542 
Digit Span-backward −.394* 0.327 
Cancellation −.548* 0.432 
 Pearson r CITC 
Digit Symbol −.657* 0.561 
TMT-A −.591* 0.492 
TMT-B −.714* 0.542 
Digit Span-backward −.394* 0.327 
Cancellation −.548* 0.432 

Notes: CITC = corrected item-total correlation; TMT = Trail Making Test.

*p< .001.

Test–Retest Reliability and Practice Effects of Individual Tests

Test–retest reliability was assessed using a single-measure, random model ICC from baseline to the 3-month follow-up assessment for individual tests. Practice effects were measured by comparing data collected at baseline to those collected 3 months apart with within-group t-tests. Table 3 shows data for 111 patients and 20 controls which were assessed at both time-points. Digit Symbol provides acceptably high values for both samples (>0.70). However, all but the Cancellation Test showed marked practice effects in the patients’ sample. As can be seen in Table 3, z-scores are consistently higher in a second administration. Controls performed significantly better in the second administration only on TMT-A, although this may be due to the lack of power to detect differences.

Table 3.

Mean performance and reliability of test–retest z-scores of cognitive tests in patients and controlsa

Tests FEP patients (n = 111)
 
Controls (n = 20)
 
 Time 1
 
Time 2
 
p-value r-value ICC Time 1
 
Time 2
 
p-value r-value ICC 
 Mean SD Mean SD    Mean SD Mean SD    
Digit Symbol −1.38 0.92 −1.04 1.00 <.001 .849 0.846 0.17 0.91 0.49 0.70 .030 .764 0.739 
TMT-A −0.83 1.68 −0.42 1.53 .006 .560 0.558 0.11 0.88 0.82 0.61 <.001 .606 0.566 
TMT-B −1.97 2.50 −1.26 2.52 <.001 .728 0.728 0.06 1.10 0.33 0.85 .144 .689 0.667 
Digit Span-backward −0.83 0.75 −0.61 0.74 .002 .543 0.543 0.15 1.11 0.04 0.87 .530 .721 0.699 
Cancellation −1.00 1.94 −0.82 1.64 .256 .608 0.600 0.01 1.17 0.18 0.55 .510 .577 0.442 
Tests FEP patients (n = 111)
 
Controls (n = 20)
 
 Time 1
 
Time 2
 
p-value r-value ICC Time 1
 
Time 2
 
p-value r-value ICC 
 Mean SD Mean SD    Mean SD Mean SD    
Digit Symbol −1.38 0.92 −1.04 1.00 <.001 .849 0.846 0.17 0.91 0.49 0.70 .030 .764 0.739 
TMT-A −0.83 1.68 −0.42 1.53 .006 .560 0.558 0.11 0.88 0.82 0.61 <.001 .606 0.566 
TMT-B −1.97 2.50 −1.26 2.52 <.001 .728 0.728 0.06 1.10 0.33 0.85 .144 .689 0.667 
Digit Span-backward −0.83 0.75 −0.61 0.74 .002 .543 0.543 0.15 1.11 0.04 0.87 .530 .721 0.699 
Cancellation −1.00 1.94 −0.82 1.64 .256 .608 0.600 0.01 1.17 0.18 0.55 .510 .577 0.442 

Notes: TMT = trail making test; ICC = intraclass correlation coefficient;

aData were standardized to z-scores (with an average of 0 and SD = 1) using baseline healthy control group data.

ROC Curve Analysis of Individual Cognitive Tests

Figure 1 shows the ROC curve for discriminating between patients with and without marked cognitive deficits (i.e., GDS > 0.5) as a function of their individual test scores.

Fig. 1.

ROC curves showing the sensitivity and 1-specificity for five neuropsychological tests. The diagonal reference line (in red) represents a prediction value of 0.5, equivalent to chance.

Fig. 1.

ROC curves showing the sensitivity and 1-specificity for five neuropsychological tests. The diagonal reference line (in red) represents a prediction value of 0.5, equivalent to chance.

The AUC serves as an overall measure of discrimination; perfect discrimination would yield an area of 1, whereas a test that failed to discriminate would have an area of 0.50. According to Hosmer and Lemeshow (2000), an AUC between 0.70 and 0.80 indicates acceptable discrimination, and at least 0.80 indicates excellent discrimination. All but one test, the Digit Span-backward, yielded an AUC over 0.70 (Table 4). Pairwise comparison of ROC curves revealed no statistical differences between AUCs, except for Digit Span-backward with Digit Symbol (▵AUC = 0.139; z = 2.030; p = .042), TMT-A (▵AUC = 0.158; z = 2.181; p = .029), and TMT-B (▵AUC = 0.206; z = 3.188; p = .001). The more balanced cut-off scores of the tests, based on the Youden index, provided reasonable specificity values (>0.65), but TMT-A and Digit Span-backward had poor sensitivity values. All cognitive tests had high PPV (>85%), which is the probability that a person whose screening test result is positive really does have has the condition of interest (i.e., marked cognitive impairment). This result, related to the specificity of the test, is particularly high for TMT-A and TMT-B. However, NPV, which is the probability that a person whose test result is negative does not have marked cognitive impairment, reaches, in the best of the cases, 57% (Table 4).

Table 4.

Sensitivity, Specificity, PPV, NPV, and AUC for cognitive tests optimal cutoff z-scores

Tests Optimal cut-off z-score Se Sp PPV NPP AUC (95% CI) 
Digit Symbol Less than −1.12 0.78 0.67 0.87 0.51 0.798 (0.713–0.882)** 
TMT-A Less than −0.74 0.60 0.94 0.97 0.46 0.817 (0.743–0.890)** 
TMT-B Less than −0.80 0.77 0.85 0.94 0.57 0.865 (0.800–0.929)** 
Digit Span-backward Less than −0.84 0.56 0.74 0.86 0.37 0.659 (0.549–0.768)* 
Cancellation Less than −0.18 0.71 0.71 0.87 0.46 0.774 (0.687–0.862)** 
Tests Optimal cut-off z-score Se Sp PPV NPP AUC (95% CI) 
Digit Symbol Less than −1.12 0.78 0.67 0.87 0.51 0.798 (0.713–0.882)** 
TMT-A Less than −0.74 0.60 0.94 0.97 0.46 0.817 (0.743–0.890)** 
TMT-B Less than −0.80 0.77 0.85 0.94 0.57 0.865 (0.800–0.929)** 
Digit Span-backward Less than −0.84 0.56 0.74 0.86 0.37 0.659 (0.549–0.768)* 
Cancellation Less than −0.18 0.71 0.71 0.87 0.46 0.774 (0.687–0.862)** 

Notes: TMT = Trail Making Test; ROC = receiver operating characteristic; Se = sensitivity; Sp = specificity; PPV = positive predictive value; NPV = negative predictive value; AUC = area under the curve; CI = confidence interval.

*p < .01.

**p < .001.

Best Subset Selection

We used Mallow's Cp to rank single variable models and to select the best subset of all possible combinations of test variables. Cp is expected to be approximately equal to p + 1, where p is the number of predictors in the fitted model, with smaller values preferred. If the model fitted is not adequate, the larger values of Cp are expected (King, 2003). On the basis of Mallow's Cp statistic, Digit Symbol yields the best-fit model for a subset of 1 variable (Cp = 6.53; AUC = 0.80; Nagelkerke R2 = .31), followed by TMT-B (Cp = 12.22; AUC = 0.87; Nagelkerke R2 = .48), TMT-A (Cp = 13.65; AUC = 0.82; Nagelkerke R2 = .36), Cancellation (Cp = 19.29; AUC = 0.78; Nagelkerke R2 = .29), and Digit Span-backward (Cp = 26.93; AUC = 0.66; Nagelkerke R2 = .08). Furthermore, Digit Symbol was also part of the best subset of two variables with TMT-A (Cp = 1.41; AUC = 0.87; Nagelkerke R2 = .47) and was also part of the best combination of three variables with TMT-A and TMT-B (Cp = 2.01; AUC = 0.90; Nagelkerke R2= .54). The best of all possible subsets was the combination of Digit Symbol and TMT-A for the patients’ sample; and Digit Symbol, TMT-A, and TMT-B for the whole sample (i.e., patients and controls; Cp = 1.60; AUC = 0.91; Nagelkerke R2 = .61).

Correlations Between Individual Test and a Global Measure of Social Disability

The Spearman correlations between cognitive tests and DAS global score indicated a small negative relationship with Digit Symbol (r = −.25; p = .007), TMT-A (r = −.23; p = .010), and TMT-B (r = −.28; p = .002), but a lack of significant relationship with Cancellation and Digit Span-backward tests.

Discussion

All the tests included in the present study are pen-and-paper tasks with desirable characteristics for screening tools, such as being easy to administer and score so that they will be more widely accepted among busy clinicians, and are also well tolerated by patients. Furthermore, these neuropsychological tests are already used worldwide in research and clinical settings. The data in this study indicate that single neuropsychological tests such as Digit Symbol, TMT (parts A and B), and Cancellation are potentially useful screening instruments in a sample of first-episode psychosis patients. The results for Digit Span-backward were rather disappointing; suggesting that this particular test should only be used in combination with other measures in order to obtain a valid estimation of general cognitive ability. On the other hand, based on ROC analyses, there were no significant differences in the accuracy for detecting marked cognitive impairment between the tests used, except Digit Span-backward. Depending on the criteria used the order of the rank will vary, but in all cases Digit Symbol, TMT-A, and TMT-B have the best indices of correlation with GDS, variance explained, and AUC. According to Mallow's Cp, Digit Symbol was the best single test for screening cognitive impairment, and the addition of TMT-A further increased the accuracy. The combination of Digit Symbol, TMT-A, and TMT-B provided excellent indices of AUC and Mallow's Cp and accounted for a substantial amount of the variance of the GDS. A caveat regarding the results of the present study is that, in the absence of a real cognitive improvement (seeGonzález-Blanch et al., 2006; Rodríguez-Sánchez et al., 2008), patients’ scores on the three tests were significantly higher in the second administration, 3 months apart. However, the consistency between the two assessments, as measured by ICC, was particularly good for Digit Symbol. These three subtests (Digit Symbol, TMT-A, and TMT-B) were the only ones that showed significant associations with a global measure of social and occupational functioning, but the magnitude of these correlations represented small to medium effect sizes (rho ≈ −0.25).

Further limitations of the present study have to be noted. First, although all these tests have alternate forms available, we did not use alternate forms to control for the assessment of potential practice effects and test–retest reliability. Second, it is likely that other speeded tasks that require information processing, visual scanning, flexibility, and motor abilities (or even related to other cognitive domains) may provide similar results. It is worth noting that WAIS Digit Span-backward involves cognitive functions of a different type (such as storage and reversing operations as well as auditory attention) and its correlation with GDS and accuracy in detecting cognitive impairment as a single test was notably poorer than the other tests used. Third, we used a global measure of functional outcome instead of specific measures of social or occupational performance. Although we have questioned the associations between neurocognition and functional status elsewhere (Gonzalez-Blanch et al., 2010), it is possible, however, that more specific measures may provide stronger correlations than those observed in the present study. Fourth, the clinical sample used in this study was mainly composed of subjects with a diagnosis of schizophrenia and schizophreniform disorder (85% of the whole sample). Thus, it is possible that our results cannot be generalized to those first-episode patients with other diagnoses. Finally, since cognitive assessment was conducted after clinical stabilization, the influence of antipsychotics on cognitive functions cannot be analyzed. However, previous analyses suggest the stability of cognitive deficits during the first weeks of treatment (González-Blanch et al., 2006, 2008) and in 1-year follow-up (Rodríguez-Sánchez et al., 2008). In a drug-naïve first-episode schizophrenia study, Hong and colleagues (2002) suggested that most of the cognitive functions remain stable during the early phase of treatment (i.e., the first 8 weeks).

From the theoretical point of view, there are two complementary interpretations for the fact that several single test scores account for much of the variance of global cognitive functioning and can correctly classify most of the patients. On one hand, it can be assumed that basic information processing represents an essential feature of cognitive dysfunction among individuals with schizophrenia. This is in agreement with previous research (Badcock, Dragovic, Waters, & Jablensky, 2005; Mohamed, Paulsen, O'Leary, Arndt, & Andreasen, 1999; Niendam et al., 2003; Rodríguez-Sánchez et al., 2007). At the same time, it can be interpreted to mean that the generalized cognitive deficit is a fundamental manifestation of schizophrenia. These multifaceted tasks may well be capturing this essential aspect of the illness. Some factorial studies in schizophrenia report moderate correlations between cognitive domains (González-Blanch et al., 2007) and between cognitive domains and a global composite score (Keefe et al., 2006). Likewise, Dickinson, Iannone, Wilk, and Gold (2004) showed that about two-thirds of the diagnosis-related variance in cognitive performance was mediated through a general factor, with more specific domains (such as verbal memory and processing speed) having small direct effects. In this line, Dickinson (2008) suggests that generalized cognitive impairment should not be dismissed as an artifact that obscures more cognitive domain-specific effects. Coding tasks will be particularly sensitive to the biological underpinnings of the illness, which may be general and systemic in nature, rather than focal abnormalities in specific brain structures (Dickinson, 2008). This point of view is largely consistent with pathophysiological models of schizophrenia such as the “cognitive dysmetria” model. This model postulates that diverse symptoms reflect abnormalities in connectivity in the cortical–cerebellar–thalamic–cortical circuit (Andreasen, Paradiso, & O'Leary, 1998). An abnormality in this circuitry may lead to difficulties in coordinating both motor and mental activities. Thus, processing speed deficits may be reflecting a generalized dysfunction that causes the widespread cognitive failures in attention, memory, motor dexterity, and executive functions.

From the clinical perspective, the results from this study suggest that it is possible to detect cognitive deficits by means of a brief test that accounts for a considerable amount of the variance associated with a more comprehensive assessment of cognitive functioning in first-episode cases. TMT and Digit Symbol have also been included in other cognitive screening batteries which are created by selecting a small number of standardized neuropsychological tests. For example, the BCA (Velligan et al., 2004) includes as part of the battery the Trails A and B, and the B-CATS (Hurford et al., in press) includes Trails B and Digit Symbol substitution. Although using a set of two or three tests can improved the psychometric properties of the screening instrument, this study has shown that the Digit Symbol test alone had correlations with the global score and had test–retest reliability similar to that of BCA and B-CATS. These very short administration times may be particularly valuable for busy, time-pressured clinicians interested in identifying those cases with marked cognitive impairment which would benefit from a more comprehensive neuropsychological assessment or in need of special rehabilitation programs.

On the other hand, it could be assumed that if the processing speed deficit could be improved, then the other deficits would be less severe (Duff & Grabowski, 2008). Importantly, growing evidence suggests that such deficits in speed of processing may be open to amelioration through cognitive remediation therapies (Hogarty, Greenwald, & Eack, 2006; Wykes, Reeder, Corner, Williams, & Everitt, 1999). As global measures of cognition may be more relevant for the determination of functional outcomes than specific neurocognitive constructs (Green, Kern, Braff, & Mintz, 2000), the improvement in individual tests that reflect global cognitive deficits may be particularly important for functional recovery in people with schizophrenia and related disorders.

In conclusion, identification of early cognitive impairment in schizophrenia spectrum disorders is important but often missed in clinical settings. Full neuropsychological assessments are time-costly and not feasible in routine practice, whereas brief screening tools such as the above cited are still not widely applicable to date. The use of a single standardized test that taps information processing, visual scanning, flexibility, and motor abilities may represent a suitable and cost-effective tool for the screening of cognitive impairment in clinical settings.

Funding

This work was supported by Instituto de Salud Carlos III (FIS 00/3095 and G03/032) and SENY Fundació Research Grant (2005-0308007), Fundación Marqués de Valdecilla (A/02/07). These institutions had no further role in study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the paper for publication.

Conflict of interest

J.L.V.-B. and B.C.-F. have received unrestricted research funding from AstraZeneca, Pfizer, Bristol-Myers Squibb, and Johnson & Johnson, which has been deposited into research accounts at the University of Cantabria. J.L.V.-B. has received honoraria for his participation as a speaker at educational events from Johnson & Johnson. B.C.-F. has received honoraria for his participation as a speaker at educational events from Pfizer, Bristol-Myers Squibb, and Johnson & Johnson and consultant fees from Pfizer. R.P.-I. has received honoraria for his participation as a speaker at educational events from Bristol-Myers Squibb-Otsuka.

The remaining authors report no competing interests.

Acknowledgements

We gratefully acknowledge the participation of patients and controls in the present study. The authors wish to thank Dr Mario Álvarez-Jiménez of ORYGEN Research Centre, University of Melbourne, for his methodological advice, and the contributions made by the staff of the Cantabria Intervention Programme of First-Episode Psychosis (PAFIP).

References

Andreasen
N.C.
Scale for the Assessment of Negative Symptoms (SANS)
1981
Iowa City, IA
University of Iowa
Andreasen
N.C.
Scale for the Assessment of Positive Symptoms (SAPS)
1983
Iowa City, IA
University of Iowa
Andreasen
N. C.
Flaum
M.
Arndt
S.
The Comprehensive Assessment of Symptoms and History (CASH). An instrument for assessing diagnosis and psychopathology
Archives of General Psychiatry
 , 
1992
, vol. 
49
 
8
(pg. 
615
-
623
)
Andreasen
N. C.
Paradiso
S.
O'Leary
D. S.
Cognitive dysmetria” as an integrative theory of schizophrenia: A dysfunction in cortical–subcortical–cerebellar circuitry?
Schizophrenia Bulletin
 , 
1998
, vol. 
24
 
2
(pg. 
203
-
218
)
Badcock
J. C.
Dragovic
M.
Waters
F. A.
Jablensky
A.
Dimensions of intelligence in schizophrenia: Evidence from patients with preserved, deteriorated and compromised intellect
Journal of Psychiatric Research
 , 
2005
, vol. 
39
 
1
(pg. 
11
-
19
)
Bromley
E.
Clinicians' concepts of the cognitive deficits of schizophrenia
Schizophrenia Bulletin
 , 
2007
, vol. 
33
 
3
(pg. 
648
-
651
)
Buchanan
R. W.
Davis
M.
Goff
D.
Green
M. F.
Keefe
R. S.
Leon
A. C.
, et al.  . 
A summary of the FDA-NIMH-MATRICS workshop on clinical trial design for neurocognitive drugs for schizophrenia
Schizophrenia Bulletin
 , 
2005
, vol. 
31
 
1
(pg. 
5
-
19
)
Carey
C. L.
Woods
S. P.
Gonzalez
R.
Conover
E.
Marcotte
T. D.
Grant
I.
, et al.  . 
Predictive validity of global deficit scores in detecting neuropsychological impairment in HIV infection
Journal of Clinical and Experimental Neuropsychology
 , 
2004
, vol. 
26
 
3
(pg. 
307
-
319
)
Cegalis
J.
Bowlin
J.
Vigil: Software for the Assessment of Attention
 , 
1991
Nashua, NH
Forthought
Crespo-Facorro
B.
Perez-Iglesias
R.
Ramirez-Bonilla
M.
Martinez-Garcia
O.
Llorca
J.
Luis Vazquez-Barquero
J.
A practical clinical trial comparing haloperidol, risperidone, and olanzapine for the acute treatment of first-episode nonaffective psychosis
Journal of Clinical Psychiatry
 , 
2006
, vol. 
67
 
10
(pg. 
1511
-
1521
)
Dickinson
D.
Digit symbol coding and general cognitive ability in schizophrenia: Worth another look
British Journal of Psychiatry
 , 
2008
, vol. 
193
 
5
(pg. 
354
-
356
)
Dickinson
D.
Iannone
V. N.
Wilk
C. M.
Gold
J. M.
General and specific cognitive deficits in schizophrenia
Biological Psychiatry
 , 
2004
, vol. 
55
 
8
(pg. 
826
-
833
)
Dickinson
D.
Ramsey
M. E.
Gold
J. M.
Overlooking the obvious: A meta-analytic comparison of digit symbol coding tasks and other cognitive measures in schizophrenia
Archives of General Psychiatry
 , 
2007
, vol. 
64
 
5
(pg. 
532
-
542
)
Duff
K.
Grabowski
T. J.
McNeill-Horton
A.
Wedding
D.
Normal Aging, Mild Cognitive Impairment, and Dementia
The neuropsychology handbook
 , 
2008
New York
Springer Publishing
(pg. 
571
-
602
)
First
M.B.
Spitzer
R.L.
Gibbon
M.
Williams
J.B.W.
Structured Clinical Interview for DSM-IV Axis I Disorders
1995
Biometrics Research Department. New York: State Psychiatric Institute
Gold
J. M.
Cognitive deficits as treatment targets in schizophrenia
Schizophrenia Research
 , 
2004
, vol. 
72
 
1
(pg. 
21
-
28
)
González-Blanch
C.
Álvarez-Jiménez
M.
Rodríguez-Sánchez
J. M.
Pérez-Iglesias
R.
Vázquez-Barquero
J. L.
Crespo-Facorro
B.
Cognitive functioning in the early course of first-episode schizophrenia spectrum disorders: Timing and patterns
European Archives of Psychiatry and Clinical Neuroscience
 , 
2006
, vol. 
256
 
6
(pg. 
364
-
371
)
González-Blanch
C.
Crespo-Facorro
B.
Álvarez-Jiménez
M.
Rodríguez-Sánchez
J. M.
Pérez-Iglesias
R.
Pelayo-Terán
J. M.
, et al.  . 
Lack of association between clinical and cognitive change in first-episode psychosis: The first 6 weeks of treatment
Canadian Journal of Psychiatry
 , 
2008
, vol. 
53
 
12
(pg. 
839
-
847
)
González-Blanch
C.
Crespo-Facorro
B.
Álvarez-Jiménez
M.
Rodríguez-Sánchez
J. M.
Pelayo-Terán
J. M.
Pérez-Iglesias
R.
, et al.  . 
Cognitive dimensions in first-episode schizophrenia spectrum disorders
Journal of Psychiatric Research
 , 
2007
, vol. 
41
 
11
(pg. 
968
-
977
)
Gonzalez-Blanch
C.
Perez-Iglesias
R.
Pardo-Garcia
G.
Rodriguez-Sanchez
J. M.
Martinez-Garcia
O.
Vazquez-Barquero
J. L.
, et al.  . 
Prognostic value of cognitive functioning for global functional recovery in first-episode schizophrenia
Psychological Medicine
 , 
2010
, vol. 
40
 
6
(pg. 
935
-
944
)
Green
M. F.
Kern
R. S.
Braff
D. L.
Mintz
J.
Neurocognitive deficits and functional outcome in schizophrenia: Are we measuring the “right stuff”?
Schizophrenia Bulletin
 , 
2000
, vol. 
26
 
1
(pg. 
119
-
136
)
Groth-Marnat
G.
Handbook of psychological assessment
 , 
2003
4th ed.
New York
John Wiley & Sons
Hanley
J. A.
McNeil
B. J.
A method of comparing the areas under receiver operating characteristic curves derived from the same cases
Radiology
 , 
1983
, vol. 
148
 
3
(pg. 
839
-
843
)
Harvey
P. D.
Serper
M. R.
White
L.
Parrella
M. J.
McGurk
S. R.
Moriarty
P. J.
, et al.  . 
The convergence of neuropsychological testing and clinical ratings of cognitive impairment in patients with schizophrenia
Comprehensive Psychiatry
 , 
2001
, vol. 
42
 
4
(pg. 
306
-
313
)
Heinrichs
R. W.
Zakzanis
K. K.
Neurocognitive deficit in schizophrenia: A quantitative review of the evidence
Neuropsychology
 , 
1998
, vol. 
12
 
3
(pg. 
426
-
445
)
Hofer
A.
Niedermayer
B.
Kemmler
G.
Rettenbacher
M. A.
Trebo
E.
Widschwendter
C. G.
, et al.  . 
Cognitive impairment in schizophrenia: Clinical ratings are not a suitable alternative to neuropsychological testing
Schizophrenia Research
 , 
2007
, vol. 
92
 
(1–3)
(pg. 
126
-
131
)
Hogarty
G. E.
Greenwald
D. P.
Eack
S. M.
Durability and mechanism of effects of cognitive enhancement therapy
Psychiatric Services
 , 
2006
, vol. 
57
 
12
(pg. 
1751
-
1757
)
Hong
K. S.
Kim
J. G.
Koh
H. J.
Koo
M. S.
Kim
J. H.
Lee
D.
, et al.  . 
Effects of risperidone on information processing and attention in first-episode schizophrenia
Schizophrenia Research
 , 
2002
, vol. 
53
 
(1–2)
(pg. 
7
-
16
)
Hosmer
C.
Lemeshow
S.
Applied logistic regression
 , 
2000
2nd ed.
New York
Wiley
Hurford
I. M.
Marder
S. R.
Keefe
R. S.
Reise
S. P.
Bilder
R. M.
A Brief Cognitive Assessment Tool for Schizophrenia: Construction of a Tool for Clinicians
Schizophrenia Bulletin
 
Hyman
S. E.
Fenton
W. S.
Medicine. What are the right targets for psychopharmacology?
Science
 , 
2003
, vol. 
299
 
5605
(pg. 
350
-
351
)
Keefe
R. S.
Bilder
R. M.
Harvey
P. D.
Davis
S. M.
Palmer
B. W.
Gold
J. M.
, et al.  . 
Baseline neurocognitive deficits in the CATIE schizophrenia trial
Neuropsychopharmacology
 , 
2006
, vol. 
31
 
9
(pg. 
2033
-
2046
)
Keefe
R. S.
Goldberg
T. E.
Harvey
P. D.
Gold
J. M.
Poe
M. P.
Coughenour
L.
The Brief Assessment of Cognition in Schizophrenia: Reliability, sensitivity, and comparison with a standard neurocognitive battery
Schizophrenia Research
 , 
2004
, vol. 
68
 
(2–3)
(pg. 
283
-
297
)
Keefe
R. S.
Poe
M.
Walker
T. M.
Kang
J. W.
Harvey
P. D.
The Schizophrenia Cognition Rating Scale: An interview-based assessment and its relationship to cognition, real-world functioning, and functional capacity
American Journal of Psychiatry
 , 
2006
, vol. 
163
 
3
(pg. 
426
-
432
)
King
J. E.
Running a best-subsets logistic regression: An alternative to stepwise methods
Educational and Psychological Measurement
 , 
2003
, vol. 
63
 
3
(pg. 
392
-
403
)
Lezak
M. D.
Howieson
D. B.
Loring
W.
Neuropsychological assessment
 , 
2004
4th ed.
New York
Oxford University Press
Mañá
S.
Ivorra
J.
Girón
M.
Adaptación y fiabilidad de la entrevista para la evaluación de la discapacidad social en pacientes psiquiátricos (OMS)
Revista de Psiquiatría de la Facultad de Medicina de Barcelona
 , 
1998
, vol. 
25
 
2
(pg. 
43
-
48
)
Mesholam-Gately
R. I.
Giuliano
A. J.
Goff
K. P.
Faraone
S. V.
Seidman
L. J.
Neurocognition in first-episode schizophrenia: A meta-analytic review
Neuropsychology
 , 
2009
, vol. 
23
 
3
(pg. 
315
-
336
)
Mohamed
S.
Paulsen
J. S.
O'Leary
D.
Arndt
S.
Andreasen
N.
Generalized cognitive deficits in schizophrenia: A study of first-episode patients
Archives of General Psychiatry
 , 
1999
, vol. 
56
 
8
(pg. 
749
-
754
)
Niendam
T. A.
Bearden
C. E.
Rosso
I. M.
Sanchez
L. E.
Hadley
T.
Nuechterlein
K. H.
, et al.  . 
A prospective study of childhood neurocognitive functioning in schizophrenic patients and their siblings
American Journal of Psychiatry
 , 
2003
, vol. 
160
 
11
(pg. 
2060
-
2062
)
Overall
J. E.
Gorham
D. R.
The Brief Psychiatric Rating Scale
1962
, vol. 
10
 (pg. 
799
-
812
Psychological Report
Peña-Casanova
J.
Gramunt
N.
Gich
J.
Test neuropsicológicos. Fundamentos para una neuropsicología clínica basada en evidencias
 , 
2004
Barcelona
Masson
Purdon
S. E.
The Screen for Cognitive Impairment in Psychiatry (SCIP): Instructions and three alternate forms
 , 
2005
Edmonton
PNL
Reichenberg
A.
Harvey
P. D.
Bowie
C. R.
Mojtabai
R.
Rabinowitz
J.
Heaton
R. K.
, et al.  . 
Neuropsychological function and dysfunction in schizophrenia and psychotic affective disorders
Schizophrenia Bulletin
 , 
2009
, vol. 
35
 
5
(pg. 
1022
-
1029
)
Rey
A.
Test de Copia de la Figura Compleja [Rey Complex Figure Test]
 , 
1987
Madrid
TEA Ediciones
Rodríguez-Sánchez
J. M.
Crespo-Facorro
B.
González-Blanch
C.
Pérez-Iglesias
R.
Vázquez-Barquero
J. L.
Cognitive dysfunction in first-episode psychosis: The processing speed hypothesis
The British Journal of Psychiatry: Supplement
 , 
2007
, vol. 
51
 (pg. 
s107
-
s110
)
Rodríguez-Sánchez
J. M.
Pérez-Iglesias
R.
González-Blanch
C.
Pelayo-Terán
J. M.
Mata
I.
Martínez
O.
, et al.  . 
1-year follow-up study of cognitive function in first-episode non-affective psychosis
Schizophrenia Research
 , 
2008
, vol. 
104
 
(1–3)
(pg. 
165
-
174
)
Schretlen
D.
Bobholz
J.
Brandt
J.
Development and psychometric properties of the Brief Test of Attention
The Clinical Neuropsychologist
 , 
1996
, vol. 
10
 (pg. 
80
-
89
)
Spreen
O.
Strauss
E.
A Compendium of neuropsychological test: Administration, norms, and commentary
 , 
1998
2nd ed.
New York
Oxford University Press
Velligan
D. I.
DiCocco
M.
Bow-Thomas
C. C.
Cadle
C.
Glahn
D. C.
Miller
A. L.
, et al.  . 
A brief cognitive assessment for use with schizophrenia patients in community clinics
Schizophrenia Research
 , 
2004
, vol. 
71
 
(2–3)
(pg. 
273
-
283
)
Ventura
J.
Cienfuegos
A.
Boxer
O.
Bilder
R.
Clinical global impression of cognition in schizophrenia (CGI-CogS): Reliability and validity of a co-primary measure of cognition
Schizophrenia Research
 , 
2008
, vol. 
106
 
1
(pg. 
59
-
69
)
Wechsler
D.
WAIS-III. Escala de inteligencia de Wechsler para adultos–III [Wechsler Adult Intelligence Scale-III]
 , 
1999
Madrid
TEA Ediciones
Wykes
T.
Reeder
C.
Corner
J.
Williams
C.
Everitt
B.
The effects of neurocognitive remediation on executive processing in patients with schizophrenia
Schizophrenia Bulletin
 , 
1999
, vol. 
25
 
2
(pg. 
291
-
307
)
Youden
W. J.
Index for rating diagnostic tests
Cancer
 , 
1950
, vol. 
3
 
1
(pg. 
32
-
35
)