Abstract

This study examined the validity of the four standard psychological paradigms that have been operationally defined within the CogState brief computerized cognitive assessment battery. Construct validity was determined in a large group of healthy adults. CogState measures of processing speed, attention, working memory, and learning showed strong correlations with conventional neuropsychological measures of these same constructs (r's = .49 to .83). Criterion validity was determined by examining patterns of performance on the CogState tasks in groups of individuals with mild head injury, schizophrenia, and AIDS dementia complex. Each of these groups was impaired on the CogState performance measures (Cohen's d's = −.60 to −1.80) and the magnitude and nature of this impairment was qualitatively and quantitatively similar in each group. Taken together, the results suggest that the cognitive paradigms operationally defined in the CogState brief battery have acceptable construct and criterion validity in a neuropsychological context.

Introduction

Neuropsychological tests are used routinely in clinical and research settings to identify cognitive impairment in neuropsychiatric or neurodegenerative disorders, to monitor the progression, or determine the effect of treatment on those impairments (Grant & Adams, 1996; Harrison & Owen, 2002; Lezak, Howieson, & Loring, 2004; Walsh & Darby, 2002). Whereas formal and complete neuropsychological evaluations are necessary for the accurate diagnoses of brain disease (Lezak et al., 2004; Walsh & Darby, 2002), shorter batteries of cognitive tests are also useful in guiding decisions about the presence, nature, and progression of cognitive impairment in patients. This is particularly so when cognitive performance must be measured repeatedly over short re-test intervals to identify cognitive changes in relevant time frames (e.g., mild traumatic brain injury [mTBI] or early neurodegenerative disease [Carroll et al., 2004; Darby, Maruff, Collie, & McStephen, 2002; Koss et al., 1996; Makdissi et al., 2001; Maruff, Currie, Malone, McArthur-Jackson, Mulhall, & Benson, 1994]). Rapid and simple but sensitive tests may also be useful when factors related to illness, such as fatigue, nausea, or reduced energy levels restrict the time for cognitive assessment, such as in AIDS dementia complex (ADC; Cysique, Maruff, & Brew, 2006a; Cysique, Maruff, Darby, & Brew, 2006b; Selnes et al., 1995; Thein et al., 2007a). Similarly, cognitive assessments must be brief in conditions where central nervous system (CNS) involvement is suspected, but where the optimal clinical management of the condition restricts the time or resources available for assessments (e.g., in chemotherapy or before and after coronary surgery; Phillips & Bernhard, 2003; Silbert et al., 2004). In such settings, brief tests can be used for cognitive screening. If abnormality is detected upon screening, or if deterioration in cognitive function from a previous assessment is identified, more complete neuropsychological work-up can ensue.

The CogState brief battery is comprised of a subset of tasks from the general CogState battery. The larger battery includes measures of visuomotor function, psychomotor/processing speed, visual attention/vigilance, attention/working memory, verbal learning and memory, executive function (spatial problem-solving, set-shifting), and social cognition (see www.cogstate.com; Collie, Maruff, Darby, & McStephen, 2003a; Pietrzak, Maruff, Mayes, Roman, Sosa, & Snyder, 2008). All tasks in the CogState battery have been specifically developed to assist with decisions about the presence or absence of cognitive change. The set of tasks included in the brief battery were selected because together they require about 8–10 min for administration, use very simple stimuli, and require simple decisions in response to simple rules. Although brief batteries of cognitive tests have been used previously in different settings (e.g., Koss et al., 1996; Murkin, Stump, & Blumenthal, 1997; Selnes et al., 1995), they are limited because they have few or no alternative forms, generate practice effects with repeated application over brief re-test intervals (Collie, Maruff, Makdissi, McStephen, Darby, & McCrory, 2004; Falleti, Maruff, Collie, & Darby, 2006), yield test outcome measures which produce data with less than optimal metric properties for measuring change (Collie, Maruff, Falleti, Silbert, & Darby, 2002), and/or require that individuals who take the tests have a history of educational and psychometric testing (e.g. Cairney, Clough, Jaragba, & Maruff, 2007; Rosselli & Ardila, 2003; Uzzell, Ponton, & Ardila, 2007).

In optimizing the tasks for the measurement of cognitive change, we sought to overcome these limitations. First, the stimulus set chosen for all tests were playing cards, as these provided 52 stimuli that varied on at least two dimensions and allowed for 52 factorial combinations to yield alternative forms of the tests. Although these test stimuli can be described linguistically, they are not verbal in nature, insomuch as they can be processed iconically without symbolic elaboration. Furthermore, because card games are played almost universally, these test stimuli and the context set by the test itself has been shown to be acceptable and familiar to individuals from diverse cultural and social groups (e.g., indigenous Australians; Cairney et al., 2007). The tasks themselves have been assembled according to principles used commonly in neuroimaging and cognitive neuroscience (e.g., Posner, 2004; Posner & Raichle, 1994). That is, for each trial in each task, only a single stimulus is shown (i.e., one playing card) and the presentation of the card requires either a “yes” or “no” response from the participant. The tasks are differentiated by the order in which the stimuli are presented (i.e., cards could be selected at random, the same card could appear on all trials, or specific cards could repeat in a pre-defined order), and the rule used by participants to make their decision (e.g., “Is it red?” or “have you seen it before?”).

Neuropsychological tasks that have been shown to provide robust measures of cognitive functions (e.g., simple and choice reaction time, n-back paradigms) were then reconstructed within this general framework. These tasks were presented using conventional computers to standardize the presentation of tasks and response recording (speed and accuracy). Finally, for each task, the performance measures that possessed the best metric properties for data analysis were identified (Falleti, Maruff, Collie, Darby, & McStephen, 2003; Maruff, Werth, Giordani, Caveney, Feltner, & Snyder, 2006; Mollica, Maruff, & Vance, 2004). These performance measures were chosen because their data distributions approximated interval level scalar characteristics, did not suffer from restriction of range, and were drawn from normal or corrected to normal distributions. Once complete, these tasks were challenged for their stability and resistance to practice effects by studies that required repeated administration over short as well as long re-test intervals (e.g., minutes to months; Collie, Maruff, Shafiq-Antonacci, Smith, Hallup, Schofield, Masters, & Currie, 2001; Falleti et al., 2006; Mollica et al., 2004). The sensitivity of the tests to change was determined from challenges with low doses of sedative drugs, CNS-stimulant medications, sleep deprivation, head injury, coronary surgery, and preclinical dementia (e.g., Collie, Makdissi, Maruff, Bennell, & McCrory, 2006; Collie et al., 2007; Falleti et al., 2006; Maruff, Collie, Darby, Weaver-Cargin, Masters, & Currie, 2004; Maruff et al., 1994; Mollica et al., 2004; Silbert et al., 2004; Thein et al., 2007b). Tests were removed if they had low test–retest reliability, were unstable, gave rise to practice effects with repeated assessment or were insensitive to detecting cognitive change. Thus, the CogState brief battery is now used in many settings to address the question of whether or not there has been change in cognitive function over both short (minutes) and long intervals of time (weeks, months).

The CogState brief battery consists of a simple reaction time task (Detection task), a choice reaction time task (Identification task), a one-back working memory task (One-Back task), and a continuous recognition visual learning task (Learning Task). Simple reaction time tasks have a long history in both experimental psychology and neuropsychology where they are often used as measures of attention and vigilance (Salthouse & Davis, 2006; Verhagen & Cerella, 2002). In contrast, performance on choice reaction time tasks requires greater perceptual, attentional, and motor processing and is therefore considered to reflect processing speed (Luce, 1986). One-Back tasks require individuals to maintain information in working memory for a brief time (Shallice, Marzocchi, Coser, Del Savio, Meuter, & Rumiati, 2002). The simplest condition of the n-back paradigm is most commonly used to model working memory in functional imaging and electroencephalogram (EEG) studies (Cohen & Leckman, 1994; Deiber et al., 2007; Jansma, Ramsey, Coppola, & Kahn, 2000; Owen, Iddon, Hodges, Summer, & Robbins, 1997). Continuous visual recognition learning paradigms are used commonly in functional neuroimaging studies of memory or in cognitive psychological studies of aging (Salthouse & Davis, 2006; Squire & Kandel, 2000). The task requires individuals to learn a set of stimuli on the basis of their serial and repeated exposure. Learning is operationally defined as the ability to discriminate between learned (i.e. stimuli seen previously) and novel (i.e. distractors) information. Therefore, the tasks used in the CogState brief battery can be considered to be measures of attention/vigilance, processing speed, working memory, and visual learning.

As many neuropsychiatric disorders and neurological illnesses are characterized by impairments in one or more of these domains (e.g., see Grant & Adams, 1996), we reasoned that measurement of these functions would provide the broadest opportunity for the detection of cognitive change or cognitive impairment irrespective of the condition in which it occurred. Although each of the tasks included in the CogState brief battery has been used previously to understand cognitive impairment in neuropsychiatric conditions, we sought to conduct a systematic investigation of their construct validity within a broad neuropsychological context. A cognitive task is generally developed within a theoretical context, with the validation of the task dependent on that context. Even though the different CogState tasks possess good theoretical validity (e.g., n-back tasks are well accepted measures of working memory) and adequate face validity (tasks that require participants to remember information do provide a measure of memory ability), they were selected and refined in an attempt to optimize their sensitivity to detect cognitive change. Therefore, rather than using specific theoretical models to guide test validation, we sought to establish the validity of the CogState brief battery within a more general neuropsychological framework.

The current study had two main aims. First, we sought to determine the construct validity of the each task by investigating the extent to which performance on that task was associated with performance on neuropsychological tests with established validity but which did not use linguistic stimuli or require verbal responses. We hypothesized that the CogState tasks would correlate with neuropsychological tests that assessed constructs similar to the experimental paradigms from which they were drawn. The second aim of the study was to determine the criterion validity of the CogState brief battery by establishing the nature and magnitude of impairments in performance in mTBI; chronic-medicated schizophrenia; and dementia associated with Human Immunodeficiency Virus (HIV dementia, HIV-D; see Cysique et al., 2006b; Mathias & Wheaton, 2007; Reichenberg & Harvey, 2007 for representative meta-analytic reviews of cognitive impairment in these disorders). Although quite different in their presentation and etiology, these groups were chosen as criterion samples because the cognitive deficits that characterize them manifest primarily in one or more of the domains of attention, processing speed, memory and executive function. Second, the magnitude of cognitive impairment in each of these conditions is sufficient to disrupt the ability to live independently or work but is not severe enough to limit neuropsychological assessments. Third, each condition is common in young and middle-aged adults, so cognitive impairments are uncomplicated by the effects of normal aging or age-related disorders. Fourth, cognitive impairment in each condition is important to its clinical management and none is associated with focal cortical abnormalities. The hypothesis was that each group would show impaired performance on the CogState battery. For all analyses, we followed the recommendations of Cohen (1988), Cumming and Maillardet (2006), and Zakzanis (2001) and based our conclusions on the interpretation of effect sizes and their confidence intervals (CIs) in addition to statistical significance.

Materials and Methods

Participants

Demographic characteristics of each group used and in this study are summarized in Table 1.

Table 1

Summary of demographic characteristics of the groups studied

Measure Healthy adults (construct validity study) mTBI mTBI controls Schiz Schizo controls ADC ADC controls 
Number 253 50 50 50 50 20 20 
Age 46.1 (10.1) 43.2 (5.6) 44.7 (4.9) 44.1 (8.9) 45.8 (7.6) 46.1 (12.1) 47.1 (13.1) 
Gender (% males) 83 80 80 87 82 89 91 
Years of education 13 (4.2) 12 (5.1) 12 (6.2) 11 (4.5) 11 (6.3) 13 (2.3) 12 (1.4) 
% Sample identifying themselves as Caucasian 98 100 100 96 98 95 95 
Estimated pre-morbid IQa 113.1 (23.1) 117.2 (15.3) 115.3 (12.3) 107.3 (25.3) 114.5 (25.3) 121.2 (23.1) 123.3 (29.1) 
Depressive symptoms 3.45 (5.2) 5.12 (6.3) 4.3 (4.9) 8.9 (7.2)M 5.7 (7.2)M 8.75 (8.8) 9.8 (12.7) 
Measure Healthy adults (construct validity study) mTBI mTBI controls Schiz Schizo controls ADC ADC controls 
Number 253 50 50 50 50 20 20 
Age 46.1 (10.1) 43.2 (5.6) 44.7 (4.9) 44.1 (8.9) 45.8 (7.6) 46.1 (12.1) 47.1 (13.1) 
Gender (% males) 83 80 80 87 82 89 91 
Years of education 13 (4.2) 12 (5.1) 12 (6.2) 11 (4.5) 11 (6.3) 13 (2.3) 12 (1.4) 
% Sample identifying themselves as Caucasian 98 100 100 96 98 95 95 
Estimated pre-morbid IQa 113.1 (23.1) 117.2 (15.3) 115.3 (12.3) 107.3 (25.3) 114.5 (25.3) 121.2 (23.1) 123.3 (29.1) 
Depressive symptoms 3.45 (5.2) 5.12 (6.3) 4.3 (4.9) 8.9 (7.2)M 5.7 (7.2)M 8.75 (8.8) 9.8 (12.7) 

Note: mTBI = mild traumatic brain injury; Schiz = chronic schizophrenia; ADC = AIDS dementia complex; M = depressive symptoms estimated from MADRS otherwise depressive symptoms estimated from DASS.

aEstimated pre-morbid IQ derived from National Adult Reading Test.

Construct Validity Study

A total of 215 healthy adults aged between 35 and 50 years were recruited from employment services, community advertisements, and word of mouth. The National Adult Reading Test (Nelson, 1982) was used to obtain an estimate of general intelligence. All participants were asked to define their race and 94% of the group defined their race as Caucasian. No participant had completed the CogState battery previously. Exclusion criteria pertaining to the general health of this control group were based on their self-reported responses upon examination and included a history of hematological, renal, endocrine, pulmonary, gastrointestinal, cardiovascular, hepatic, immunological, allergic, neurological, or psychiatric illness that had required medical attention in the past 10 years. Participants with routine use of sedative medication, analgesic, or other CNS active medications, or who reported current illicit use of substances such as marijuana, stimulants, opiates, or sedatives or who reported drinking more than two glasses of wine, beer, or spirits per day were also excluded. All of these participants completed the depression scale from the Depression Anxiety Stress Scales (DASS; Lovibond & Lovibond, 1995) and any participant giving a rating of 11 or greater was excluded (although none met this criterion).

Criterion Validity Study

Three patient groups were recruited from our ongoing studies of mTBI, schizophrenia, and ADC. Whereas all the participants from these groups had completed neuropsychological assessments in the past, none had performed any CogState tests. The general inclusion criteria for each clinical group was that participants were aged between 35 and 50 years, were able to give consent and perform the cognitive tests, had a confirmed diagnosis and evaluation of inclusion/exclusion criteria by an appropriately qualified clinician and did not meet DSM-IV criteria (American Psychiatric Association, 2000) for major depressive disorder at the time of assessment. General exclusion criteria included: history of alcohol or other substance abuse in their clinical record; report of illicit use of substances such as marijuana, stimulants, opiates, or sedatives; report of drinking more than two glasses of wine, beer, or spirits per day. All participants had normal or corrected to normal vision and hearing. An estimate of intelligence in controls and pre-morbid intelligence in participants in the clinical groups was obtained using the NART (Nelson, 1982). Specific inclusion/exclusion criteria were also applied for each clinical group and are described subsequently.

For each clinical group, a control group was recruited that was case-matched to the clinical group on age (within 3 years), gender, geographic location, and NART-estimated pre-morbid IQ (within 5 points). Participants were excluded from the control groups for the mTBI and schizophrenia clinical groups if they had a history of clinically significant hematological, renal, endocrine, pulmonary, gastrointestinal, cardiovascular, hepatic, immunological, allergic, neurological, or psychiatric disease. Control participants were also excluded if they reported routine sedative or other psychoactive medication use, current illicit use of substances such as marijuana, stimulants, opiates, or sedatives, or drinking more than two glasses of wine, beer, or spirits per day. The depression scale from the DASS was used to measure depressive symptoms in the mTBI and HIV control groups. The Montgomery–Åsberg Depression Rating Scale (MADRS; Montgomery & Åsberg, 1979) was used to measure depressive symptoms in the schizophrenia control group. Because subtle increases in depressive symptoms are known to occur in each of the neuropsychiatric conditions, clinical depression was an exclusion criteria for each clinical group. In addition, each clinical group had a specifically matched control group. The exclusion criteria for level of depressive symptoms was higher for the criterion validity samples. Individuals were excluded from the mTBI and HIV control groups if their DASS depression rating was greater than 12 (mild levels of depressive symptoms; Lovibond & Lovibond, 1995). Individuals were excluded from the schizophrenia control groups if their MADRS score was greater than 19 (Montgomery & Åsberg, 1979). The mean levels of depressive symptoms for each control group are shown in Table 1. The HIV control group was also matched on their sexual orientation and HIV serostatus. All participants were asked to define their race and more than 95% of each clinical group and control group defined their race as Caucasian. Data from the clinical control group participants were not used in the construct validity study. For each control group, recruitment continued until the sample size was equivalent to that of the relative clinical group. Data from the CogState tasks were not used at all to inform the classification or diagnosis of any patient in any of the clinical groups. The data in Table 1 show the equivalence between each clinical and matched control group on the demographic and mood variables. As was expected from the matching procedure no statistically significant differences were observed for any measure between each clinical and its relative control group (for brevity the results of these comparisons are not shown).

Mild Traumatic Brain Injury

Patients (n = 50) were recruited from head injury outpatient review clinics in major metropolitan hospitals. The classification of mTBI was based on guidelines by The Mild Traumatic Brain Injury Committee of the Head Injury Interdisciplinary Special Interest Group of the American Congress of Rehabilitation Medicine; Definition of Mild Traumatic Brain Injury (1993). This required patients to have been admitted to an emergency department with a history of head trauma associated with a road accident (either as a pedestrian, passenger, or driver) in the 12 hr prior to admission. Inclusion criteria were: Glasgow Coma Scale (GCS) score between 13 and 15 and either loss of consciousness (LOC) and/or post-traumatic amnesia (PTA). Exclusion criteria were: LOC > 30 min or PTA > 24 hr, seizures, focal neurological signs or symptoms or an abnormal computed tomography (CT) or magnetic resonance imaging (MRI) scan of the head. Additional exclusion criteria for the mTBI group included: a current skeleto-motor injury; history of hematological, renal, endocrine, pulmonary, gastrointestinal, cardiovascular, hepatic, immunological, allergic, or psychiatric disease; and routine use of sedative or other psychoactive medication. Patients with mTBI were also excluded if they had a history of neurological disease unrelated to the head injury or if their DASS depression rating was greater than 19. The current group was assessed at a mean of 72 days (standard deviation; SD = 14) after injury. At injury, the median GCS score of the mTBI group was 14 (range 13–15). Nineteen patients (38%) had LOC at the time of injury with the median LOC being 5 min (range: 30 s to 30 min). Forty-three patients (86%) had PTA at the time of injury with the median time of PTA being 15 min (range: ∼1 min to 63 min). The control group consisted of 50 healthy adults drawn from among the friends and family of each patient.

Schizophrenia

Fifty patients who fulfilled the Diagnostic and Statistical Manual, Forth Edition (DSM-IV; American Psychiatric Association, 2000) criteria for schizophrenia were recruited from inpatients admitted to a rehabilitation ward at a major metropolitan psychiatric service. All were receiving treatment with atypical antipsychotic medication and were judged by their primary treating clinician to have been clinically stable for longer than 6 months. No participant with schizophrenia had evidence of focal brain abnormality on head CT scan. Additional exclusion criteria for the schizophrenia group included: evidence or history of clinically significant neurological, hematological, renal, endocrine, pulmonary, gastrointestinal, cardiovascular, hepatic, immunological, or allergic disease, and routine use of sedative medication. Individuals were excluded from the schizophrenia control group if their MADRS score was greater than 19. For the schizophrenia group, the average length of illness duration was 19.8 yr (SD = 6.2). The total score on the positive and negative symptom scale (PANSS) was 59.2 (SD = 19.8), the group mean score on the withdrawal retardation factor on the Brief Psychiatric Rating Scale (BPRS) was 2.1 (SD = 1.2) and the group mean score on the BPRS thought disturbance factor was 2.1 (SD = 1.3). Hence, all patients had chronic stable illness. The matched control group was recruited from among unemployed adults attending an employment-finding service.

AIDS Dementia Complex

Twenty patients who were seropositive to infection with the HIV were recruited from HIV clinics in metropolitan hospitals. All had advanced HIV-infection and met criteria for ADC stage 1 or 2 (Price & Brew, 1988). All were homosexual or bisexual. Exclusion criteria were: existing neurological disorders; and current or past history of major depression or psychosis. The control group consisted of 20 homosexual or bisexual HIV-seropositive adults who met clinical criteria for the Acquired Immunodeficiency Syndrome (AIDS) but who did not meet criteria for ADC or the HIV-related minor neurocognitive disorder [accordingly none were rated as 0.5 on the Price Brew Dementia rating system. See Cysique et al. (2006b) for a discussion of this definition of a control group for HIV dementia]. Inclusion/exclusion criteria for the HIV control group were the same as those for the other control groups except that a history of clinical significant immunological disease was allowed. A history of significant hematological, renal, endocrine, pulmonary, gastrointestinal, cardiovascular, hepatic, allergic, neurological, or psychiatric disease was also allowed, provided any condition had resolved at least 6 months before study entry. Of most relevance to the current study was that in the HIV control group there were five individuals with a history (latest diagnosis made in 1999) of brain HIV-related diseases (including three cases of ADC stage 0.5, one case of cryptococcal meningitis, and one case with toxoplasmosis).

Test Battery

CogState brief battery

The CogState brief battery used in this study consisted of four tasks. Each has been described in detail elsewhere (Collie, Maruff, Falleti, Silbert, & Darby, 2002; Falleti et al., 2003, 2006) and is summarized subsequently. The test battery was presented on a laptop computer with headphones to minimize distracting noise. At the beginning of each task, written instructions were presented on the screen to indicate the task rules. Each participant was then given an interactive demonstration and, once they had successfully completed a sufficient number of practice trials to demonstrate their awareness of the rules, the task began. As described earlier, the CogState tasks are in the form of card games. For each task participants responded either “yes” or “no”: at the presentation of each stimulus using the “d” and “k” keys on the computer keyboard. “Yes” responses were made with the dominant hand. At the beginning of each task participants are instructed to “respond as fast and as accurately as possible.” The performance measures recorded for each trial on each task was the speed of the response (i.e., reaction time recorded in milliseconds) and the accuracy of response (i.e., correct or incorrect reported as percentages). These outcome measures were then transformed so that data distributions are normalized and to reduce the potential for correlation between mean performance and variability (see Bland & Altman, 1996 and Maruff et al., 2006 for discussions of these points). Estimates of test–retest reliability for each of these measures range between .84 and .91 (Collie et al., 2003b; Falleti et al., 2006). To facilitate interpretation of the data, both transformed and back-transformed data are shown. The tasks employed in this study, in their order of presentation, are described further.

The Detection task is a simple reaction time test in which the participant must attend to the task in the center of the screen and follow the rule “Has the card turned face up? If yes, press the ‘K’ key” (“D” key, if the participant was left-hand dominant). The task continued until 25 correct responses were recorded or the maximum allowed time (2 min) had elapsed. The primary outcome measure for this task was the speed of correct responses in milliseconds. Data distributions were normalized using a logarithmic base 10 (log 10) transformation.

The Identification Task is a choice reaction time test in which the participant must attend to the card in the center of the screen and follow the rule “Is the face-up card red? If the answer is yes press the ‘K’ key, if the answer is no press the ‘D’ key” (this was reversed for participants who were left hand dominant). This task continues until 35 correct responses have been recorded or the maximum allowed time (2 min) had elapsed. The primary outcome measure for this task was speed of correct responses in milliseconds. Data distributions were then normalized using a log 10 transformation.

The One-Back Task requires the participant to attend to the card in the center of the screen and use the rule “is this card the same as that on the immediately previous trial? If the answer is yes press the ‘K’ key, if the answer is no press the ‘D’ key” (this was reversed for participants who were left hand dominant). Forty-two cards are shown in the task in which the “yes or no” response is correct in 50% each of the trials presented. This task continued until the 42 trials have been completed or the maximum allowed time (3 min) had elapsed. The primary outcome measure for this task was the number of correct responses (i.e., true-positive and true-negative) expressed as a percentage of the total trials. Data distributions were then normalized by expressing the percentages as a decimal and applying an arcsine transformation.

The Visual Learning task requires that the participant attend to the card in the center of the screen and follow the rule “Have you seen this card before in this task? If the answer is yes press the ‘K’ key, if the answer is no press the ‘D’ key” (this was reversed for participants who were left hand dominant). Participants were required to learn a series of six cards repeated pseudo-randomly throughout the task, intermixed with 8 distracter (i.e., non-repeating) cards in epochs of 14 cards. Three 14-card epochs were presented, and this task continued until the participant had made 42 complete responses or the maximum time allowed (3 min) had elapsed. The primary outcome measure for this task was the number of correct responses (i.e., true-positive and true-negative) expressed as a proportion of the total trials. Data distributions were then normalized by expressing the percentages as a decimal and applying an arcsine transformation.

Comparator Neuropsychological Tests

The comparator neuropsychological tests used in the construct validity study were selected because their validity had been previously established and they neither required spoken responses nor use words or sentences as stimuli. This battery included the Grooved Pegboard Test for dominant (GPB-D) and non-dominant hands (GPB-ND), the Trail Making Test Part A and B (TMT-A and TMT-B), the Symbol Digit Modalities Test (SDMT), the Brief Visual Memory Test (BVMT), the 30-min delayed recall trial of the Rey Complex Figure Test (RCFT-R), and the Spatial Span Subtest (Span) from the Wechsler Memory Scale Third Edition (WMS-III). Each of these tests has been described in detail in compendia of neuropsychological tests (Darby & Walsh, 2005; Lezak et al., 2004; Spreen & Strauss, 1998). All tests were administered according to their standard instructions. For each test, a single outcome measure was used to characterize performance. Measures recorded and used for analysis were: for the GPD, the mean time (in seconds) required to place the pegs on two trials, for the TMT-A and TMT-B, the time (in seconds) to complete the trail, for the SDMT, the number of correct responses written in 90 s, for the BVMT, the total score trial 1 to 3, for the RCFT, the score for correctness of the recall, and for the Span test, the total span score.

Study Design and Data Analysis

Analysis of construct validity

The construct validity of the CogState task was determined by the extent to which performance on each task correlated with performance on the comparator neuropsychological tests. First, Pearson product–moment correlations were computed between the CogState measures and the comparator neuropsychological tests in the healthy volunteers (Winer, 1971; Table 2). Correlations that were statistically significant were then expressed as measures of effect size (e.g. Cohen, 1988) and plotted with their 95% confidence intervals (95% CI; Fig. 1). In Fig. 1, where the CIs for two effect sizes do not overlap, the statistical probability that these means are from the same distribution is less than .01. Hence differences between correlations whose 95% CIs do not overlap are statistically significant (see Cumming & Maillardet, 2006 for discussion of interpreting CIs).

Fig. 1.

Effect sizes and their 95% confidence intervals (95% CIs) for significant correlations between CogState tasks and the comparator neuropsychological tests. Where 95% CIs do not overlap effect sizes are significantly different. Open diamonds joined by a solid line indicate effect sizes for Detect task; filled circles joined by a dotted line indicate effect sizes for Ident task; Open triangles joined by a solid line indicate effect sizes for One-Back task; Filled squares joined by a dotted line indicate effect sizes for Learn task.

Fig. 1.

Effect sizes and their 95% confidence intervals (95% CIs) for significant correlations between CogState tasks and the comparator neuropsychological tests. Where 95% CIs do not overlap effect sizes are significantly different. Open diamonds joined by a solid line indicate effect sizes for Detect task; filled circles joined by a dotted line indicate effect sizes for Ident task; Open triangles joined by a solid line indicate effect sizes for One-Back task; Filled squares joined by a dotted line indicate effect sizes for Learn task.

Table 2

Pearson's product–moment correlations between each CogState performance measure and the neuropsychological measures

 GPB-D GPB-ND TMT-A TMT-B SDMT Span BVMT RCFT-R 
Detect .81** .71** .70** .52* .31 .11 .17 .12 
Identify .53* .49* .76** .78** .74* .1 .04 .05 
One-back .13 .21 .69* .71* .81* .80* .54* .39 
Learn .17 .15 .19 .59* .57* .69* .83* .79* 
 GPB-D GPB-ND TMT-A TMT-B SDMT Span BVMT RCFT-R 
Detect .81** .71** .70** .52* .31 .11 .17 .12 
Identify .53* .49* .76** .78** .74* .1 .04 .05 
One-back .13 .21 .69* .71* .81* .80* .54* .39 
Learn .17 .15 .19 .59* .57* .69* .83* .79* 

Note: GPB = grooved pegboard; -D = dominant hand; -ND = non-dominant hand; TMT-A/B = Trail Making Test parts A and B; SDMT = Symbol Digit Modalities Test; Span = WMS III spatial span task; BVMT = Brief Visual Memory Test; RCFT-R = Rey Complex Figure Test-Delayed Recall.

*p < .01; **p < .001.

Analysis of criterion validity

The criterion validity of the four CogState tasks was established by (i) determining the nature and magnitude of impairment on each CogState task for each clinical group relative to their control groups using a series of independent groups t-tests; (ii) expressing the magnitude of the difference between groups as an effect size (Cohen's d; Cohen, 1988; Zakzanis, 2001) together with their 95% CIs (Cumming & Maillardet, 2006); and (iii) computing the non-overlap statistic (non-OL%) for each difference. Measures of effect size express the difference between the clinical group mean and the control group mean as a function of their pooled standard deviations. Therefore magnitudes of effect sizes can be understood like z-scores where the units of measurement are also standard deviations. The non-OL% statistic reflects the extent to which data distributions of the clinical and control groups do not overlap. Hence, for the current study, the non-OL% reflects the proportion of the clinical group whose performance was not shared by the control group with larger values indicating a better classification (Zakzanis, 2001).

Experiment-Wise Error Rates

Because multiple comparisons and correlations were computed in the current analyses, there was an increased risk of Type I error. That is, with many analyses of the same data, the chance of finding results that are statistically significant but not true, increases as the number of tests increases. In order to protect against this risk, the probability required for the result of any comparison or correlation to be considered as statistically significant was set at p < .01. This level was considered to balance the risk of Type I error against the potential for the outcome measures to be correlated with one another. Further, the computation of effect sizes and their CIs was used to guide interpretation about the potential for Type I and Type II errors. For example, comparisons or correlations where statistical significance occurs but where effect sizes are very small (i.e., < d = .2) will be considered to have been Type I errors and effect sizes greater than .8 but which are not statistically significant will be considered to have been Type II errors.

Results

Construct Validity

Correlations between CogState and comparator measures are shown in Table 2; significant associations are summarized in Fig. 1.

Each of the outcome measures from the CogState tasks correlated with performance on more than one neuropsychological test. In Fig. 1, the greater the effect size (d), the better the correlation between the CogState task and the comparator task. Each comparator neuropsychological test is shown on the x-axis and the lines shown link the effect sizes for each CogState task. In general, small 95% CIs for the effect sizes indicate that each test provides a good estimate of the relationship in the population and also reflect the relatively large sample size in which the interrelationships between tests were studied. Effect sizes whose 95% CIs did not overlap are significantly different from one another.

Performance on the Detect task correlated significantly with performance on the GP-D, GP-ND, TMT-A, and TMT-B tests (Table 2). Consideration of the effect sizes and their 95% CIs on Fig. 1 shows the strongest associations were with the GPB-D, GP-ND, and TMT-A tests. Performance on the Identify task correlated significantly with performance on the GP-D, GP-ND, TMT-A, TMT-B, and SDMT tests (Table 2). Fig. 1 suggests that the strongest associations with the Identify task were for the TMT-A, TMT-B, and SDMT tests and that the magnitude of the correlation between performance on the TMT-B task and the Identify task was greater than that observed between performance on the TMT-B and Detect tasks.

Performance on the One-Back task correlated significantly with performance in the TMT-A, TMT-B, SDMT, Span, and BVMT. Fig. 1 shows that the strongest associations with the One-Back task were for the SDMT and Span tests and that these were better than the associations observed between these same comparator tests and performance on the Identify or Learn tasks. Performance on the Learn task correlated significantly with performance on the TMT-B, SDMT, BVMT, and RCFT-R. Fig. 1 shows that the strongest associations were for the BVMT and RCFT-R tests.

Criterion Validity

Table 3 shows the mean performance and variability for each performance measure from the CogState tasks in each of the three clinical groups and their matched controls. The statistical significance of each comparison and the non-%OL statistic for each performance measure is also shown on Table 3. For all tasks except Detect in the ADC group, performance in the clinical group was significantly worse than in matched controls. Table 3 also shows the group raw (back-transformed) means and ranges of scores on the different outcome measures. For the control groups, the average speed of performance on the Detect task ranged from 250 ms to 350 ms and for the Identify task, average group performance speed ranged from 480 ms to 580 ms. The accuracy of performance on the One-Back Task ranged from 85%–94% and the accuracy of performance on the Learn task ranged from 67% to 87%. For the clinical groups the slowing in speed of performance on the Detect and Identify tasks raged from 50 ms to 100 ms and decreases in the accuracy of performance on the One-Back and Learn tasks ranged from 7% to 23%.

Table 3

Group means and statistical significance for comparison of each patient group with their respective control group on CogState tasks

Clinical group Task Transformed data
 
Back-transformed data
 
  Mean control group (SDMean clinical group (SDt p %N-OL Mean control group Low 95% CI Up 95% CI Mean clinical group Low 95% CI Up 95% CI 
mTBI Detect 2.46 (.06) 2.59 (.16) 5.9 <.0001 62 288.4 251.2 331.1 389 269.2 562.3 
Identify 2.69 (.09) 2.78 (.1) 4.8 <.0001 53 489.8 398.1 602.6 602.6 478.6 758.6 
One-back 1.22 (.12) 0.93 (.39) 5.7 <.0001 60 0.94 .89 .97 0.8 .51 .97 
Learn 1.06 (.15) 0.69 (.25) 9.3 <.0001 78 0.87 .79 .94 0.64 .43 .81 
Schiz Detect 2.49 (.07) 2.59 (.15) 4.6 <.0001 52 309 263 363.1 389 275.4 549.5 
Identify 2.73 (.08) 2.84 (.11) 5.8 <.0001 61 537 446.7 645.7 691.8 537 891.3 
One-back 1.22 (.11) 1.03 (.23) 5.6 <.0001 60 0.94 .9 .97 0.86 .72 .95 
Learn 0.84 (.14) 0.61 (.18) 7.2 <.0001 69 0.74 .64 .83 0.57 .42 .71 
ADC Detect 2.54 (.09) 2.6 (.09) 2.1 <.05 41 346.7 281.8 426.6 398.1 323.6 489.8 
Identify 2.76 (.11) 2.9 (.13) 3.7 <.001 62 575.4 446.7 741.3 794.3 588.8 1071.5 
One-back 1.02 (.12) 0.9 (.12) 3.2 <.01 55 0.85 .78 .91 0.78 .7 .85 
Learn 0.73 (.15) 0.57 (.2) 2.9 <.01 52 0.67 .55 .77 0.54 .36 .7 
Clinical group Task Transformed data
 
Back-transformed data
 
  Mean control group (SDMean clinical group (SDt p %N-OL Mean control group Low 95% CI Up 95% CI Mean clinical group Low 95% CI Up 95% CI 
mTBI Detect 2.46 (.06) 2.59 (.16) 5.9 <.0001 62 288.4 251.2 331.1 389 269.2 562.3 
Identify 2.69 (.09) 2.78 (.1) 4.8 <.0001 53 489.8 398.1 602.6 602.6 478.6 758.6 
One-back 1.22 (.12) 0.93 (.39) 5.7 <.0001 60 0.94 .89 .97 0.8 .51 .97 
Learn 1.06 (.15) 0.69 (.25) 9.3 <.0001 78 0.87 .79 .94 0.64 .43 .81 
Schiz Detect 2.49 (.07) 2.59 (.15) 4.6 <.0001 52 309 263 363.1 389 275.4 549.5 
Identify 2.73 (.08) 2.84 (.11) 5.8 <.0001 61 537 446.7 645.7 691.8 537 891.3 
One-back 1.22 (.11) 1.03 (.23) 5.6 <.0001 60 0.94 .9 .97 0.86 .72 .95 
Learn 0.84 (.14) 0.61 (.18) 7.2 <.0001 69 0.74 .64 .83 0.57 .42 .71 
ADC Detect 2.54 (.09) 2.6 (.09) 2.1 <.05 41 346.7 281.8 426.6 398.1 323.6 489.8 
Identify 2.76 (.11) 2.9 (.13) 3.7 <.001 62 575.4 446.7 741.3 794.3 588.8 1071.5 
One-back 1.02 (.12) 0.9 (.12) 3.2 <.01 55 0.85 .78 .91 0.78 .7 .85 
Learn 0.73 (.15) 0.57 (.2) 2.9 <.01 52 0.67 .55 .77 0.54 .36 .7 

Note: mTBI = mild traumatic brain injury; Schiz = chronic schizophrenia; ADC = AIDS dementia complex; abbreviations for CogState tasks as for Table 2; Cont = control; group means and standard deviation (SD) computed on transformed data described in methods; back-transformed data = group mean of transformed data converted back to original units and 95% CIs; for Detect and Ident tasks units are milliseconds and for One-Back and Learn tasks units are percent correct responses. t = t Value from between-group comparison; p = probability of t; %N-OL = percentage of data distributions that do not overlap.

To allow direct comparison between measures within clinical groups, the difference in mean performance between each clinical group and its control was also expressed as a measure of effect size and 95%CIs were also computed for these estimates (Fig. 2). The data in Fig. 2 indicate that the magnitude of impairment on the Identify and One-Back tasks was equivalent across the three groups. In the mTBI group, impairment was greatest for the Learn task. For the schizophrenia group, the magnitude of impairment across all tasks was equivalent and for the ADC group, impairment on the Detect task was less than that for the other three tasks.

Fig. 2.

Effect sizes and their 95% confidence intervals for the magnitude of cognitive impairment on the CogState outcome measures for mild traumatic brain injury (mTBI), schizophrenia (schiz), and the AIDS dementia complex (ADC). GPB = grooved pegboard; -D = dominant hand; -ND = non-dominant hand; TMT-A/B = Trail Making Test parts A and B; SDMT = Symbol Digit Modalities Test; Span = WMS III spatial span task; BVMT = Brief Visual Memory Test; RCFT-R = Rey Complex Figure Test-Delayed Recall.

Fig. 2.

Effect sizes and their 95% confidence intervals for the magnitude of cognitive impairment on the CogState outcome measures for mild traumatic brain injury (mTBI), schizophrenia (schiz), and the AIDS dementia complex (ADC). GPB = grooved pegboard; -D = dominant hand; -ND = non-dominant hand; TMT-A/B = Trail Making Test parts A and B; SDMT = Symbol Digit Modalities Test; Span = WMS III spatial span task; BVMT = Brief Visual Memory Test; RCFT-R = Rey Complex Figure Test-Delayed Recall.

Consideration of the data for the %N-OL statistic (Table 3) indicates that despite the robust significance of the differences between group means for all but one measure (Detect task in ADC group) and the large effect sizes for these comparisons (Fig. 1), the extent to which any single measure could identify cognitive impairment in individuals from each clinical group ranged from 78% at best (Learn task in mTBI group) to 41% (Detect task in ADC group). In general, %N-OL statistics ranged between 50% and 60%.

Discussion

The results of this study demonstrate the construct validity of the CogState brief battery in measuring attention/vigilance, processing speed, memory, and working memory functions. They also demonstrate the criterion validity of the CogState brief battery in detecting cognitive impairment in mTBI, schizophrenia, and ADC. In the current study, we inferred that associations between the computerized and comparator neuropsychological tests were statistically significant if their associated p-value was less than .01 and that the magnitude of the correlations were different to one another when the 95% CIs for their effect sizes did not overlap. The strongest correlations observed suggested that performance on the computerized measure task of attentional function (Detect task) were with performance on comparator tasks that required simple visual attentional, vigilance, or psychomotor functions (i.e., the GPB-D, GPB-ND, TMT-A). The strongest correlations observed for the computerized measure of processing speed (Identify task) were with tasks that required complex visual processing and divided attention (TMT-A, TMT-B, and SDMT). The strongest correlations observed for the computerized test of working memory (One-Back task) were with comparator tasks that required visual scanning and working memory (Span). The strongest correlations observed for the computerized test of continuous visual recognition learning (Learn task) was with the comparator tasks that required visual learning and memory (BVMT, ROCF-R). Importantly, although the magnitudes of the effect sizes shown in Fig. 1 are very large, they represent only the mathematical transformation of the correlations shown in Table 2 into d values using the formulae outlined in Zakzanis and coworkers (1998). Thus the effect sizes in Fig. 1 do not reflect the sensitivity of the tests to cognitive impairment, but only the strength of the association.

Each of the performance measures from the computerized tests correlated significantly with performance on more than one of the comparator neuropsychological tests. This pattern of multiple associations was expected, as performance on all neuropsychological tests require the coordination of multiple and different cognitive operations (Chan, Shum, Toulopoulou, & Chen, 2008; Lezak et al., 2004; Royall et al., 2002). None of the computerized tasks studied here operationally define a single neuropsychological construct. When considered qualitatively, however, the broader patterns of significant associations between the CogState tasks and the comparator tests were generally consistent with the theoretical models from which the tasks were derived. Thus, performance on the tasks requiring simple but speeded decisions about visual stimuli (Detect and Identify tasks) correlated strongly, although to a greater or lesser extent, with performance on the comparator neuropsychological tests which also emphasized speeded performance, visual attention, scanning, and divided attention. Performance on the computerized working memory task correlated with performance on other tasks that have a working memory component and performance on the visual learning task was associated with performance on other tasks that require memory for visual material. When considered together, the correlations presented here suggest that the psychological paradigms used to measure attention, processing speed, working memory, and visual learning within the CogState brief battery do indeed assess those functions. However, the observation that the CogState tasks correlated with performance on multiple neuropsychological tests also underscores that even relatively simple cognitive tasks depend on multiple cognitive operations. This overlap between attention, working memory, and visual learning is consistent with theoretical models that also have difficulty disentangling complex attentional and simple working memory functions (e.g., Chan et al., 2008; Posner, 2004).

In neuropsychological settings, the ability of a test to detect impairment in a specific group can be considered an important aspect of criterion validity where the criterion is the abnormal cognition associated with a specific state or disease. In this study, we sought to evaluate the criterion validity of each of the cognitive psychological paradigms constructed within the CogState brief battery by measuring the nature and magnitude of cognitive impairment in three clinical groups: mTBI, schizophrenia, and ADC. These three groups were selected because each is characterized by subtle impairments in attention, processing speed, working memory and visual memory compared to the impairments in similar functions in conditions characterized by more severe cognitive impairment (e.g., Alzheimer's disease). Selection of these groups also allowed us to assess the independence in performance on the different CogState tasks from a perspective different to that gained from the analysis of construct validity.

Compared to their matched control groups, performance in each of the clinical groups was impaired significantly on all but one of the computerized tasks. In general, the nature and magnitude of impairment on the CogState tasks in the three clinical groups was similar. The performance of the mTBI group indicated that despite the minimum time from injury being six months, robust but subtle impairments persisted in attention, processing speed and working memory, with a relatively larger impairment evident in visual learning. This finding of ongoing difficulty with visual learning and impairment in attention, processing speed and working memory at more than six months of injury is consistent with findings from reviews of cognitive impairment in patients recovering from mTBI (Carroll et al., 2004). These generalized impairments are hypothesized to arise from diffuse axonal injury (DAI) to the white matter tracts that link cortex to brainstem, which is caused by shearing forces generated in the brain by sudden deceleration (Bigler, 2008).

Subtle disruption to cortical-striatal pathways is also hypothesized to be at the basis of cognitive impairment in chronic schizophrenia (Lewis, 2002). Whereas cortical-striatal disruption is proposed to be neurochemical in nature, there is evidence of reductions in grey cortical matter especially in heteromodal association cortices and in the hippocampus (Lewis, 2002). Cognitive impairment in the current schizophrenia group manifested as a generalized but subtle disruption to processing speed and attentional, working memory and memory processes. The magnitude of impairment in visual learning was slightly larger than for other cognitive measures, although the 95%CIs were relatively large as well. The finding of a generalized impairment across cognitive domains with slightly greater impairments in memory is consistent with meta-analyses of cognitive impairment in schizophrenia, which found magnitudes of impairment ranging from 1 to 1.5 SD below healthy controls (Heinrichs & Zakzanis, 1998; Reichenberg & Harvey, 2007). Interestingly, the generalized impairment in the current chronic schizophrenia group was quantitatively and qualitatively similar to that observed in the mTBI group, with the main difference being that impairment in visual memory was relatively greater in the mTBI group than in the schizophrenia group.

In the ADC group, cognitive impairments were greatest for processing speed, working memory, and learning. Impairment in attention was not large enough to meet the adjusted significance level used in the current study (e.g., p < .01; Table 3). However, the effect size for this impairment was, by convention, large and consistent with the impairments on other related measures in the same group. Thus, it is possible that impairment on the Detect task in the ADC group was true but more subtle than the impairments observed for other tasks. Despite this finer point, the impairment in cognitive function observed in the ADC was qualitatively and quantitatively similar to that observed in the mTBI and schizophrenia groups. The physical fatigue and general ill-health associated with advanced immunosuppression was controlled by comparing patients with ADC to individuals who were cognitively normal but who met clinical criteria for AIDS. A limited appreciation of the indirect effect of these non-cognitive factors on performance can be gained by comparing performance between the mTBI, schizophrenia, and ADC control groups. Performance in the ADC control group was greater than 1 SD below the healthy controls for the schizophrenia and mTBI groups (Table 3). There is agreement in the literature that the pathophysiology of ADC is primarily subcortical in nature (Anthony & Bell, 2008; Cysique et al., 2006a). Thus, the current data suggest that cognitive impairment in the ADC group, who were all receiving highly active antiretroviral therapy, is characterized by impairment in processing speed, working memory, and learning. Impairments in these same cognitive functions have been observed previously in patients with ADC and are consistent with the predilection of HIV CNS disease for subcortical brain areas (Cysique et al., 2006a, b).

Taken together, these data suggest that although the mTBI, schizophrenia, and ADC are quite different clinically, when compared with carefully matched controls, the nature and magnitude of cognitive impairment observed in each group is relatively similar. This similarity may reflect a limitation in the extent to which cognitive impairment can be characterized in detail on the basis of a 10–12-min computerized battery. With more tests and performance measures, differences between the cognitive impairment in each group would likely become more evident. However, the similarity between groups in their cognitive impairment may also reflect similarity in the nature of brain dysfunction in these three conditions. For example, although their etiologies are different, each of these conditions is associated with diffuse damage or disruption to sub-cortical regions and their cortical projections, but not focal cortical abnormalities. Irrespective of the reason, these data suggest that the cognitive paradigms used in the CogState brief battery are sensitive to cognitive impairment in the different clinical groups. However, the brevity of the battery restricts its specificity to dissociate the cognitive impairments that characterize mTBI, chronic schizophrenia, and ADC.

An appreciation of the clinical significance of the cognitive impairments observed in the current study can be gained by reference to the %N-OL statistic. This statistic gives the percentage of participants in the clinical group whose scores are not shared with that of the control group. The magnitude of impairment on the Identify and One-Back task was similar across the clinical groups. The %N-OL statistics for these differences indicate that on average 50%–60% of the clinical group did not share performance scores with the controls. Zakzanis (2001) provides data to indicate that a %N-OL of 98% is required for diagnostic certainty although a good clinical marker would provide a %N-OL of 93%. Thus, by themselves, the Identify or One-Back task cannot be used to classify group membership. The best discriminator in the current study was the Learn task in the mTBI group, where 78% of the clinical group's performance on this task was not shared by controls. However, even at this level, the rate of false-positive classifications would be unacceptably high. The Learn task was also the best discriminating test in the schizophrenia group, with 69% of patients having performance scores not shared by the matched controls. However, this rate is again too low for this test, by itself, to be considered a good clinical marker. For the ADC group, the best discriminating task was the Identification task, where 62% of the clinical group provided performance that was not shared by controls. Thus, although statistically reliable and experimentally large impairments were observed in the three clinical groups on the CogState measures, none of these measures by themselves possessed that ability to classify group membership with an acceptable accuracy. Importantly, though the magnitudes of the impairments observed in the current study are consistent with the impairments found in meta-analyses of experimental data in similar participant groups. Therefore, the inability of the measures presented here to classify group membership occurs because cognitive impairments in these disorders are quite subtle not because the tests or the paradigms they define are insensitive.

In summary, results of the current study suggest that the brief CogState battery has adequate construct validity and is sensitive to detecting subtle cognitive impairment in mTBI, schizophrenia, and ADC. The tasks in the CogState brief battery were selected and modified so that they would have good sensitivity to changes in cognitive function. The current data suggest that these same measures have good sensitivity to cognitive impairment, although the specificity of the different tasks in disentangling cognitive deficits specific to these conditions must be understood further. Despite the strong correlations observed between the CogState tasks and the comparator neuropsychological tests, we recommend that assessments of attention, processing speed, memory, and working memory based only on the CogState brief battery can support only broad conclusions about these different cognitive functions. Furthermore, because the CogState tasks are so brief, and performed on a computer, examiners cannot derive a great deal of qualitative information about the manner in which tasks are performed, as is important in neuropsychological assessment. Thus, the CogState brief battery should be considered as another tool in the library of neuropsychologists that can be used to screen for the presence of cognitive impairment in large groups of people, to assess disease progression, and to monitor treatment response. It can also be used in group studies of particular diseases or disorder to generate hypotheses about the nature and severity of cognitive impairment. These hypotheses can then be examined in detail with more focused neuropsychological assessment.

Finally, it is important to point out the limitations of the current study. First, all of the clinical samples studies here and their control groups were convenience samples. Therefore, we do not know as yet the extent to which data collected here can be generalized more broadly. Second, construct validity was estimated from the performance of a large group of healthy adults. These data require replication in a clinical group to determine whether the relationships observed are maintained in the context of true cognitive impairment.

Conflict of Interest

P.M. is a full-time employee of CogState Ltd. and P.S. is a consultant to CogState Ltd. CogState Ltd. provides the CogState tasks reported in this study.

References

Anthony
I. C.
Bell
J. E.
The neuropathology of HIV/AIDS
International Review of Psychiatry
 , 
2008
, vol. 
20
 (pg. 
15
-
24
)
American Psychiatric Association
Diagnostic and statistical manual of mental disorders
 , 
2000
4th ed.
Washington, DC
American Psychiatric Press
Bland
J. M.
Altman
D. G.
Measurement error
British Medical Journal
 , 
1996
, vol. 
313
 pg. 
774
 
Bigler
E. D.
Neuropsychology and clinical neuroscience of persistent post-concussive syndrome
Journal of the International Neuropsychological Society
 , 
2008
, vol. 
14
 (pg. 
1
-
22
)
Cairney
S.
Clough
A.
Jaragba
M.
Maruff
P.
Cognitive impairment in Aboriginal people with heavy episodic patterns of alcohol use
Addiction
 , 
2007
, vol. 
102
 (pg. 
909
-
915
)
Carroll
L. J.
Cassidy
J. D.
Peloso
P. M.
Borg
J.
von Holst
H.
Holm
L.
, et al.  . 
Prognosis for mild traumatic brain injury: Results of the WHO Collaborating Centre Task Force on Mild Traumatic Brain Injury
Journal of Rehabilitation Medicine
 , 
2004
, vol. 
43
 
Suppl.
(pg. 
84
-
105
)
Chan
C. K.
Shum
D.
Touloupoulou
T.
Chen
E. Y. H.
Assessment of executive functions: Review of instruments and identification of critical issues
Archives of Clinical Neuropsychology
 , 
2008
, vol. 
23
 
2
(pg. 
201
-
216
)
Cohen
J.
Statistical Power for the Behavioral Sciences
 , 
1988
New York, NY
Lawrence Erlbaum
Cohen
D. J.
Leckman
J. F.
Developmental psychopathology and neurobiology of Tourette's syndrome
Journal of the American Academy of Child and Adolescent Psychiatry
 , 
1994
, vol. 
33
 (pg. 
2
-
15
)
Collie
A.
Darekar
A.
Weissgerber
G.
Toh
M. K.
Snyder
P. J.
Maruff
P.
, et al.  . 
Cognitive testing in early-phase clinical trials: development of a rapid computerized test battery and application in a simulated Phase I study
Contemporary Clinical Trials
 , 
2007
, vol. 
28
 
4
(pg. 
391
-
400
)
Collie
A.
Makdissi
M.
Maruff
P.
Bennell
K.
McCrory
P.
Cognition in the days following concussion: Comparison of symptomatic versus asymptomatic athletes
Journal of Neurology Neurosurgery and Psychiatry
 , 
2006
, vol. 
77
 
2
(pg. 
241
-
245
)
Collie
A.
Maruff
P.
Darby
D. G.
McStephen
M.
The effects of practice on the cognitive test performance of neurologically normal individuals assessed at brief test-retest intervals
Journal of the International Neuropsychological Society
 , 
2003
, vol. 
a 9
 (pg. 
419
-
428
)
Collie
A.
Maruff
P.
Falleti
M.
Silbert
B.
Darby
D. G.
Determining the extent of cognitive change following coronary artery bypass grafting: A review of available statistical procedures
Annals of Thoracic Surgery
 , 
2002
, vol. 
73
 (pg. 
2005
-
2011
)
Collie
A.
Maruff
P.
Makdissi
M.
McCrory
P.
McStephen
M.
Darby
D.
CogSport: Reliability and correlation with conventional cognitive tests used in postconcussion medical evaluations
Clinical Journal of Sport Medicine
 , 
2003
, vol. 
b 13
 (pg. 
28
-
32
)
Collie
A.
Maruff
P.
Makdissi
M.
McStephen
M.
Darby
D. G.
McCrory
P.
Statistical procedures for determining the extent of cognitive change following concussion
British Journal of Sports Medicine
 , 
2004
, vol. 
38
 (pg. 
273
-
278
)
Collie
A.
Maruff
P.
Shafiq-Antonacci
R.
Smith
M.
Hallup
M.
Schofield
P. R.
, et al.  . 
Memory decline in healthy older people: Implications for identifying mild cognitive impairment
Neurology
 , 
2001
, vol. 
56
 (pg. 
1533
-
1538
)
Cumming
G.
Maillardet
R.
Confidence intervals and replication: Where will the next mean fall?
Psychological Methods
 , 
2006
, vol. 
11
 
3
(pg. 
217
-
227
)
Cysique
L.
Maruff
P.
Brew
B.
The neuropsychological profile of symptomatic, AIDS and ADC patients in the pre-HAART era: A meta-analysis
Journal of the International Neuropsychological Society
 , 
2006
, vol. 
a 12
 (pg. 
368
-
382
)
Cysique
L. A.
Maruff
P.
Darby
D.
Brew
B. J.
The assessment of cognitive function in advanced HIV-1 infection and AIDS dementia complex using a new computerized cognitive test battery
Archives of Clinical Neuropsychology
 , 
2006
, vol. 
b 21
 (pg. 
185
-
194
)
Darby
D.
Maruff
P.
Collie
A.
McStephen
M.
Mild cognitive impairment can be detected by multiple assessments in a single day
Neurology
 , 
2002
, vol. 
59
 (pg. 
1042
-
1046
)
Darby
D.
Walsh
K.
Clinical neuropsychology
 , 
2005
London
Churchill Livingston
Deiber
M. P.
Missonnier
P.
Bertrand
O.
Gold
G.
Fazio-Costa
L.
Ibanez
V.
, et al.  . 
Distinction between perceptual and attentional processes in working memory tasks: A study of phase-locked and induced oscillatory brain dynamics
Journal of Cognitive Neuroscience
 , 
2007
, vol. 
19
 (pg. 
158
-
172
)
Falleti
M. G.
Maruff
P.
Collie
A.
Darby
D. G.
Practice effects associated with the repeated assessment of cognitive function using the CogState battery at 10-minute, one week and one month test-retest intervals
Journal of Clinical and Experimental Neuropsychology
 , 
2006
, vol. 
28
 
7
(pg. 
1095
-
1112
)
Falleti
M. G.
Maruff
P.
Collie
A.
Darby
D. G.
McStephen
M.
Qualitative similarities in cognitive impairment associated with 24 h of sustained wakefulness and a blood alcohol concentration of 0.05%
Journal of Sleep Research
 , 
2003
, vol. 
12
 (pg. 
265
-
274
)
Grant
I.
Adams
K. M.
Neuropsychological assessment of neuropsychiatric disorders
 , 
1996
2nd ed.
Oxford
Oxford University Press
Harrison
J.
Owen
A. M.
Cognitive deficits in brain disorders
 , 
2002
London
Martin Dunitz Publishers
Heinrichs
R. W.
Zakzanis
K. K.
Neurocognitive deficit in schizophrenia: a quantitative review of the evidence
Neuropsychology
 , 
1998
, vol. 
12
 (pg. 
426
-
445
)
Jansma
J. M.
Ramsey
N. F.
Coppola
R.
Kahn
R. S.
Specific versus nonspecific brain activity in a parametric n-Back task
NeuroImage
 , 
2000
, vol. 
12
 (pg. 
688
-
697
)
Koss
E.
Edland
S.
Fillenbaum
G.
Mohs
R.
Clark
C.
Galasko
D.
, et al.  . 
Clinical and neuropsychological differences between patients with earlier and later onset of Alzheimer's disease: A CERAD analysis part XII
Neurology
 , 
1996
, vol. 
46
 (pg. 
136
-
141
)
Lewis
D.
Davis
K. L.
Davis
L.
Charney
D.
Coyle
J. T.
Nemeroff
C.
Neural circuitry approaches to understanding the pathophysiology of schizophrenia
Neuropsychopharmacology – 5th generation of progress
 , 
2002
Washington, DC
American College of Neuropsychopharmacology
Lezak
M. D.
Howieson
D. B.
Loring
D. W.
Neuropsychological assessment
 , 
2004
4th ed.
New York, NY
Oxford University Press
Lovibond
P. F.
Lovibond
S. H.
The structure of negative emotional states: Comparison of the Depression Anxiety Stress Scales (DASS) with the Beck Depression and Anxiety Inventories
Behavior Research Therapy
 , 
1995
, vol. 
33
 (pg. 
335
-
343
)
Luce
R. D.
Response times: Their role in inferring elementary mental organization
 , 
1986
New York, NY
Oxford University Press
Makdissi
M.
Collie
A.
Maruff
P.
Darby
D. G.
Bush
A.
McCrory
P.
, et al.  . 
Computerized cognitive assessment of concussed Australian Rules footballers
British Journal of Sports Medicine
 , 
2001
, vol. 
35
 (pg. 
354
-
360
)
Maruff
P.
Collie
A.
Darby
D.
Weaver-Cargin
J.
Masters
C.
Currie
J.
Subtle memory decline over 12 months in mild cognitive impairment
Dementia and Geriatric Cognitive Disorders
 , 
2004
, vol. 
18
 
3–4
(pg. 
342
-
348
)
Maruff
P.
Currie
J.
Malone
V.
McArthur-Jackson
C.
Mulhall
B.
Benson
E.
Neuropsychological characterization of the AIDS dementia complex and rationalization of a test battery
Archives of Neurology
 , 
1994
, vol. 
51
 (pg. 
689
-
695
)
Maruff
P.
Werth
J.
Giordani
B.
Caveney
A. F.
Feltner
D.
Snyder
P. J.
A statistical method for measuring cognitive response to treatment in individual patients
Psychopharmacology
 , 
2006
, vol. 
186
 (pg. 
7
-
17
)
Mathias
J. L.
Wheaton
P.
Changes in attention and information-processing speed following severe traumatic brain injury: a meta-analytic review
Neuropsychology
 , 
2007
, vol. 
21
 (pg. 
212
-
223
)
Mollica
C.
Maruff
P.
Vance
A.
Development of a statistical approach for classifying treatment response in individual children with ADHD
Human Psychopharmacology
 , 
2004
, vol. 
19
 (pg. 
445
-
456
)
Montgomery
S. A.
Åsberg
M.
A new depression scale designed to be sensitive to change
British Journal of Psychiatry
 , 
1979
, vol. 
134
 (pg. 
382
-
389
)
Murkin
J. M.
Stump
D. A.
Blumenthal
J. A.
Defining dysfunction: group means versus incidence analysis—a statement of consensus
Annals of Thoracic Surgery
 , 
1997
, vol. 
64
 (pg. 
904
-
905
)
Nelson
H. E.
National Adult Reading Test (NART): Test manual
 , 
1982
Windsor
UK Nelson Publishing
Owen
A. M.
Iddon
J. L.
Hodges
J. R.
Summer
B. A.
Robbins
T. W.
Spatial and non-spatial working memory at different stages of Parkinson's disease
Neuropsychologia
 , 
1997
, vol. 
35
 (pg. 
519
-
532
)
Phillips
K.
Bernhard
J.
Adjuvant breast cancer treatment and cognitive function: Current knowledge and research directions
Journal of the National Cancer Institute
 , 
2003
, vol. 
95
 (pg. 
190
-
197
)
Pietrzak
R. H.
Maruff
P.
Mayes
L. C.
Roman
S. A.
Sosa
J. A.
Snyder
P. J.
An examination of the construct validity and factor structure of the Groton Maze Learning Test, a new measure of spatial working memory, learning efficiency, and error monitoring
Archives of Clinical Neuropsychology
 , 
2008
, vol. 
23
 (pg. 
433
-
445
)
Posner
M. I.
Cognitive neuroscience of attention
 , 
2004
New York, NY
Guilford Press
Posner
M. I.
Raichle
M. E.
Images of mind
 , 
1994
New York, NY
Scientific American Books
Price
R. W.
Brew
B. J.
The AIDS dementia complex
The Journal of Infectious Diseases
 , 
1988
, vol. 
158
 (pg. 
1079
-
1083
)
Reichenberg
A.
Harvey
P. D.
Neuropsychological impairments in schizophrenia: Integration of performance-based and brain imaging findings
Psychological Bulletin
 , 
2007
, vol. 
133
 (pg. 
833
-
858
)
Rosselli
M.
Ardila
A.
The impact of culture and education on non-verbal neuropsychological measurements: A critical review
Brain and Cognition
 , 
2003
, vol. 
52
 
3
(pg. 
326
-
333
)
Royall
D. R.
Lauterbach
E. C.
Cummins
J. L.
Reeve
A.
Rummans
T.
Kaufer
D. I.
, et al.  . 
Executive control function: A review of its promise and challenges for clinical research
Journal of Neuropsychiatry and Clinical Neurosciences
 , 
2002
, vol. 
14
 
4
(pg. 
377
-
405
)
Salthouse
T. A.
Davis
H. P.
Organization of cognitive abilities and neuropsychological variables across the lifespan
Developmental Review
 , 
2006
, vol. 
26
 (pg. 
31
-
54
)
Selnes
O. A.
Galai
N.
Bacellar
H.
Miller
E. N.
Becker
J. T.
Wesch
J.
, et al.  . 
Cognitive performance after progression to AIDS: a longitudinal study from the Multicentre AIDS cohort study
Neurology
 , 
1995
, vol. 
45
 (pg. 
267
-
275
)
Shallice
T.
Marzocchi
G. M.
Coser
S.
Del Savio
M.
Meuter
R. F.
Rumiati
R. I.
Executive function profile in children with attention deficit hyperactivity disorder
Developmental Neuropsychology
 , 
2002
, vol. 
21
 
1
(pg. 
43
-
71
)
Silbert
B. S.
Maruff
P.
Evered
L. A.
Scott
D. A.
Kalpokas
M.
Martin
K. J.
, et al.  . 
Detection of cognitive decline after coronary surgery: a comparison of computerized and conventional tests
British Journal of Anaesthesia
 , 
2004
, vol. 
92
 (pg. 
814
-
820
)
Spreen
O.
Strauss
E.
A Compendium of neuropsychological tests: Administration, norms, and commentary
 , 
1998
2nd ed
Oxford
Oxford University Press
Squire
L. R.
Kandel
E. R.
Memory: From mind to molecules
 , 
2000
New York, NY
W.H. Freeman & Co
The Mild Traumatic Brain Injury Committee of the Head Injury Interdisciplinary Special Interest Group of the American Congress of Rehabilitation Medicine
Definition of mild traumatic brain injury
Journal of Head Trauma Rehabilitation
 , 
1993
, vol. 
8
 (pg. 
86
-
87
)
Thein
H.
Maruff
P.
Krahn
M.
Kaldor
J.
Koorey
D.J.
Brew
B.
, et al.  . 
Cognitive function, mood and health-related quality of life in hepatitis C virus (HCV)-monoinfected and HIV/HCV-coinfected individuals commencing HCV treatment
HIV Medicine
 , 
2007
, vol. 
a 8
 (pg. 
192
-
202
)
Thein
H. H.
Maruff
P.
Krahn
M. D.
Kaldor
J. M.
Koorey
D. J.
Brew
B. J.
, et al.  . 
Improved cognitive function as a consequence of hepatitis C virus treatment
HIV Medicine
 , 
2007
, vol. 
b 8
 
8
(pg. 
520
-
528
)
Uzzell
B. P.
Ponton
M. O.
Ardila
A.
International handbook of cross-cultural neuropsychology
 , 
2007
New York, NY
Routledge
Verhagen
P.
Cerella
J.
Aging, executive control, and attention: a review of meta-analyses
Neuroscience and Biobehavioral Reviews
 , 
2002
, vol. 
26
 (pg. 
846
-
857
)
Walsh
K.
Darby
D. G.
Neuropsychology
 , 
2002
4th ed.
London
Churchill Livingstone
Winer
B. J.
Statistical principles in experimental design
 , 
1971
London
McGraw-Hill
Zakzanis
K. K.
Statistics to tell the truth, the whole truth, and nothing but the truth: formulae, illustrative numerical examples, and heuristic interpretation of effect sizes analyses for neuropsychological researchers
Archives of Clinical Neuropsychology
 , 
2001
, vol. 
16
 (pg. 
653
-
667
)