Abstract

The study purpose was to compare the diagnostic utility of the Brief Cognitive Status Exam (BCSE) to that of the Mini-Mental State Examination (MMSE) and to develop equated scores to facilitate comparisons. One hundred and eighty-two patients underwent cognitive evaluation and were placed into three groups: dementia (DEM), cognitive impairment, no dementia (CIND), and no cognitive impairment (NCI). One hundred and eighty-two healthy controls from the BCSE standardization sample served as a comparison group. On both measures, the DEM group obtained significantly lower scores than the other two groups, and the CIND group scored significantly lower than the NCI group. The BCSE was more sensitive in all clinical groups, although at extremely low scores, the two tests displayed similar sensitivity. Results indicate the BCSE has diagnostic utility as a cognitive screening measure in a mixed clinical sample and is more sensitive at detecting cognitive impairment, particularly milder levels, than the MMSE.

Introduction

Screening for cognitive impairment is important in clinical and research settings. In clinical settings, complaints of cognitive problems are common, particularly in older adults and individuals with psychiatric conditions. If present, cognitive dysfunction may negatively affect health outcomes as cognitively impaired patients may fail to remember (or remember incorrectly) important details about their condition, treatment regimen, and/or healthcare providers' recommendations; they may also lack insight into the need for treatment (Gaviria, Pliskin, & Kney, 2011; Han et al., 2011; Martinez-Aran et al., 2009; Valldeoriola et al., 2011; Zogg et al., 2010). In research settings, such as clinical trials, cognitive impairment often is an inclusion or an exclusion criterion, depending on the study purpose, and the need to identify appropriate study subjects is important for ensuring the best chance of detecting the targeted outcome or treatment effect. Because cognitive outcomes are increasingly recognized as important in treatment studies, the need for brief yet sensitive assessment tools is critical.

Although there are multiple brief cognitive screening instruments available for use with adults aged 60 and older for dementia screening (Brodaty, Fay, Gibson, & Burns, 2006; Ismail, Rajji, & Shulman, 2010), few have been developed for use with a broader age range. In fact, most of the instruments currently used with patients under age 60 were developed for older adults and then applied to younger adults. For example, the Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005) was developed and validated on older adults with mild cognitive impairment (MCI) and Alzheimer disease (AD) but has been applied to a variety of younger populations, including aneurysmal subarachnoid hemorrhage (Schweizer, Al-Khindi, & Macdonald, 2012), brain tumor (Olson, Chhanabhai, & McKenzie, 2008), psychiatric inpatients (Jones, Perlman, Hirdes, & Scott, 2010), and stroke (Dong et al., 2010; Pendlebury, Cuthbertson, Welch, Mehta, & Rothwell, 2010). However, using tests developed primarily for dementia screening in older adults may lack sensitivity for detecting cognitive impairment in younger individuals. Furthermore, broad age-norms for most screening instruments, including the MoCA, are still relatively lacking. Thus, there is a need for a cognitive screening measure that was developed to detect impairment in a wide age range of individuals and is adequately normed.

The purpose of the current study was to compare the diagnostic utility of the Brief Cognitive Status Exam (BCSE; Wechsler, 2009), a relatively new screening measure, to that of the Mini-Mental State Examination (MMSE; Folstein, Folstein, & McHugh, 1975) in a mixed-diagnosis clinical sample. The BCSE was standardized and normed on a nationally representative sample of 1,400 adults aged 16–90 years and uses a multivariate base rate model to classify an individual's level of cognitive functioning. The multivariate base rate model takes advantage of the tendency for healthy individuals to have low performance on one or two cognitive measures while patient populations typically have multiple low scores. Using this model can reduce false positive errors while maintaining sensitivity to clinically meaningful cognitive impairment. Additionally, the BCSE assesses multiple cognitive domains, including executive functioning. The MMSE was selected as the “gold standard” comparison because it remains the most widely used cognitive screening test (Iracleous et al., 2010; Milne, Culverwell, Guss, Tuppen, & Whelton, 2008; Shulman et al., 2006) despite criticisms of limited sensitivity, especially in patients with MCI and/or executive dysfunction, and false positive results in persons of older age and lower education (Ismail et al., 2010; Lonie, Tierney, & Ebmeier, 2009; Mitchell, 2009).

Method

Participants

Participants were 182 patients (68% men) who underwent cognitive evaluation in one of the following settings: a Veterans Administration (VA) hospital (n = 110), an academic medical center (n = 43), a state long-term care facility (n = 19), and a private nursing home (n = 10). No formal inclusion or exclusion criteria were delineated; consecutive patients who were referred for cognitive evaluation and deemed capable of completing the BCSE and the MMSE were approached for study participation. The average age of the sample was 62.5 years (SD = 13.8), and the average education was 13.3 years (SD = 3.0). Approximately 66% were non-Hispanic White, 21% were Hispanic, and 13% were African American. Average estimated premorbid IQ of the sample based on sight-word reading ability and/or demographics was in the average range (i.e., standard score = 95.8; SD = 12.0), as was current IQ as estimated by the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV; Wechsler, 2008) General Ability Index (GAI) (i.e., standard score = 92.4; SD = 13.7).

The sample represented a wide age range (22–97 years old) and a variety of clinical diagnoses that were grouped into three cognitive impairment categories based on Diagnostic and Statistical Manual of Mental Disorders-Fourth Edition-Text Revision (DSM-IV-TR) diagnostic criteria: dementia (DEM; 35%), cognitive impairment, no dementia (CIND; 46%), and no cognitive impairment (NCI; 19%). Determination of which cognitive impairment category to place each participant was made by the examining clinician based on results of the cognitive evaluation, which included information from a clinical interview, available medical records, and neuropsychological measures. Importantly, placement into cognitive impairment categories was not based on performance on the BCSE or the MMSE. Etiologies of cognitive impairment in the DEM and CIND groups were: neurodegenerative diseases (19%), neurologic medical conditions (18%), MCI (15%), stroke (13%), non-neurologic (usually multiple) chronic medical conditions (12%), psychiatric disorders (10%), traumatic brain injury (7%), and multiple sclerosis (6%). Comorbid psychiatric disorders, with the exception of schizophrenia spectrum disorders, were common and spread relatively evenly across all three groups, with depressive disorder noted in 42%, post-traumatic stress disorder in 14%, other anxiety disorders in 26%, and other mental health disorders in 15%. Schizophrenia spectrum disorders comprised 5% of the sample and were significantly more common in the CIND group.

In order to compare the sensitivity of the BCSE in the patient samples to individuals in the general population, 182 healthy controls (HCs) from the BCSE standardization sample were selected to match the demographic characteristics of the clinical groups. The average age of the HC group was 62.4 years (SD = 14.3), and the education level was 17.3% 11 years or less, 30.8% 12 years, 24.9% 13 to 15 years, and 27.0% 16 or more years. Approximately 70% were non-Hispanic White, 16% were Hispanic, 13% were African American, and 1% was “other.” Average estimated premorbid IQ and current ability levels of the control sample were in the average range (premorbid = 100.2 ± 15.3 and current = 99.9 ± 15.2) using the same measures obtained on the clinical sample.

Measures

The BCSE was developed as a cognitive screening instrument designed to assess general cognitive functioning in approximately 10 min. The BCSE consists of 12 items assessing orientation to time, time estimation, mental control, incidental recall, clock drawing, inhibition, and verbal fluency. For each item set, the examinee's performance is compared with the general population. Applying multivariate base rates across the subtests distinguishes typical from atypical performance (Brooks, Holdnack, & Iverson, 2011; Brooks, Iverson, Holdnack, & Feldman, 2008). Raw scores are converted into weighted raw scores (i.e., a modified ability score) based on the percentage of the normative sample achieving specific raw scores (i.e., ≥25%, 10%–24%, 3%–9%, and ≤2%). As part of the multivariate base rate model, item test scores focus only on impaired performance and not the full ability range. Additional weighting is made on items requiring processing speed and mental control to increase the sensitivity of the measure, since these cognitive skills are often impaired in persons with cognitive dysfunction. The sum of the weighted item scores represents a multivariate base rate of low scores. Having one or two low scores is not necessarily indicative of cognitive difficulties (i.e., a higher score on the BCSE total), whereas having many low scores is quite rare (i.e., a low score on BCSE). Total weighted raw scores range from 0 to 58 and are adjusted based on four age bands (16–29, 30–44, 45–69, 70–90) and five categories of educational attainment (≤8 years, 9–11 years, high school diploma or GED, 13–15 years, and ≥16 years) to arrive at classification levels representing percentiles of cases within the specified age and education bands. Classification levels are “Average” (25th through 100th percentiles), “Low average” (10th through 24th percentiles), “Borderline” (5th through 9th percentiles), “Low” (2nd through 4th percentiles), and “Very low” (<2nd percentile).

Normative data are based on a national sample of 1,400 individuals stratified on age, sex, race/ethnicity, education level, and geographic region according to data collected in 2005 by the U.S. Bureau of the Census. Stability of the BCSE is excellent, with test–retest decision consistency in classification levels ranging from 0.96 to 0.98 across the four age bands, and initial validity studies with selected clinical populations (e.g., probable AD, major depressive disorder, traumatic brain injury) showed promising sensitivity, specificity, and positive (PPV) and negative predictive values (NPV; Wechsler, 2009). Although BCSE classification levels are not diagnostic by themselves, scores in the “Very low” range are often obtained by examinees diagnosed with dementia or at least mild intellectual disability, and scores in the “Low” or “Very low” ranges are obtained by less than 5% of the normative sample.

The MMSE is a well-known test of global cognitive functioning that provides a brief snapshot of aspects of orientation, repetition, attention, language, visuoconstruction, and short-term recall. Scores are influenced by age and education level, and normative corrections have long been available (Crum, Anthony, Bassett, & Folstein, 1993; Folstein, Folstein, & McHugh, 2002; Tombaugh, McDowell, Kristjansson, & Hubley, 1996), allowing for the use of demographically corrected T-scores. However, many clinicians and researchers continue to use the traditional cutoff score method (Mast & Gerstenecker, 2010). Recommended cutoff scores denoting impairment have ranged from ≤24 for patients with dementia or stroke (Dong et al., 2010; Toglia, Fitzgerald, O'Dell, Mastrogiovanni, & Lin, 2011) to ≤26 for persons with higher levels of education (O'Bryant et al., 2008).

In addition to the BCSE and MMSE, participants were administered some combination of the following cognitive measures: Animal Naming (Benton, 1969), Boston Naming Test (Kaplan, Goodglass, & Weintraub, 2001), California Verbal Learning Test (Delis, Kramer, Kaplan, & Ober, 2000), Controlled Oral Word Association Test (Benton, Sivan, Hamsher, Varney, & Spreen 1994), Finger Tapping Test (Reitan, & Wolfson, 1985), Grooved Pegboard (Klove, 1963), Repeatable Battery for the Assessment of Neuropsychological Status (Randolph, 1998), Rey Complex Figure Test (Rey, 1941), Stroop Color and Word Test (Stroop; Golden & Freshwater, 2002), Test of Memory Malingering (Tombaugh, 1996), Texas Functional Living Scale (Cullum et al., 2001), Trail Making Test (Reitan & Wolfson, 1985), WAIS-IV (Wechsler, 2008), Wechsler Memory Scale-Fourth Edition (Wechsler, 2009), Wechsler Test of Adult Reading (Wechsler, 2001), and the Wisconsin Card Sorting Test (Kongs, Thompson, Iverson, & Heaton, 2000). Because cognitive evaluations were tailored individually based on the referral question and some participants were severely demented, not every participant was administered the same neuropsychological measures.

Procedures

All participants in the clinical sample were administered the BCSE and the MMSE at the beginning of their clinical evaluation. Administration of the BCSE and the MMSE was counterbalanced to minimize order and fatigue effects. Study procedures were approved by the institutional review board at each site. For the HC participants, the BCSE was administered at either the beginning of a testing session for standardization of memory and IQ measures or after IQ testing was completed but prior to memory assessment (refer to Wechsler, 2009, for more information about standardization procedures).

Statistical Analysis

Descriptive statistics were used to characterize the study sample, and Spearman's correlation coefficients were conducted to show the relationships between BCSE and MMSE scores and between each measure and demographic variables. Spearman's correlation coefficients were preferred over Pearson's in this case because distributions of both measures were negatively skewed. One-way analyses of covariance were conducted to compare the mean BCSE and MMSE scores among the three cognitive impairment groups. Numbers of participants obtaining scores below the two commonly used MMSE cutoffs (i.e., ≤24 and ≤26) were grouped by the BCSE classification level to illustrate overlap in the categorization of cognitive impairment, and BCSE and MMSE scores were equated by equipercentiles and also by sensitivity to dementia in order to provide a means for comparing scores of the two screening tests. Diagnostic statistics, that is, sensitivity (the proportion of impaired individuals correctly classified as impaired), specificity (the proportion of unimpaired individuals correctly classified as unimpaired), PPVs (the probability that the individual classified as impaired actually is impaired), NPVs (the probability that the individual classified as unimpaired is actually unimpaired), and odds ratios (ORs), were used to compare the functioning of the two tests among the three clinical groups and for the BCSE between the clinical and HC groups. Population estimates based on the theoretical distribution of scores in the general population were used to estimate the sensitivity of the tests to distinguish impaired from normal cognitive functioning. Significance was set at .05 for all analyses and was two-tailed.

Results

Demographic characteristics, including IQ estimates and BCSE and MMSE scores, for the total sample and each cognitive impairment group are displayed in Table 1. There were significant group differences in age, F (2) = 12.8; p < .001, and education, F (2) = 6.0; p = .003, with the DEM group being significantly older and less educated than the other two groups, which did not differ significantly from each other on either variable. The three groups did not differ significantly in gender, χ2(2) = 0.70; p = .71, or racial/ethnic composition, χ2(6) = 2.2; p = .90. There were significant group differences in estimated premorbid IQ, F(2, 139) = 5.7; p = .004, and WAIS-IV GAI scores, F(2, 120) = 21.2; p < .001. In the former case, the DEM group obtained a significantly lower estimated premorbid IQ than the other two groups, which did not differ significantly from each other, although there were no group differences after controlling for education. In the latter case, all three groups differed significantly from each other, with the DEM group obtaining a significantly lower WAIS-IV GAI score than the other two groups, and the CIND group obtaining a significantly lower WAIS-IV GAI score than the NCI group. Group differences in current IQ remained significant after controlling for education.

Table 1.

Demographic characteristics and BCSE and MMSE total raw scores by group

 DEM (n = 64) CIND (n = 84) NCI (n = 34) Total sample (n = 182) 
Age (years; M ± SD69.0 (11.2)a 58.3 (12.1)b 60.7 (17.4)b 62.5 (13.8) 
Sex (%) 
 Men 72 66 68 68 
 Women 28 34 32 32 
Education (years; M ± SD12.4 (3.5)a 13.5 (2.4)b 14.5 (2.8)b 13.3 (3.0) 
 11 years or less (%) 31 17 20 
 12 years or GED (%) 28 27 24 27 
 13–15 years (%) 19 30 26 25 
 16 years or more (%) 22 26 41 28 
Ethnicity (%) 
 Non-Hispanic white 66 64 70 66 
 Hispanic 22 23 15 21 
 African American 12 13 15 13 
Est premorbid IQ (M ± SD)a 91.6 (12.5)a 96.5 (11.4)b 100.9 (10.2)b 95.8 (12.0) 
WAIS-IV GAI (M ±SD)b 81.3 (11.1)a 94.6 (12.0)b 101.0 (12.7)c 92.4 (13.7) 
BCSE score (M ± SD30.1 (12.0)a 44.9 (10.0)b 49.4 (7.4)c 40.5 (13.0) 
MMSE score (M ± SD23.4 (5.1)a 27.1 (3.5)b 28.7 (1.3)c 26.1 (4.4) 
 DEM (n = 64) CIND (n = 84) NCI (n = 34) Total sample (n = 182) 
Age (years; M ± SD69.0 (11.2)a 58.3 (12.1)b 60.7 (17.4)b 62.5 (13.8) 
Sex (%) 
 Men 72 66 68 68 
 Women 28 34 32 32 
Education (years; M ± SD12.4 (3.5)a 13.5 (2.4)b 14.5 (2.8)b 13.3 (3.0) 
 11 years or less (%) 31 17 20 
 12 years or GED (%) 28 27 24 27 
 13–15 years (%) 19 30 26 25 
 16 years or more (%) 22 26 41 28 
Ethnicity (%) 
 Non-Hispanic white 66 64 70 66 
 Hispanic 22 23 15 21 
 African American 12 13 15 13 
Est premorbid IQ (M ± SD)a 91.6 (12.5)a 96.5 (11.4)b 100.9 (10.2)b 95.8 (12.0) 
WAIS-IV GAI (M ±SD)b 81.3 (11.1)a 94.6 (12.0)b 101.0 (12.7)c 92.4 (13.7) 
BCSE score (M ± SD30.1 (12.0)a 44.9 (10.0)b 49.4 (7.4)c 40.5 (13.0) 
MMSE score (M ± SD23.4 (5.1)a 27.1 (3.5)b 28.7 (1.3)c 26.1 (4.4) 

Notes: DEM = dementia; CIND = cognitive impairment, no dementia; NCI = no cognitive impairment; GED = General Equivalency Diploma; SD = Standard Deviation; WAIS-IV GAI = Wechsler Adult Intelligence Scale-Fourth Edition General Ability Index; MMSE = Mini Mental State Examination; BCSE = Brief Cognitive Status Examination.

Variables with differing superscripts in each row were significantly different from one another. Specifically, a differs significantly from b, and where applicable, both a and b differ significantly from c.

an = 45 in the DEM group, 70 in the CIND group, and 27 in the NCI group.

bn = 32 in the DEM group, 68 in the CIND group, and 23 in the NCI group.

Relationship of BCSE and MMSE Scores

The average BCSE total weighted raw score for the sample was 40.5 (SD = 13.0) with a range from 1 to 58, and the average MMSE total raw score was 26.1 (SD = 4.4), with a range from 5 to 30. BCSE and MMSE scores correlated significantly with each other at .80 (p < .001) and both correlated significantly with age (−.27 and −.23, respectively; p < .001 and .002, respectively), education (.42 and .41, respectively; both ps < .001), estimated premorbid IQ (.40 and .41, respectively; both ps < .001), and estimated current IQ (.63 and .47, respectively; both ps < .001), but not sex (.08 and .02, respectively) or ethnicity (−.09 and −.12, respectively). There were significant group differences on both the BCSE and MMSE total scores by group even after adjusting for age and education—F(2, 177) = 32.9 and F(2, 177) = 12.5, respectively (both ps < .001). On both measures, the DEM group obtained significantly lower scores than the other two groups (all ps < .001), and the CIND group obtained significantly lower scores than the NCI group (both ps = .04) (Table 1). A larger effect size was found for the BCSE, however, with a partial eta squared of 0.27 for the BCSE and 0.12 for the MMSE.

Table 2 shows numbers of patients within each BCSE classification level according to commonly used MMSE cutoff scores of ≤24 and ≤26. As seen in Table 2, when using the cutoff score of ≤24 as indicative of cognitive impairment, the BCSE identified an additional 48 patients (26%) as probably cognitively impaired (i.e., classified in the “Low” and “Very low” ranges) than did the MMSE, whereas the MMSE identified only 6 patients (3%) as impaired that were not classified as impaired by the BCSE. When applying the cutoff score of ≤26, the BCSE identified an additional 30 patients (16%) as probably cognitively impaired, and the MMSE identified 14 patients (8%) as impaired that were not classified as impaired on the BCSE.

Table 2.

Number of patients within BCSE classification levels by MMSE cutoff scores

BCSE classification level MMSE ≤24 (impaired) MMSE >24 (not impaired) MMSE ≤26 (impaired) MMSE >26 (not impaired) 
Normal (25th–100th percentiles) 49 48 
Low average (10–24th percentiles) 25 21 
Borderline (5th–9th percentiles) 15 12 
Low (2nd–4th percentiles) 33 20 22 
Very low (<2nd percentile) 30 15 37 8 
BCSE classification level MMSE ≤24 (impaired) MMSE >24 (not impaired) MMSE ≤26 (impaired) MMSE >26 (not impaired) 
Normal (25th–100th percentiles) 49 48 
Low average (10–24th percentiles) 25 21 
Borderline (5th–9th percentiles) 15 12 
Low (2nd–4th percentiles) 33 20 22 
Very low (<2nd percentile) 30 15 37 8 

Notes: BCSE = Brief Cognitive Status Exam; MMSE = Mini-Mental Status Exam; Numbers bolded and italicized indicate the additional cases of likely cognitive impairment identified by the BCSE that were missed by the MMSE.

Because the BCSE is a relatively new screening instrument clinicians will not have as much familiarity with interpreting performance on this test. Therefore, BCSE weighted raw scores were equated to MMSE raw scores, presented in Table 3, to facilitate the interpretation of BCSE weighted raw scores. For example, BCSE scores of 35–38 are equivalent to an MMSE score of 26 and BCSE scores of 28–32 equate to an MMSE score of 24. Clinicians using specific MMSE cut scores could apply the equated BCSE score ranges and obtain similar results. BCSE and MMSE scores equated by sensitivity to dementia can be viewed in Table 4. For example, an MMSE cutoff of 26 has sensitivity of 70% to dementia as does a BCSE cutoff of 36. A BCSE score range of 35–38 is associated with an MMSE score of 26; the sensitivity range for those BCSE scores is 65%–78%.

Table 3.

BCSE and MMSE raw scores equated by percentiles

BCSE MMSE 
2–3 
4–5 
10 
11 
 12 
10 13 
 14 
 15 
11 16 
12–13 17 
14 18 
15–16 19 
17–20 20 
21 21 
22–23 22 
24–27 23 
28–32 24 
33–34 25 
35–38 26 
39–41 27 
42–46 28 
47–51 29 
52–55 30 
BCSE MMSE 
2–3 
4–5 
10 
11 
 12 
10 13 
 14 
 15 
11 16 
12–13 17 
14 18 
15–16 19 
17–20 20 
21 21 
22–23 22 
24–27 23 
28–32 24 
33–34 25 
35–38 26 
39–41 27 
42–46 28 
47–51 29 
52–55 30 

Notes: BCSE = Brief Cognitive Status Exam; MMSE = Mini-Mental Status Exam.

Table 4.

BCSE and MMSE equated by sensitivity to dementia

Sensitivity (%) BCSE Raw MMSE Raw 
90 45 28 
80 40 27 
70 36 26 
60 34 25 
50 32 24 
40 27 23 
30 22 22 
20 19 20 
10 11 17 
Sensitivity (%) BCSE Raw MMSE Raw 
90 45 28 
80 40 27 
70 36 26 
60 34 25 
50 32 24 
40 27 23 
30 22 22 
20 19 20 
10 11 17 

Notes: BCSE = Brief Cognitive Status Exam; MMSE = Mini-Mental Status Exam.

Comparison of BCSE to MMSE in the Specific Clinical Samples

Table 5 shows sensitivity of the BCSE and MMSE in the DEM, CIND, and NCI groups. In this table, BCSE age- and education-adjusted classifications are compared with age- and education-adjusted MMSE T-scores. The classification levels applied were comparable with the percentage of HCs that would be expected to have impaired scores (<25%, <10%, <5%, and <2%) on both tests. The BCSE was generally more sensitive in all of the clinical groups, although at extremely low scores, the two tests were similar in their sensitivity to dementia (0.48 vs. 0.47). At the recommended BCSE classification of “Low and below” (<5% false positive rate), the BCSE had a sensitivity of 0.83 for detection of dementia compared with 0.55 for the MMSE. Because some test statistics are affected by the base rate of the disorder (PPV and NPV), Table 6 shows sensitivity and specificity of each test at two base rates, 50% and 10%. Data show that the BCSE has superior diagnostic statistics for sensitivity, PPV, NPV, and OR at all classification levels; however, the two tests are essentially equivalent for the “Very low” classification level. The BCSE had optimal diagnostic statistics at the “Low and below” level consistent with previous research in dementia populations (Wechsler, 2009); however, the “Borderline” level had acceptable false positive rates with slightly better sensitivity. The MMSE operated best in terms of sensitivity and specificity at the “Borderline” level.

Table 5.

Sensitivity (95% CI) of BCSE and MMSE to cognitive groups using age- and education-adjusted classifications

BCSE classification or MMSE T-score Dementia
 
CIND
 
NCI
 
BCSE MMSE BCSE MMSE BCSE MMSE 
Low average and below (<25%) 0.95 (0.91–0.99) 0.73 (0.69–0.77) 0.65 (0.62–0.68) 0.39 (0.36–0.42) 0.47 (0.42–0.52) 0.09 (0.04–0.14) 
Borderline and below (< 10%) 0.88 (0.84–0.92) 0.64 (0.60–0.68) 0.48 (0.45–0.51) 0.29 (0.26–0.32) 0.21 (0.16–0.26) 0.03 (0–0.08) 
Low and below (<5%) 0.83 (0.79–0.87) 0.55 (0.51–0.59) 0.35 (0.32–0.38) 0.24 (0.21–0.27) 0.15 (0.10–0.20) 0.00 (0–0.05) 
Very low and below (<2%) 0.48 (0.44–0.52) 0.47 (0.43–0.51) 0.12 (0.09–0.15) 0.17 (0.14–0.20) 0.12 (0.07–0.17) 0.00 (0–0.05) 
BCSE classification or MMSE T-score Dementia
 
CIND
 
NCI
 
BCSE MMSE BCSE MMSE BCSE MMSE 
Low average and below (<25%) 0.95 (0.91–0.99) 0.73 (0.69–0.77) 0.65 (0.62–0.68) 0.39 (0.36–0.42) 0.47 (0.42–0.52) 0.09 (0.04–0.14) 
Borderline and below (< 10%) 0.88 (0.84–0.92) 0.64 (0.60–0.68) 0.48 (0.45–0.51) 0.29 (0.26–0.32) 0.21 (0.16–0.26) 0.03 (0–0.08) 
Low and below (<5%) 0.83 (0.79–0.87) 0.55 (0.51–0.59) 0.35 (0.32–0.38) 0.24 (0.21–0.27) 0.15 (0.10–0.20) 0.00 (0–0.05) 
Very low and below (<2%) 0.48 (0.44–0.52) 0.47 (0.43–0.51) 0.12 (0.09–0.15) 0.17 (0.14–0.20) 0.12 (0.07–0.17) 0.00 (0–0.05) 

Notes: BCSE = Brief Cognitive Status Exam; MMSE = Mini-Mental Status Exam; CIND = cognitive impairment, no dementia; NCI = no cognitive impairment.

Table 6.

Diagnostic statistics (95% CI) for BCSE and MMSE classifications for dementia versus population estimates using 50% and 10% base rate of dementia

Classification BCSE
 
MMSE
 
Sens Spec PPV NPV Odds Ratio Sens Spec PPV NPV Odds ratio 
Base rate of dementia (50%) 
 ≤Low average 0.95 (±0.04) 0.75 (±0.04) 0.79 (±0.04) 0.94 (±0.04) 61.00 0.73 (±0.04) 0.75 (±0.04) 0.75 (±0.04) 0.74 (±0.04) 8.29 
 ≤Borderline 0.88 (±0.04) 0.89 (±0.04) 0.89 (±0.04) 0.88 (±0.04) 57.00 0.64 (±0.04) 0.89 (±0.04) 0.85 (±0.04) 0.71 (±0.04) 14.52 
 ≤Low 0.83 (±0.04) 0.97 (±0.04) 0.96 (±0.04) 0.85 (±0.04) 149.36 0.55 (±0.04) 0.97 (±0.04) 0.95 (±0.04) 0.68 (±0.04) 37.41 
 Very low 0.48 (±0.04) 0.98 (±0.04) 0.97 (±0.04) 0.66 (±0.04) 59.20 0.47 (±0.04) 0.98 (±0.04) 0.97 (±0.04) 0.65 (±0.04) 55.59 
Base rate of dementia (10%) 
 ≤Low average 0.95 (±0.02) 0.75 (±0.02) 0.30 (±0.02) 0.99 (±0.02) 61.00 0.73 (±0.02) 0.75 (±0.02) 0.25 (±0.02) 0.96 (±0.02) 8.29 
 ≤Borderline 0.88 (±0.02) 0.89 (±0.02) 0.47 (±0.02) 0.98 (±0.02) 57.00 0.64 (±0.02) 0.89 (±0.02) 0.39 (±0.02) 0.96 (±0.02) 14.52 
 ≤Low 0.83 (±0.02) 0.97 (±0.02) 0.75 (±0.02) 0.98 (±0.02) 149.36 0.55 (±0.02) 0.97 (±0.02) 0.66 (±0.02) 0.95 (±0.02) 37.41 
 Very low 0.48 (±0.02) 0.98 (±0.02) 0.78 (±0.02) 0.95 (±0.02) 59.20 0.47 (±0.02) 0.98 (±0.02) 0.77 (±0.02) 0.94 (±0.02) 55.59 
Classification BCSE
 
MMSE
 
Sens Spec PPV NPV Odds Ratio Sens Spec PPV NPV Odds ratio 
Base rate of dementia (50%) 
 ≤Low average 0.95 (±0.04) 0.75 (±0.04) 0.79 (±0.04) 0.94 (±0.04) 61.00 0.73 (±0.04) 0.75 (±0.04) 0.75 (±0.04) 0.74 (±0.04) 8.29 
 ≤Borderline 0.88 (±0.04) 0.89 (±0.04) 0.89 (±0.04) 0.88 (±0.04) 57.00 0.64 (±0.04) 0.89 (±0.04) 0.85 (±0.04) 0.71 (±0.04) 14.52 
 ≤Low 0.83 (±0.04) 0.97 (±0.04) 0.96 (±0.04) 0.85 (±0.04) 149.36 0.55 (±0.04) 0.97 (±0.04) 0.95 (±0.04) 0.68 (±0.04) 37.41 
 Very low 0.48 (±0.04) 0.98 (±0.04) 0.97 (±0.04) 0.66 (±0.04) 59.20 0.47 (±0.04) 0.98 (±0.04) 0.97 (±0.04) 0.65 (±0.04) 55.59 
Base rate of dementia (10%) 
 ≤Low average 0.95 (±0.02) 0.75 (±0.02) 0.30 (±0.02) 0.99 (±0.02) 61.00 0.73 (±0.02) 0.75 (±0.02) 0.25 (±0.02) 0.96 (±0.02) 8.29 
 ≤Borderline 0.88 (±0.02) 0.89 (±0.02) 0.47 (±0.02) 0.98 (±0.02) 57.00 0.64 (±0.02) 0.89 (±0.02) 0.39 (±0.02) 0.96 (±0.02) 14.52 
 ≤Low 0.83 (±0.02) 0.97 (±0.02) 0.75 (±0.02) 0.98 (±0.02) 149.36 0.55 (±0.02) 0.97 (±0.02) 0.66 (±0.02) 0.95 (±0.02) 37.41 
 Very low 0.48 (±0.02) 0.98 (±0.02) 0.78 (±0.02) 0.95 (±0.02) 59.20 0.47 (±0.02) 0.98 (±0.02) 0.77 (±0.02) 0.94 (±0.02) 55.59 

Notes: BCSE = Brief Cognitive Status Exam; MMSE = Mini-Mental Status Exam; PPV = positive predictive values; NPV = negative predictive values; Sens = sensitivity; Spec = specificity.

Diagnostic Statistics for BCSE Raw Scores

Table 7 displays the diagnostic statistics for the three clinical groups compared with matched HC participants. In the DEM group, a raw score cutoff of 44 yielded similar sensitivity to the “Borderline” classification range; however, the false positive rate was about 8% higher. At a false positive rate of 10% (i.e., raw score of 38), the sensitivity of the test was only 72%. For both the CIND and NCI groups, the false positive rate was much lower than observed for the dementia matched controls. The sensitivity and specificity data for a cutoff of 44 were similar to using the “Borderline” classification level with a marginally lower false positive rate.

Table 7.

Diagnostic statistics (95% CI) for BCSE raw scores in dementia, CIND, and NCI groups

BCSE raw score Sens Spec PPV NPV Odds ratio 
Dementia versus Matched Controls 
 BCSE ≤44 0.89 (±0.04) 0.81 (±0.04) 0.83 (±0.04) 0.88 (±0.04) 35.30 
 BCSE ≤42 0.86 (±0.04) 0.84 (±0.04) 0.85 (±0.04) 0.86 (±0.04) 33.00 
 BCSE ≤38 0.72 (±0.04) 0.91 (±0.04) 0.88 (±0.04) 0.76 (±0.04) 24.70 
 BCSE ≤36 0.70 (±0.04) 0.92 (±0.04) 0.90 (±0.04) 0.76 (±0.04) 27.95 
 BCSE ≤28 0.44 (±0.04) 0.98 (±0.04) 0.97 (±0.04) 0.64 (±0.04) 49.00 
CIND versus Matched Controls 
 BCSE ≤44 0.44 (±0.03) 0.93 (±0.03) 0.86 (±0.03) 0.62 (±0.03) 10.21 
 BCSE ≤42 0.31 (±0.03) 0.95 (±0.03) 0.87 (±0.03) 0.58 (±0.03) 8.96 
 BCSE ≤38 0.18 (±0.03) 0.98 (±0.03) 0.88 (±0.03) 0.54 (±0.03) 8.91 
 BCSE ≤36 0.14 (±0.03) 0.99 (±0.03) 0.92 (±0.03) 0.54 (±0.03) 13.83 
 BCSE ≤28 0.05 (±0.03) 1.00 (±0.03) 1.00 (±0.03) 0.53 (±0.03) N/A 
NCI versus Matched Controls 
 BCSE ≤44 0.24 (±0.05) 0.91 (±0.05) 0.73 (±0.05) 0.54 (±0.05) 3.18 
 BCSE ≤42 0.21 (±0.05) 0.94 (±0.05) 0.78 (±0.05) 0.54 (±0.05) 4.14 
 BCSE ≤38 0.12 (±0.05) 0.97 (±0.05) 0.80 (±0.05) 0.52 (±0.05) 2.20 
 BCSE ≤36 0.09 (±0.05) 1.00 (±0.05) 1.00 (±0.05) 0.52 (±0.05) N/A 
 BCSE ≤28 0.00 (±0.05) 1.00 (±0.05) 1.00 (±0.05) 0.50 (±0.05) N/A 
BCSE raw score Sens Spec PPV NPV Odds ratio 
Dementia versus Matched Controls 
 BCSE ≤44 0.89 (±0.04) 0.81 (±0.04) 0.83 (±0.04) 0.88 (±0.04) 35.30 
 BCSE ≤42 0.86 (±0.04) 0.84 (±0.04) 0.85 (±0.04) 0.86 (±0.04) 33.00 
 BCSE ≤38 0.72 (±0.04) 0.91 (±0.04) 0.88 (±0.04) 0.76 (±0.04) 24.70 
 BCSE ≤36 0.70 (±0.04) 0.92 (±0.04) 0.90 (±0.04) 0.76 (±0.04) 27.95 
 BCSE ≤28 0.44 (±0.04) 0.98 (±0.04) 0.97 (±0.04) 0.64 (±0.04) 49.00 
CIND versus Matched Controls 
 BCSE ≤44 0.44 (±0.03) 0.93 (±0.03) 0.86 (±0.03) 0.62 (±0.03) 10.21 
 BCSE ≤42 0.31 (±0.03) 0.95 (±0.03) 0.87 (±0.03) 0.58 (±0.03) 8.96 
 BCSE ≤38 0.18 (±0.03) 0.98 (±0.03) 0.88 (±0.03) 0.54 (±0.03) 8.91 
 BCSE ≤36 0.14 (±0.03) 0.99 (±0.03) 0.92 (±0.03) 0.54 (±0.03) 13.83 
 BCSE ≤28 0.05 (±0.03) 1.00 (±0.03) 1.00 (±0.03) 0.53 (±0.03) N/A 
NCI versus Matched Controls 
 BCSE ≤44 0.24 (±0.05) 0.91 (±0.05) 0.73 (±0.05) 0.54 (±0.05) 3.18 
 BCSE ≤42 0.21 (±0.05) 0.94 (±0.05) 0.78 (±0.05) 0.54 (±0.05) 4.14 
 BCSE ≤38 0.12 (±0.05) 0.97 (±0.05) 0.80 (±0.05) 0.52 (±0.05) 2.20 
 BCSE ≤36 0.09 (±0.05) 1.00 (±0.05) 1.00 (±0.05) 0.52 (±0.05) N/A 
 BCSE ≤28 0.00 (±0.05) 1.00 (±0.05) 1.00 (±0.05) 0.50 (±0.05) N/A 

Notes: BCSE = Brief Cognitive Status Exam; MMSE = Mini-Mental Status Exam; CIND = cognitive impairment, no dementia; NCI = no cognitive impairment; N/A = not applicable; PPV = positive predictive values; NPV = negative predictive values; Sens = sensitivity; Spec = specificity.

Discussion

The BCSE was developed to provide clinicians with a brief cognitive screening tool for use with a general clinical population age 16–90. The BCSE covers a broad range of abilities related to general cognitive functioning, including: orientation, time estimation, memory, visual-spatial, planning, working memory, inhibitory control, processing speed, and verbal productivity. The use of weighted raw scores at the item level enables the clinician to identify the domains of cognitive functioning that are relatively impaired.

The current study compared BCSE and MMSE scores in a mixed clinical sample with varying levels of cognitive functioning. Total scores on the BCSE and the MMSE were highly correlated, indicating that there is strong agreement between the relative standing among the patients in the sample. The correlation does not, however, indicate the rate of agreement across instruments regarding impairment status. At two common and fairly liberal cutoff scores for the MMSE (i.e., 24 and 26), the BCSE correctly identified more patients as cognitively impaired than the MMSE. Further, when BCSE raw scores are equated to MMSE scores, the equated BCSE range for the MMSE value tended to be in the lower end of the sensitivity range, suggesting that BCSE scores are more sensitive than the MMSE. The equating tables provided enable the clinician to use the BCSE total weighted raw score in a manner similar to how the MMSE has been used. For example, if the examiner typically uses an MMSE of ≤24 for further evaluation, this would equate to a BCSE score of 32 (e.g., use of the upper range value increases sensitivity to impairment) and an MMSE of 26 would equate to a BCSE of 38.

The BCSE appears to be more sensitive to cognitive deficits in a general clinical sample; however, it is difficult to know which cases are false positives without further evaluation. In a sample of patients diagnosed with dementia, there should be very few to no false positives. Diagnostic statistics indicated that at the same level of specificity, the BCSE was more sensitive at all classification levels except the most restrictive (i.e., Very low and below). For example, at a base rate of 10% false positives (i.e., Borderline and below), the BCSE identified 24% more cases of dementia, and at a false positive rate of 3%–4% (i.e., Low and below), the BCSE identified 28% more of the dementia cases. Thus, the BCSE classification of Low and below yielded the highest rates of sensitivity and specificity to dementia. Similarly, the BCSE was more sensitive to cognitive deficits in the CIND and NCI groups. The CIND and NCI groups tended to show less severe impairment than the DEM group, indicating more specific rather than general cognitive impairment. It is important to note, also, that the specificity of the BCSE in these groups remained high.

Bouman, Hendriks, Aldenkamp, and Kessels (2014) also recently compared the BCSE to the MMSE, this time in a sample of 88 patients with MCI and dementia. The correlation between the BCSE and MMSE in their sample was almost exactly the same as that obtained in the current sample (i.e., .79 and .80, respectively). In contrast to the current study, however, the BCSE did not show clear classification superiority to the MMSE in the study by Bouman and colleagues. Differences in study findings are likely due to differences in study samples. The cognitively impaired subjects in the current study were younger, less cognitively impaired, and more heterogeneous in their etiologies of cognitive impairment. The homogenous nature of the Bouman and colleagues sample, particularly the restricted age range, is likely also the reason they did not see meaningful correlations between the BCSE and MMSE and age and education.

In some clinical settings, the use of a specific raw score cutoff is more efficient than deriving classification scores. The use of BCSE weighted raw score cutoffs, which do not adjust for age and education level, yielded less specificity than the classification model. However, the clinician can choose the level of sensitivity desired to identify individuals needing further evaluation. In general, the classification model will yield better sensitivity and specificity than raw score models for the BCSE; however, on the MMSE, the raw score and T-score models have similar levels of sensitivity.

This study demonstrates the diagnostic utility of the BCSE as a screener for general cognitive impairment in a diverse clinical population where psychiatric comorbidity was common. In this sample, the BCSE was more sensitive than the MMSE in general and particularly when matched for specificity. In other words, in situations where identification of possible cognitive impairment is more important than the potential for false positives, such as in medical and rehabilitation settings where identifying individuals that may benefit from services is a primary objective, the BCSE is a better option given its superior sensitivity in individuals with less obvious cognitive impairment. In addition, because the BCSE is normed for individuals aged 16–90, it can be used in a wide variety of patient populations (e.g., traumatic brain injury, stroke, neurodegenerative diseases) and settings (e.g., inpatient hospital and rehabilitation, outpatient clinics, and research). As with all cognitive screening instruments, more subtle deficits may not be identified by the BCSE, although its PPVs in clinical samples were generally high (i.e., 0.73–0.86%). The relatively high PPVs in this sample may also lend confidence that the BCSE is able to identify true cognitive impairment in spite of notable psychiatric comorbidity, which was spread evenly across the three clinical groups in this study.

The study has a number of limitations which influence the interpretation of the findings. There was no specific control group for the MMSE, which may have underestimated its sensitivity and made the BCSE appear more superior than it truly is. To this point, the sensitivity and specificity of the MMSE in this particular study was lower than that found in the general literature on the MMSE. In addition, the specificity rates were estimated using percentages derived from T-score distributions. This assumes that the actual percentage of cases used to derive the T-scores is the same as expected population based percentages. Furthermore, while adequate and demographically diverse, sample sizes of the clinical groups were relatively small with a larger percentage of men than typically found in dementia samples due to the majority of participants being from a VA setting; generalizability of findings needs verification in larger samples. In particular, examination of the diagnostic utility of the BCSE in specific clinical groups, such as MCI, different etiologies of neurodegenerative disease, traumatic brain injury, and stroke, is recommended, especially in comparison with a comprehensive neuropsychological battery. Finally, diagnoses were based on clinical evaluations performed at one point in time, and there was no mechanism in place to verify the accuracy of clinical diagnoses.

In conclusion, results of the current study indicate that the BCSE has utility as a cognitive screening measure in a mixed clinical sample and is more sensitive at detecting cognitive impairment, particularly milder levels of cognitive impairment, than the MMSE. This finding is also important because whether a patient is referred for further assessment often is based on cognitive screening measures, and the current results indicate that the BCSE is likely to identify more cases of cognitive impairment than the MMSE, without increasing false positives. Additional advantages of the BCSE include normative data from age 16 to 90 and ability to identify cognitive domains that may benefit from further evaluation. Further validation and comparison with other cognitive screening measures (e.g., MoCA) and more detailed neuropsychological evaluations is recommended. In addition, an item analysis of the BCSE in patients with various conditions would be helpful to identify which items have the greatest sensitivity.

Funding

This work was supported by Pearson Inc., the publisher of the Brief Cognitive Status Examination (BCSE), who provided BCSE and Mini-Mental State Examination (MMSE) test manuals and response forms but no monetary support.

Conflict of Interest

Drs Holdnack, Whipple Drozdick, and Wahlstrom were employed by Pearson, the publisher of the BCSE, at the time this study was conducted. However, none of these individuals receive royalties on sales of the BCSE.

References

Benton
A. L.
(
1969
).
Development of a multilingual aphasia battery: Progress and problems
.
Journal of the Neurological Sciences
 ,
9
,
39
48
.
Benton
A. L.
,
Sivan
A. B.
,
Hamsher
K. deS.
,
Varney
N. R.
,
Spreen
O.
(
1994
).
Contributions to neuropsychological assessment
 .
Orlando, FL
:
Psychological Assessment Resources
.
Bouman
Z.
,
Hendriks
M. P. H.
,
Aldenkamp
A. P.
,
Kessels
R. P. C.
(
2014
).
Clinical validaiton of the WMS-IV-NL brief cognitive status exam (BCSE) in older adults with MCI or dementia
.
International Psychogeriatrics
 ,
27
,
221
229
.
Brodaty
H.
,
Fay
L. L.
,
Gibson
L.
,
Burns
K.
(
2006
).
What is the best dementia screening instrument for general practitioners to use?
American Journal of Geriatric Psychiatry
 ,
14
,
391
400
.
Brooks
B. L.
,
Holdnack
J. A.
,
Iverson
G. L.
(
2011
).
Advanced clinical interpretation of the WAIS-IV and WMS-IV: Prevalence of low scores varies by level of intelligence and years of education
.
Assessment
 ,
18
,
156
167
.
Brooks
B. L.
,
Iverson
G. L.
,
Holdnack
J. A.
,
Feldman
H. H.
(
2008
).
The potential for misclassification of mild cognitive impairment: A study of memory scores on the Wechsler Memory Scale-III in healthy older adults
.
Journal of the International Neuropsychological Society
 ,
14
,
463
478
.
Crum
R. M.
,
Anthony
J. C.
,
Bassett
S. S.
,
Folstein
M. F.
(
1993
).
Population-based norms for the Mini-Mental State Examination by age and education level
.
Journal of the American Medical Association
 ,
269
,
2386
2391
.
Cullum
C. M.
,
Saine
K.
,
Chan
L. D.
,
Martin-Cook
K.
,
Gray
K. F.
,
Weiner
M. F.
(
2001
).
Performance-based instrument to assess functional capacity in dementia: The Texas Functional Living Scale
.
Neuropsychiatry, Neuropsychology, and Behavioral Neurology
 ,
14
,
103
108
.
Delis
D. C.
,
Kramer
J. H.
,
Kaplan
E.
,
Ober
B. A.
(
2000
).
California Verbal Learning Test-Second Edition
 .
San Antonio, TX
:
The Psychological Corporation
.
Dong
Y.
,
Sharma
V. K.
,
Chan
B. P.
,
Venketasubramanian
N.
,
Teoh
H. L.
,
Seet
R. C.
et al
. (
2010
).
The Montreal Cognitive Assessment (MoCA) is superior to the Mini-Mental State Examination (MMSE) for the detection of vascular cognitive impairment after acute stroke
.
Journal of Neurological Sciences
 ,
299
,
15
18
.
Folstein
M. F.
,
Folstein
S. E.
,
McHugh
P. R.
(
1975
).
Mini Mental State: A practical method for grading the cognitive state of patients for the clinician
.
Journal of Psychiatric Research
 ,
12
,
189
198
.
Folstein
M. F.
,
Folstein
S. E.
,
McHugh
P. R.
(
2002
).
Mini-Mental State Examination
 .
Lutz, FL
:
Psychological Assessment Resources
.
Gaviria
M.
,
Pliskin
N.
,
Kney
A.
(
2011
).
Cognitive impairment in patients with advanced heart failure and its implications on decision-making capacity
.
Congestive Heart Failure
 ,
17
,
175
179
.
Golden
C. J.
,
Freshwater
S. M.
(
2002
).
Stroop color and word test: Revised examiners manual.
 
Wood Dale, IL
:
Stoelting
.
Han
J. H.
,
Bryce
S. N.
,
Ely
E. W.
,
Kripalani
S.
,
Morandi
A.
,
Shintani
A.
et al
. (
2011
).
The effect of cognitive impairment on the accuracy of the presenting complaint and discharge instruction comprehension in older emergency department patients
.
Annals of Emergency Medicine
 ,
57
,
662
671
.
Iracleous
P.
,
Nie
J. X.
,
Tracy
C. S.
,
Moineddin
R.
,
Ismail
Z.
,
Shulman
K. I.
et al
. (
2010
).
Primary care physicians’ attitudes toward cognitive screening: Findings from a national postal survey
.
International Journal of Geriatric Psychiatry
 ,
25
,
23
29
.
Ismail
Z.
,
Rajji
T. K.
,
Shulman
K. I.
(
2010
).
Brief cognitive screening instruments: An update
.
International Journal of Geriatric Psychiatry
 ,
25
,
111
120
.
Jones
K.
,
Perlman
C. M.
,
Hirdes
J. P.
,
Scott
T.
(
2010
).
Screening cognitive performance with the Resident Assessment Instrument for Mental Health Cognitive Performance Scale
.
Canadian Journal of Psychiatry
 ,
55
,
736
740
.
Kaplan
E. F.
,
Goodglass
H.
,
Weintraub
S.
(
2001
).
The Boston Naming Test
  (
2nd ed.
).
Philadephia
:
Lippincott Williams & Wilkins
.
Klove
H.
(
1963
).
Clinical neuropsychology
. In
Forster
F. M.
(Ed.),
The medical clinics of North America
  (pp.
110
125
).
New York
:
W.B. Saunders
.
Kongs
S. K.
,
Thompson
L. L.
,
Iverson
G. L.
,
Heaton
R. K.
(
2000
).
Wisconsin Card Sorting Test-64 card version
 .
Lutz, FL
:
Psychological Assessment Resources
.
Lonie
J. A.
,
Tierney
K. M.
,
Ebmeier
K. P.
(
2009
).
Screening for mild cognitive impairment: A systematic review
.
International Journal of Geriatric Psychiatry
 ,
24
,
902
915
.
Martinez-Aran
A.
,
Scott
J.
,
Colom
F.
,
Torrent
C.
,
Tabares-Seisdedos
R.
,
Daban
C.
et al
. (
2009
).
Treatment nonadherence and neurocognitive impairment in bipolar disorder
.
Journal of Clinical Psychiatry
 ,
70
,
1017
1023
.
Mast
B. T.
,
Gerstenecker
A.
(
2010
).
Screening instruments and brief batteries for dementia
. In
Lichtenberg
P. A.
(Ed.),
Handbook of assessment in clinical gerontology
  (
2nd ed
. pp.
503
530
).
Boston, MA
:
Elsevier
.
Milne
A.
,
Culverwell
A.
,
Guss
R.
,
Tuppen
J.
,
Whelton
R.
(
2008
).
Screening for dementia in primary care: A review of the use, efficacy and quality of measures
.
International Psychogeriatrics
 ,
20
,
911
926
.
Mitchell
A. J.
(
2009
).
A meta-analysis of the accuracy of the Mini-Mental State Examination in the detection of dementia and mild cognitive impairment
.
Journal of Psychiatric Research
 ,
43
,
411
431
.
Nasreddine
Z. S.
,
Phillips
N. A.
,
Bédirian
V.
,
Charbonneau
S.
,
Whitehead
V.
,
Collin
I.
et al
. (
2005
).
The Montreal Cognitive Assessment, MoCA: A brief screening tool for mild cognitive impairment
.
Journal of the American Geriatrics Society
 ,
53
,
695
699
.
O'Bryant
S. E.
,
Humphreys
J. D.
,
Smith
G. E.
,
Ivnik
R. J.
,
Graff-Radford
N. R.
,
Petersen
R. C.
et al
. (
2008
).
Detecting dementia with the Mini-Mental State Examination in highly educated individuals
.
Archives of Neurology
 ,
65
,
963
967
.
Olson
R. A.
,
Chhanabhai
T.
,
McKenzie
M.
(
2008
).
Feasibility study of the Montreal Cognitive Assessment (MoCA) in patients with brain metastases
.
Supportive Care in Cancer
 ,
16
,
1273
1278
.
Pendlebury
S. T.
,
Cuthbertson
F. C.
,
Welch
S. J.
,
Mehta
Z.
,
Rothwell
P. M.
(
2010
).
Underestimation of cognitive impairment by Mini-Mental State Examination versus the Montreal Cognitive Assessment in patients with transient ischemic attack and stroke: A population-based study
.
Stroke
 ,
41
,
1290
1293
.
Randolph
C.
(
1998
).
Repeatable Battery for the Assessment of Neuropsychological Status (RBANS)
 .
San Antonio, TX
:
The Psychological Corporation
.
Reitan
R. M.
,
Wolfson
D.
(
1985
).
The Halstead-Reitan Neuropsychological Test Battery
 .
Tucson, AZ
:
Neuropsychology Press
.
Rey
A.
(
1941
).
L'examen psychologique dans les cas d'encephalopathies traumatique
.
Archives de Pschologie
 ,
28
,
286
340
.
Schweizer
T. A.
,
Al-Khindi
T.
,
Macdonald
R. L.
(
2012
).
Mini-Mental State Examination versus Montreal Cognitive Assessment: Rapid assessment tools for cognitive and functional outcome after aneurysmal subarachnoid hemorrhage
.
Journal of Neurological Sciences
 ,
316
,
137
140
.
Shulman
K. I.
,
Herrmann
N.
,
Brodaty
H.
,
Chiu
H.
,
Lawlor
B.
,
Ritchie
K.
et al
. (
2006
).
IPA survey of brief cognitive screening instruments
.
International Psychogeriatrics
 ,
18
,
281
294
.
Toglia
J.
,
Fitzgerald
K. A.
,
O'Dell
M. W.
,
Mastrogiovanni
A. R.
,
Lin
C. D.
(
2011
).
The Mini-Mental State Examination and Montreal Cognitive Assessment in persons with mild subacute stroke: Relationship to functional outcome
.
Archives of Physical Medicine and Rehabilitation
 ,
92
,
792
798
.
Tombaugh
T. N.
(
1996
).
Test of Memory Malingering (TOMM)
 .
New York
:
Multi Health Systems
.
Tombaugh
T. N.
,
McDowell
I.
,
Kristjansson
B.
,
Hubley
A. M.
(
1996
).
Mini-Mental State Examination (MMSE) and the modified MMSE (3MS): A psychometric comparison and normative data
.
Psychological Assessment
 ,
8
,
48
59
.
Valldeoriola
F.
,
Coronell
C.
,
Pont
C.
,
Buongiorno
M. T.
,
Cámara
A.
,
Gaig
C.
et al
. (
2011
).
Socio-demographic and clinical factors influencing the adherence to treatment in Parkinson's disease: The ADHESON study
.
European Journal of Neurology
 ,
18
,
980
987
.
Wechsler
D.
(
2008
).
Wechsler Adult Intelligence Scale-Fourth Edition
 .
San Antonio, TX
:
Pearson
.
Wechsler
D.
(
2009
).
Wechsler Memory Scale-Fourth Edition
 .
San Antonio, TX
:
Pearson
.
Wechsler
D.
(
2001
).
Wechsler Test of Adult Reading
 .
San Antonio, TX
:
The Psychological Corporation
.
Zogg
J. B.
,
Woods
S. P.
,
Weber
E.
,
Iudicello
J. E.
,
Dawson
M. S.
,
Grant
I.
, &
the HIV Neurobehavioral Research Center Group
et al.   (
2010
).
HIV-associated prospective memory impairment in the laboratory predicts failures on a semi-naturalistic measure of health care compliance
.
The Clinical Neuropsychologist
 ,
24
,
945
962
.