## Abstract

Objective

Detecting cognitive decline in presymptomatic Alzheimer's disease (AD) and early mild cognitive impairment (MCI) is challenging, but important for treatments targeting AD-related neurodegeneration. The current study aimed to investigate the utility and performance of internally developed robust norms and standard norms in identifying cognitive impairment in late middle-age (baseline age range = 36–68; M = 54).

Method

Robust norms were developed for neuropsychological measures based on longitudinally confirmed cognitively normal (CN) participants (n= 476). Seven hundred and seventy-nine participants enriched for AD risk were classified as psychometric MCI (pMCI) or CN based on standard and robust norms and “single-test” versus “multi-test” criteria.

Results

Prevalence of pMCI ranged from 3% to 49% depending on the classification scheme used. Those classified as pMCI using robust norms exhibited greater subjective cognitive complaints, diagnostic stability, and mild clinical symptoms at follow-up.

Conclusions

Results suggest that identifying early clinically relevant cognitive decline in late middle-age is feasible using robust norms and multi-test criteria.

## Introduction

Recognition that Alzheimer's disease (AD) pathology develops decades before clinical symptoms has gained broad acceptance. As a result, substantial effort has focused on characterization of a preclinical stage of AD via neuroimaging, cerebrospinal fluid, and genetic biomarkers in individuals at risk for AD. This preclinical stage is posited to precede a diagnosis of mild cognitive impairment (MCI), and is characterized by accumulating amyloid pathology, neurodegeneration, and subsequent subtle cognitive decline that can potentially be detected via sensitive neuropsychological measures (Sperling et al., 2011). In addition to biomarker changes, studies have also shown that cognitive measures are important outcomes in secondary prevention trials of proposed AD-modifying interventions. Specifically, cognitive measures are closely related to the core symptoms of disease progression and are sensitive to treatment effects (Donohue et al., 2014). Developing sensitive and reliable methods to identify mild cognitive decline is needed for identification of individuals in preclinical stages of AD who may benefit most from intervention, as well as for use as outcome measures to assess cognitive change following interventions.

Applying strategies used to improve the MCI construct may be helpful in developing methods to identify decline in the preclinical AD timeframe. In 2011, the NIA-Alzheimer's Association workgroup defined MCI as: (i) reported concern regarding a change in cognition; (ii) objective evidence of impairment in one or more cognitive domains, often including memory; (iii) preserved independence of functional abilities; and (iv) do not meet criteria for a diagnosis of dementia, with greater confidence of MCI diagnosis in the presence of positive biomarkers (Albert et al., 2011). However, substantial heterogeneity remains across studies of MCI, due in part to a lack of uniform methods to operationalize this construct (Bondi & Smith, 2014; Ward, Arrighi, Michels, & Cedarbaum, 2012), as well as to the varying factors that may contribute to mild impairments on cognitive testing.

For example, objective evidence of impairment necessary for study entry can be defined using performance on a single memory measure (i.e., Alzheimer's Disease Neuroimaging Initiative (ADNI) (Petersen et al., 2010); Alzheimer's Disease Cooperative Study Vitamin E trial (Petersen et al., 2005)). However, evidence suggests that a “single-test” approach may be more susceptible to false-positive diagnostic errors (Brooks, Iverson, Holdnack, & Feldman, 2008; Clark et al., 2013; Edmonds et al., 2015; Saxton et al., 2009), and may miss individuals in early stages of MCI or those with impairment in non-memory domains (Chang et al., 2011; Howieson et al., 2008). Applying neuropsychological approaches that examine patterns of cognitive performance across a battery of tests, in contrast to a single-test approach, demonstrates increased reliability and stability of the diagnosis over time (Jak et al., 2009; Loewenstein et al., 2009). Accordingly, Bondi and colleagues (2014) found that application of these neuropsychological criteria to the ADNI sample improved diagnostic stability and associations with AD biomarkers. Further, their results indicated that up to one-third of the MCI sample in ADNI may actually be CN overall, and at lower risk for progression to AD.

Furthermore, available standard normative data used to define an “impaired” versus a “normal” score on neuropsychological indices in late middle-aged and older adults may be confounded by the lack of longitudinal verification of cognitive normality. In other words, the normative sample may include individuals with incipient disease who later progress to MCI or dementia (De Santi et al., 2008; Pedraza et al., 2010; Sliwinski, Lipton, Buschke, & Stewart, 1996). Inclusion of these individuals may reduce the normative mean and increase the variance, resulting in cut-offs that may be too low to detect preclinical decline (Sliwinski et al., 1996). Alternatively, developing “robust” normative samples that follow individuals longitudinally, and that exclude from the normative distribution those who develop MCI or dementia, provides improved sensitivity to mild deficits (De Santi et al., 2008). The prevalence of amnestic pMCI in the Wisconsin Registry for Alzheimer's Prevention (WRAP) cohort was previously examined using robust normative data developed for factor scores derived from the Rey Auditory Verbal Learning Test (RAVLT) (Koscik et al., 2014). The present study was designed to expand on the prior contribution by developing robust normative data for additional memory and non-memory measures.

The current study applies the concepts described above which we operationalized as a multi-test approach that takes into account a pattern of performance across a battery of tests and utilizes robust normative comparison distributions. We applied these criteria to the WRAP cohort, a sample of late middle-aged individuals enriched with risk factors for AD, to identify early cognitive changes expected to precede a clinical diagnosis of MCI (referred to here as psychometric MCI; pMCI). Additional aims included comparing differences in classification of pMCI when using a robust normative dataset and standard normative datasets available in published manuals, establishing the prevalence of pMCI in WRAP using single-test and multi-test criteria, and quantifying differences among participants classified as pMCI using various criteria on demographics, cognitive performance, subjective cognitive complaints, clinical status, parental history of AD, and diagnostic stability. We hypothesized that individuals classified with pMCI using robust normative data and a multi-test approach would exhibit greater correlations with clinical MCI symptoms, AD risk factors, diagnostic stability, and global cognitive impairment at follow-up than those classified using standard normative data or a “single-test” approach.

## Method

### Neuropsychological and Clinical Measures

The neuropsychological measures assess a broad range of cognitive functions including episodic memory, language, executive functioning, attention/processing speed, and visuospatial abilities. The measures included in development of internal robust norms are displayed in Table 1. The Wide Range Achievement Test-3rd Edition reading subtest (WRAT-III Reading) (Wilkinson, 1993) was used as an estimate of literacy and premorbid functioning. Each measure was examined for normality and transformed if necessary prior to inclusion in normative regression models.

Table 1.

Cognitive performance (mean(SD)) and regression coefficients used to calculate robust z-scores in robust normative sample (n= 476)

Cognitive domain Mean (raw) β0 βage βgender βliteracy RMSE
Immediate memory
RAVLT Total Trials 1–5a 52.9 (7.3) 49.620192 −0.151643 4.704805 2.615045 6.415946
WMS-R LM immediatea 29.7 (6.0) 29.618445 −0.107647 0.840342 1.984174 5.581112
BVMT-R immediatea 24.8 (5.4) 24.169550 −0.255258 2.496791 1.130315 4.918924
Delayed memory
RAVLT delayed recalla 11.0 (2.6) 9.823554 −0.072392 1.675560 0.857910 2.242370
WMS-R logical memory delaya 26.5 (6.6) 26.21908 −0.156043 1.404768 2.462468 6.046947
BVMT-R delaya 9.7 (1.8) 9.509670 −0.057683 0.665952 0.397521 1.716219
Executive functioning
TMT Part Ba,b 61.7 (28.2) 1.785677 0.007041 −0.036295 −0.058415 0.132620
Stroop Color-Word Interferencea 110.6 (19.4) 107.867511 −0.788895 3.948321 7.155325 17.230303
WAIS-R Digit Symbola 58.1 (9.8) 56.253828 −0.525599 5.849860 2.066808 8.637696
Attention
TMT Part Ab 26.2 (8.1) 1.423509 0.004700 −0.033402 −0.016354 0.117865
WAIS-III Digit Span Forward 10.6 (2.2) 10.959106 −0.004301 −0.489343 0.861879 2.017822
WAIS-III Digit Span Backward 7.1 (2.2) 7.500588 −0.002464 −0.507588 0.969086 1.999305
WAIS-III L-N Sequencing 11.0 (2.4) 10.835015 −0.025144 0.193495 1.069053 2.126190
Language
Boston Naming Testb 56.9 (3.6) 0.414110 0.000020 0.091045 −0.166332 0.290785
Animal Fluency 23.4 (5.1) 24.004379 −0.091021 −0.619654 1.062367 5.151716
Letter Fluency 43.9 (11.4) 43.346797 −0.084162 0.821358 4.696764 10.415596
Visuospatial functioning
Clock Drawing Testb 9.5 (0.9) 0.110274 0.000931 0.030377 −0.042823 0.189542
Cognitive domain Mean (raw) β0 βage βgender βliteracy RMSE
Immediate memory
RAVLT Total Trials 1–5a 52.9 (7.3) 49.620192 −0.151643 4.704805 2.615045 6.415946
WMS-R LM immediatea 29.7 (6.0) 29.618445 −0.107647 0.840342 1.984174 5.581112
BVMT-R immediatea 24.8 (5.4) 24.169550 −0.255258 2.496791 1.130315 4.918924
Delayed memory
RAVLT delayed recalla 11.0 (2.6) 9.823554 −0.072392 1.675560 0.857910 2.242370
WMS-R logical memory delaya 26.5 (6.6) 26.21908 −0.156043 1.404768 2.462468 6.046947
BVMT-R delaya 9.7 (1.8) 9.509670 −0.057683 0.665952 0.397521 1.716219
Executive functioning
TMT Part Ba,b 61.7 (28.2) 1.785677 0.007041 −0.036295 −0.058415 0.132620
Stroop Color-Word Interferencea 110.6 (19.4) 107.867511 −0.788895 3.948321 7.155325 17.230303
WAIS-R Digit Symbola 58.1 (9.8) 56.253828 −0.525599 5.849860 2.066808 8.637696
Attention
TMT Part Ab 26.2 (8.1) 1.423509 0.004700 −0.033402 −0.016354 0.117865
WAIS-III Digit Span Forward 10.6 (2.2) 10.959106 −0.004301 −0.489343 0.861879 2.017822
WAIS-III Digit Span Backward 7.1 (2.2) 7.500588 −0.002464 −0.507588 0.969086 1.999305
WAIS-III L-N Sequencing 11.0 (2.4) 10.835015 −0.025144 0.193495 1.069053 2.126190
Language
Boston Naming Testb 56.9 (3.6) 0.414110 0.000020 0.091045 −0.166332 0.290785
Animal Fluency 23.4 (5.1) 24.004379 −0.091021 −0.619654 1.062367 5.151716
Letter Fluency 43.9 (11.4) 43.346797 −0.084162 0.821358 4.696764 10.415596
Visuospatial functioning
Clock Drawing Testb 9.5 (0.9) 0.110274 0.000931 0.030377 −0.042823 0.189542

Notes: Age was centered prior to inclusion in regression models using the mean baseline age of 53.775. The WRAT-III reading raw score was transformed into a z-score prior to inclusion in regression models using the mean and standard deviation at baseline (50.448, 5.0507). For gender, males were coded as 0 and females were coded as 1. RMSE = Root-Mean-Squared Error; RAVLT = Rey Auditory Verbal Learning Test; WMS-R LM = Wechsler Memory Scale-Revised Logical Memory subtest; BVMT-R = Brief Visuospatial Memory Test-Revised; TMT = Trailmaking Test; WAIS-R = Wechsler Adult Intelligence Scale-Revised; WAIS-III = Wechsler Adult Intelligence Scale-Third Edition; L-N Sequencing = Letter-Number Sequencing.

aNine measures included in MCI classification and statistical analyses.

bDue to non-normality, these measures were log transformed to improve skewness or kurtosis prior to inclusion in regression models. Raw means and standard deviations are displayed here.

Because a clinical diagnosis of MCI also requires a subjective memory complaint and intact functional activities, these characteristics were also examined. The presence of a subjective memory complaint was defined as a “yes” response to “Do you think you have a problem with your memory?” or “Has someone told you that you have a problem with your memory?”, and/or memory decline endorsed by an informant on the Informant Questionnaire on Cognitive Decline in the Elderly-Short Form (IQCODE; Jorm, 1994). The IQCODE includes 16 items in which the informant indicates if the participant exhibits change in each item over the past 2–3 years (e.g., much improvement [score = 1], a bit improved [score = 2], no change [score = 3], a bit worse [score = 4], or much worse [score = 5]) and an IQCODE ≥ 55 [mean rating = 3.44] indicated the presence of a subjective cognitive complaint [Jorm, 2004]. Instrumental activities of daily living (IADLs) were assessed by an informant using a modified version of the Lawton Instrumental Activities of Daily Living Scale (Lawton & Brody, 1969) that assesses the same eight domains as the original scale (using the telephone, transportation, grocery shopping, meal preparation, housework, laundry, medications, and managing finances). An informant rates if the participant is “unable to do at all” (0 points), “needs some help” (1 point), or “needs no help” (2 points) for each domain resulting in a total score ranging from 0 to 16. A score of 14 or lower was defined as impaired to reflect an inability to independently perform at least one activity of daily living, or needing assistance with at least two activities of daily living. Clinical symptoms and functional abilities at follow-up were measured using the Clinical Dementia Rating (CDR) Scale; (Morris, 1997) in which a score of 0 suggests cognitive normality and a score of 0.5 indicates mild impairment. This measure was added to the study at the fourth visit and therefore was not available to examine at earlier visits.

### Robust Normative Sample Selection

In a previous study (Koscik et al., 2014), robust normative data based on WRAP participants without a parental history of AD were developed for two factor scores representing learning and memory and two factors representing executive function. Those who exhibited at least one factor score at or below −1.5 SDs (adjusted for age, gender, and literacy) at both baseline and Visit 2 were removed from the normative sample.

The robust normative sample in the current study (n = 476) differed from that of the previous study in that it included additional participants that had enrolled in WRAP in the interim, and included participants with both positive and negative parental history of AD, since ∼70% of the WRAP cohort has a positive parental history and exclusion of these participants would result in a smaller normative sample. In an attempt to exclude participants at higher likelihood for preclinical AD, while maintaining a large normative sample size, individuals at genetic risk for AD were excluded from the normative sample in the current study. The specific criteria for inclusion in the robust normative sample were: (i) non-carrier of the APOE ϵ4 allele, (ii) no history of neurological or major psychiatric disorders by self-report, and (iii) classified as CN at baseline and all follow-up visits (94% completed two visits, 71% completed three visits, and 36% completed four visits) using normative factor score data from the previous study. Six percent of participants (n = 29) with only baseline visit data were included in the normative sample to allow inclusion of a greater proportion of racial or ethnic minorities. Because these participants were enrolled more recently in the WRAP study, they did not yet have follow-up data. Additionally, to avoid violating the assumption of independence necessary for regression models due to multiple members from the same family enrolled in WRAP, we allowed only one randomly selected member from each family to be included in the normative sample. However, if the family included two siblings of opposite gender, both participants were retained because the models were adjusted for gender (n= 15 sibling pairs were included in the final robust normative sample).

### Development of Normative Data Sets

Two sets of normative z-scores were developed for nine neuropsychological measures included in the classification of pMCI; that is, one set based on values published in test manuals or similar sources (“Standard Norms”), and a second set based on the internal robust normative sample developed in the current study (“Robust Norms”). Tests in which higher raw scores indicated poorer performance (e.g., time to completion on Trail Making Test) were multiplied by −1 such that higher z-scores indicated better performance for all neuropsychological measures.

#### Standard norms

Means and SDs for various cells representing age, gender, and/or education-level combinations were identified. Standard normative datasets were gathered from the Wechsler Adult Intelligence Scale-Revised manual (WAIS-R Digit Symbol (Wechsler, 1981), Wechsler Memory Scales-Revised (WMS-R Logical Memory (LM) (Wechsler, 1987), RAVLT manual (Schmidt, 1996), Brief Visuospatial Memory Test-Revised manual (BVMT-R; Benedict, 1997), Stroop Neuropsychological Screening Test manual (Stroop; Trenerry, Crosson, DeBoe, & Leber, 1989), and the Revised Comprehensive Norms for an Expanded Halstead-Reitan Battery (Trailmaking Test [TMT]; Heaton, Miller, Taylor, & Grant, 2004). For each observed test score, a z-score was calculated by subtracting the normative mean from the observed score and then dividing by the normative SD. For the standard z-scores, age was the only demographic variable available in the calculation of normative scores for the RAVLT, Stroop, WAIS-R Digit Symbol, WMS-R Logical Memory, and BVMT-R tests. The TMT normative scores were adjusted for age, gender, and education as these data were available in the Revised Comprehensive Norms manual.

#### Robust norms

Procedures conducted to generate the normative z-scores using robust norms were similar to those described in De Santi and colleagues (2008) and Shirk and colleagues (2011), and adjusted for age, gender, and estimated literacy. First, multiple regression analyses were conducted within the robust normative sample (n= 476) that included predictors of age (centered by mean age of robust normative sample at baseline; M= 53.775), gender (coded 0 for male and 1 for female), and an estimate of literacy (WRAT-III reading raw score transformed to z-score using mean and SD of robust normative sample; M= 50.448, SD= 5.0507). The outcome variable in each model was the raw score on each neuropsychological measure at the first administration.

From each model, the root-mean-squared error (RMSE) and the β-coefficients associated with the intercept and each predictor (βAge,βGender, and βLiteracy) were saved. These parameter estimates were used to calculate a predicted score for all participants on each neuropsychological measure (see Equation 1).

(1)
$Predicted score=β0+βAGE×Age(centered)+βGENDER×Gender(0=male; 1=\,female)+βLITERACY×Literacy.$

Third, robust z-scores were created for each neuropsychological measure by subtracting the predicted score from the observed score and dividing by the RMSE (see Equation 2). The RMSE is the square root of the average squared differences between the observed score and the predicted score, providing an estimate of the average deviation around predicted scores for each model (Shirk et al., 2011).

(2)
$Robustz-score=(Observed score−Predicted score)Root-Mean-Squared-Error.$

### Classification of pMCI Versus CN

pMCI was defined as objective performance 1.5 SD below the expected mean and did not require a subjective memory complaint. A subset of nine measures that assessed episodic memory and executive functioning, two cognitive domains that have been shown to demonstrate decline in presymptomatic AD and early MCI (Albert et al., 2014; Albert, Moss, Tanzi, & Jones, 2001; Chen et al., 2001; Donohue et al., 2014; Elias et al., 2000) were included in the classification of pMCI. Measures represented cognitive domains of immediate recall memory (RAVLT Trials 1–5 Total, LM Immediate Recall, BVMT-R Immediate Recall), delayed memory (RAVLT Long-Delay Free Recall, LM Delayed Recall, BVMT-R Delayed Recall), and executive functioning (TMT Part B, Stroop Color-Word, Digit Symbol). Standard and robust norms z-scores were examined relative to two classifications schemes (single-test and multi-test). For each method, a test score was identified as in the impaired range if the corresponding z-score was ≤−1.5.

A classification of pMCI using single-test criteria required at least one impaired measure on the nine measures included in the classification procedures. Classification of pMCI using multi-test criteria required at least two impaired measures within a cognitive domain (immediate memory, delayed memory, or executive functioning), or at least one impaired measure in each of the three cognitive domains (Jak et al., 2009). All participants classified as pMCI using multi-test criteria also met criteria for pMCI via single-test criteria, whereas the opposite was not necessarily true. The multi-test criteria used in the current study differed from the comprehensive neuropsychological criteria used in the Jak and colleagues (2009) study in inclusion of only memory and executive functioning domains (rather than additional domains of visuospatial, language, and attention) and application of a more conservative threshold of 1.5 SDs below the normative mean (rather than 1.0 SD) to define impairment. Cognitive domains that included at least three measures at multiple study visits (e.g., animal fluency was not initiated until Visit 3 leaving only two measures within the language domain) and have evidence of decline in preclinical AD (memory and executive functioning) were included in the current study. The rationale for applying a 1.5 SD threshold was due to concern that using a 1 SD threshold with potentially more sensitive robust norms may overclassify individuals with pMCI.

Combining the classification methods with the two sets of normative z-scores resulted in four pMCI classification variables (1 = met pMCI criteria for that method, 0 = did not meet criteria): Standard/Single-Test, Robust/Single-Test, Standard/Multi-Test, and Robust/Multi-Test.

### Statistical Analyses

All statistical analyses were performed in SPSS version 22 and statistical significance was defined as p< .05. Statistical analyses were conducted on WRAP participants who were not included in the robust normative sample and had completed a baseline visit and at least one follow-up visit (Wave 2) at the time this study was conducted (n= 779). Analyses below were conducted on the Wave 2 data because three of the measures included in the classification of pMCI were not administered at baseline (LM, BVMT, and Digit Symbol).

#### Prevalence of pMCI

Cross-sectional prevalence of pMCI was identified via frequency analyses. McNemar χ2 tests for paired binomial proportions tested differences in the proportion classified as pMCI depending on the criteria applied.

#### Comparison of pMCI and CN groups

Because participants could be classified as pMCI using various criteria (see Fig. 1 for overlap among criteria), statistical comparisons among the four pMCI groups were not conducted. Rather, comparisons between the pMCI and CN group formed based on each classification scheme were conducted. t-Tests and χ2 analyses were conducted to compare the CN and pMCI groups resulting from the four classification schemes on demographic variables (age, gender, education, and literacy), parental history of AD, and clinical symptoms (the presence of subjective memory complaint, the presence of decline in complex IADLs). One-way analyses of covariance (ANCOVAs) compared neuropsychological raw scores between CN and pMCI groups resulting from each classification scheme, including concurrent age, gender, and literacy scores as covariates.

Fig. 1.

Ovals depicting pMCI status by each classification method. Lettered areas represent overlap among the classification schemes (a = pMCI Standard norms/Single-Test only [n= 2], b = pMCI all schemes [n= 21]; c = pMCI Standard norms/Single-Test and Robust norms/Single-Test [n= 43]; d = pMCI Robust norms/Single-Test and Robust norms/Multi-Test [n= 57], e = pMCI Standard norms/Single-Test, Robust norms/Single-Test, and Robust norms/Multi-Test [n= 76], f = pMCI Robust norms/Single-Test only [n= 183]). Space outside the ovals represents the subset of the sample classified as CN across all classification schemes.

Fig. 1.

Ovals depicting pMCI status by each classification method. Lettered areas represent overlap among the classification schemes (a = pMCI Standard norms/Single-Test only [n= 2], b = pMCI all schemes [n= 21]; c = pMCI Standard norms/Single-Test and Robust norms/Single-Test [n= 43]; d = pMCI Robust norms/Single-Test and Robust norms/Multi-Test [n= 57], e = pMCI Standard norms/Single-Test, Robust norms/Single-Test, and Robust norms/Multi-Test [n= 76], f = pMCI Robust norms/Single-Test only [n= 183]). Space outside the ovals represents the subset of the sample classified as CN across all classification schemes.

#### Stability of classification

Frequency analyses examined stability of pMCI and CN classifications over two follow-up visits. Specifically, individuals were classified as “stable” if the same classification was given at Waves 3 and 4 as was given at Wave 2, “revert” if pMCI criteria were met at Wave 2, but not at Waves 3 and 4, “progress” if CN criteria were met at Wave 2, but a pMCI classification was given at Waves 3 and 4, and “fluctuating” if the classification changed at each visit (e.g., pMCI, CN, pMCI or CN, pMCI, and CN). Because the likelihood of performing below expectations on one measure at any visit may be high (Binder, Iverson, & Brooks, 2009), the definition of stability for single-test criteria required that the same test exhibit impairment at follow-up visits.

#### Clinical follow-up

χ2 analyses compared the CN and pMCI groups resulting from the four classification schemes on CDR and IQCODE scores at most recent follow-up visit.

## Results

### Participant Characteristics

There were no significant differences between the robust normative sample (n= 476) and the test sample included in statistical analyses (n= 779) on age at baseline visit (p= .70), gender (p= .91), years of education completed (p= .36), and an estimate of literacy (p= .20) (see Table 2). There were differences in race/ethnicity (p< .001), with the robust normative sample including fewer Caucasian and more Hispanic participants. The robust normative sample intentionally included significantly fewer individuals with a parental history of AD (participants with an APOE ϵ4 allele were excluded from this sample) (p< .001). Mean performance on neuropsychological measures within the robust normative sample, as well as regression coefficients used for calculation of predicted and robust z-scores, are presented in Table 1.

Table 2.

Demographic and clinical characteristics at WRAP baseline visit (mean(SD))

Robust normative sample Analysis sample Significance (p
N 476 779
Age 53.8 (6.7) 53.6 (6.5) .70
Female 333 (70%) 550 (71%) .91
Education (years) 16.2 (2.9) 16.3 (2.9) .36
Race/ethnicity 90% Caucasian 95% Caucasian <.001
5% African American 4% African American
4% Hispanic 1% Othera
1% Otherb
WRAT-III Reading 104.8 (10.5) 105.6 (9.4) .20
Positive parental history of AD 295 (62%) 623 (80%) <.001
APOE ϵ4 carrier 0 (0%) 464 (60%) <.001
Robust normative sample Analysis sample Significance (p
N 476 779
Age 53.8 (6.7) 53.6 (6.5) .70
Female 333 (70%) 550 (71%) .91
Education (years) 16.2 (2.9) 16.3 (2.9) .36
Race/ethnicity 90% Caucasian 95% Caucasian <.001
5% African American 4% African American
4% Hispanic 1% Othera
1% Otherb
WRAT-III Reading 104.8 (10.5) 105.6 (9.4) .20
Positive parental history of AD 295 (62%) 623 (80%) <.001
APOE ϵ4 carrier 0 (0%) 464 (60%) <.001

Notes: Sample included in analyses was not included in robust normative sample. WRAT-III = Wide Range Achievement Test-Third Edition. AD = Alzheimer's disease.

aHispanic, Asian American, and Native American participants.

bAsian American and Native American participants.

### Prevalence of pMCI

Using single-test criteria, 18% (n= 142) of the sample met criteria for pMCI when standard norms were applied, whereas 49% (n= 380) were classified as pMCI using robust norms (McNemar χ2 = 232.10, p< .001). The majority of participants classified as pMCI using standard norms were also classified as pMCI using the robust norms (n= 140 of 142; 98%). In contrast, only 37% of those classified as pMCI using robust norms (n= 140 of 380) were also considered impaired based on standard norms.

Using multi-test criteria, 3% (n= 21) were classified as pMCI using standard norms, and 20% of the sample (n= 154) were classified as pMCI when the robust norms were applied (McNemar χ2 = 131.01, p< .001). All participants classified as pMCI using standard norms (n= 21) were also classified as pMCI using robust norms, whereas only 14% (n = 21 of 154) of those classified as pMCI using robust norms were also classified as pMCI using standard norms.

The four ovals in Fig. 1 depict the samples of pMCI formed by each classification scheme. The lettered areas represent overlap among the classification schemes. The space outside the ovals represents the subset of the sample classified as CN across all classification schemes.

### Demographic and Clinical Characteristics of pMCI and CN Groups

The pMCI and CN groups resulting from each of the four classification schemes were compared on demographic and clinical characteristics. Sample characteristics across the pMCI and CN groups are depicted for each classification scheme in Table 3. Regardless of the method used, there were no differences in parental history of AD across pMCI and CN groups.

Table 3.

Demographic and clinical characteristics (mean(SD)) at Wave 2 for pMCI and CN participants classified via four classification schemes (Standard/Single-Test, Robust/Single-Test, Standard/Multi-Test, Robust/Multi-Test criteria)

Single-test

Multi-test criteria

CN pMCI Statistic (t or χ2df p Effect size CI 95% CN pMCI Statistic (t or χ2df p Effect size CI 95%
Standard norms
N 637 142 (18%)      758 21 (3%)
Age mean (SD57.8 (6.4) 58.6 (6.7) −1.24 777 .22 0.09 −1.91; 0.43 58.0 (6.4) 57.5 (7.3) 0.31 777 .76 0.02 −2.4; 3.2
Gender (Proportion F) 0.74 0.57 15.39 <.001 0.14 0.08; 0.26 0.71 0.62 0.79 .38 0.03 −0.1; 0.3
Education (years) 16.5 (2.8) 15.6 (2.9) 3.24 777 .001 0.23 0.34; 1.38 16.4 (2.9) 14.4 (2.0) 3.05 777 <.01 0.22 0.7; 3.2
WRAT-III Reading (standard score) 108.1 (8.4) 104.0 (10.5) 4.95 772 <.001 0.36 2.47; 5.71 107.5 (8.9) 100.9 (9.9) 3.30 772 .001 0.24 2.7; 10.6
Parental history of AD (proportion positive) 0.80 0.78 0.35 .55 0.02 −0.05; 0.10 0.80 0.86 0.44 .51 0.02 −0.2; 0.2
Subjective complaint (proportion positive) 0.35 0.40 1.42 .23 0.05 −0.03; 0.15 0.35 0.47 1.17 .28 0.04 −0.1; 0.3
IQCODE 48.1 (4.8) 48.2 (5.5) −0.24 662 .81 0.02 −1.13; 0.89 48.0 (5.0) 50.7 (3.8) −2.11 662 <.05 0.16 −5.3; −0.2
IADL questionnaire (proportion impaired) 0.02 0.04 2.39 .12 0.06 −0.01; 0.08 0.02 0.13 7.84 <.01 0.11 0.0; 0.4
Robust norms
N 399 380 (49%)      625 154 (20%)
Age mean (SD57.4 (6.4) 58.5 (6.4) −2.33 777 <.05 0.17 −2.0; −0.2 57.7 (6.4) 59.2 (6.5) −2.64 777 <.01 0.19 −2.7; −0.4
Gender (proportion F) 0.70 0.71 0.07 .79 0.01 −0.1; 0.1 0.71 0.68 0.54 .46 0.03 −0.1; 0.1
Education (years) 16.4 (2.7) 16.2 (3.0) 1.0 777 .32 0.02 −0.2; 0.6 16.4 (2.8) 16.1 (3.2) 1.20 777 .23 0.09 −0.2; 0.8
WRAT-III reading (standard score) 107.4 (8.6) 107.3 (9.3) 0.17 772 .86 0.01 −1.2; 1.4 107.3 (8.9) 107.6 (9.2) −0.42 772 .68 0.03 −1.9; 1.3
Parental history of AD (proportion positive) 0.82 0.77 3.15 .08 0.06 0.0; 0.1 0.81 0.75 3.37 .07 0.07 0.0; 0.2
Subjective complaint (proportion positive) 0.33 0.39 3.01 .08 0.07 0.0; 0.1 0.33 0.45 6.40 .01 0.10 0.0; 0.2
IQCODE 48.2 (4.4) 48.0 (5.5) 0.46 662 .65 0.04 −0.6; 0.9 47.9 (4.9) 48.7 (5.3) −1.51 662 .13 0.12 −1.7; 0.2
IADL scale (proportion impaired) 0.02 0.03 1.66 .20 0.05 −.01; .04 0.02 0.05 3.87 <.05 0.08 0.0; 0.1
Single-test

Multi-test criteria

CN pMCI Statistic (t or χ2df p Effect size CI 95% CN pMCI Statistic (t or χ2df p Effect size CI 95%
Standard norms
N 637 142 (18%)      758 21 (3%)
Age mean (SD57.8 (6.4) 58.6 (6.7) −1.24 777 .22 0.09 −1.91; 0.43 58.0 (6.4) 57.5 (7.3) 0.31 777 .76 0.02 −2.4; 3.2
Gender (Proportion F) 0.74 0.57 15.39 <.001 0.14 0.08; 0.26 0.71 0.62 0.79 .38 0.03 −0.1; 0.3
Education (years) 16.5 (2.8) 15.6 (2.9) 3.24 777 .001 0.23 0.34; 1.38 16.4 (2.9) 14.4 (2.0) 3.05 777 <.01 0.22 0.7; 3.2
WRAT-III Reading (standard score) 108.1 (8.4) 104.0 (10.5) 4.95 772 <.001 0.36 2.47; 5.71 107.5 (8.9) 100.9 (9.9) 3.30 772 .001 0.24 2.7; 10.6
Parental history of AD (proportion positive) 0.80 0.78 0.35 .55 0.02 −0.05; 0.10 0.80 0.86 0.44 .51 0.02 −0.2; 0.2
Subjective complaint (proportion positive) 0.35 0.40 1.42 .23 0.05 −0.03; 0.15 0.35 0.47 1.17 .28 0.04 −0.1; 0.3
IQCODE 48.1 (4.8) 48.2 (5.5) −0.24 662 .81 0.02 −1.13; 0.89 48.0 (5.0) 50.7 (3.8) −2.11 662 <.05 0.16 −5.3; −0.2
IADL questionnaire (proportion impaired) 0.02 0.04 2.39 .12 0.06 −0.01; 0.08 0.02 0.13 7.84 <.01 0.11 0.0; 0.4
Robust norms
N 399 380 (49%)      625 154 (20%)
Age mean (SD57.4 (6.4) 58.5 (6.4) −2.33 777 <.05 0.17 −2.0; −0.2 57.7 (6.4) 59.2 (6.5) −2.64 777 <.01 0.19 −2.7; −0.4
Gender (proportion F) 0.70 0.71 0.07 .79 0.01 −0.1; 0.1 0.71 0.68 0.54 .46 0.03 −0.1; 0.1
Education (years) 16.4 (2.7) 16.2 (3.0) 1.0 777 .32 0.02 −0.2; 0.6 16.4 (2.8) 16.1 (3.2) 1.20 777 .23 0.09 −0.2; 0.8
WRAT-III reading (standard score) 107.4 (8.6) 107.3 (9.3) 0.17 772 .86 0.01 −1.2; 1.4 107.3 (8.9) 107.6 (9.2) −0.42 772 .68 0.03 −1.9; 1.3
Parental history of AD (proportion positive) 0.82 0.77 3.15 .08 0.06 0.0; 0.1 0.81 0.75 3.37 .07 0.07 0.0; 0.2
Subjective complaint (proportion positive) 0.33 0.39 3.01 .08 0.07 0.0; 0.1 0.33 0.45 6.40 .01 0.10 0.0; 0.2
IQCODE 48.2 (4.4) 48.0 (5.5) 0.46 662 .65 0.04 −0.6; 0.9 47.9 (4.9) 48.7 (5.3) −1.51 662 .13 0.12 −1.7; 0.2
IADL scale (proportion impaired) 0.02 0.03 1.66 .20 0.05 −.01; .04 0.02 0.05 3.87 <.05 0.08 0.0; 0.1

Note: Effect size refers to Cramer's V for χ2 analyses on dichotomous variables and Cohen's d for t-test analyses on continuous variables; WRAT-III = Wide Range Achievement Test Third Edition; IQCODE = Informant Questionnaire on Cognitive Decline in the Elderly-Short Form; IADL scale = Modified version of Lawton Instrumental Activities of Daily Living scale.

Bolded values indicate statistically significant (p < .05) differences between CN and pMCI groups determined via each classification scheme.

Individuals classified as pMCI using single-test criteria and standard norms were comprised of a significantly greater proportion of males and had fewer years of education and lower literacy estimates compared with those classified as CN using the same criteria. Participants classified as pMCI using single-test criteria and robust norms were significantly older than the CN group. There were no differences between CN and pMCI groups across either scheme in subjective memory complaints per participant or informant questionnaire, or proportion of participants with decline in complex IADLs.

Individuals classified as pMCI using multi-test criteria and standard norms had significantly fewer years of education, lower literacy estimates, greater reported memory decline on the informant questionnaire (IQCODE), and a higher proportion of participants with mild IADL impairment as reported by an informant compared with those classified as CN using the same criteria. Participants classified as pMCI using multi-test criteria and robust norms were significantly older and comprised a higher proportion of individuals with subjective cognitive complaints and mild IADL impairment.

### Neuropsychological Performance

As expected, the pMCI groups across all classification schemes performed lower on average than the CN groups on all neuropsychological measures included in the diagnosis of pMCI (p's < .001). As compared by examination of non-overlapping confidence intervals in Table 4, the pMCI group identified using Standard/Multi-Test criteria demonstrated lower scores on list-learning (RAVLT Trials 1–5), story memory (LM Delayed Recall), visual memory (BVMT-R Delayed Recall), and executive functioning (TMT B, Stroop) compared with the pMCI groups identified using the other three criteria. In contrast, the pMCI group identified using the Robust/Single-Test criteria exhibited higher scores across the neuropsychological battery compared with the other three pMCI groups, with the exception of Digit Symbol. The remaining two pMCI groups (Standard/Single-Test; Robust/Multi-test) exhibited similar raw scores across the test battery.

Table 4.

Adjusted means (SE), 95% confidence intervals, and partial eta-squared η2p for comparison of Wave 2 neuropsychological performance between pMCI and CN groups

Visit 2 Standard norms

Robust norms

Single-test

Multi-test

Single-test

Multi-test

CN pMCI $ηp2$ CN pMCI $ηp2$ CN pMCI $ηp2$ CN pMCI $ηp2$
N 637 142  758 21  399 380  625 154
Immediate memory
RAVLT Trials 1–5 51.2 (0.3) [50.7, 51.8] 42.2 (0.6) [40.9, 43.4] .18 50.0 (0.3) [49.4, 50.5] 35.7 (1.7) [32.3, 39.0] .08 53.7 (0.3) [53.0, 54.4] 45.3 (0.4) [44.6, 46.0] .27 51.4 (0.3) [50.9, 52.0] 42.1 (0.6) [41.0, 43.3] .21
LM Immediate 29.4 (0.2) [29.0, 29.9] 24.3 (0.5) [23.3, 25.2] .11 28.7 (0.2) [28.3, 29.1] 19.9 (1.3) [17.4, 22.5] .06 31.0 (0.3) [30.4, 31.5] 25.9 (0.3) [25.4, 26.4] .18 29.8 (0.2) [29.4, 30.3] 23.0 (0.4) [22.2, 23.9] .21
BVMT-R Immediate 24.9 (0.2) [24.6, 25.3] 18.3 (0.4) [17.5, 19.0] .23 23.9 (0.2) [23.6, 24.3] 15.8 (1.1) [13.5, 18.0] .06 26.0 (0.2) [25.5, 26.4] 21.3 (0.2) [20.9, 21.8] .20 24.7 (0.2) [24.4, 25.1] 19.5 (0.4) [18.8, 20.3] .16
Delayed memory
RAVLT Long-Delay 10.7 (0.1) [10.5, 10.9] 7.1 (0.2) [6.7, 7.6] .21 10.1 (0.1) [9.9, 10.3] 5.6 (0.6) [4.4, 6.8] .06 11.5 (0.1) [11.2, 11.7] 8.5 (0.1) [8.3, 8.8] .26 10.7 (0.1) [10.5, 10.9] 7.3 (0.2) [6.9–7.7] .22
LM Delay 26.3 (0.2) [25.8, 26.7] 19.6 (0.5) [18.5, 20.6] .14 25.3 (0.2) [24.9, 25.8] 14.5 (1.5) [11.7, 17.4] .07 28.0 (0.3) [27.4, 28.6] 22.0 (0.3) [21.4, 22.6] .20 26.7 (0.2) [26.2, 27.4] 18.4 (0.5) [17.5, 19.4] .24
BVMT-R Delay 9.8 (0.1) [9.7, 9.9] 7.6 (0.1) [7.3, 7.8] .20 9.5 (0.1) [9.3, 9.6] 6.3 (0.4) [5.5, 7.1] .07 10.2 (0.1) [10.0, 10.3] 8.6 (0.1) [8.4, 8.8] .17 9.8 (0.1) [9.6, 9.9] 7.9 (0.1) [7.6, 8.1] .17
Executive functioning
TMT B 59.1 (0.9) [57.3, 60.1] 80.9 (2.0) [76.9, 84.8] .11 61.9 (0.9) [60.2, 63.6] 103.1 (5.3) [92.7,113.6] .07 55.8 (1.2) [53.5, 58.1] 70.5 (1.2) [68.2, 72.9] .09 59.5 (0.9) [57.6, 61.3] 77.2 (1.9) [73.4, 80.9] .08
Stroop Color-Word 111.5 (0.7) [110.1, 113.0] 96.9 (1.6) [93.7, 100.1] .08 109.6 (0.7) [108.3, 111.0] 82.2 (4.4) [73.6, 90.8] .05 114.9 (0.9) [113.1, 116.8] 102.6 (1.0) [100.7, 104.5] .10 111.0 (0.7) [110.5, 113.4] 96.6 (1.5) [93.6, 99.6] .10
Digit Symbol 57.9 (0.4) [57.2, 58.6] 51.6 (0.8) [50.0, 53.2] .06 57.0 (0.3) [56.4, 57.7] 47.4 (2.1) [43.2, 51.5] .03 60.1 (0.5) [59.2, 61.0] 53.3 (0.5) [52.4, 54.2] .13 58.2 (0.4) [57.5, 58.9] 50.9 (0.7) [49.5, 52.4] .09
Visit 2 Standard norms

Robust norms

Single-test

Multi-test

Single-test

Multi-test

CN pMCI $ηp2$ CN pMCI $ηp2$ CN pMCI $ηp2$ CN pMCI $ηp2$
N 637 142  758 21  399 380  625 154
Immediate memory
RAVLT Trials 1–5 51.2 (0.3) [50.7, 51.8] 42.2 (0.6) [40.9, 43.4] .18 50.0 (0.3) [49.4, 50.5] 35.7 (1.7) [32.3, 39.0] .08 53.7 (0.3) [53.0, 54.4] 45.3 (0.4) [44.6, 46.0] .27 51.4 (0.3) [50.9, 52.0] 42.1 (0.6) [41.0, 43.3] .21
LM Immediate 29.4 (0.2) [29.0, 29.9] 24.3 (0.5) [23.3, 25.2] .11 28.7 (0.2) [28.3, 29.1] 19.9 (1.3) [17.4, 22.5] .06 31.0 (0.3) [30.4, 31.5] 25.9 (0.3) [25.4, 26.4] .18 29.8 (0.2) [29.4, 30.3] 23.0 (0.4) [22.2, 23.9] .21
BVMT-R Immediate 24.9 (0.2) [24.6, 25.3] 18.3 (0.4) [17.5, 19.0] .23 23.9 (0.2) [23.6, 24.3] 15.8 (1.1) [13.5, 18.0] .06 26.0 (0.2) [25.5, 26.4] 21.3 (0.2) [20.9, 21.8] .20 24.7 (0.2) [24.4, 25.1] 19.5 (0.4) [18.8, 20.3] .16
Delayed memory
RAVLT Long-Delay 10.7 (0.1) [10.5, 10.9] 7.1 (0.2) [6.7, 7.6] .21 10.1 (0.1) [9.9, 10.3] 5.6 (0.6) [4.4, 6.8] .06 11.5 (0.1) [11.2, 11.7] 8.5 (0.1) [8.3, 8.8] .26 10.7 (0.1) [10.5, 10.9] 7.3 (0.2) [6.9–7.7] .22
LM Delay 26.3 (0.2) [25.8, 26.7] 19.6 (0.5) [18.5, 20.6] .14 25.3 (0.2) [24.9, 25.8] 14.5 (1.5) [11.7, 17.4] .07 28.0 (0.3) [27.4, 28.6] 22.0 (0.3) [21.4, 22.6] .20 26.7 (0.2) [26.2, 27.4] 18.4 (0.5) [17.5, 19.4] .24
BVMT-R Delay 9.8 (0.1) [9.7, 9.9] 7.6 (0.1) [7.3, 7.8] .20 9.5 (0.1) [9.3, 9.6] 6.3 (0.4) [5.5, 7.1] .07 10.2 (0.1) [10.0, 10.3] 8.6 (0.1) [8.4, 8.8] .17 9.8 (0.1) [9.6, 9.9] 7.9 (0.1) [7.6, 8.1] .17
Executive functioning
TMT B 59.1 (0.9) [57.3, 60.1] 80.9 (2.0) [76.9, 84.8] .11 61.9 (0.9) [60.2, 63.6] 103.1 (5.3) [92.7,113.6] .07 55.8 (1.2) [53.5, 58.1] 70.5 (1.2) [68.2, 72.9] .09 59.5 (0.9) [57.6, 61.3] 77.2 (1.9) [73.4, 80.9] .08
Stroop Color-Word 111.5 (0.7) [110.1, 113.0] 96.9 (1.6) [93.7, 100.1] .08 109.6 (0.7) [108.3, 111.0] 82.2 (4.4) [73.6, 90.8] .05 114.9 (0.9) [113.1, 116.8] 102.6 (1.0) [100.7, 104.5] .10 111.0 (0.7) [110.5, 113.4] 96.6 (1.5) [93.6, 99.6] .10
Digit Symbol 57.9 (0.4) [57.2, 58.6] 51.6 (0.8) [50.0, 53.2] .06 57.0 (0.3) [56.4, 57.7] 47.4 (2.1) [43.2, 51.5] .03 60.1 (0.5) [59.2, 61.0] 53.3 (0.5) [52.4, 54.2] .13 58.2 (0.4) [57.5, 58.9] 50.9 (0.7) [49.5, 52.4] .09

Note: RAVLT = Rey Auditory Verbal Learning Test; LM = Wechsler Memory Scale-Revised Logical Memory subtest; BVMT-R = Brief Visuospatial Memory Test-Revised; TMT = Trailmaking Test. ANCOVA model included covariates of age at Wave 2, gender, and literacy estimate (WRAT-III reading subtest Wave 2 raw score). Higher scores on TMT B indicate worse performance. All pMCI versus CN comparisons were statistically significant at the p< .001 level.

### Stability of pMCI Classification

Cognitive statuses were available for three consecutive visits for 303 participants. For each classification scheme, Table 5 displays the number (%) of participants that retained the same diagnosis across three visits, reverted from pMCI to CN at both follow-up visits or progressed from CN to pMCI at both follow-up visits, or exhibited fluctuations in diagnosis across the three visits (e.g., pMCI, CN, and pMCI).

Table 5.

Stability of classification at follow-up visits (n; percentage; [95% confidence interval])

Initial classification (Wave 2)

Outcome (Waves 3 and 4)

Stable Reversion (pMCI to CN) Progression (CN to pMCI) Fluctuating
CN
Standard norms/Single-Test 257 231; 90% [86.3, 93.6]  6; 2% [0.5, 4.2] 20; 8% [4.5, 11.1]
Standard norms/Multi-Test 297 289; 97% [95.3, 99.0]  0; 0% [0, 0.6] 8; 3% [0.9, 4.5]
Robust norms/Single-Test 165 118; 72% [64.6, 78.4]  11; 7% [2.9, 10.5] 36; 22% [15.5, 28.1]
Robust norms/Multi-Test 254 215; 85% [80.2, 89.1]  15; 6% [3.0, 8.8] 24; 9% [5.9, 13.1]
pMCI
Standard norms/Single-Test 47 11; 23.5% [11.3, 35.5] 27; 57.5% [43.3, 71.6]  9; 19% [7.9, 30.4]
Standard norms/Multi-Test 1; 17% [0, 46.5] 3; 50% [10.0, 90.0]  2; 33% [0, 71.1]
Robust norms/Single-Test 139 65; 47% [48.5, 55.1] 58; 42% [33.5, 49.9]  16; 11% [6.2, 16.8]
Robust norms/Multi-Test 50 20; 40% [26.4, 53.6] 12; 24% [12.2, 35.8]  18; 36% [22.7, 49.3]
Initial classification (Wave 2)

Outcome (Waves 3 and 4)

Stable Reversion (pMCI to CN) Progression (CN to pMCI) Fluctuating
CN
Standard norms/Single-Test 257 231; 90% [86.3, 93.6]  6; 2% [0.5, 4.2] 20; 8% [4.5, 11.1]
Standard norms/Multi-Test 297 289; 97% [95.3, 99.0]  0; 0% [0, 0.6] 8; 3% [0.9, 4.5]
Robust norms/Single-Test 165 118; 72% [64.6, 78.4]  11; 7% [2.9, 10.5] 36; 22% [15.5, 28.1]
Robust norms/Multi-Test 254 215; 85% [80.2, 89.1]  15; 6% [3.0, 8.8] 24; 9% [5.9, 13.1]
pMCI
Standard norms/Single-Test 47 11; 23.5% [11.3, 35.5] 27; 57.5% [43.3, 71.6]  9; 19% [7.9, 30.4]
Standard norms/Multi-Test 1; 17% [0, 46.5] 3; 50% [10.0, 90.0]  2; 33% [0, 71.1]
Robust norms/Single-Test 139 65; 47% [48.5, 55.1] 58; 42% [33.5, 49.9]  16; 11% [6.2, 16.8]
Robust norms/Multi-Test 50 20; 40% [26.4, 53.6] 12; 24% [12.2, 35.8]  18; 36% [22.7, 49.3]

Note: Stable = same classification given at Waves 2, 3, and 4; Reversion = pMCI at Wave 2, CN at Waves 3 and 4; Progress = CN at Wave 2, pMCI at Waves 3 and 4; Fluctuating = Classification changed across Waves 2, 3, and 4 (pMCI, CN, pMCI or CN, pMCI, CN).

Using single-test criteria, a greater percentage of participants classified as pMCI remained stable at two follow-up visits when robust normative data were used to define impairment (47%) compared with standard norms (23.5%). Additionally, fewer participants reverted from pMCI to CN using robust norms (42%) compared with standard norms (57.5%). However, a greater percentage of participants initially classified as CN remained normal at follow-up visits using standard norms (90% vs. 72%), and fewer progressed to a pMCI diagnosis at follow-up (2% vs. 7%).

Using multi-test criteria, a greater proportion of participants initially classified as pMCI using robust norms were also classified as pMCI at two follow-up visits (40%) compared with those classified using standard norms (17%). Additionally, fewer participants classified using robust norms reverted from pMCI to CN diagnoses at any follow-up visits (24%) compared with those classified using standard norms (50%). Similar to above, more CN participants remained cognitively stable at follow-up visits when standard norms were applied (97%) compared with when robust norms were used (85%).

### Global Cognition and Functional Ability at Most Recent Visit

Participants classified as pMCI at Wave 2 using robust norms comprised a greater proportion of participants that were rated as mildly impaired (0.5) on the CDR at their most recent follow-up visit than participants classified as CN, although this difference did not reach statistical significance (Robust/Single-Test: χ2(1,N= 360) = 3.64, p= .06; Robust/Multi-test: χ2(1,N= 360) = 2.90, p= .09).

There were no differences in CDR rating at the most recent visit between those initially classified as pMCI or CN at Wave 2 using standard norms (either single-test or multi-test criteria; p's > .16). Furthermore, there were no differences between pMCI and CN participants in subjective cognitive complaints reported by an informant on the IQCODE at the most recent visit among any of the four classification schemes (p's > .25).

## Discussion

The current study aimed to combine two approaches that demonstrate evidence to improve sensitivity and reliability of MCI diagnoses: (i) Use of robust norms, and (ii) application of multi-test criteria for classification of impairment. We extended previous studies by applying these methods to the identification of very early MCI in late middle-aged adults enriched for a parental history of AD. Robust normative data were developed for neuropsychological measures of memory, executive functioning, attention/processing speed, language, and visuospatial functioning based on an internal sample of middle-aged individuals who remained CN across multiple visits. The parameter estimates presented in Table 1 (β-coefficients and RMSE) can be used with similar samples to create predicted scores and robust z-scores for each neuropsychological test available.

To investigate the impact of using various MCI diagnostic criteria in late middle-age, participants were classified as impaired via single-test criteria (e.g., at least one test impaired) and multi-test criteria (e.g., at least two impaired tests in a domain, or one test in each of three domains). As CN participants may perform below normal thresholds on one neuropsychological test in a large battery (Binder et al., 2009; Schretlen, Testa, Winicki, Pearlson, & Gordon, 2008), single-test criteria may overclassify MCI (Clark et al., 2013; Edmonds et al., 2015; Saxton et al., 2009). Similar to previous studies, a larger percentage of the WRAP cohort was identified as pMCI when using single-test versus multi-test criteria. In particular, when single-test criteria were used in conjunction with robust norms, a much larger than expected proportion of participants was classified as impaired (49%), suggesting that the combination of potentially sensitive norms and a single test to define impairment is overly inclusive. This pMCI group (Robust/Single-Test), on average, exhibited higher scores on cognitive tests. Conversely, the combination of multi-test criteria and standard norms may have reduced the ability to detect mild deficits in late middle-age, classifying only 3% of the sample as pMCI. Additionally, this pMCI group (Standard/Multi-Test) exhibited lower scores on cognitive tests, suggesting that only the most significantly impaired participants were identified with this method. It should be noted that our application of multi-test criteria differs in some respects from previous studies that have used this approach (Bondi et al., 2014; Clark et al., 2013; Jak et al., 2009). Our use of a more conservative threshold (e.g., 1.5 SD instead of 1.0 SD), although applied in an attempt to reduce false-positive pMCI classification, may have excluded individuals who demonstrate mild decline predictive of an MCI diagnosis.

Previous studies indicate that robust norms may be more sensitive to early decline in MCI or AD due to reduced confounding by individuals in preclinical stages of AD or other dementia syndromes (De Santi et al., 2008; Ritchie, Frerichs, & Tuokko, 2007; Sliwinski et al., 1996). Supporting this point, use of robust norms identified most participants classified as impaired by standard norms, while also identifying an additional subset as impaired that were classified as CN by standard norms. These results suggest that robust norms may identify milder deficits not captured by the standard norms used in this study. Whenever possible, we used standard norms published in test manuals, but for some tests (e.g., TMT, RAVLT) there are several sets of norms that vary in sample characteristics, which may affect the sensitivity of the norms to impairment in this study. Moreover, the standard norms did not include as many demographic corrections as the robust norms. To examine if the classification differences between the norms were due only to greater demographic corrections, rather than the robust nature of the normative sample, we calculated the frequency of participants who demonstrated impaired performances based on the robust normative sample, but only adjusted for age. Results of these post hoc analyses demonstrated that differences in the frequency of impaired performances existed between the robust and standard norms even when both were only adjusted for age (see Supplementary material online, TableS1). Additionally, more participants were classified as impaired on each measure when using robust norms adjusted for age, gender, and literacy than those only adjusted for age. These findings suggest that use of robust norms with demographic corrections may provide greater ability to identify very mild deficits on these measures.

Furthermore, demographic and clinical characteristics varied depending on criteria used to classify pMCI. Individuals classified as pMCI using standard norms (and either single-test or multi-test criteria) exhibited lower estimates of literacy compared with those classified as CN using the same criteria. Those with pMCI did not differ from CN in proportion of participants with parental history of AD across all classification schemes. Individuals classified as pMCI using multi-test criteria (and either standard or robust norms) comprised a greater proportion of participants with clinical MCI symptoms compared with those classified as CN using the same criteria, including subjective cognitive complaints (by participant or informant) and mild difficulties with IADLs. These latter findings suggest a possibility that use of multi-test criteria may identify participants more likely to also meet criteria for clinical MCI; however, as subjective complaints have previously been reported to correlate more strongly with emotional factors than with objective memory performance (Buckley et al., 2013; Slavin et al., 2010), further investigation will be necessary to understand the implications of these findings. Importantly, the present study focused on identification of objective measures of impairment expected to precede a clinical diagnosis of MCI, and therefore did not require a subjective cognitive complaint to be classified as pMCI. Nevertheless, a clinical evaluation including medical and psychological factors, symptom course and frequency of presentation, and collateral information regarding independent functioning is essential for accurate clinical diagnoses of MCI.

Moreover, participants classified using robust normative data compared with standard normative data exhibited greater stability of pMCI classification (40%–47% vs. 17%–23% of participants remained pMCI) and less reversion to CN (24%–42% vs. 50%–57% reverted to CN) at two follow-up visits. Additionally, a greater proportion of participants classified as pMCI using multi-test criteria (and either set of norms) demonstrated a trend towards mild clinical symptoms and functional decline at follow-up, as measured by a 0.5 rating on the CDR scale. These observations suggest that applying robust normative datasets and multi-test criteria to define impairment in late middle-age may allow for detection of early/mild cognitive deficits, as well as provide improved reliability of the classification over time. However, it is also possible that the cost of improved detection of milder deficits is a greater false-positive classification rate that remains false-positive over time. Additionally, performance validity tests were not included and it is possible that sub-optimal effort or engagement at one or all evaluations could influence classification stability. However, the participants enrolled in WRAP return for multiple follow-up visits and are a highly motivated cohort.

Rates of clinical MCI in the literature are varied (e.g., 3%–42%; Ward et al., 2012), with reported rates in adults over age 70 appearing to converge ∼14%–18% (Petersen et al., 2009). An estimated prevalence rate of pMCI is unknown; however, a study of preclinical AD based on biomarkers reported a 31% prevalence rate in an older adult sample, with the rate slightly lower for those under the age of 72 (26%) (Vos et al., 2013). However, the current sample is younger than these older samples, and it is important to note that the impairment which led to a classification of pMCI in our sample may reflect other etiologies that can interfere with cognition besides preclinical AD, such as depression. Ultimately, validation of these methods awaits further longitudinal follow-up to clinical endpoints that are currently ongoing. Prior investigations of preclinical AD report gradual memory decline in the earliest stages, followed by a steeper decline in memory and other cognitive functions nearer to a clinical MCI diagnosis (Snyder et al., 2014). Other studies of cognitive decline in presymptomatic AD indicate the importance of using sensitive memory measures and longitudinal designs to reliably identify early changes (Sperling et al., 2011). For example, Caselli and colleagues (2014) demonstrated longitudinal memory decline in middle-aged APOE ϵ4 carriers compared with non-carriers, but no differences in memory performance at baseline. Moreover, a previous WRAP study indicated an association between memory decline and age occurred only in participants classified as amnestic pMCI at multiple visits (Koscik et al., 2014). Additional follow-up studies in WRAP will include comparisons of longitudinal cognitive trajectories depending on classification criteria, and development of longitudinal change norms.

Important limitations of the robust normative sample include: (i) the absence of clinical endpoints of WRAP participants (e.g., although individuals were at low genetic risk for AD and have not exhibited decline over a 4- to 10-year period, it is not known which individuals will avoid AD in the future); (ii) examination of only learning/memory and executive functioning factors in defining cognitive stability for inclusion in the robust normative sample; and (iii) current sample characteristics comprise 90% Caucasian, highly educated, middle-aged adults from urban or rural areas of Wisconsin and the Midwest region, which may differ from sample characteristics in the standard normative datasets used. Due to the second limitation, it is possible that the classification approaches were less accurate at identifying pMCI participants with more atypical cognitive patterns of decline; however, identification of preclinical AD (rather than non-AD dementia syndromes) was the aim of the current study. Due to the third limitation, the robust norms may be less accurate at identifying pMCI in non-Caucasian WRAP participants at this time. However, data collection from participants representing greater diversity is currently ongoing and we plan to update the robust normative sample as a larger sample of follow-up data for these participants becomes available. Finally, Wave 2 was used as the “baseline” in our analyses since some measures included in the pMCI classification scheme were not administered at Wave 1. However, it is possible that tests administered at Waves 1 and 2 (e.g., RAVLT, TMT B, and Stroop) were less sensitive to impairment than those first administered at Wave 2 due to practice effects.

In conclusion, the present investigation provides normative data on a late middle-aged sample with reduced confounding by presymptomatic AD. Comparisons of individuals at risk for MCI to this normative dataset or other available robust norms may be more sensitive to early or mild cognitive deficits occurring during middle-age. Compared with standard normative datasets, our results suggest that use of robust norms in late middle-age in combination with multi-test criteria may identify reliable mild deficits prior to clinical symptoms of MCI or dementia.

## Supplementary material

Supplementary material is available at Archives of Clinical Neuropsychology online.

## Funding

This work was supported by the Clinical Translational Science Award (CTSA) program, through the National Institutes of Health National Center for Advancing Translational Sciences (NCATS), and grant UL1TR00427. Funding support was also provided by the National Institutes of Health (NIH) (R01 AG027161 to S.C.J., R01 AG021155 to S.C.J., ADRC P50 AG033514 to S.A.) and the Wisconsin Alzheimer's Institute (WAI) Holland Research Fund. The content is solely the responsibility of the authors and does not represent the official views of NIH.

None declared.

## Acknowledgements

The authors gratefully acknowledge the assistance of Janet Rowley, Amy Hawley, Allen Wenzel, Shawn Bolin, Lisa Bluder, Diane Wilkinson, Emily Groth, Susan Schroeder, Laura Hegge, Chuck Illingworth, and Jen Oh. Most importantly, we wish to thank our dedicated participants of the WRAP for their continued support and participation in this research.

## References

Albert
M.
,
Soldan
A.
,
Gottesman
R.
,
McKhann
G.
,
Sacktor
N.
,
Farrington
L.
et al
. (
2014
).
Cognitive changes preceding clinical symptom onset of mild cognitive impairment and relationship to ApoE genotype
.
Current Alzheimer Research
,
11
,
773
784
.
Albert
M. S.
,
DeKosky
S. T.
,
Dickson
D.
,
Dubois
B.
,
Feldman
H. H.
,
Fox
N. C.
et al
. (
2011
).
The diagnosis of mild cognitive impairment due to Alzheimer's disease: Recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease
.
Alzheimer's & Dementia
,
7
,
270
279
.
Albert
M. S.
,
Moss
M. B.
,
Tanzi
R.
,
Jones
K.
(
2001
).
Preclinical prediction of AD using neuropsychological tests
.
Journal of the International Neuropsychological Society
,
7
,
631
639
.
Benedict
R. H.
(
1997
).
Brief visuospatial memory test-revised
.
Odessa, FL
:
Psychological Assessment Resources
.
Binder
L. M.
,
Iverson
G. L.
,
Brooks
B. L.
(
2009
).
To err is human: “abnormal” neuropsychological scores and variability are common in healthy adults
.
Archives of Clinical Neuropsychology
,
24
,
31
46
.
Bondi
M. W.
,
Edmonds
E. C.
,
Jak
A. J.
,
Clark
L. R.
,
Delano-Wood
L.
,
McDonald
C. R.
et al
. (
2014
).
Neuropsychological criteria for mild cognitive impairment improves diagnostic precision, biomarker associations, and progression rates
.
Journal of Alzheimer's Disease
,
42
,
275
289
.
Bondi
M. W.
,
Smith
G. E.
(
2014
).
Mild cognitive impairment: A concept and diagnostic entity in need of input from neuropsychology
.
Journal of the International Neuropsychological Society
,
20
,
129
134
.
Brooks
B. L.
,
Iverson
G. L.
,
Holdnack
J. A.
,
Feldman
H. H.
(
2008
).
Potential for misclassification of mild cognitive impairment: A study of memory scores on the Wechsler Memory Scale-III in healthy older adults
.
Journal of the International Neuropsychological Society
,
14
,
463
478
.
Buckley
R.
,
Saling
M. M.
,
Ames
D.
,
Rowe
C. C.
,
Lautenschlager
N. T.
,
Macaulay
S. L.
et al
. (
2013
).
Factors affecting subjective memory complaints in the AIBL aging study: Biomarkers, memory, affect, and age
.
International Psychogeriatrics
,
25
,
1307
1315
.
Caselli
R. J.
,
Locke
D. E.
,
Dueck
A. C.
,
Knopman
D. S.
,
Woodruff
B. K.
,
Hoffman-Snyder
C.
et al
. (
2014
).
The neuropsychology of normal aging and preclinical Alzheimer's disease
.
Alzheimer's & Dementia
,
10
,
84
92
.
Chang
Y. L.
,
Bondi
M. W.
,
McEvoy
L. K.
,
Fennema-Notestine
C.
,
Salmon
D. P.
,
D.
et al
. (
2011
).
Global clinical dementia rating of 0.5 in MCI masks variability related to level of function
.
,
76
,
652
659
.
Chen
P.
,
Ratcliff
G.
,
Belle
S. H.
,
Cauley
J. A.
,
DeKosky
S. T.
,
Ganguli
M.
(
2001
).
Patterns of cognitive decline in presymptomatic Alzheimer disease: A prospective community study
.
Archives of General Psychiatry
,
58
,
853
858
.
Clark
L. R.
,
Delano-Wood
L.
,
Libon
D. J.
,
McDonald
C. R.
,
Nation
D. A.
,
Bangen
K. J.
et al
. (
2013
).
Are empirically-derived subtypes of mild cognitive impairment consistent with conventional subtypes?
Journal of the International Neuropsychological Society
,
19
,
635
645
.
De Santi
S.
,
Pirraglia
E.
,
Barr
W.
,
Babb
J.
,
Williams
S.
,
Rogers
K.
et al
. (
2008
).
Robust and conventional neuropsychological norms: Diagnosis and prediction of age-related cognitive decline
.
Neuropsychology
,
22
,
469
484
.
Donohue
M. C.
,
Sperling
R. A.
,
Salmon
D. P.
,
Rentz
D. M.
,
Raman
R.
,
Thomas
R. G.
et al
. (
2014
).
The preclinical Alzheimer cognitive composite: Measuring amyloid-related decline
.
JAMA Neurology
,
71
,
961
970
.
Edmonds
E. C.
,
Delano-Wood
L.
,
Clark
L. R.
,
Jak
A. J.
,
Nation
D. A.
,
McDonald
C. R.
et al
. (
2015
).
Susceptibility of the conventional criteria for mild cognitive impairment to false-positive diagnostic errors
.
Alzheimer's & Dementia
,
11
,
415
424
.
Elias
M. F.
,
Beiser
A.
,
Wolf
P. A.
,
Au
R.
,
White
R. F.
,
D'Agostino
R. B.
(
2000
).
The preclinical phase of Alzheimer disease: A 22-year prospective study of the Framingham Cohort
.
Archives of Neurology
,
57
,
808
813
.
Heaton
R. K.
,
Miller
S. W.
,
Taylor
M. J.
,
Grant
I.
(
2004
).
Revised comprehensive norms for an expanded Halstead-Reitan battery: Demographically adjusted neuropsychological norms for African American and Caucasian Adults
.
Lutz, FL
:
Psychological Assessment Resources Inc
.
Howieson
D. B.
,
Carlson
N. E.
,
Moore
M. M.
,
Wasserman
D.
,
Abendroth
C. D.
,
Payne-Murphy
J.
et al
. (
2008
).
Trajectory of mild cognitive impairment onset
.
Journal of the International Neuropsychological Society
,
14
,
192
198
.
Jak
A. J.
,
Bondi
M. W.
,
Delano-Wood
L.
,
Wierenga
C.
,
Corey-Bloom
J.
,
Salmon
D. P.
et al
. (
2009
).
Quantification of five neuropsychological approaches to defining mild cognitive impairment
.
The American Journal of Geriatric Psychiatry
,
17
,
368
375
.
Jorm
A. F.
(
1994
).
A short form of the Informant Questionnaire on Cognitive Decline in the Elderly (IQCODE): Development and cross-validation
.
Psychological Medicine
,
24
,
145
153
.
Jorm
A. F.
(
2004
).
The Informant Questionnaire on cognitive decline in the elderly (IQCODE): A review
.
International Psychogeriatrics
,
16
,
275
293
.
Kawas
C.
,
Segal
J.
,
Stewart
W. F.
,
M.
,
Thal
L. J.
(
1994
).
A validation study of the Dementia Questionnaire
.
Archives of Neurology
,
51
,
901
906
.
Koscik
R. L.
,
La Rue
A.
,
Jonaitis
E. M.
,
Okonkwo
O. C.
,
Johnson
S. C.
,
Bendlin
B. B.
et al
. (
2014
).
Emergence of mild cognitive impairment in late middle-aged adults in the Wisconsin registry for Alzheimer's prevention
.
Dementia and Geriatric Cognitive Disorders
,
38
,
16
30
.
Lawton
M. P.
,
Brody
E. M.
(
1969
).
Assessment of older people: Self-maintaining and instrumental activities of daily living
.
The Gerontologist
,
9
,
179
186
.
Loewenstein
D. A.
,
Acevedo
A.
,
Small
B. J.
,
Agron
J.
,
Crocco
E.
,
Duara
R.
(
2009
).
Stability of different subtypes of mild cognitive impairment among the elderly over a 2- to 3-year follow-up period
.
Dementia and Geriatric Cognitive Disorders
,
27
,
418
423
.
McKhann
G.
,
Drachman
D.
,
Folstein
M.
,
Katzman
R.
,
Price
D.
,
E. M.
(
1984
).
Clinical diagnosis of Alzheimer's disease: Report of the NINCDS-ADRDA Work Group under the auspices of Department of Health and Human Services Task Force on Alzheimer's Disease
.
,
34
,
939
944
.
Morris
J. C.
(
1997
).
Clinical dementia rating: A reliable and valid diagnostic and staging measure for dementia of the Alzheimer type
.
International Psychogeriatrics
,
9
(Suppl. 1)
,
173
176
;
discussion 177–178
.
Pedraza
O.
,
Lucas
J. A.
,
Smith
G. E.
,
Petersen
R. C.
,
N. R.
,
Ivnik
R. J.
(
2010
).
Robust and expanded norms for the Dementia Rating Scale
.
Archives of Clinical Neuropsychology
,
25
,
347
358
.
Petersen
R. C.
,
Aisen
P. S.
,
Beckett
L. A.
,
Donohue
M. C.
,
Gamst
A. C.
,
Harvey
D. J.
et al
. (
2010
).
Alzheimer's Disease Neuroimaging Initiative (ADNI): Clinical characterization
.
,
74
,
201
209
.
Petersen
R. C.
,
Roberts
R. O.
,
Knopman
D. S.
,
Boeve
B. F.
,
Geda
Y. E.
,
Ivnik
R. J.
et al
. (
2009
).
Mild cognitive impairment: Ten years later
.
Archives of Neurology
,
66
,
1447
1455
.
Petersen
R. C.
,
Thomas
R. G.
,
Grundman
M.
,
Bennett
D.
,
Doody
R.
,
Ferris
S.
et al
. (
2005
).
Vitamin E and donepezil for the treatment of mild cognitive impairment
.
The New England Journal of Medicine
,
352
,
2379
2388
.
Ritchie
L. J.
,
Frerichs
R. J.
,
Tuokko
H.
(
2007
).
Effective normative samples for the detection of cognitive impairment in older adults
.
Archives of Clinical Neuropsychology
,
21
,
863
874
.
Sager
M. A.
,
Hermann
B.
,
La Rue
A.
(
2005
).
Middle-aged children of persons with Alzheimer's disease: APOE genotypes and cognitive function in the Wisconsin Registry for Alzheimer's Prevention
.
Journal of Geriatric Psychiatry and Neurology
,
18
,
245
249
.
Saxton
J.
,
Snitz
B. E.
,
Lopez
O. L.
,
Ives
D. G.
,
Dunn
L. O.
,
Fitzpatrick
A.
et al
. (
2009
).
Functional and cognitive criteria produce different rates of mild cognitive impairment and conversion to dementia
.
Journal of Neurology, Neurosurgery, and Psychiatry
,
80
,
737
743
.
Schmidt
M.
(
1996
).
Rey Auditory Verbal Learning Test: A Handbook
.
Los Angeles, CA
:
Western Psychological Services
.
Schretlen
D. J.
,
Testa
S. M.
,
Winicki
J. M.
,
Pearlson
G. D.
,
Gordon
B.
(
2008
).
Frequency and bases of abnormal performance by healthy adults on neuropsychological testing
.
Journal of the International Neuropsychological Society
,
14
,
436
445
.
Shirk
S. D.
,
Mitchell
M. B.
,
Shaughnessy
L. W.
,
Sherman
J. C.
,
Locascio
J. J.
,
Weintraub
S.
et al
. (
2011
).
A web-based normative calculator for the uniform data set (UDS) neuropsychological test battery
.
Alzheimer's Research & Therapy
,
3
,
32
.
Slavin
M. J.
,
Brodaty
H.
,
Kochan
N. A.
,
Crawford
J. D.
,
Trollor
J. N.
,
Draper
B.
et al
. (
2010
).
Prevalence and predictors of “subjective cognitive complaints” in the Sydney Memory and Ageing Study
.
The American Journal of Geriatric Psychiatry
,
18
,
701
710
.
Sliwinski
M.
,
Lipton
R. B.
,
Buschke
H.
,
Stewart
W.
(
1996
).
The effects of preclinical dementia on estimates of normal cognitive functioning in aging
.
The Journals of Gerontology. Series B, Psychological Sciences and Social Sciences
,
51
,
P217
P225
.
Snyder
P. J.
,
Kahle-Wrobleski
K.
,
Brannan
S.
,
Miller
D. S.
,
Schindler
R. J.
,
DeSanti
S.
et al
. (
2014
).
Assessing cognition and function in Alzheimer's disease clinical trials: Do we have the right tools?
Alzheimer's & Dementia
,
10
,
853
860
.
Sperling
R. A.
,
Aisen
P. S.
,
Beckett
L. A.
,
Bennett
D. A.
,
Craft
S.
,
Fagan
A. M.
et al
. (
2011
).
Toward defining the preclinical stages of Alzheimer's disease: Recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease
.
Alzheimer's & Dementia
,
7
,
280
292
.
Trenerry
M. R.
,
Crosson
B.
,
DeBoe
J.
,
Leber
W. R.
(
1989
).
The Stroop Neuropsychological Screening Test
.
Odessa, FL
:
Psychological Assessment Resources
.
Vos
S. J.
,
Xiong
C.
,
Visser
P. J.
,
Jasielec
M. S.
,
Hassenstab
J.
,
Grant
E. A.
et al
. (
2013
).
Preclinical Alzheimer's disease and its outcome: A longitudinal cohort study
.
Lancet Neurology
,
12
,
957
965
.
Ward
A.
,
Arrighi
H. M.
,
Michels
S.
,
Cedarbaum
J. M.
(
2012
).
Mild cognitive impairment: Disparity of incidence and prevalence estimates
.
Alzheimer's & Dementia
,
8
,
14
21
.
Wechsler
. (
1981
).
.
San Antonio
:
The Psychological Corporation, Harcourt Brace & Co for The Psychological Corporation
.
Wechsler
. (
1987
).
Wechsler Memory Scale-Revised
.
New York
:
The Psychological Corporation, Harcourt Brace Jovanovich, Inc. for Psychological Corp
.
Wilkinson
G.
(
1993
).
The wide range achievement test: Manual
(
3rd ed.
).
Wilmington, DE
:
Jastak Association
.