Abstract

The Symbol--Digit Modalities Test (SDMT) is widely used to assess processing speed in MS patients. We developed a computerized version of the SDMT (c-SDMT) that scored participants' performance during subintervals over the course of the usual 90-s time period and also added an incidental learning test (c-ILT) to assess how well participants learned the symbol-digit associations while completing the c-SDMT. Patients with MS (n = 65) achieved lower scores than healthy controls (n = 38) on both the c-SDMT and c-ILT, and the scores on the two tests were correlated. However, no increase in the rate of item completion occurred for either group over the course of the c-SDMT, and the difference between groups was the same during each subinterval. Therefore, it seems implausible that controls completed more items on the c-SDMT because they were more adept at learning the symbol-digit associations as the test ensued. Instead, MS patients' poorer incidental learning performance appears to reflect the greater attentional burden that tasks requiring rapid serial processing of information impose upon them.

Introduction

The primary cognitive deficit characterizing patients with multiple sclerosis (MS) appears to be a generalized slowing in the speed of information processing (de Sonneville et al., 2002; DeLuca, Chelune, Tulsky, Lengenfelder, & Chiaravalloti, 2004; Denney, Gallagher, & Lynch, 2011; Kujala, Portin, Revonsuo, & Ruutiainen, 1994; Reicker, Tombaugh, Walker, & Freedman, 2007). Compared with healthy controls, patients with MS complete substantially fewer items on rapid serial processing tests such as the Paced Auditory Serial Addition Test (PASAT; Kujala, Portin, Revonsuo, & Ruutiainen, 1995; Litvan, Grafman, Vendrell, & Martinez, 1988) or the Symbol–Digit Modalities Test (SDMT; Parmenter, Weinstock-Guttman, Garg, Munschauer, & Benedict, 2007), and also exhibit substantially longer response times on a variety of reaction time measures (Bodling, Denney, & Lynch, 2012; de Sonneville et al., 2002; Kujala et al., 1994, Reicker et al., 2007; Tombaugh, Berrigan, Walker, & Freedman, 2010).

The SDMT (Rao, 1990; Smith, 1982) has emerged as one of the most commonly used tests of information processing speed for patients with MS. The items of this test consist of geometric symbols arranged in rows on a printed test form. A reference key at the top of the form shows the nine symbols used on the test and the digit associated with each symbol. Subjects proceed along the rows of the test, reporting the digit corresponding to each symbol. The score is the number of correct items completed during the 90-s trial. In addition to its greater brevity, ease of administration, and acceptability to patients (Walker et al., 2012), recent studies have shown scores on the SDMT to be more highly related to structural MRI measures bearing on MS such as lesion load and atrophy than the older PASAT (Batista et al., 2012; Papadopoulou et al., 2013; Rao et al., 2014). Several investigators have suggested replacing the PASAT with the SDMT as the measure of cognitive impairment on the MS Functional Composite Index (e.g., Brochet et al., 2008; Drake, Weinstock-Guttman, Morrow, Hojnacki, Munschauer, & Benedict, 2010).

A number of computerized versions of the SDMT have been developed (Akbar, Honarmand, Kou, & Feinstein, 2011; Hughes, Denney, Owens, & Lynch, 2013). The purpose of these versions has usually been either to adapt the test for use in conjunction with fMRI studies (DeLuca, Genova, Hillary, & Wylie, 2008; Forn et al., 2009,, 2013; Genova, Hillary, Wylie, Rypma, & Deluca, 2009) or to expedite its administration as part of an automated testing routine in a clinic setting (Lapshin, Lanctôt, O'Connor, & Feinstein, 2013; Lapshin, Audet, & Feinstein, 2014; Ruet, Deloire, Charré-Morin, Hamel, & Brochet, 2013). In the first instance, substantial differences have been introduced on fMRI-adapted versions, such as having item presentation paced by the computer instead of the subject (Forn et al., 2009; Rypma et al., 2006), thereby compromising the test as a measure of processing speed. When the test has been computerized for clinical application, the differences from the original paper version are relatively trivial, such as having the items appear on the screen individually (Hughes et al., 2013) or providing a blinking cursor to help the subject locate each successive item to be solved (Ruet et al., 2013). The clinical adaptations perform well in comparison with the original SDMT in terms of distinguishing impaired from non-impaired MS patients (Akbar et al., 2011; Lapshin et al., 2013; Ruet et al., 2013), and correlations of 0.88 (Ruet et al., 2013) and 0.86 (Hughes et al., 2013) have been reported between patients' scores on these computerized forms and the original SDMT.

A computerized test can often be designed to evaluate additional features of subjects' performance which might otherwise be difficult to capture. For example, one of the modifications made in the present study was to have the computer divide the 90-s work time into three 30-s intervals and record the number of items completed in each interval as well as the total score. As elaborated upon later, we believed the interval scores would provide useful information bearing on the hypotheses of the study.

We also appended an incidental learning test immediately following the computerized SDMT (c-SDMT) to assess the number of symbol–digit pairings the subject managed to learn over the course of working on the c-SDMT (Although some authorities (e.g., Reber, 2008) point to subtle distinctions between incidental learning/memory and implicit learning/memory, here the terms are considered synonymous, and accordingly, previous studies of either incidental learning or implicit memory were reviewed in preparing this article.). Similar tests have been used with the Digit–Symbol subtest of the Wechsler intelligent scales, mainly to examine the contribution of incidental learning to performance on this subtest (Joy, Kaplan, & Fein, 2003, 2004). The conclusion from these studies is that incidental learning makes a modest contribution to overall performance on the Digit–Symbol subtest, one that is clearly secondary to processing speed. However, this conclusion is based primarily on studies of healthy adults. We were interested in the relevance of incidental learning to the observed disparity between MS patients and healthy individuals in performance on the SDMT.

It seems reasonable that individuals who manage to learn the symbol–digit pairings over the course of working on the SDMT would be able to achieve higher scores by not having to repeatedly shift their gaze between the test items and the reference key at the top of the test. If healthy controls are more adept at learning these pairs than MS patients, they should achieve higher scores on an incidental learning test following the SDMT. Furthermore, the disparity between the two groups' performance during the SDMT should increase over the course of the 90-s trial as incidental acquisition of the symbol–digit pairings occurs. Therefore, in the present study, we compared the performance of MS patients and healthy controls on a computerized version of the SDMT that scored performance during three successive intervals of the overall trial and was followed by an incidental learning test. The hypotheses were that (a) patients would achieve lower scores on the c-SDMT than controls, (b) patients would achieve lower scores on the subsequent incidental learning test, and (c) the disparity between patients and controls in performance on the c-SDMT would increase over the course of the trial.

Materials and Methods

Participants

Sixty-five patients (37 females, 28 males) with clinically definite MS (Polman et al., 2011) and 38 healthy controls of comparable demographics (age, sex, and education) were enrolled in this study. Patients were recruited during their regularly scheduled appointment at an MS Clinic in a large University Medical Center, had been under the care of the same neurologist for at least 1 year, and were diagnosed with relapsing-remitting (RRMS: n = 24), primary progressive (PPMS: n = 17), or secondary progressive MS (SPMS: n = 24). Patients ranged in age from 26 to 65 (M = 49.1, SD = 9.8) and had 12–20 years of education (M = 15.1, SD = 2.0). They had been diagnosed with MS for 1–33 years (M = 11.9, SD = 6.5); disability ratings determined by the EDSS (Kurtzke, 1983) at the time of recruitment ranged from 1.0 to 8.5 (Md = 5.0). Exclusionary criteria for the patients were: neurological disorder other than MS; history of drug or alcohol abuse, premorbid psychological disorder, mental retardation, or head injury; current use of narcotics; visual acuity >20/50 (corrected) or impaired color vision; disabling symptomatic involvement of the hands (e.g., paresthesia, paralysis); MS relapse within the past 30 days; or cognitive impairment of sufficient severity to interfere with comprehension of testing instructions.

A convenience sample of 38 healthy controls (25 females, 13 males) was recruited through personal contacts of the MS patients or staff members at the medical center. In addition to the exclusionary criteria applied to patients, controls were also excluded if they had any chronic medical condition or were taking any medications other than nutritional supplements, birth control, or low-dose aspirin. Controls ranged in age from 24 to 65 (M = 54.3, SD = 10.9) and had between 12 and 18 years of education (M = 15.4, SD = 1.7).

Procedure

The study was approved by the Human Subject Committee of the Medical Center, and all participants provided informed consent. After consenting to the study, participants were scheduled for an individual testing session during which they completed four questionnaires and four computerized neuropsychological tests in addition to the c-SDMT. Testing was conducted in a quiet, private room designed for neuropsychological assessment. The time required to complete the full battery was ∼60 min, and participants were invited to take breaks as needed between each test. The c-SDMT was the third test in the battery, preceded by the Rey Auditory Verbal Learning Test and the Tower of London, and was introduced ∼40 min into the testing session. Because the scores for the other tests were not relevant to the aims of the present study, they are not reported here.

Measures

Computerized symbol–digit modalities test

This version of the c-SDMT developed by Hughes and colleagues (2013) consisted of a single 90-s trial, preceded by 10 practice items. Instructions identical to those used with the original SDMT (Smith, 1982) were displayed on the screen and read aloud by the examiner. Throughout the instruction period, practice, and test trial, a reference key was located at the top of the computer screen displaying nine nondescript geometric symbols and their corresponding digit (i.e., 1–9). Stimulus items consisted of symbols presented individually at the center of the computer screen. The participant stated the digit associated with each stimulus and immediately pressed the space bar to display the next item. The computer timed the trial, divided the trial into three 30-s intervals, and recorded the number of items completed in each interval along with the total. The examiner recorded any errors, but these occurred rarely on the c-SDMT and were not included in the analysis.

Computerized Incidental Learning Test

The incidental learning test immediately followed the c-SDMT and was designed to assessing symbol recognition and the acquisition of symbol–digit pairs. Participants were shown a set of 22 symbols appearing individually at the center of the computer screen. For each item, they were asked whether the symbol was one used on the c-SDMT. If the participant responded “yes,” a grid with nine digits was displayed beneath the symbol and the participant was asked to select the digit that corresponded to the symbol. The Computerized Incidental Learning Test (c-ILT) yielded three scores: first, the number of symbols correctly identified as being included or not included on the c-SDMT (Recognition score); second, the number of symbols included on the c-SDMT for which the participant correctly supplied the corresponding digit (Pairing score); and third, a combined Point score, wherein one point was awarded for each symbol identified as having been included on the c-SDMT and an additional point was awarded for each of these symbols if the participant could also correctly supply the corresponding digit. Scores could range from 0 to 22 on the Recognition score, 0 to 9 on the Pairing score, and 0 to 18 on the Point score.

Statistical Analyses

Analyses were performed using SPSS 20.0. Patients and controls were compared using the appropriate parametric or nonparametric test for independent samples—depending on whether the scores on the measure were distributed normally. Participants' scores for the three intervals of the c-SDMT were examined using a Group × Interval mixed factorial analysis.

Results

Patients and controls did not differ in education (t(98) = 0.74, p = .459) or gender distribution (χ2(1) = 0.79, Fisher's exact test: p = .411), but the controls were older than the patients (M = 54.3 vs. 49.1; t(99) = 2.45, p = .016). Because of this difference in age, all comparisons between patients and controls on the outcome measures of the study were first performed with age entered as a covariate. Age was retained as a covariate whenever it was determined to be significant, and the comparison reported here is based on covariate-adjusted scores. However, when age was not a significant covariate, it was dropped from the analysis and the result without the covariate is reported.

Table 1 presents the means and standard deviations for patients and controls on the outcome measures of the study as well as the results for simple comparisons between patients and controls on each measure. Age was a significant covariate in all comparisons involving the scores from the c-SDMT, and therefore the table presents the results for one-way analyses of covariance performed on these scores. However, age was unrelated to performance on the c-ILT, and moreover Kolmogorov-Smirnov tests showed that the distributions of these scores diverged significantly from normal. Therefore, group comparisons on these measures were based on the nonparametric Mann–Whitney U test. As shown in the table, patients completed significantly fewer items overall and during each interval of the c-SDMT and had lower scores on each of the measures derived from the c-ILT.

Table 1.

Comparison between multiple sclerosis (MS) patients and healthy controls on computerized Symbol–Digit Modalities Test (c-SDMT) and Incidental Learning Test (c-ILT)

 MS patients
 
Controls
 
Between-group comparison
 
M SD M SD Fa P Partial η2 
c-SDMT 
 Total score 46.3 14.6 59.7 10.9 32.07 <.001 0.247 
 Interval        
  1 (0–30) 15.5 5.0 19.7 3.4 27.65 <.001 0.220 
  2 (31–60) 15.3 4.8 19.6 4.2 28.32 <.001 0.224 
  3 (61–90) 15.5 5.2 20.5 3.8 33.52 <.001 0.255 
 M SD M SD Zb P R2 
c-ILT 
 Recognition score 20.3 1.1 21.1 1.0 3.61 <.001 .112 
 Pairing score 4.3 1.9 6.0 2.2 3.67 <.001 .151 
 Point score 12.5 2.2 14.7 2.4 4.33 <.001 .183 
 MS patients
 
Controls
 
Between-group comparison
 
M SD M SD Fa P Partial η2 
c-SDMT 
 Total score 46.3 14.6 59.7 10.9 32.07 <.001 0.247 
 Interval        
  1 (0–30) 15.5 5.0 19.7 3.4 27.65 <.001 0.220 
  2 (31–60) 15.3 4.8 19.6 4.2 28.32 <.001 0.224 
  3 (61–90) 15.5 5.2 20.5 3.8 33.52 <.001 0.255 
 M SD M SD Zb P R2 
c-ILT 
 Recognition score 20.3 1.1 21.1 1.0 3.61 <.001 .112 
 Pairing score 4.3 1.9 6.0 2.2 3.67 <.001 .151 
 Point score 12.5 2.2 14.7 2.4 4.33 <.001 .183 

aBased on one-way analysis of covariance with age entered as a covariate. The effect size statistic (partial η2) indicates the proportion of the variability in the measure that is accounted for by group membership (i.e., patient vs. control).

bBased on Mann–Whitney U test. The effect size statistic is the square of the point biserial correlation between the grouping variable and the measure; like η2, it indicates the proportion of the variability in the measure that is accounted for by group membership.

A more informative examination of participants' performance on the c-SDMT was obtained with a 2 (Group) × 3 (Interval) mixed factorial analysis of covariance with age serving as the covariate. This analysis yielded only a significant main effect for Group (F(1,98) = 32.07, p < .001, (partial η2 = 0.247). Neither the main effect for Interval (F < 1) nor the Group × Interval interaction (F(2,196) = 1.72, p = .182) was significant. Figure 1 presents the c-SDMT scores pertaining to this analysis.

Fig. 1.

Number of items completed by multiple sclerosis patients and healthy controls on three successive 30-s intervals of the computerized Symbol–Digit Modalities Test.

Fig. 1.

Number of items completed by multiple sclerosis patients and healthy controls on three successive 30-s intervals of the computerized Symbol–Digit Modalities Test.

Spearman rank-order correlations were computed between the c-SDMT and each of the c-ILT scores for the sample of 65 patients. The total score on the c-SDMT was related to the Recognition score (rs = .277, p = .026), Pairing score (rs = .242, p = .052), and Point score (rs = .295, p = .017).

A simultaneous binary logistic regression analysis was performed to determine which scores made unique contributions toward distinguishing MS patients from controls. Group membership was entered as the dependent variable in this analysis, with the control group serving as the reference. The predictors were the total score on the c-SDMT and the Point score on the c-ILT. The resulting model was significant (χ2(2) = 30.0, p < .001, Nagelkerke R2 = .346), accurately classifying 53/68 (78%) patients and 19/38 (50%) controls. Both the c-SDMT score (Wald = 7.76, df = 1, p = .005) and the Point score (Wald = 6.02, df = 1, p = .014) were significant predictors of group membership.

Discussion

Patients completed fewer items on the c-SDMT and achieved lower scores on all measures of incidental learning on the c-ILT. Furthermore, their scores on the two tests were correlated. Although one might conclude from these results that patients fail to complete as many items on the c-SDMT because they are less able to acquire the symbol–digit associations while working on the test, this interpretation is belied by a finer dissection of participants' performance. Patients completed fewer items than controls during each of the 30-s intervals of the c-SDMT, including the first interval when presumably little incidental learning could have occurred. Furthermore, no appreciable increase was observed in the rate of item completion over the course of the 90-s trial for either group, and the disparity in performance between the groups remained about the same for each interval. One can only conclude that neither group benefitted much from acquiring the symbol–digit pairings in implicit memory over the course of the usual trial time allocated to the SDMT. Had the trial been extended by another 30 or 60 s, incidental learning might have had a larger impact on the scores, and it would be interesting to see whether the performance of the two groups diverged to a greater degree in the wake of this extended time. However, within the customary trial time of the SDMT, incidental learning seems to have had little influence on performance for either group.

A more plausible explanation for these results is that the cognitive challenge arising from performing the SDMT adversely impacts incidental learning. A substantial component of that challenge involves the demand for rapid serial processing of information. As a result of their deficit in information processing speed, this burden is greater for patients and therefore their capacity remaining for incidental learning is more limited. In this vein, the scores on the c-ILT are just another manifestation of the difficulty patients with MS have when confronted with tasks that require rapid serial processing of information. DeLuca and colleagues (2004) has proposed the Relative Consequence Model, suggesting that at least some of the deficits exhibited by MS patients in various cognitive domains stem from their more fundamental problem with information processing speed. The impairment in patients' incidental learning observed in the present study would seem to be consistent with this model. It may also explain why previous studies of MS patients have found no evidence of impairment in implicit memory when non-speeded measures such as the word stem completion task have been used (Beatty, Goodkin, Monson, & Beatty, 1990; Blum et al., 2002; Latchford, Morley, Peace, & Boyd, 1993; Scarrabelotti & Carroll, 1999). Absent the burden imposed by time limits, patients' implicit memory performance would seem to be similar to that of healthy individuals. On the other hand, Filley, Heaton, Nelson, Burks, and Franklin (1989) found substantial deficits in implicit memory scores when MS patients completed the Tactual Performance Test, a speeded test included in the Halstead-Reitan battery.

While we find this alternative account centering on cognitive burden to be the more plausible explanation for the current results, two difficulties must be acknowledged. First, although patients' scores on the c-SDMT were correlated with their scores on the c-ILT, the magnitude of these correlations was modest and similar to those reported by Joy and colleagues (2004) for healthy adults. If the disproportionately greater cognitive burden imposed by the c-SDMT were exclusively responsible for patients' poor performance on the c-ILT, one would expect these correlations to be stronger. Similarly, in the logistic regression analysis, scores on both tests made significant contributions toward distinguishing patients from controls. Here again, if patients' comparatively poor performance on the incidental learning task were exclusively a “relative consequence” of their deficit in information processing speed as indexed by the c-SDMT, one might expect only one score (more likely the c-SDMT score) to emerge as a significant predictor in the logistic regression analysis. The fact that scores from both tests contributed to the distinction between patients and controls suggests that deficits in separate cognitive domains may be implicated in MS.

Based on a recent review of the literature, Reber (2013) has argued that, unlike declarative memory with its well-established alignment with the medial temporal lobe, no specific localization has been established for implicit memory. Instead, implicit memory should be viewed as an emergent property of diffuse cortical–subcortical circuitry that confers a “pervasive plasticity” to the brain. This might serve to explain why impairment in incidental learning, like deficits in information processing speed, could stem from widely distributed pathology common to neurodegenerative diseases such as MS. Further investigation of patients' incidental learning during tasks posing various levels of burden on information processing speed and perhaps accompanied by functional imaging studies would be helpful to determine whether separate domains are in play in conjunction with MS.

Some limitations of the present study must be acknowledged. The c-SDMT and c-ILT were part of a larger battery of neuropsychological tests that were administered in a fixed order for all participants. The tests featured in this paper were preceded by the Rey Auditory Verbal Learning Test and the Tower of London, and the impact these earlier tests might have had on participants' scores on the c-SDMT and c-ILT is unknown. Certainly one must consider the possibility that fatigue might have impacted these latter scores, although participants were encouraged to take breaks between tests in order to re-marshal their energy. The results of the present study need to be confirmed without the potential confounds posed by the fuller test battery.

More importantly, it must also be noted that, owing to their slower performance, the MS patients in the present study encountered fewer items on the c-SDMT than did controls, and fewer exposures could well have contributed to reduced opportunities for incidental learning and therefore lower scores on the c-ILT. Rapid serial processing tests such as the SDMT are often structured as time trials, where the score is based on the number of items completed in a fixed period of time. Since our intention was to use a computerized version of the SDMT that replicated the original test as closely as possible, this was the format we adopted. Alternatively, the number of items could be fixed and the score consist of the time to completion. The latter would afford exposure to an equal number of items for patients and controls and would constitute a better format for examining the relationships between information processing speed and incidental learning being explored in the present study. Accordingly, additional studies incorporating variations of the SDMT differing in both format and length are likely to prove useful in advancing this topic.

Funding

This work was supported by a grant from the National Multiple Sclerosis Society (RG 4495-A-4; PI: SGL).

Conflict of Interest

None declared.

Acknowledgements

We thank JoAnn Lierman for her assistance on this project.

References

Akbar
N.
Honarmand
K.
Kou
N.
Feinstein
A.
(
2011
).
Validity of a computerized version of the Symbol Digit Modalities Test in multiple sclerosis
.
Journal of Neurology
 ,
258
,
373
379
.
Batista
S.
Zivadinov
R.
Hoogs
M.
Bergsland
N.
Heininen-Brown
M.
Dwyer
M. G.
et al
(
2012
).
Basal ganglia, thalamus and neocortical atrophy predicting slowed cognitive processing in multiple sclerosis
.
Journal of Neurology
 ,
259
,
139
146
.
Beatty
W.
Goodkin
D.
Monson
N.
Beatty
P.
(
1990
).
Implicit learning in patients with chronic progressive multiple sclerosis
.
International Journal of Clinical Neuropsychology
 ,
12
,
166
172
.
Blum
D.
Yonelinas
A. P.
Luks
T.
Newitt
D.
Oh
J.
Lu
Y.
et al
(
2002
).
Dissociating perceptual and conceptual implicit memory in multiple sclerosis patients
.
Brain and Cognition
 ,
50
,
51
61
.
Bodling
A. M.
Denney
D. R.
Lynch
S. G.
(
2012
).
Individual variability in speed of information processing: An index of cognitive impairment in multiple sclerosis
.
Neuropsychology
 ,
26
,
357
367
.
Brochet
B.
Deloire
M. S.
Bonnet
M.
Salort-Campana
E.
Ouallet
J. C.
Petry
K. G.
et al
(
2008
).
Should SDMT substitute for PASAT in MSFC? A 5-year longitudinal study
.
Multiple Sclerosis
 ,
14
,
1242
1249
.
de Sonneville
L. M. J.
Boringa
J. B.
Reuling
I. E. W.
Lazeron
R. H. C.
Ader
H. J.
Polman
C. H.
(
2002
).
Information processing characteristics in subtypes of multiple sclerosis
.
Neuropsychologia
 ,
40
,
1751
1765
.
DeLuca
J.
Chelune
G. J.
Tulsky
D. S.
Lengenfelder
J.
Chiaravalloti
N. D.
(
2004
).
Is speed of processing or working memory the primary information processing deficit in multiple sclerosis?
Journal of Clinical Experimental Neuropsychology
 ,
26
,
550
562
.
DeLuca
J.
Genova
H. M.
Hillary
F. G.
Wylie
G.
(
2008
).
Neural correlates of cognitive fatigue in multiple sclerosis using functional MRI
.
Journal of the Neurological Sciences
 ,
270
,
28
39
.
Denney
D. R.
Gallagher
K. S.
Lynch
S. G.
(
2011
).
Deficits in processing speed in patients with multiple sclerosis: Evidence from explicit and covert measures
.
Archives of Clinical Neuropsychology
 ,
26
,
110
119
.
Drake
A. S.
Weinstock-Guttman
B.
Morrow
S. A.
Hojnacki
D.
Munshauer
F. E.
Benedict
R. H. B.
(
2010
).
Psychometrics and normative data for the Multiple Sclerosis Functional Composite: Replacing the PASAT with the Symbol Digit Modalities Test
.
Multiple Sclerosis
 ,
16
,
228
237
.
Filley
C. M.
Heaton
R. K.
Nelson
L. M.
Burks
J. S.
Franklin
G. M.
(
1989
).
A comparison of dementia in Alzheimer's disease and multiple sclerosis
.
Archives of Neurology
 ,
46
,
157
161
.
Forn
C.
Belloch
V.
Bustamante
J. C.
Garbin
G.
Parcet-Ibars
M. A.
Sanjuan
A.
et al
(
2009
).
A symbol digit modalities test version suitable for functional MRI studies
.
Neuroscience Letters
 ,
456
,
11
14
.
Forn
C.
Ripolles
P.
Cruz-Gómez
A. J.
Belenguer
A.
González-Torre
J. A.
Avila
C.
(
2013
).
Task-load manipulation in the Symbol Digit Modalities Test: An alternative measure of information processing speed
.
Brain and Cognition
 ,
82
,
152
160
.
Genova
H. M.
Hillary
F. G.
Wylie
G.
Rypma
B.
Deluca
J.
(
2009
).
Examination of processing speed deficits in multiple sclerosis using functional magnetic resonance imaging
.
Journal of the International Neuropsychological Society
 ,
15
,
383
393
.
Hughes
A. J.
Denney
D. R.
Owens
E. M.
Lynch
S. G.
(
2013
).
Procedural variations in the Stroop and Symbol Digit Modalities Test in patients with multiple sclerosis
.
Archives of Clinical Neuropsychology
 ,
28
,
452
462
.
Joy
S.
Kaplan
E.
Fein
D.
(
2003
).
Digit Symbol – Incidental Learning in the WAIS-III: Construct validity and clinical significance
.
The Clinical Neuropsychologist
 ,
17
,
182
194
.
Joy
S.
Kaplan
E.
Fein
D.
(
2004
).
Speed and memory in the WAIS-III Digit Symbol-Coding subtest across the adult lifespan
.
Archives of Clinical Neuropsychology
 ,
19
,
759
767
.
Kujala
P.
Portin
R.
Revonsuo
A.
Ruutiainen
J.
(
1995
).
Attention related performance in two cognitively different subgroups of patients with multiple sclerosis
.
Journal of Neurology, Neurosurgery, and Psychiatry
 ,
59
,
77
82
.
Kujala
P.
Portin
R.
Revonsuo
A.
Ruutiainen
J.
(
1994
).
Automatic and controlled information processing in multiple sclerosis
.
Brain
 ,
117
,
1115
1126
.
Lapshin
H.
Audet
B.
Feinstein
A.
(
2014
).
Detecting cognitive dysfunction in a busy multiple sclerosis clinical setting: A computer generated approach
.
European Journal of Neurology
 ,
21
,
281
286
.
Lapshin
H.
Lanctot
K. L.
O'Connor
P.
Feinstein
A.
(
2013
).
Assessing the validity of a computer-generated cognitive screening instrument for patients with multiple sclerosis
.
Multiple Sclerosis
 ,
19
,
1905
1912
.
Latchford
G.
Morley
S.
Peace
K.
Boyd
J.
(
1993
).
Implicit memory in multiple sclerosis
.
Behavioural Neurology
 ,
6
,
129
133
.
Litvan
I.
Grafman
J.
Vendrell
P.
Martinez
J. M.
(
1988
).
Slowed information processing speed in multiple sclerosis
.
Archives of Neurology
 ,
45
,
281
285
.
Papadopoulou
A.
Müller-Lenke
N.
Naegelin
Y.
Kalt
G.
Bendfeldt
K.
Kuster
P.
et al
(
2013
).
Contribution of cortical and white matter lesions to cognitive impairment in multiple sclerosis
.
Multiple Sclerosis
 ,
19
,
1290
1296
.
Parmenter
B. A.
Weinstock-Guttman
B.
Garg
N.
Munschauer
F.
Benedict
R. H.
(
2007
).
Screening for cognitive impairment in multiple sclerosis using the Symbol Digit Modalities Test
.
Multiple Sclerosis
 ,
13
,
52
57
.
Polman
C. H.
Reingold
S. C.
Banwell
B.
Clanet
M.
Cohen
J. A.
Filippi
M.
et al
(
2011
).
Diagnostic criteria for multiple sclerosis: 2010 revisions to the McDonald Criteria
.
Annals of Neurology
 ,
69
,
292
302
.
Rao
S. M.
(
1990
).
A manual for the brief repeatable battery of neuropsychological test in multiple sclerosis
 .
New York
:
National Multiple Sclerosis Society
.
Rao
S. M.
Martin
A. L.
Huelin
R.
Wissinger
E.
Khankhel
Z.
Kim
E.
et al
(
2014
).
Correlations between MRI and information processing speed in MS: A meta-analysis
.
Multiple Sclerosis International
 ,
2014
,
975803
.
Reber
P. J.
(
2008
).
Cognitive neuroscience of declarative and nondeclarative memory
. In
Aaron
S.
Benjamin
A. S.
De Belle
J. S.
Etnyre
B.
Polk
T. A.
(Eds.),
Advances in Psychology, 135, 23–49
.
Reber
P. J.
(
2013
).
The neural basis of implicit learning and memory: A review of neuropsychological and neuroimaging research
.
Neuropsychologia
 ,
51
,
2026
2042
.
Reicker
L. I.
Tombaugh
T. N.
Walker
L.
Freedman
M. S.
(
2007
).
Reaction time: An alternative method for assessing the effects of multiple sclerosis on information processing speed
.
Archives of Clinical Neuropsychology
 ,
22
,
655
664
.
Ruet
A.
Deloire
M. S.
Charré-Morin
J.
Hamel
D.
Brochet
B.
(
2013
).
A new computerised cognitive test for the detection of information processing speed impairment in multiple sclerosis
.
Multiple Sclerosis
 ,
19
,
1665
1672
.
Rypma
B.
Berger
J. S.
Prabhakaran
V.
Bly
B. M.
Kimberg
D. Y.
Biswal
B. B.
et al
(
2006
).
Neural correlates of cognitive efficiency
.
Neuroimage
 ,
33
,
969
979
.
Scarrabelotti
M.
Carroll
M.
(
1999
).
Memory dissociation and metamemory in multiple sclerosis
.
Neuropsychologia
 ,
37
,
1335
1350
.
Smith
A.
(
1982
).
Symbol digit modalities test- revised
 .
Los Angeles
:
Western Psychological Services
.
Tombaugh
T. N.
Berrigan
L. I.
Walker
L. A. S.
Freedman
M. S.
(
2010
).
The Computerized Test of Information Processing (CTIP) offers an alternative to the PASAT for assessing cognitive processing speed in individuals with multiple sclerosis
.
Cognitive and Behavioral Neurology
 ,
23
,
192
198
.
Walker
L. A.
Cheng
A.
Berard
J.
Berrigan
L. I.
Rees
L. M.
Freedman
M. S.
(
2012
).
Tests of information processing speed: What do people with multiple sclerosis think about them?
International Journal of MS Care
 ,
14
,
92
99
.