Abstract

Decreased information processing speed is often cited as the primary cognitive deficit occurring in conjunction with multiple sclerosis (MS). Two common tools for assessing this deficit are the Stroop Test and the Symbol Digit Modalities Test (SDMT). However, there are procedural variations in these rapid serial processing (RSP) tests pertaining to the response format (e.g., verbal or manual) and the administration format (e.g., paper-based or computerized). The present study was designed to assess whether such variations impact MS patients' and healthy individuals' performance on these tests. In Experiment 1, we showed that response formats in which either the experimenter or the participant was responsible for advancing the items on computerized versions of the Stroop Test and the SDMT were basically equivalent in terms of distinguishing between patients and controls. In Experiment 2, we found differences between administration formats that appear to interact with some of the disease-related features of MS. Understanding how procedural variations differentially impact patients and controls can be useful for interpreting what RSP tests reveal about the cognitive impact of MS.

Introduction

A decrease in information processing speed is frequently cited as the earliest and most pervasive cognitive deficit associated with multiple sclerosis (MS; Archibald & Fisk, 2000; Bergendal, Fredrikson, & Almkvist, 2007; Bodling, Denney, & Lynch, 2008, 2009, 2012; DeLuca, Chelune, Tulsky, Lengenfelder, & Chiaravalloti, 2004; Demaree, DeLuca, Gaudino, & Diamond, 1999; Denney, Gallagher, & Lynch, 2011; Denney & Lynch, 2009; de Sonneville et al., 2002; Kail, 1997, 1998; Kujala, Portin, Revonsuo, & Ruutiainen, 1994; Lengenfelder et al., 2006; Macniven et al., 2008; Rao, St. Aubin-Faubert, & Leo, 1989; Reicker, Tombaugh, Walker, & Freedman, 2007; Schulz, Kopp, Kunkel, & Faiss, 2006). This deficit has been demonstrated with a variety of neuropsychological measures, most of them based on either reaction time (RT) tests (e.g., Bodling et al., 2012; de Sonneville et al., 2002; Kujala et al., 1994; Reicker et al., 2007; Tombaugh, Berrigan, Walker, & Freedman, 2010) or tests requiring rapid serial processing (RSP). On RSP tests, a series of items is presented sequentially with little or no variation in the operation to be performed on each item. The operation itself is typically not very difficult, but must be executed quickly, with the objective being to complete as many items as possible in some allotted period of time.

Hughes, Denney, and Lynch (2011) recently compared RT and RSP tests and found the latter to be more successful at differentiating MS patients from healthy controls in terms of information processing speed. The relatively larger effect sizes on RSP measures were seen as resulting from an additive factor termed the “compounding effect.” Whereas RT tests provide an averaged measure of the time required to perform a single information processing operation, RSP measures focus instead on the number of such operations successfully executed during an overall period of time. In the latter case, small differences between subjects' information processing speed on each item are compounded by the necessity of having to execute the same operation numerous times over the course of the period. This compounding appears to enhance the sensitivity of RSP measures for distinguishing between MS patients and controls.

Two of the most common RSP measures are the Stroop Test (Denney & Lynch, 2009; Macniven et al., 2008) and the Symbol Digit Modalities Test (SDMT; Benedict et al., 2006; Drake et al., 2010; Parmenter, Weinstock-Guttman, Garg, Munschauer, & Benedict, 2007). Although both are highly effective at distinguishing MS patients from controls, the testing formats adopted for these tests have varied considerably across studies (e.g., Akbar, Honarmand, Kou, & Feinstein, 2011; Bodling et al., 2012; Brochet et al., 2008; Denney & Lynch, 2009; Drake et al., 2010; Lynch, Dickerson, & Denney, 2010; Macniven et al., 2008; Portaccio et al., 2009), and the impact of these variations has not been examined empirically. For example, in the only direct comparison between MS patients' performance on the Stroop Test and the SDMT, Hughes and colleagues (2011) chose to format each test in the way typically adopted in the MS literature. The SDMT featured a paper-based format and required subjects to make only a verbal response to each stimulus. The Stroop Test, on the other hand, employed a computerized format and required subjects to respond verbally and then press the space bar to expose the next item. As a result, the comparison between the tests themselves was confounded by possible differences in the “administration format” (paper-based vs. computerized) and the “response format” (verbal-only vs. verbal-manual).

Examining the impact of such differences in format is especially important in the case of RSP tests because of the compounding effect. Format differences potentially implicate disparities between MS patients and controls in ancillary visual and motor functions (Arnett, Smith, Barwick, Benedict, & Ahlstrom, 2008; Bruce, Bruce, & Arnett, 2007; Jennekens-Schinkel, Lanser, Van Der Velde, & Sanders, 1990; Smith & Arnett, 2007), and like differences in central processing speed, these ancillary differences may also be subject to compounding across the items of the test. Whereas most RT tests used in MS research are computer administered, the RSP measures are still divided between paper-based and computerized tests. The recent trend, however, is toward adapting RSP measures to computer administration to take advantage of the greater convenience, experimental precision, and standardization afforded by this technology (Akbar et al., 2011; Salo, Henik, & Robertson, 2001). Issues involving format readily arise when designing such adaptations.

Although a few studies have sought to compare paper-based and computerized versions of the Stroop Test (Potter, Jory, Bassett, Barrett, & Mychalkiw, 2002; Salo et al., 2001; Seignourel et al., 2005) or the SDMT (Akbar et al., 2011), the results of these studies have only a tangential bearing on the present paper because of the way these investigators chose to adapt the tests to the computer. In every instance, the computerized version was altered from an RSP measure to an RT measure. New items did not appear immediately upon the subject's completion of a preceding item, so it was no longer possible to consider the number of items completed in an allotted period of time. Instead, a fixed inter-item interval was part of the computer program, and RTs to each item were recorded and “averaged” across the items. Then, in order to provide a score on the paper-based version of the test that was deemed comparable with these RT scores, the investigators calculated what might be termed a “derived RT” score by dividing the number of items completed by the length of the period. Thus, each of these previous studies features comparisons between RT scores from a computerized test and derived RT scores from the paper-based version of the same test. To the extent that compounding effects add to the sensitivity of RSP measures, RT scores and derived RT scores are not entirely comparable. Our goal was to examine paper-based and computerized versions of the Stroop Test and the SDMT in their original RSP formulation, thereby preserving the compounding effects inherent to these measures. In order to accomplish this, it was first important to resolve an issue concerning the response format employed in the computerized versions of these tests.

When RSP tests are adapted for the computer, the test items are typically presented individually, the subject gives a verbal response to an item, and then a key is immediately pressed to display the next item. The two response formats compared in Experiment 1 center on whether the experimenter (E-press) or the subject (S-press) executes the key press to advance the next item. In early studies of the computerized Stroop Test (e.g., Denney, Lynch, Parmenter, & Horne, 2004; Denney, Sworowski, & Lynch, 2005), the experimenter was responsible for this action, based on the assumption that patients' problems with manual dexterity might affect their performance and thereby confound the test as a measure of information processing speed. However, this practice has been criticized for its possible vulnerability to experimenter bias. Despite much speculation, no previous studies have compared measures of information processing speed in MS patients and controls obtained under these two alternative response formats. We felt it best to examine this matter empirically. The response for the S-press condition was exclusively executed by the participant, whereas the E-press condition required the participant's verbal response to be followed by the experimenter's key press. On the basis of simple logistics, it seemed reasonable to expect the S-press condition would result in higher scores than the E-press condition. The principal concern, however, was whether the disparity between patients and controls would differ for the two conditions, thereby indicating that the measure of central processing speed was confounded by experimenter bias or, alternatively, by patients' ancillary motor problems.

After addressing the issue of response format, Experiment 2 was conducted to determine whether the administration format differentially impacts performance on RSP tests. Whereas computerized tests typically present items individually in the center of the screen, paper-based tests present all items together on a single page in what some have characterized as a “cluttered field” (Salo et al., 2001; Seignourel et al., 2005). Subjects must shift their focus from item to item down the page, a requirement that may be more taxing for patients with the kinds of ocular-motor problems (e.g., nystagmus) commonly encountered in MS. The study by Salo and colleagues (2001) included a comparison between two conditions on a computerized Stroop Test, one in which multiple items were presented together in rows and columns and another in which the items were presented individually in the center of the screen. RTs were shorter for the former condition, indicating that the cluttered field did not impede performance and that instead the availability of additional items may have allowed participants to begin processing the next item before completing their response on the preceding one—a process that we will refer to as “forward anticipation.” The results from Salo and colleagues led us to hypothesize that scores would be greater for the paper-based formatted tests in Experiment 2. However, here again, the principal concern was whether the disparity between patients and controls would differ between the two administration formats. No prediction regarding this Group × Format interaction was possible based on Salo and colleagues' study because only healthy individuals were examined. Due to the ancillary ocular-motor problems encountered in MS, one might expect greater disparity between groups on the paper-based format employing a cluttered field of stimulus items.

The overall goal for these experiments was to assess the impact of procedural variations commonly utilized with the Stroop Test and the SDMT on the information processing speed of patients with MS. Examining how variations in response formats (Experiment 1: E-press vs. S-press) and administration formats (Experiment 2: Computerized vs. paper-based) impact performance may allow one to draw more informed conclusions about the deficits that MS patients evidence on these tests.

General Method

Participants

Different samples of patients with clinically definite MS (Polman et al., 2005) and healthy individuals of comparable demographics (i.e., age, sex, education) were recruited for each of the two experiments. All patients were under the care of the same neurologist (Sharon G. Lynch) at the University of Kansas Medical Center and had been diagnosed with MS at least 1 year prior to recruitment. Exclusionary criteria for the patients were the following: Neurological disorder other than MS; history of drug or alcohol abuse, premorbid psychological disorder, mental retardation, or head injury; current use of stimulants, narcotics, or benzodiazepines; visual acuity greater than 20/50 (corrected) or impaired color vision; symptomatic involvement of the hands; MS relapse within the past 30 days; or cognitive impairment of sufficient severity to interfere with comprehension of testing instructions. The convenience samples of healthy controls for the two experiments were recruited through personal contacts of the research staff, employees, or volunteer staff members at the medical center. The same exclusionary criteria were applied to controls. In addition, controls were excluded if they had any chronic medical condition or were taking any medications other than nutritional supplements, birth control, and low-dose aspirin.

Procedure

These studies were approved by the Human Subjects Committee at the University of Kansas Medical Center. Eligible patients were recruited during the course of their regularly scheduled appointment at the MS Clinic. After the patient provided signed consent, the neurologist assessed the patient's disability status using the Disease Steps Scale (DSS; Hohol, Orav, & Weiner, 1995; Experiment 1) or the Expanded Disability Status Scale (EDSS; Kurtzke, 1983; Experiment 2). The research session followed the patients' clinic appointment and lasted 40–45 min.

Experiment 1

The focus of Experiment 1 was a comparison between the E-press and the S-press response format in conjunction with the computerized versions of the Stroop Test and the SDMT. The principal aims of the study were to replicate previous findings concerning a deficit in information processing speed in patients with MS and to determine whether this deficit was more prominent for one response format as opposed to the other.

Methods

The participants in this study were 40 patients with relapsing-remitting (RRMS; N = 27) or secondary progressive MS (SPMS; N = 13) and 40 healthy controls. The duration of patients' illness ranged from 1 to 29 years (M = 10.4, SD= 6.4) and disease severity based on the DSS ranged from 0 to 6 (Md = 2.0). Information concerning gender, age, and education for patients and controls is provided in the upper portion of Table 1. Each participant completed the two computerized versions of the Stroop Test and the SDMT—one incorporating the E-press condition, the other the S-press condition. The order of these conditions was randomized and counterbalanced across participants. Within each condition, the Stroop Test was administered first followed by the SDMT.

Table 1.

Demographic variables for patients and controls in Experiments 1 and 2

 MS patients (N = 40) Controls (N = 40) 
Experiment 1 
Sex (F/M) 30/10 26/14 
Age   
M 48.7 46.2 
SD 8.6 10.1 
 Range 28–60 22–58 
Education (years)   
M 15.3 15.6 
SD 1.9 1.8 
 Range 12–18 12–18 
 MS patients (N = 42) Controls (N = 40) 
Experiment 2 
Sex (F/M) 29/11 33/9 
Age 
M 44.8 42.6 
SD 12.1 10.7 
 Range 22–64 22–62 
Education (years) 
M 15.1 16.4 
SD 2.1 2.6 
 Range 12–19 12–20 
AmNART 
M 30.4 32.2 
SD 8.0 5.8 
FSS 
M 37.9 17.3 
SD 15.1 9.2 
CES-D 
M 34.4 26.3 
SD 10.5 5.1 
 MS patients (N = 40) Controls (N = 40) 
Experiment 1 
Sex (F/M) 30/10 26/14 
Age   
M 48.7 46.2 
SD 8.6 10.1 
 Range 28–60 22–58 
Education (years)   
M 15.3 15.6 
SD 1.9 1.8 
 Range 12–18 12–18 
 MS patients (N = 42) Controls (N = 40) 
Experiment 2 
Sex (F/M) 29/11 33/9 
Age 
M 44.8 42.6 
SD 12.1 10.7 
 Range 22–64 22–62 
Education (years) 
M 15.1 16.4 
SD 2.1 2.6 
 Range 12–19 12–20 
AmNART 
M 30.4 32.2 
SD 8.0 5.8 
FSS 
M 37.9 17.3 
SD 15.1 9.2 
CES-D 
M 34.4 26.3 
SD 10.5 5.1 

Notes: MS = multiple sclerosis; AmNART = North American Adult Reading Test; FSS = Fatigue Severity Scale; CES-D = Center for Epidemiological Studies, Depression Scale.

Computerized Stroop Test

This test was adapted from the paper-based Stroop Color-Word Interference Test (Golden, 1978) and consisted of three 60-s trials, each of which was preceded by a set of eight practice items to insure the participant understood the task to follow. In the first trial (word reading, W), participants read color words (RED, BLUE, YELLOW, GREEN) printed in black letters; in the second trial (color naming, C), they named the colors used to print a row of four Xs; and in the third trial (color-word naming, CW), they named the color of the print for a set of incongruent Stroop stimuli (e.g., BLUE printed with red letters). In each trial, the stimulus item appeared in the center of the computer screen, the participant responded verbally to the item, and either the experimenter (E-press condition) or the participant (S-press condition) immediately pressed the space bar to display the next item. The computer timed each trial and recorded the number of items completed in the allotted time. Errors averaged less than one per trial for patients or controls and were not considered on the Stroop Test in this or the subsequent study. In addition to the W, C, and CW scores, the combined score for word-reading and color-naming (W + C) was examined as a particularly good measure of processing speed, as recommended by Denney and Lynch (2009). Also, a measure of relative interference (RI) was computed by dividing the difference between the color-naming and the color-word naming scores by the color-naming score (RI = [C − CW]/C) (Denney & Lynch, 2009; Macniven et al., 2008).

Computerized SDMT

This test was adapted from the paper-based SDMT (Smith, 1982) and consisted of a single 90-s trial, preceded by 10 practice items. Throughout the trial, a reference key was located at the top of the computer screen displaying nine non-descript geometric symbols with their corresponding digit (i.e., 1–9). Stimulus items consisted of symbols presented individually at the center of the computer screen. Participants stated the number associated with each stimulus, and either the experimenter (E-press condition) or the participant (S-press condition) immediately pressed the space bar to display the next item. Two alternate forms of the SDMT (Hinton-Bayre & Geffen, 2005) were counterbalanced with the two conditions so that participants' performance on the second administration was not affected by their recalling the symbol-digit associations from the first administration. The computer timed the trial and recorded the number of responses. The score on the SDMT was the number of correct responses made during the 90-s trial. As in the case of the Stroop Test, errors occurred rarely, averaging less than 1.5 for patients or controls, and were not analyzed in either study.

Results

Preliminary analyses revealed that the patient and control samples were comparable in terms of gender composition (Fisher's exact test, p = .47), age (t = 1.2, p = .25), and education (t < 1, p = .59). Experiment 1 conformed to a 2 (Group) × 2(Order) × 2 (Format) factorial analysis of variance (ANOVA) with the last factor being a repeated-measures factor involving the participant's score under the E-press vs. the S-press condition. The order of the two conditions was counterbalanced across participants for both the Stroop Test and the SDMT, and preliminary ANOVAs revealed no significant main effect or interactions involving order. Therefore, this factor was eliminated, and the main analysis performed on each test score consisted of a Group × Format ANOVA. Means, standard deviations, and between-group effect sizes (Cohen's d) for the SDMT and for each score on the Stroop Test are presented by group and by response format in Table 2.

Table 2.

Experiment 1: Comparison between groups and response formats on information processing speed measures

  S-press
 
E-press
 
M SD da M SD da 
Stroop 
 W 
  Patients 77.6 16.1 0.84 76.3 10.9 1.25 
  Controls 91.05 16.1 90.1 11.3 
 C 
  Patients 67.7 14.9 0.81 64.5 10.6 1.05 
  Controls 77.3 12.0 74.5 8.4 
 CW 
  Patients 48.8 11.8 0.69 47.7 9.2 0.68 
  Controls 56.3 9.7 53.8 8.6 
 W + C 
  Patients 145.3 29.8 0.82 140.8 20.2 1.24 
  Controls 168.3 26.1 164.6 18.2 
 RI 
  Patients 27.25 11.35 0.05 25.89 8.02 0.22 
  Controls 26.69 9.68 27.76 8.87 
SDMT 
  Patients 54.5 12.7 1.07 51.5 11.1 1.20 
  Controls 67.3 11.1 63.9 9.5 
  S-press
 
E-press
 
M SD da M SD da 
Stroop 
 W 
  Patients 77.6 16.1 0.84 76.3 10.9 1.25 
  Controls 91.05 16.1 90.1 11.3 
 C 
  Patients 67.7 14.9 0.81 64.5 10.6 1.05 
  Controls 77.3 12.0 74.5 8.4 
 CW 
  Patients 48.8 11.8 0.69 47.7 9.2 0.68 
  Controls 56.3 9.7 53.8 8.6 
 W + C 
  Patients 145.3 29.8 0.82 140.8 20.2 1.24 
  Controls 168.3 26.1 164.6 18.2 
 RI 
  Patients 27.25 11.35 0.05 25.89 8.02 0.22 
  Controls 26.69 9.68 27.76 8.87 
SDMT 
  Patients 54.5 12.7 1.07 51.5 11.1 1.20 
  Controls 67.3 11.1 63.9 9.5 

Notes: W = word reading score (Trial 1); C = color naming score (Trial 2); CW = color-word naming score (Trial 3); W + C = combined score for word reading and color naming; RI = relative interference score; SDMT = Symbol Digit Modalities Test.

aEffect size concerning the difference between patients and controls based on Cohen's d.

Patients achieved significantly lower scores than controls on all measures of information processing speed derived from the Stroop Test–W: F(1, 78) = 26.8, p < .001; C: F(1, 78) = 17.8, p < .001; CW: F(1, 78) = 10.6, p < .005; W + C: F(1, 78) = 25.0, p < .001—and the SDMT—F(1, 78) 28.2, p < .001. Effect sizes for group differences were large for all measures except the CW scores (Table 2). Participants completed significantly fewer items under the E-press condition than under the S-press condition for the color-naming and color-word naming trials of the Stroop Test—C: F(1,78) = 6.2, p < .05; CW: F(1, 78) = 6.1, p < .05—and for the SDMT—F(1, 78) = 17.6, p < .001. However, there were no significant Group × Format interactions (all p's > .30). With respect to RI, neither the main effects nor the interaction was significant (all p's > .33).

The correlations between scores under the E-press and S-press conditions were computed separately for patients and controls and then combined across groups using Fisher's z-transformation procedure. The combined correlations were as follows: W, 0.48; C, 0.61; CW, 0.79; W + C, 0.57; RI, 0.32; SDMT, 0.84 (all p's < .001, except RI: p = .004). The correlations between the SDMT score and the combined W + C score from the Stroop Test were also computed separately for patients and controls and then combined across groups. The combined correlation was .54 under the E-press condition and .55 under the S-press condition (both p's < .001). All correlations were slightly higher for patients owing to their somewhat greater variability on the tests.

Discussion

Experiment 1 demonstrated that MS patients performed more slowly than controls on the computerized versions of the Stroop Test and the SDMT regardless of which response format was used. All participants performed faster under the S-press condition than under the E-press condition. Indeed, as each trial ensued, some participants adopted the practice of pressing the space bar at the same time they gave their verbal response, whereas in the E-press condition, there was an inevitable delay between the participant's verbal response and the experimenter's key press. On the other hand, this delay was virtually the same whether patients or controls were being tested and therefore not contaminated by experimenter bias. No significant interaction between Group and Format occurred on any of the measures of information processing speed.

Finally, inspection of Table 2 reveals numerically larger effect sizes for group differences under the E-press condition than under the S-press condition. Although some may be tempted to view these effect size differences as evidence of experimenter bias, the more cogent explanation lies in the relatively greater variability in all participants' performance under the S-press condition. Because an effect size is calculated by dividing the difference between the group means by the pooled standard deviation, the greater variability is responsible for lowering the effect size for the S-press condition. If experimenter bias were in play here, the Group × Format interactions would have accounted for more variance in the study. With the exception of performance on the CW trial of the Stroop Test and the RI score derived from that trial, the effect sizes (partial eta squared) for these interaction terms were ≤.001 (all p's > .80).

Experiment 2

Having demonstrated basic equivalence between response formats for distinguishing between MS patients and controls on computerized versions of the Stroop Test and the SDMT, we turn now to differences in the administration format. Experiment 2 featured a comparison between paper-based and computerized formats of these tests. The principal aims of this study were again to replicate the deficit in information processing speed observed in previous studies of patients with MS and to determine whether this deficit was more prominent under one administrative format than the other, especially given that the computerized format eliminates the burden of visually scanning stimulus items that occurs due to the cluttered field of paper-based tests.

Method

Participants were 42 patients with RRMS (N = 31) or SPMS (N = 11) and 40 healthy controls, none of whom participated in the first experiment. The duration of patients' illness ranged from 1 to 38 years (M = 10.1, SD = 8.0) and disease severity based on the EDSS ranged from 0 to 8.5 (Md = 3.0). Information concerning gender, age, education, premorbid intelligence, depression, and fatigue for patients and controls is provided in the lower portion of Table 1. Each participant completed both paper-based and computerized versions of the Stroop Test and the SDMT. Both the administration format and the order of tests within each format were randomized and counterbalanced across participants. To avoid participants' learning the symbol-digit associations on the SDMT, alternate forms of this test (Hinton-Bayre & Geffen, 2005) were used, with the forms counterbalanced between the administration formats.

The computerized versions of the Stroop Test and the SDMT were the same as those described in Experiment 1. For the present study, the participant pressed the space bar to advance the items on each test (i.e., S-press response format). The paper-based versions of the Stroop Test (Golden, 1978) and the SDMT (Smith, 1982) were the standard versions of these tests with the stimulus items presented in columns and rows on a single sheet of paper. The items for each trial of the Stroop Test were arranged in 20 rows, with 5 items per row. Those for the SDMT were arranged in 8 rows with 15 items per row, and the reference key was located at the top of the sheet. On both tests, participants were instructed to respond to items from left to right, beginning with the top row and moving down the page.

Prior to working on the Stroop Test and the SDMT, participants completed the North American Adult Reading Test (AmNART; Friend & Grattan, 1998; Nelson, 1982) to provide an estimate of their premorbid intelligence. Participant's experiences with fatigue and depression during the week preceding their research session were assessed using the Fatigue Severity Scale (Krupp, LaRocca, Muir-Nash, & Steinberg, 1989) and the Center for Epidemiological Studies-Depression Scale (Radloff, 1977).

Results

The two groups did not differ in sex (Fisher's exact test, p = .61) or age (t < 1, p = .38). Patients had fewer years of education (t = 2.4, p = .02), although their scores on the AmNART did not differ from those of controls (t = 1.2, p = .24). Patients also reported higher scores on fatigue (t = 7.5, p < .001) and depression (t = 4.5, p < .001). However, when education, fatigue, and depression scores were included in the analysis of the information processing measures, none of these variables emerged as significant covariates, and therefore, they were dropped. All results reported below are for the analyses without covariates.

Preliminary analyses of the measures of information processing speed also revealed no significant effects for either the order in which the administration formats (paper-based, computerized) or the order in which the tests (Stroop Test, SDMT) were introduced to participants, and therefore, factors representing these order variables were also eliminated from the final analyses. The final analyses consisted of a 2 (Group) × 2 (Administration Format) mixed factorial ANOVA performed on each measure derived from the Stroop Test and the SDMT. Means, standard deviations, and between-groups effect sizes (Cohen's d) for these measures are presented by group and administration format in Table 3.

Table 3.

Experiment 2: Comparison between groups and administration formats on information processing speed measures

  Paper-based
 
Computerized
 
M SD da M SD da 
Stroop 
 W 
  Patients 109.40 26.18 1.24 70.38 18.12 1.15 
  Controls 139.38 21.75 90.48 16.92 
 C 
  Patients 78.55 17.17 1.47 59.69 11.17 1.37 
  Controls 101.68 14.03 75.53 11.97 
 CW 
  Patients 48.90 14.24 1.03 43.71 10.89 1.00 
  Controls 62.43 11.75 54.13 9.81 
 W + C 
  Patients 187.95 41.32 1.44 130.07 28.17 1.29 
  Controls 241.05 31.33 166.00 27.64 
 RI 
  Patients 36.36 11.37 0.11 26.47 8.56 0.03 
  Controls 37.54 8.84 26.70 9.17 
SDMT 
  Patients 47.02 12.34 1.49 51.43 10.90 1.48 
  Controls 62.90 8.46 66.60 9.52 
  Paper-based
 
Computerized
 
M SD da M SD da 
Stroop 
 W 
  Patients 109.40 26.18 1.24 70.38 18.12 1.15 
  Controls 139.38 21.75 90.48 16.92 
 C 
  Patients 78.55 17.17 1.47 59.69 11.17 1.37 
  Controls 101.68 14.03 75.53 11.97 
 CW 
  Patients 48.90 14.24 1.03 43.71 10.89 1.00 
  Controls 62.43 11.75 54.13 9.81 
 W + C 
  Patients 187.95 41.32 1.44 130.07 28.17 1.29 
  Controls 241.05 31.33 166.00 27.64 
 RI 
  Patients 36.36 11.37 0.11 26.47 8.56 0.03 
  Controls 37.54 8.84 26.70 9.17 
SDMT 
  Patients 47.02 12.34 1.49 51.43 10.90 1.48 
  Controls 62.90 8.46 66.60 9.52 

Notes: W = word reading score (Trial 1); C = color naming score (Trial 2); CW = color-word naming score (Trial 3); W + C = combined score for word reading and color naming; RI = relative interference score; SDMT = Symbol Digit Modalities Test.

aEffect size concerning the difference between patients and controls based on Cohen's d.

As in Experiment 1, patients had significantly lower scores on all measures of information processing speed derived from the Stroop Test—W: F(1, 80) = 37.4, p < .001; C: F(1, 80) = 50.5, p < .001; CW: F(1, 80) = 24.6, p < .001; W + C: F(1, 80) = 46.7, p < .001—and the SDMT—F(1, 80) = 50.0, p < .001. Effect sizes for group differences were large for all scores, although lowest for the CW scores (Table 3). These effect sizes were slightly larger for the paper-based version of the Stroop Test, but virtually the same for the two administration formats of the SDMT.

Participants achieved significantly higher scores on the paper-based version compared with the computerized version of the Stroop Test—W: F(1, 80) = 391.2, p < .001; C: F(1,80) = 285.3, p < .001; CW: F(1, 80) = 46.0, p < .001; W + C: F(1, 80) = 458.8, p < .001. Conversely, they had higher scores on the computerized version than on the paper-based version of the SDMT—F(1, 80) = 32.5, p < .001.

In addition to the main effects, significant Group × Format interactions were found for scores derived from the first two trials of the Stroop Test—W: F(1, 80) = 4.9, p < .05; C: F(1, 80) = 7.5, p < .01; W + C: F(1, 80) = 7.7, p < .01. These interactions were due to greater disparity between patients and controls on the paper-based format than on the computerized format of this test. This disparity was also evident (though not significant) on the third trial of the Stroop Test (p = .12) and virtually absent on the SDMT (p = .62). In terms of RI, there was only a significant main effect for Format, F(1, 80) = 94.4, p < .001, due to comparatively greater interference on the paper-based version of the Stroop Test. Neither the main effect for Group nor the Group × Format interaction was significant (all p's > .66).

The correlations between scores under the two administration formats were computed separately for patients and controls and then combined across groups. These correlations were as follows: W, 0.57; C, 0.66; CW, 0.73; W + C, 0.66; RI, 0.50; SDMT, 0.82 (all p's < .001). The correlation between the SDMT score and the W + C score on the Stroop Test were also computed separately for patients and controls and then combined across groups. The combined correlation was .57 for the paper-based format and .71 for the computerized format (both p's < .001). As in Experiment 1, all correlations were slightly higher for patients owing to their somewhat greater variability on the tests.

Discussion

Experiment 2 replicated the findings of Experiment 1 in that MS patients performed substantially more slowly than controls on the Stroop Test and the SDMT regardless of the procedural format. Unlike Experiment 1, however, the administration format appeared to impact performance differently on the two tasks. Although participants generally completed more items on the paper-based version than on the computerized version of the Stroop Test, the opposite was true for the SDMT. The difference could be explained by the process earlier referred to as “forward anticipation,” in which participants begin processing the next item while completing their response to the current one. The paper-based Stroop Test may be more conducive to forward anticipation than the paper-based SDMT because, on the latter, participants must frequently shift their gaze between the stimulus item and the reference key at the top of the page. Furthermore, the reason that more items were completed on the computerized than on the paper-based version of the SDMT may be that the computerized format made it easier to return to the stimulus item after consulting the reference key, since only one such item was present in the center of the screen rather than being embedded in a cluttered field of items.

Forward anticipation would also seem to be more likely when the operation to be performed on each item is a relatively automatic one such as reading words or naming colors on the first two trials of the Stroop Test. This would explain why the differences between formats are greatest for these two trials of the Stroop Test, especially the word-reading trial. It might also help to account for the significant interactions between Group and Format that occurred on these two trials. The cognitive burden imposed by RSP appears to be greater for patients with MS than for controls, and this may limit patients' ability to exploit forward anticipation as a strategy even on relatively simple tasks.

General Discussion

The procedure variations investigated here do affect scores pertaining to information processing speed on both the Stroop Test and the SDMT; it should not be surprising that the speed with which participants can perform a task is dependent on how the task is formatted. Thus, for example, in Experiment 1, if participants are allowed to press the space bar to advance to the next item, they can do so while simultaneously giving their verbal response and therefore move through the items more quickly than if they must wait for the brief delay required for the experimenter to press the space bar. A more interesting finding is that the paper and computerized administration formats examined in Experiment 2 had a different impact on the SDMT than on the Stroop Test. Compared with their respective computer-formatted tests, performance for all participants was “faster” on the paper-based Stroop Test, but “slower” on the paper-based SDMT. The presence of multiple items on paper-based tests offers the possibility of forward anticipation, but this strategy appears to be more readily exploited when the task to be performed on each item is relatively simple and automatic, such as those involved in the first two trials of the Stroop Test. On the other hand, both the color-word naming trial of the Stroop Test and the SDMT constitute more demanding tasks. CW naming requires participants to suppress the more automatic word-reading response and to resist interference arising from the incongruity between the word and color of each item. The SDMT requires participants to shift their visual focus between each stimulus item and the reference key, thereby demanding greater visual scanning. Although stemming from different sources, the greater cognitive burden of these trials seems to attenuate the use of forward anticipation by participants in both groups.

In terms of the primary aim of these studies, the most pertinent findings originate from the interactions between Group and Format. When these interactions are not significant, as in the case of all measures in Experiment 1 as well as the SDMT in Experiment 2, one can feel assured that the procedural variation encompassed by the format does not impact the information processing performance of patients differently than that of controls and therefore one is safe in adopting either variation. However, when significant Group × Format interactions occur, they potentially inform about underlying features of the individuals' performance on the task, features that may not equally characterize the performance of patients and controls. Such was the case in Experiment 2 on the preliminary word-reading and color-naming trials of the Stroop Test. Here, differences in the abilities of patients and controls to exploit a forward anticipation strategy in the cluttered field presented by the paper-based format may possibly account for the greater disparity between patients and controls on these relatively simple information processing tasks.

The relevance of the present results to those of other studies is complicated by the fact that previous investigators who have examined computerized versus paper-based versions of the Stroop Test (Potter et al., 2002; Salo et al., 2001; Seignourel et al., 2005) or the SDMT (Akbar et al., 2011) have consistently altered the tests from measures of RSP to that of RT. Furthermore, the Stroop studies have focused primarily on interference rather than information processing speed and have involved normal individuals alone (Salo et al., 2001) or in comparison with patients with traumatic head injury (Potter et al., 2002; Seignourel et al., 2005). The most comparable study is the one by Akbar and colleagues (2011) comparing the performance of MS patients and healthy controls on both paper-based and computerized versions of the SDMT, but here too, differences abound. The computerized SDMT designed by these investigators presented rows of nine items below the reference key instead of just single items as in the present study, so the interplay of forward anticipation and scanning differs between the two studies. Furthermore, Akbar and colleagues failed to analyze their results as a Group × Format ANOVA, reporting only that the patients performed more poorly than controls on both versions of the SDMT. Although no comparison between formats and no examination of the Group × Format interaction were offered, it is possible to glean some information from mean RTs reported in Table 3 of their paper. As in the present study, it appears that performance by all participants was slower on the paper-based version of the SDMT and no interaction between group and format occurred in this earlier study.

Comparisons to other studies are possible by considering the effect sizes for patients versus controls on various tests and trials. It is important to emphasize that these effect size differences do not constitute “statistically significant” differences between the measures; nevertheless, they are useful to consider when comparing different neuropsychological tests reputably evaluating the same attribute (Zakzanis, 2001). When Hughes and colleagues (2011) examined performance on both the SDMT and the Stroop Test, they found that the SDMT yielded a numerically larger effect size between MS patients and controls than any of the measures derived from the Stroop Test. They discussed this advantage accruing to the SDMT as a possible “augmentation effect”: the sensitivity with which a measure is able to distinguish between patients and controls is augmented when the measure evaluates more than one attribute on which patients have deficiencies relative to healthy individuals. While primarily a measure of information processing speed, the SDMT also entails a substantial amount of scanning, and this additional requirement may serve to augment the test's ability to differentiate MS patients from controls relative to that of the Stroop Test. Evidence of augmentation is also seen in the present results. When scores are averaged across both variations in each experiment and effect sizes are computed for the combined samples of patients and controls, the effect size for the SDMT score (d = 112.3) was again larger than those for the scores derived from the preliminary trials of the Stroop Test (W, d = 0.85; C, d = 0.96; W + C, d = 0.93). Thus, in the present study, the better RSP measure for differentiating between patients and controls was the SDMT, though its advantage may stem from the added requirement it places on scanning and not due solely to its efficacy as a measure of information processing speed.

The computerized and paper-based versions of the Stroop Test and the SDMT compared in Experiment 2 differ in several ways, and a shortcoming of this experiment is that it permits only speculations as to which of these differences might constitute important determinants of participants' performance. In particular, it would be useful to have included a cluttered field condition within the computerized format of each test, as previous studies of the Stroop Test (Salo et al., 2001) and the SDMT (Akbar et al., 2011) have done. In such a condition, more than one stimulus item would be presented on the screen at the same time, probably as one or more rows of items that the participant completes from left to right. The condition would represent something of a “compromise” between the two formats examined in Experiment 2, offering greater opportunity for forward anticipation than a single item presentation, but also partly limiting the burden on visual scanning that occurs when all stimulus items are presented together. Various permutations of such compromise conditions could lead to greater understanding of the roles of both anticipation and scanning as determinants of performance on these RSP measures.

Although deficits in information processing speed are certainly not the only cognitive impairments observed in patients with MS, the present results support the consensus in the current literature that processing speed is the predominant area impacted by this disease. The results also show that impairments in this domain can be effectively detected using a variety of procedural variations. Although no single response or administration format is significantly more effective in identifying slowed processing speed, researchers should be cautious to use consistent methodology within a given study, as slight differences emerge between formats for both patients and controls. Furthermore, in the case of studies that include patients with significant problems with manual dexterity, having an experimenter (E-press) operate response keys appears to be a viable alternative to reduce confounds due to physical disability, without introducing measurable experimenter bias. Also, because paper-based administration formats introduce problems stemming from the cluttered field of stimulus items and the elicitation of forward anticipation strategies, future studies should take advantage of the convenience and reliability of computerized administration formats that preserve the original RSP formats of the Stroop Test and the SDMT.

Funding

The research presented in this article was unfunded and based largely on work completed by the first author in conjunction with her master's thesis.

Conflict of Interest

None declared.

References

Akbar
N.
Honarmand
K.
Kou
N.
Feinstein
A.
Validity of a computerized version of the Symbol Digit Modalities Test in multiple sclerosis
Journal of Neurology
 , 
2011
, vol. 
258
 (pg. 
373
-
379
)
Archibald
C. J.
Fisk
J. D.
Information processing efficiency in patients with multiple sclerosis
Journal of Clinical and Experimental Neuropsychology
 , 
2000
, vol. 
22
 (pg. 
686
-
701
)
Arnett
P. A.
Smith
M. M.
Barwick
F. H.
Benedict
R. H. B.
Ahlstrom
B. P.
Oralmotor slowing in multiple sclerosis: Relationship to neuropsychological tasks requiring an oral response
Journal of the International Neuropsychological Society
 , 
2008
, vol. 
14
 (pg. 
454
-
462
)
Benedict
R. H. B.
Cookfair
D.
Gavett
R.
Gunther
M.
Munschauer
F.
Garg
N.
, et al.  . 
Validity of the Minimal Assessment of Cognitive Function in Multiple Sclerosis (MACFIMS)
Journal of the International Neuropsychological Society
 , 
2006
, vol. 
12
 (pg. 
549
-
558
)
Bergendal
G.
Fredrikson
S.
Almkvist
O.
Selective decline in information processing in subgroups of multiple sclerosis: An 8-year longitudinal study
European Neurology
 , 
2007
, vol. 
57
 (pg. 
193
-
202
)
Bodling
A. M.
Denney
D. R.
Lynch
S. G.
Rapid serial processing in patients with multiple sclerosis: The contribution of peripheral deficits
Journal of the International Neuropsychological Society
 , 
2008
, vol. 
14
 (pg. 
646
-
650
)
Bodling
A. M.
Denney
D. R.
Lynch
S. G.
Cognitive aging in patients with multiple sclerosis: A cross-sectional analysis of speeded processing
Archives of Clinical Neuropsychology
 , 
2009
, vol. 
24
 (pg. 
761
-
767
)
Bodling
A. M.
Denney
D. R.
Lynch
S. G.
Individual variability in speed of information processing: An index of cognitive impairment in multiple sclerosis
Neuropsychology
 , 
2012
, vol. 
26
 (pg. 
357
-
367
)
Brochet
B.
Deloire
M. S.
Bonnet
M.
Salort-Campana
E.
Quallet
J. C.
Petry
K. G.
, et al.  . 
Should SDMT substitute for PASAT in MSFC? A 5-year longitudinal study
Multiple Sclerosis
 , 
2008
, vol. 
14
 (pg. 
1242
-
1249
)
Bruce
J. M.
Bruce
A. S.
Arnett
P. A.
Mild visual acuity disturbances are associated with performance on tests of complex attention in MS
Journal of the International Neuropsychological Society
 , 
2007
, vol. 
13
 (pg. 
544
-
548
)
DeLuca
J.
Chelune
G. J.
Tulsky
D. S.
Lengenfelder
J.
Chiaravalloti
N. D.
Is speed of processing or working memory the primary information processing deficit in multiple sclerosis?
Journal of Clinical and Experimental Neuropsychology
 , 
2004
, vol. 
26
 (pg. 
550
-
562
)
Demaree
H. A.
DeLuca
J.
Gaudino
E. A.
Diamond
B. J.
Speed of processing as the key deficit in multiple sclerosis: Implications for rehabilitation
Journal of Neurology, Neurosurgery, and Psychiatry
 , 
1999
, vol. 
67
 (pg. 
661
-
663
)
Denney
D. R.
Gallagher
K. A.
Lynch
S. G.
Deficits in processing speed in patients with multiple sclerosis: Evidence from explicitly-timed and covertly-timed measures
Archives of Clinical Neuropsychology
 , 
2011
, vol. 
26
 (pg. 
110
-
119
)
Denney
D. R.
Lynch
S. G.
The impact of multiple sclerosis on patients' performance on the Stroop Test: Processing speed vs. interference
Journal of the International Neuropsychological Society
 , 
2009
, vol. 
15
 (pg. 
451
-
458
)
Denney
D. R.
Lynch
S. G.
Parmenter
B. A.
Horne
N.
Cognitive impairment in relapsing and primary progressive multiple sclerosis: Mostly a matter of speed
Journal of the International Neuropsychological Society
 , 
2004
, vol. 
10
 (pg. 
948
-
956
)
Denney
D. R.
Sworowski
L. A.
Lynch
S. G.
Cognitive impairment in three subtypes of multiple sclerosis
Archives of Clinical Neuropsychology
 , 
2005
, vol. 
20
 (pg. 
967
-
981
)
de Sonneville
L. M.
Boringa
J. B.
Reuling
I. E.
Lazeron
R. H.
Adèr
H. J.
Polman
C. H.
Information processing characteristics in subtypes of multiple sclerosis
Neuropsychologia
 , 
2002
, vol. 
40
 (pg. 
1751
-
1765
)
Drake
A. S.
Weinstock-Guttman
B.
Morrow
S. A.
Hojnacki
D.
Munshauer
F. E.
Benedict
R. H. B.
Psychometrics and normative data for the Multiple Sclerosis Functional Composite: Replacing the PASAT with the Symbol Digit Modalities Test
Multiple Sclerosis
 , 
2010
, vol. 
16
 (pg. 
228
-
237
)
Friend
K. B.
Grattan
L.
Use of the North American Adult Reading Test to estimate premorbid intellectual function in patients with multiple sclerosis
Journal of Clinical and Experimental Neuropsychology
 , 
1998
, vol. 
20
 (pg. 
846
-
851
)
Golden
C. J.
The Stroop Color and Word Test
 , 
1978
Wood Dale, IL
Stoelting Company
Hinton-Bayre
A.
Geffen
G.
Comparability, reliability, and practice effects on alternate forms of the Digit Symbol Substitution and Symbol Digit Modalities Tests
Psychological Assessment
 , 
2005
, vol. 
17
 (pg. 
237
-
241
)
Hohol
M. J.
Orav
E. J.
Weiner
H. L.
Disease steps in multiple sclerosis: A simple approach to evaluate disease progression
Neurology
 , 
1995
, vol. 
45
 (pg. 
251
-
255
)
Hughes
A. J.
Denney
D. R.
Lynch
S. G.
Reaction time and rapid serial processing measures of information processing speed in multiple sclerosis: Complexity, compounding, and augmentation
Journal of the International Neuropsychological Society
 , 
2011
, vol. 
17
 (pg. 
1113
-
1121
)
Jennekens-Schinkel
A.
Lanser
J. B. K.
Van Der Velde
E. A.
Sanders
E. A. C. M.
Performances of multiple sclerosis patients in tasks requiring language and visuocontruction: Assessment of outpatients in quiescent disease stages
Journal of the Neurological Sciences
 , 
1990
, vol. 
95
 (pg. 
89
-
103
)
Kail
R.
The neural noise hypothesis: Evidence from processing speed in adults with multiple sclerosis
Aging, Neuropsychology, and Cognition
 , 
1997
, vol. 
4
 (pg. 
157
-
165
)
Kail
R.
Speed of information processing in patients with multiple sclerosis
Journal of Clinical and Experimental Neuropsychology
 , 
1998
, vol. 
20
 (pg. 
98
-
106
)
Krupp
L. B.
LaRocca
N. G.
Muir-Nash
J.
Steinberg
A. D.
The fatigue severity scale: Application to patients with multiple sclerosis and systemic lupus erythematosus
Archives of Neurology
 , 
1989
, vol. 
46
 (pg. 
1121
-
1123
)
Kujala
P.
Portin
R.
Revonsuo
A.
Ruutiainen
J.
Automatic and controlled information processing in multiple sclerosis
Brain
 , 
1994
, vol. 
117
 (pg. 
1115
-
1126
)
Kurtzke
M. D.
Rating neurologic impairment in multiple sclerosis: An expanded disability status scale (EDSS)
Neurology
 , 
1983
, vol. 
33
 pg. 
1444
 
Lengenfelder
J.
Bryant
D.
Diamond
B.
Kalmar
J.
Moor
N.
DeLuca
J.
Processing speed interacts with working memory efficiency in multiple sclerosis
Archives of Clinical Neuropsychology
 , 
2006
, vol. 
21
 (pg. 
229
-
238
)
Lynch
S. G.
Dickerson
K. J.
Denney
D. R.
Evaluating speed in multiple sclerosis: A comparison of two rapid serial processing measures
The Clinical Neuropsychologist
 , 
2010
, vol. 
24
 (pg. 
963
-
976
)
Macniven
J. A.
Davis
C.
Ho
M. Y.
Bradshaw
C. M.
Szabadi
E.
Constantinescu
C. S.
Stroop performance in multiple sclerosis: Information processing, selective attention, or executive functioning?
Journal of the International Neuropsychological Society
 , 
2008
, vol. 
14
 (pg. 
805
-
814
)
Nelson
H. E.
National Adult Reading Test (NART): Test manual
 , 
1982
Windsor, UK
NFER Nelson
Parmenter
B. A.
Weinstock-Guttman
B.
Garg
N.
Munschauer
F.
Benedict
R. H. B.
Screening for cognitive impairment in multiple sclerosis using the Symbol Digit Modalities Test
Multiple Sclerosis
 , 
2007
, vol. 
13
 (pg. 
52
-
57
)
Polman
C. H.
Reingold
S. C.
Edan
G.
Filippi
M.
Hartung
H. P.
Kappos
L.
, et al.  . 
Diagnostic criteria for multiple sclerosis: 2005 revisions to the “McDonald criteria”
Annals of Neurology
 , 
2005
, vol. 
58
 (pg. 
840
-
846
)
Portaccio
E.
Goretti
B.
Zipoli
V.
Siracusa
G.
Sorbi
S.
Amato
M. P.
A short version of Rao's Brief Repeatable Battery as a screening tool for cognitive impairment in multiple sclerosis
The Clinical Neuropsychologist
 , 
2009
, vol. 
23
 (pg. 
268
-
275
)
Potter
D. D.
Jory
S. H.
Bassett
M. R. A.
Barrett
K.
Mychalkiw
W.
Effect of mild head injury on event-related potential correlates of Stroop task performance
Journal of the International Neuropsychological Society
 , 
2002
, vol. 
8
 (pg. 
828
-
837
)
Radloff
L. S.
The CES-D Scale: A self-report depression scale for research in the general population
Applied Psychological Measurement
 , 
1977
, vol. 
1
 (pg. 
385
-
401
)
Rao
S. M.
St. Aubin-Faubert
P.
Leo
G. L.
Information processing speed in patients with multiple sclerosis
Journal of Clinical and Experimental Neuropsychology
 , 
1989
, vol. 
11
 (pg. 
471
-
477
)
Reicker
L. I.
Tombaugh
T. N.
Walker
L.
Freedman
M. S.
Reaction time: An alternative method for assessing the effects of multiple sclerosis on information processing speed
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22
 (pg. 
655
-
664
)
Salo
R.
Henik
A.
Robertson
L. C.
Interpreting Stroop interference: An analysis of differences between task versions
Neuropsychology
 , 
2001
, vol. 
15
 (pg. 
462
-
471
)
Schulz
D.
Kopp
B.
Kunkel
A.
Faiss
J. H.
Cognition in the early stage of multiple sclerosis
Journal of Neurology
 , 
2006
, vol. 
253
 (pg. 
1002
-
1010
)
Seignourel
P. J.
Robins
D. L.
Larson
M. J.
Demery
J. A.
Cole
M.
Perlstein
W. M.
Cognitive control in closed head injury: Context maintenance dysfunction or prepotent response inhibition deficit?
Neuropsychology
 , 
2005
, vol. 
19
 (pg. 
578
-
590
)
Smith
A.
Symbol Digit Modalities Test-Revised
 , 
1982
Los Angeles
Western Psychological Services
Smith
M. M.
Arnett
P. A.
Dysarthria predicts poorer performance on cognitive tasks required speeded oral response in an MS population
Journal of Clinical and Experimental Neuropsychology
 , 
2007
, vol. 
29
 (pg. 
804
-
812
)
Tombaugh
T. N.
Berrigan
L. I.
Walker
L. A. S.
Freedman
M. S.
The Computerized Test of Information Processing (CTIP) offers an alternative to the PASAT for assessing cognitive processing speed in individuals with multiple sclerosis
Cognitive and Behavioral Neurology
 , 
2010
, vol. 
23
 (pg. 
192
-
198
)
Zakzanis
K. K.
Statistics to tell the truth, the whole truth, and nothing but the truth: Formulae, illustrative numerical examples, and heuristic interpretation of effect size analyses for neuropsychological researchers
Archives of Clinical Neuropsychology
 , 
2001
, vol. 
16
 (pg. 
653
-
667
)