Abstract

We developed a multiple-form list learning test appropriate for use with the Greek population and generated norms for clinical and research use. This task, the Greek Verbal Learning Test (GVLT), was based on the California Verbal Learning Test. We administered the standard version (Form A) to a sample of 354 healthy individuals, as well as two alternative forms (B and C) to a subgroup of the initial sample. Performance on the three forms was equivalent, and each test presented excellent internal consistency. We found good sensitivity and specificity in the testãs (Form A) utility in differentiating individuals with schizophrenia (n = 50) and individuals with traumatic brain injury (n = 53) from healthy adults. A multiple regression analysis indicated that age, education and sex predicted performance. Regression-based norms are also provided. Taken together, these data provide preliminary support for the reliability and construct validity of the GVLT.

Rapidly growing interest in neuropsychological assessment across the globe has led to the need for the development of culturally tailored tests (Nell, 2000; Puente & Perez-Garcia, 2000; Uzzell, Ponton, and Ardila, 2007). Given the need to assess individuals' functioning in terms of their ability to learn, retain, and recall or recognize verbal information, a number of tasks have been adapted or translated and normed in non-English-speaking cultures. Examples include the Rey Auditory-Verbal Learning Test (RAVLT; Rey, 1964), which has been translated and normed in several languages and cultures (e.g., Malloy-Diniz, Lasmar, Gazinelli, Fuentes, & Salgado, 2007; Mueller, Hasse-Sander, Horn, Helmsteadter, & Elger, 1997), including Greek (Messinis, Tsakona, Malefaki, & Papathanasopoulos, 2007), and the Hopkins Verbal Learning Test-Revised (HVLT-R; Brandt & Benedict, 2001), which has also been adapted to French (Rieu, Bachoud-Levi, Laurent, Jurion, & Barba, 2006) and Spanish (Cherner et al., 2007).

Interestingly, the importance of adapting verbal-learning tasks to the culture in which they will be used has been demonstrated previously, and is independent of limitations posed by differences in language or dialect. Barker-Collo, Clarkson, Cribb, & Grogan (2002) administered an American list-learning test and a version modified to reflect New Zealand-relevant content to 90 healthy individuals aged 17 to 81 years. They found that New Zealanders performed significantly more poorly on the American version of the task than on the culturally relevant, New Zealand version, despite the fact that both versions of the task were in English. This highlights the nontrivial influence of cultural factors above and beyond language.

One of the most widely used verbal memory tasks, both clinically and in research protocols, is the California Verbal Learning Test in both its original (CVLT; Delis et al., 1987) and revised forms (CVLT-II; Delis, Kramer, Kaplan, & Ober, 2000). In comparison with the aforementioned verbal memory measures, which assess learning, retention, and recall of rote verbal information, this test additionally offers the opportunity to assess the use of strategies in learning, such as semantic, serial, and subjective clustering. The individual's ability to develop strategies for the purpose of increasing the amount of verbal information learned and retained has been associated with executive-type abilities (Brooks, Weaver, & Scialfa, 2006; Nolan, 2006; Ribeiro, Guerreiro, & De Mendonca, 2007; Tremont, Halpert, Javorsky, & Stern, 2000). Therefore, selecting the verbal stimuli of a list-learning task so that they can be organized into semantic categories provides the additional opportunity to assess the use of learning strategies, an executive verbal memory ability. Although beyond the scope of this study, there is considerable literature supporting the clinical utility of the English version of the test in terms of differential diagnosis, as well as sufficient evidence on typical patterns of performance associated with a variety of neurological and psychiatric diagnoses. Furthermore, one study compared the CVLT with the HVLT-R in terms of sensitivity to diagnosis, and concluded that the CVLT is a more helpful tool for the evaluation of certain recall errors common in Alzheimer's disease (Lacritz, Cullum, Weiner, & Rosenberg, 2001). Another study compared the CVLT to the Wechsler Memory Scale-III (WMS-III) word list test and found the former to be a more sensitive tool in detecting impairment than the latter (McDowell, Bayless, Moser, Meyers, & Paulsen, 2004). Thus, the inclusion of semantically related material appears to augment its clinical utility.

Several suggestions have been made regarding methods for appropriate neuropsychological assessment in non-English-speaking countries. Direct translation may be inadequate as the actual stimuli may be less familiar in the culture to which the test is being adapted than in the culture of the test's origin (Kosmidis, 2008; Nell, 2000). This is particularly problematic with word lists, as word frequency in each language may differ considerably, changing the level of difficulty of the test from one language to another. A more sound approach would be to create tests that target the cognitive component assessed in traditional neuropsychological tests, while using culturally relevant stimuli and developing appropriate norms for the specific population. Consequently, a growing number of new tests are being developed specifically for the Greek culture inspired by existing tests in English (e.g., Folia & Kosmidis, 2003; Giaglis et al., 2008, Giaglis, Kyriazidou, Paraskevopoulou, Tascos, & Kosmidis, 2010; Hiou, Vagia, Haritidou, Karakostas, & Kosmidis, 2004; Kosmidis et al., 2004a, 2004b; Vlahou, Chatzikallia, & Kosmidis, 2011; Vlahou, Kosmidis, Dardagani, & Tsotsi, 2011).

Despite its clinical utility, studies presenting culturally adapted versions and/or norms for the CVLT, CVLT-II, or California Verbal Learning Test-Children's Version (CVLT-C) for non-English-speaking populations are limited. Rosselli, Ardila, Bateman, & Guzman (2001) administered the CVLT-C to 290 Spanish-speaking children in Colombia. Adaptation of the test to Spanish involved a direct translation of the original word list, and subsequent modification of 8 out of the 16 words. Furthermore, plural words were used to avoid the use of articles in the Spanish language. The culturally adapted version resulted in higher scores in comparison with the performance of the normative sample on the translated English version. The authors attributed this to the more simplified nature of the Spanish stimuli and the lack of an interference trial. Nevertheless, this finding also offers additional evidence underlining the importance of adapting neuropsychological tests to both language and culture. Although other tests assessing list-learning abilities and verbal memory are available in Greek (RAVLT; Messinis et al., 2007), to our knowledge, there have been no previous efforts to model a culturally appropriate test for Greek-speaking populations after the CVLT or CVLT-II, in terms of semantic categories and assessment of verbal executive functioning.

Besides culture and language effects, previous studies have indicated that demographic factors such as gender, age, and educational level can have a significant effect on participants' performance on the adult version of the CVLT-II. Kramer, Yaffe, Legenfelder, & Delis (2003) reported an interaction between age and gender in a large sample of 573 young (16–47 years old) and 446 older (55–89 years old) men and women. Specifically, verbal memory was negatively correlated with age for the sample of younger and that of older men, as well as the sample of older women, but the same pattern did not hold for the sample of younger women, whose performance was not related to age. This finding was attributed to the potential protective effects of estrogen in the younger women. Similarly, Rosselli and colleagues (2001) also found a significant effect for age and gender on test performance among children on the CVLT-C, with better performance among older, when compared with younger, children, and among girls when compared with boys. Therefore, women seem to perform better on this test across the lifespan; thus, while age does affect performance, this effect appears to be mediated by gender.

Our aim in the current study was to present a model for the development of culturally appropriate tests for non-English-speaking populations. Specifically, our goal was to develop a Greek Verbal Learning Test (“GVLT”) based on the CVLT that is culturally relevant and to present regression-based norms (RBNs) for clinical and research use. Based on previous investigations, we hypothesized that demographic factors, mainly age, education, and sex, would influence performance. Therefore, we explored the potential influence of demographic variables on performance, as well as the test's psychometric properties, including diagnostic validity of the test and its utility in differentiating neurological and/or psychiatric patients from healthy individuals. Furthermore, previous studies have supported the use of an alternate test form when retesting is expected, as when evaluating an intervention in order to ascertain progress, or in the case of a deteriorating condition (Benedict, 2005; Woods et al., 2005). Given the frequent need for repeat testing of individuals with memory decline, we developed two alternate forms of the test. Therefore, an additional goal was to explore the equivalence of the three forms of the test (A, B, and C) for use in follow-up assessments.

Method

Participants

Three hundred and fifty-four healthy community-dwelling individuals (55% women) completed the standard form (Form A) of the GVLT. Participants were recruited from a large pool of regular volunteers for our lab and were members of the community in a metropolitan area in the Northern region of Greece. Men and women did not differ in age [t(352) = − 1.23, p = .221] or education level [t(338) = 1.46, p = .144]. Out of the total sample, 101 participants (53% women) were also administered an alternate form of the test (Form B). Furthermore, a smaller subgroup of individuals from the same sample (n = 45, 47% women) were administered a second alternate form of the test (Form C) in addition to the other two—a total of three different forms. Participants to whom Forms B and C were also administered were selected through sequential sampling. All participants included in the study were literate, having received at least 1 year of schooling, and their native language was Greek. Exclusion criteria for healthy participants were a history of a neurological or psychiatric disorder, closed head injury, or other medical conditions potentially compromising the central nervous system (based on self-report). Participants over 65 years of age with a Mini-Mental State Examination score less than 24 were excluded from the study (cutoff based on a normative study in Greece; Fountoulakis, Tsolaki, Mohs, & Kazis, 1998). Demographic characteristics for the sample and each subsample separately are presented in Table 1.

Table 1.

Demographic characteristics of the main sample (Form A) and subsamples

  N Age
 
Education (years)
 
Range Mean (SD) Range Mean (SD) 
Form A only 354 15–88 50.6 (19.9) 1–21 11.3 (4.7) 
Form A and B 101 18–78 39.2 (15.1) 6–18 13.5 (3.1) 
Form A, B, and C 45 18–69 33.8 (11.3) 6–17 14.3 (2.6) 
  N Age
 
Education (years)
 
Range Mean (SD) Range Mean (SD) 
Form A only 354 15–88 50.6 (19.9) 1–21 11.3 (4.7) 
Form A and B 101 18–78 39.2 (15.1) 6–18 13.5 (3.1) 
Form A, B, and C 45 18–69 33.8 (11.3) 6–17 14.3 (2.6) 

To evaluate the diagnostic validity of the test, the standard form (Form A) was also administered to 50 individuals (40% women) diagnosed with schizophrenia [age range 20–65 years (M = 35.9, SD = 9.9), education range 6–17 years (M = 11.3, SD = 2.9)], and 53 individuals (26% women) who had suffered a traumatic brain injury (TBI) [age range 18–53 years (M = 31.3, SD = 10), education range 6–21 years (M = 12.4, SD = 3.4)]. These individuals were recruited through a large regional hospital. Those with schizophrenia were consecutive psychiatric admissions invited to participate in the study shortly before discharge and once their condition had stabilized. Medical history, including diagnosis, was retrieved from medical records with the participants' consent. Individuals with TBI at least 6 months post injury were identified through hospital records and invited to participate in a broader study. Exclusion criteria for the sample of individuals with schizophrenia were a history of other disorders affecting the central nervous system as well as a head injury with loss of consciousness longer than 10 min, while for the head injury group the exclusion criteria were any psychiatric or neurological diagnoses. The clinical characteristics of these patient groups are presented in Table 2.

Table 2.

Clinical characteristics of patients with schizophrenia and with TBI

Schizophrenia
 
Traumatic brain injury 
 Percentage  Percentage 
Illness subtype Severity 
 Paranoid 45  Mild 32.1 
 Residual 35  Moderate 22.6 
 Undifferentiated 10  Severe 41.5 
 Brief psychotic episode 10 Type 
   Closed 90.6 
   Open 5.7 
Medications (antipsychotics)    
 Typical   
 Atypical 100   
   Mean (SD) 
Severity Mean (SD) Time since injury (months) 37.00 (24.98) 
 PANSS 66.16 (22.12) Glascow Coma Scale 11.70 (4.51) 
Illness duration (months) 10.05 (8.10)   
Age at first diagnosis 25.50 (7.32)   
Number of hospital admissions 2.94 (3.49)   
Schizophrenia
 
Traumatic brain injury 
 Percentage  Percentage 
Illness subtype Severity 
 Paranoid 45  Mild 32.1 
 Residual 35  Moderate 22.6 
 Undifferentiated 10  Severe 41.5 
 Brief psychotic episode 10 Type 
   Closed 90.6 
   Open 5.7 
Medications (antipsychotics)    
 Typical   
 Atypical 100   
   Mean (SD) 
Severity Mean (SD) Time since injury (months) 37.00 (24.98) 
 PANSS 66.16 (22.12) Glascow Coma Scale 11.70 (4.51) 
Illness duration (months) 10.05 (8.10)   
Age at first diagnosis 25.50 (7.32)   
Number of hospital admissions 2.94 (3.49)   

Materials

The format of the newly developed test was based on the shopping list design of the original CVLT (Delis et al., 1987). Three forms of the test were created, a standard form (A) and two alternative forms (B and C). Every form of the test comprised two lists, each containing 16 items total: four categories of four words each. None of the categories were repeated on different forms of the test. The test was divided into 10 recall trials and a recognition task.

Procedure

Test Development

Since we were developing a new test, we were not confined to following the format of either the original or the revised CVLT. Despite the limitations of the original CVLT pertaining to shopping list learning (Delis et al., 2000), we retained this format, as it appeared more ecologically valid for the Greek culture. Given the relatively large proportion of elderly individuals in the Greek population with minimal or even no education, we were concerned about the cultural appropriateness of a list-learning test resembling school-based tasks with little relevance to daily life. Previous studies have attributed poor performance among individuals with no or minimal formal schooling to artifacts of the test format (Folia & Kosmidis, 2003), highlighting the importance of task familiarity (Nell, 2000; Yassuda et al., 2009), as well as school-based acculturation (Kosmidis, Tsapkini, & Folia, 2006; Kosmidis, Zafiri, & Politimou, 2011) (i.e., placing value on successful performance of tests, training in cognitive strategies to maximize learning efficiency). In an attempt to minimize any gender bias on the test, we selected categories that were balanced with respect to their relevance to each gender, within (and according to) the Greek culture (this selection was done at face value). In this way we hoped to overcome the problems identified in the original CVLT. The category of words in the recognition condition that rhyme with those in the target list determined the choice of words within the four categories of the target list.

As in the CVLT, we created an initial shopping list (“Monday list”) and a second, interference, list (“Tuesday list”). The 16 words in the interference list also belonged to four categories; two of those categories were the same as in the Monday list (i.e., clothing and fruit) and two belonged to two new categories, i.e., “fish” and “vegetables.” The four words selected for each semantic category were all different from those in the previous list.

Finally, for the recognition task, we selected 44 words that were either: (a) from the Monday list (16 words), (b) from the Tuesday list, category in common with the Monday list (4 words), (c) from the Tuesday list, category not in common with the Monday list (4 words), (d) new word from a similar semantic category to one in the Tuesday list (4 words), (e) new word from one of the same semantic categories as the Monday list (4 words), (f) new word from an irrelevant semantic category (4 words), or (g) new word, phonetically similar to a word on the Monday list (8 words). We should note that the words on the Monday list were in part determined by the existence, in Greek, of a phonetically similar word to use as a foil on the recognition condition. In fact, we followed the same procedure in developing the two alternative forms (B and C) for use in follow-up assessments to avoid practice effects. The word lists for each of the three forms developed are presented in the Appendix.

We initially created one form of the test (Form A, created by MHK and CHV) and administered only this in the early part of the project. As the need for alternate versions arose in other parallel investigations, we created the second and, later, the third forms (Form B, created by EZ and MHK; and Form C, created by MG and MHK) in the same manner as we did for Form A. These latter forms were added to the administration procedure at the time of their development and were thus given to all subsequent participants (sequential sampling).

Test Administration

Five learning trials were provided in which the examiner read the Monday shopping list, and the examinee was requested to recall as many of the items as possible regardless of order. The interference (Tuesday) list was read subsequent to the five Monday learning trials, followed first by an immediate free, then by a cued recall trial (wherein the examiner indicated the semantic categories one at a time and asked the participants to recall all items belonging to each) of the Monday list. During the 20-min delay, other tests were administered. Finally, delayed recall was examined in a free and a cued format (as previously), followed by a yes/no recognition trial in which examinees indicated which of the new list of words were included in the Monday shopping list and which were not.

Scoring

The scoring procedure involved writing each word recalled by the participant on each trial in the order in which it was recalled and then adding the number of words recalled on each trial. In order to calculate the learning slope, which reflects the amount of new learning on each trial for the five learning trials, we computed the least-squares regression line which fits the number of words learned across trials (according to the guidelines provided in the CVLT manual, Delis et al., 1987).

Study Protocol

The study was approved by the head of the Research Committee of the Aristotle University of Thessaloniki, the ethics board of the hospital from which individuals with TBI or schizophrenia were recruited, and the Hellenic Data Protection Authority (regarding the handling of personal data). All participants volunteered to participate and did not receive financial compensation. Healthy participants were assessed individually in a quiet, distraction-free room in their homes, or in our lab; individuals with TBI or schizophrenia were assessed in their respective clinics, in a room appropriate for testing.

The data were collected as part of a larger, European-funded project, over the period 2009–2011. The test was administered by undergraduate and graduate psychology students (trained and supervised by MHK, a licensed clinical neuropsychologist). Alternative versions were administered in the same order for all participants (A–B–C) and in three separate sessions at 1-week intervals. The test was administered in a standardized manner to all participants, in Greek, among other brief non-verbal neuropsychological tests which were administered during the 20-min delay phase (e.g., Trail Making Test, Judgment of Line Orientation). Initially, all participants were provided with information regarding the study, and asked to sign an informed consent form. Healthy participants provided basic information on demographics (e.g., date of birth, education level) and medical history (e.g., history of head injury with loss of consciousness, psychiatric or neurological diagnosis, medication regimens, etc.) as part of a semi-structured screening interview. Given the time limitation in the medical settings and to assure the accuracy of the information, we retrieved demographic information and the medical history of individuals with TBI or schizophrenia from their medical records, with their consent.

Variables of interest included performance on trial 1, as a measure of working memory, and trial 5, as a measure of working memory and learning; total number of words recalled over the five trials and learning slope across trials 1–5, both indicators of participants' learning curve; also, of interest were performance on the interference trial (retroactive interference), short-delay recall with and without cues, long-delay recall with and without cues, and recognition hits.

Data Analysis

We analyzed predictor variables to explore whether assumptions of normality of distribution were met. Histograms and normal P–P plots indicated normally distributed residuals for all variables, with the exception of recognition, which was positively skewed and failed to yield a normal distribution. Therefore, parametric analyses were indicated.

Results

Statistical analyses were performed using IBM PASW Statistics v18.0 (SPSS Inc., 2009). Pearson r correlation coefficients of variables on the standard form (Form A) (n = 354) showed all items to be significantly correlated with each other (Table 3). Internal consistency (based on the full sample for each test form) was adequate for each of the three forms of the test (Form A: Cronbach's alpha = 0.84; Form B: Cronbach's alpha = 0.81; Form C: Cronbach's alpha = 0.90).

Table 3.

Correlations among GVLT variables

Variable Trial 1 Trial 5 Total words Learning slope List B Short delay free recall Short delay cued recall Long delay free recall Long delay cued recall Recognition 
Trial 1 — .64* .83* .68* .67* .63* .57* .64* .63* .29* 
Trial 5  — .90* .91* .64* .82* .79* .85* .81* .42* 
Total words   — .97* .71* .85* .79* .85* .82* .42* 
Learning slope    — .66* .86* .81* .86* .83* .44* 
List B     — .58* .54* .53* .55* .28* 
Short delay free recall      — .85* .89* .86* .44* 
Short delay cued recall       — .86* .89* .51* 
Long delay free recall        — .89* .45* 
Long delay cued recall         — .54* 
Recognition          — 
Variable Trial 1 Trial 5 Total words Learning slope List B Short delay free recall Short delay cued recall Long delay free recall Long delay cued recall Recognition 
Trial 1 — .64* .83* .68* .67* .63* .57* .64* .63* .29* 
Trial 5  — .90* .91* .64* .82* .79* .85* .81* .42* 
Total words   — .97* .71* .85* .79* .85* .82* .42* 
Learning slope    — .66* .86* .81* .86* .83* .44* 
List B     — .58* .54* .53* .55* .28* 
Short delay free recall      — .85* .89* .86* .44* 
Short delay cued recall       — .86* .89* .51* 
Long delay free recall        — .89* .45* 
Long delay cued recall         — .54* 
Recognition          — 

*p < .01.

Inter-Form Equivalence

In order to assess the inter-form equivalence of our test forms, we conducted a one-way repeated-measures ANOVA comparing the performance of only those participants (n = 45) who had completed all three forms of the test (standard form A, and alternative forms B and C). Mauchly's test indicated that the assumption of sphericity had not been violated in any of these analyses. The results showed that performance did not differ between the three forms on learning, long-delay recall or recognition [total number of words recalled over the five learning trials: F(2,88) = 1.06, p > .35; long-delay free recall: F(2,88) = 0.08, p > .926; long-delay cued recall: F(2,88) = 2.46, p > .091; recognition: F(2,84) = 0.42, p > .657]. We found a form difference in participants' performance on short-delay recall [short-delay free recall: F(2,88) = 4.56, p < .013; short-delay cued recall: F(2,86) = 5, p < .009]; a Bonferroni post hoc analysis indicated that performance on short-delay free and cued recall was significantly greater on Form C, in comparison with Form A.

Equivalence between tests was also assessed by a Pearson r correlation analysis (including only those participants who had completed all three test forms); variables of interest were found to be significantly correlated across the three forms, with the exception of the recognition item of Form C, which did not correlate with the recognition items of the other two forms. The mean performance of healthy participants on variables of interest, as well as correlations of each variable across each form (A, B, and C) are presented in Table 4.

Table 4.

Mean (standard deviation) performance on each test form (A, B, and C) of healthy participants and correlations of variables across each form of the GVLT

Variable Form A
 
Form B
 
Form A and B
 
Form C
 
Form A and C
 
Form B and C
 
M (SD) M (SD) r M (SD) r R 
Trial 1 9.82 (0.42) 9.62 (0.41) .65* 9.56 (0.47) .42* .61* 
Trial 5 14.4 (0.37) 14.93 (0.27) .76* 15.02 (0.29) .51* .51* 
Total words 65.13 (1.66) 66.71 (1.36) .79* 66.16 (1.50) .79* .75* 
List B 7.78 (0.38) 9.33 (0.42) .63* 8.24 (0.41) .59* .69* 
Short delay free recall 13.67 (0.44) 14.4 (0.32) .82* 14.51 (0.32) .68 .66* 
Short delay cued recall 14.36 (0.29) 14.98 (0.21) .75* 15.00 (0.23) .68* .51* 
Long delay free recall 14.56 (0.27) 14.64 (0.25) .84* 14.60 (0.33) .71* .66* 
Long delay cued recall 14.69 (0.25) 15.27 (0.17) .75* 14.82 (0.31) .59* .54* 
Recognition 15.54 (0.09) 15.65 (0.11) .75* 15.63 (0.12) .20 .16 
Variable Form A
 
Form B
 
Form A and B
 
Form C
 
Form A and C
 
Form B and C
 
M (SD) M (SD) r M (SD) r R 
Trial 1 9.82 (0.42) 9.62 (0.41) .65* 9.56 (0.47) .42* .61* 
Trial 5 14.4 (0.37) 14.93 (0.27) .76* 15.02 (0.29) .51* .51* 
Total words 65.13 (1.66) 66.71 (1.36) .79* 66.16 (1.50) .79* .75* 
List B 7.78 (0.38) 9.33 (0.42) .63* 8.24 (0.41) .59* .69* 
Short delay free recall 13.67 (0.44) 14.4 (0.32) .82* 14.51 (0.32) .68 .66* 
Short delay cued recall 14.36 (0.29) 14.98 (0.21) .75* 15.00 (0.23) .68* .51* 
Long delay free recall 14.56 (0.27) 14.64 (0.25) .84* 14.60 (0.33) .71* .66* 
Long delay cued recall 14.69 (0.25) 15.27 (0.17) .75* 14.82 (0.31) .59* .54* 
Recognition 15.54 (0.09) 15.65 (0.11) .75* 15.63 (0.12) .20 .16 

*p < .01.

Effect of Demographic Variables on Test Performance

Stepwise multiple regression analyses (MRAs) for Form A (including the full sample of healthy adults) were performed in order to explore the effect of demographic variables on the dependent variables. Age, education, and sex were entered as the independent variables. The following trials and composites comprised the dependent variables: number of words recalled in the first learning trial (trial 1), number of words recalled in the final learning trial (trial 5), total number of words recalled over the five learning trials, learning slope, number of words recalled on interference trial (List B), number of words recalled in the short-delay free and cued recall trials, and in the long-delay free and cued recall trials. As described in the Data Analysis section, histograms and normal P–P plots indicated normally distributed residuals for all variables, with the exception of recognition, which was positively skewed. A plot of the regression standardized residual against the standardized predicted value produced a random array of dots evenly dispersed around zero, showing that the assumption of homoscedasticity was met. Finally, the predictor variables did not show multicollinearity (all VIFs < 2), and the Durbin–Watson test statistic indicated that the assumption of independent errors was met (Field, 2009) (trial 1 = 1.65; trial 5 = 1.46; total number of words = 1.38; List B = 1.73; short delay free Recall = 1.58; Short Delay Cued Recall: 1.62; Long Delay Free Recall = 1.62; Long Delay Cued Recall: 1.75; Recognition = 1.95). Recognition was not included in the analysis as it failed to yield a normal distribution and was therefore considered inappropriate for MRA. We found that age affected all dependent variables, wherein performance decreased with age, with the exception of learning slope; education affected learning (trial 5, total number of words recalled over the five learning trials, learning slope, list B), and long-delay recall, wherein performance increased with more years of education. Sex affected all variables, with women performing better than men; an exception to this was the interference trial (List B) and short-delay cued recall, on which gender did not have an effect. The results of these analyses, as well as means and standard deviations of the full sample's performance on each item, are presented in Table 5.

Table 5.

Form A: Results of step-wise multiple regression analyses for demographic factors (i.e., age, education, sex) predicting test items on the GVLT

Variable Predictor β Standard Error β t Standardized β Total R2 p 
Trial 1 (M = 7.27, SD = 3.07) Constant 9.50 0.89 10.66    
Age −0.08 0.01 −9.10 −0.51   
Education 0.08 0.04 2.19 0.12   
Sex 0.55 0.27 2.02 0.09 .35 .044 
  
Trial 5 (M = 12.73, SD = 3.39) Constant 12.98 0.96 13.50    
Age −0.08 0.01 −8.05 −0.44   
Education 0.16 0.04 4.03 0.22   
Sex 1.14 0.30 3.86 0.17 .37 <.001 
  
Total words (M = 53.02, SD = 15.55) Constant 58.36 4.07 14.35    
Age −0.41 0.04 −10.48 −0.53   
Education 0.70 0.17 4.20 0.21   
Sex 5.00 1.25 4.02 0.16 .47 <.001 
  
Learning slope (M = 1.33, SD = 0.67) Constant 0.89 0.15 5.84    
Education 0.02 0.01 2.01 0.11   
Sex 0.17 0.07 2.38 0.13 .03 .045 
  
List B (M = 6.37, SD = 2.55) Constant 6.94 0.81 8.58    
Age −0.05 0.01 −6.48 −0.39   
Education 0.124 0.03 3.67 0.22 .32 <.001 
  
Short delay free recall (M = 11.57, SD = 3.60) Constant 13.64 1.03 13.25    
Age −0.09 0.01 −9.39 −0.52   
Education 1.09 0.32 3.44 0.15   
Sex 0.09 0.04 2.12 0.12 .37 .034 
  
Short delay cued recall (M = 13.02, SD = 2.8) Constant 16.60 0.36 45.61    
Age −0.07 0.01 −10.55 −0.50 .25 <.001 
  
Long delay free recall (M = 12.17, SD = 3.75) Constant 13.17 1.07 12.32    
Age −0.09 0.01 −9.10 −0.49   
Education 0.19 0.04 4.21 0.23   
Sex 0.91 0.32 2.80 0.12 .45 .005 
  
Long delay cued recall (M = 13.20, SD = 3.00) Constant 14.55 0.93 15.70    
Age −0.07 0.01 −7.79 −0.46   
Education 0.10 0.04 2.63 0.16   
Sex 0.57 0.20 2.08 0.10 .34 .045 
Variable Predictor β Standard Error β t Standardized β Total R2 p 
Trial 1 (M = 7.27, SD = 3.07) Constant 9.50 0.89 10.66    
Age −0.08 0.01 −9.10 −0.51   
Education 0.08 0.04 2.19 0.12   
Sex 0.55 0.27 2.02 0.09 .35 .044 
  
Trial 5 (M = 12.73, SD = 3.39) Constant 12.98 0.96 13.50    
Age −0.08 0.01 −8.05 −0.44   
Education 0.16 0.04 4.03 0.22   
Sex 1.14 0.30 3.86 0.17 .37 <.001 
  
Total words (M = 53.02, SD = 15.55) Constant 58.36 4.07 14.35    
Age −0.41 0.04 −10.48 −0.53   
Education 0.70 0.17 4.20 0.21   
Sex 5.00 1.25 4.02 0.16 .47 <.001 
  
Learning slope (M = 1.33, SD = 0.67) Constant 0.89 0.15 5.84    
Education 0.02 0.01 2.01 0.11   
Sex 0.17 0.07 2.38 0.13 .03 .045 
  
List B (M = 6.37, SD = 2.55) Constant 6.94 0.81 8.58    
Age −0.05 0.01 −6.48 −0.39   
Education 0.124 0.03 3.67 0.22 .32 <.001 
  
Short delay free recall (M = 11.57, SD = 3.60) Constant 13.64 1.03 13.25    
Age −0.09 0.01 −9.39 −0.52   
Education 1.09 0.32 3.44 0.15   
Sex 0.09 0.04 2.12 0.12 .37 .034 
  
Short delay cued recall (M = 13.02, SD = 2.8) Constant 16.60 0.36 45.61    
Age −0.07 0.01 −10.55 −0.50 .25 <.001 
  
Long delay free recall (M = 12.17, SD = 3.75) Constant 13.17 1.07 12.32    
Age −0.09 0.01 −9.10 −0.49   
Education 0.19 0.04 4.21 0.23   
Sex 0.91 0.32 2.80 0.12 .45 .005 
  
Long delay cued recall (M = 13.20, SD = 3.00) Constant 14.55 0.93 15.70    
Age −0.07 0.01 −7.79 −0.46   
Education 0.10 0.04 2.63 0.16   
Sex 0.57 0.20 2.08 0.10 .34 .045 

Regression-Based Norms

Given recent studies demonstrating the increased clinical utility of RBNs relative to traditional normative data (Chelune, 2010; Parmenter, Testa, Schretlen, Weinstock-Guttman, & Benedict, 2010; Smerbeck et al., 2011; Testa, Winicki, Pearlson, Gordon, & Schretlen, 2009), we estimated RBNs for Form A (using the full healthy adult sample) according to Testa and colleagues (2009). We initially selected variables of clinical interest (i.e., total number of words recalled over the five learning trials, number of words recalled in the short-delay free and cued recall trials, and in the long-delay free and cued recall trials) and converted raw scores to scaled scores (M = 10, SD = 3) using the cumulative frequency distribution for each variable (Table 6). This generated a scaled score value that could then be used for comparison with the predicted scaled score value. Subsequently, whereas raw scores were used for all previous analyses, we conducted multiple regression analyses using the scaled score values and entering age, age-squared, education, and sex as predictors, en bloc. Age squared was added as a predictor variable in order to also evaluate nonlinear effects of this variable on performance (Parmenter et al., 2010; Smerbeck et al., 2011).The following formula was used for calculating the predicted scaled score of participants on each of these variables (adopted from Smerbeck et al., 2011): 

formula

Table 6.

Raw to scaled score conversion

Scaled score Total Short delay free recall Short delay cued recall Long delay free recall Long delay cued recall 
>19 79     
18      
17 78     
16 77     
15 75–76     
14 71–74  16   
13 67–70 16  16 16 
12 62–66 15 15 15  
11 58–61 14  14 15 
10 53–57 13 14 13 14 
48–52 11 12–13 11–12 12–13 
40–47 10 11 9–10 11 
34–39 10 10 
28–33 6–7 
23–27 7–8 4–5 7–8 
15–22 5–6 2–3 4–6 
9–14 3–4 
  
  
Scaled score Total Short delay free recall Short delay cued recall Long delay free recall Long delay cued recall 
>19 79     
18      
17 78     
16 77     
15 75–76     
14 71–74  16   
13 67–70 16  16 16 
12 62–66 15 15 15  
11 58–61 14  14 15 
10 53–57 13 14 13 14 
48–52 11 12–13 11–12 12–13 
40–47 10 11 9–10 11 
34–39 10 10 
28–33 6–7 
23–27 7–8 4–5 7–8 
15–22 5–6 2–3 4–6 
9–14 3–4 
  
  

Finally, the predicted scaled score was subtracted from the actual scaled score and divided by the standard error of the estimate. The following formula was used in order to derive a z-score for each of an individual's scores on these test trials: 

formula

The final regression models for calculation of RBNs of selected GVLT variables are presented in Table 7.

Table 7.

Final regression models for RBNs on variables of the GVLT

Variable Predictor β Standard error β t Standardized β Total R2 SD of residuals p 
Total words Constant 6.371 0.947 6.727     
Age 0.023 0.038 0.604 0.148    
Age2 −0.001 0.000 −1.868 −0.461    
Education 0.198 0.030 6.597 0.296    
Sex 1.179 0.232 5.092 0.194 .311 2.534 <.000 
Short delay free recall Constant 8.593 1.367 6.285     
Age −0.089 0.055 −1.597 −0.413    
Age2 0.000 0.001 0.600 0.157    
Education 0.239 0.043 5.511 0.263    
Sex 1.584 0.334 4.737 0.191 .226 3.655 <.000 
Short delay cued recall Constant 7.099 1.019 6.969     
Age −0.010 0.041 −0.247 −0.067    
Age2 0.000 0.000 −0.410 −0.111    
Education 0.170 0.032 5.253 0.261    
Sex 0.933 0.248 3.759 0.158 .166 2.710 <.000 
Long delay free recall Constant 6.206 0.946 6.560     
Age 0.003 0.038 0.091 0.023    
Age2 0.000 0.000 −0.973 −0.253    
Education 0.218 0.030 7.235 0.344    
Sex 0.984 0.230 4.271 0.172 .283 2.421 <.000 
Long delay cued recall Constant 5.746 0.985 5.835     
Age 0.026 0.040 0.649 0.176    
Age2 −0.001 0.000 −1.233 −0.338    
Education 0.190 0.031 6.079 0.306    
Sex 1.023 0.239 4.273 0.181 .209 2.505 <.000 
Variable Predictor β Standard error β t Standardized β Total R2 SD of residuals p 
Total words Constant 6.371 0.947 6.727     
Age 0.023 0.038 0.604 0.148    
Age2 −0.001 0.000 −1.868 −0.461    
Education 0.198 0.030 6.597 0.296    
Sex 1.179 0.232 5.092 0.194 .311 2.534 <.000 
Short delay free recall Constant 8.593 1.367 6.285     
Age −0.089 0.055 −1.597 −0.413    
Age2 0.000 0.001 0.600 0.157    
Education 0.239 0.043 5.511 0.263    
Sex 1.584 0.334 4.737 0.191 .226 3.655 <.000 
Short delay cued recall Constant 7.099 1.019 6.969     
Age −0.010 0.041 −0.247 −0.067    
Age2 0.000 0.000 −0.410 −0.111    
Education 0.170 0.032 5.253 0.261    
Sex 0.933 0.248 3.759 0.158 .166 2.710 <.000 
Long delay free recall Constant 6.206 0.946 6.560     
Age 0.003 0.038 0.091 0.023    
Age2 0.000 0.000 −0.973 −0.253    
Education 0.218 0.030 7.235 0.344    
Sex 0.984 0.230 4.271 0.172 .283 2.421 <.000 
Long delay cued recall Constant 5.746 0.985 5.835     
Age 0.026 0.040 0.649 0.176    
Age2 −0.001 0.000 −1.233 −0.338    
Education 0.190 0.031 6.079 0.306    
Sex 1.023 0.239 4.273 0.181 .209 2.505 <.000 

Diagnostic Validity

We explored the diagnostic validity of the test through a receiver operator characteristic (ROC) curve in order to calculate sensitivity and specificity. This analysis was run separately for Form A, for each of the two patient populations, i.e., individuals with schizophrenia and individuals with TBI, with a subsample of the healthy control group matched with each patient group based on age and education level, and for three of the test variables, i.e., total number of words recalled over the five learning trials, short-delay free recall and long-delay free recall. In order for the results to be of greater clinical value, we ran these analyses using normative scores (z-scores). The area under the ROC curve was approximately 0.9 for individuals with schizophrenia on the three variables (i.e., total number of words = 0.905; short-delay free recall = 0.857; long-delay free recall = 0.867) suggesting good specificity for a large range of sensitivities and good diagnostic utility. For individuals with TBI, the area under the ROC curve was approximately 0.7 on the three variables (total number of words = 0.69; short-delay free recall = 0.691; long-delay free recall = 0.661), supporting the fair diagnostic utility of the test. Table 8 lists group means (SD), cutoff scores, sensitivities, specificities, and overall hit rate for scores derived from RBNs for each of the three variables of the test for individuals with schizophrenia and for individuals with TBI separately.

Table 8.

Cutoff scores, sensitivity, specificity and overall hit rate for scores derived from regression-based norms for three variables of the GVLT for patients with schizophrenia and patients with TBI

  Schizophrenia
 
TBI
 
Total words Short delay free recall Long delay free recall Total words Short delay free recall Long delay free recall 
Mean (SD) 35.4 (13.4) 7.7 (3.7) 7.8 (4.1) 51.6 (12.1) 10.6 (3.8) 11.0 (4.3) 
Cutoff z-score −0.600 −0.481 −0.774 0.022 −0.139 −0.304 
Sensitivity 0.90 0.84 0.90 0.70 0.70 0.70 
Specificity 0.74 0.74 0.70 0.60 0.60 0.60 
Overall hit rate 74% 72% 69% 59% 59% 57% 
  Schizophrenia
 
TBI
 
Total words Short delay free recall Long delay free recall Total words Short delay free recall Long delay free recall 
Mean (SD) 35.4 (13.4) 7.7 (3.7) 7.8 (4.1) 51.6 (12.1) 10.6 (3.8) 11.0 (4.3) 
Cutoff z-score −0.600 −0.481 −0.774 0.022 −0.139 −0.304 
Sensitivity 0.90 0.84 0.90 0.70 0.70 0.70 
Specificity 0.74 0.74 0.70 0.60 0.60 0.60 
Overall hit rate 74% 72% 69% 59% 59% 57% 

Discussion

In the current study, we present a model for the development of a word list-learning test, based on the CVLT, which is relevant to a non-English-speaking population and culture, namely, Greek. All three forms (the standard form and two alternate forms for re-testing purposes) of the newly developed GVLT were found to have satisfactory psychometric properties. Furthermore, with respect to the diagnostic utility of the test, we found fair to good sensitivity and specificity in its ability to differentiate individuals with a psychiatric (i.e., schizophrenia) or neurological (i.e., TBI) condition from healthy adults. This evidence supports the utility of the GVLT in clinical practice, both as a tool for assessing deficits in verbal learning and memory and verbal learning strategies, as well as a bedside screening tool to determine the need for further neuropsychological follow-up. We estimated and present norms for our sample in the form of RBNs for clinical and research use.

Our initial research hypothesis that demographic factors, such as age, education, and sex, would be related to performance was confirmed. Consistently with previous studies on English versions of verbal learning tasks (Kramer et al., 2003; Weible, Nuest, Welty, Pate, & Turner, 2002), we found that age affected performance, as younger participants demonstrated better performance on all variables. An exception to this was the learning slope, which was unaffected by age. This finding might reflect the fact that as age increases, the number of words encoded and learned decrease, without, however, affecting the rate of learning. Individuals with a higher level of education learned more words over the course of learning trials, and more years of education resulted in better performance on long-delay free and cued recall, but were not related to recall after a short delay when assistance with cues was provided. This might reflect improved cognitive organization of the new information in those with an extensive education, thus increasing capacity for verbal learning and permitting a more durable storage of the information. Women outperformed men on nearly all trials of the test, a finding which is consistent with previous studies (Wiens, Tindall, & Crossen, 1994).

Although developed and normed in northern Greece, we have no reason to believe that the test might not be appropriate for use in other parts of the country with native speakers of Greek. In fact, the vast majority of the population speaks standard modern Greek, which is taught in the school system and is used in all forms of media (despite the potential use of any regional dialects in remote areas or among elderly individuals). Caution must be exercised, however, in using the current stimuli and normative data with populations from neighboring Greek-speaking countries (i.e., Cyprus), immigrants to Greece, or Greeks who have been raised outside the country or in non-Greek-speaking families. Clinicians should rely on their expertise and clinical judgment in order to assess each individual's background, level of familiarity with the language and culture, and, thus, the applicability of the current test stimuli and the appropriateness of the present normative data.

Our study also presents some limitations, which should be taken into consideration in future research endeavors for the development of culturally appropriate tests. Information on the participants' pre-morbid IQ was not collected, although one could argue that education could be considered as a predictor. The GVLT was not administered as part of a standardized battery. Therefore, we do not have data from other measures in order to provide information on convergent and/or divergent validity. Participants were not tested in the same setting, and this might have affected standardization. We established that healthy participants had an unremarkable neurological and psychiatric history based on their self-report through a semi-structured screening interview and did not use a standardized clinical interview. Furthermore, the cultural relevance of the words and categories selected during the test development phase was based on face validity. A final limitation is that although comparison among the three forms of the test showed equivalent performance in learning, long-delay recall and recognition, alternate forms were administered in a standard (A–B–C) and not randomized order, which may have resulted in a practice effect on one variable [i.e., participants' performance on short-delay recall was better on the form that was administered last (i.e., Form C), in comparison with the first (i.e., Form A)]. As practice effects after administrations of different forms of a test are inevitable, however, and as initial learning, and long-term recall and recognition were unaffected, we consider Form C to be equivalent to the other two forms and clinically useful. Overall, we do not consider that these limitations negatively impact the interpretability and clinical utility of the GVLT.

In conclusion, research in cross-cultural neuropsychology has emphasized the need for developing strategies for accurately assessing cognitive skills within various cultures. Direct translation of the material from the English language can compromise the validity and reliability of the test. Alternatively, current directions are to develop new stimuli that are culturally relevant to the language and culture (Nell, 2000), or even completely new tests (Kosmidis, 2008). As a direction for future research, provided the need to extend the practice of neuropsychology to non-English-speaking countries, we suggest that countries with language and cultural commonalities (e.g., Greece and Cyprus, Scandinavian countries, or Arabic countries) join efforts in developing tests the different forms of which can be pilot-tested, adapted, and normed within each of their populations in order to be culturally appropriate. Furthermore, tests with alternative, culturally adapted forms that can be shared among countries can be developed in multi-center investigations, in order to facilitate the assessment of populations that have relocated and thus have multiple native languages and cultural influences, a need that is growing with the increase in globalized population mobility.

Funding

This work was supported by a Marie Curie International Reintegration Grant (FP7-PEOPLE-2007-4-3-IRG) awarded by the European Commission to the first author with the scientific coordination of the second.

Conflict of interest

None declared.

Acknowledgements

The authors would like to thank Dr. Eleni Aretouli for her statistical assistance, and Ioanna Antoniou, Nantia Chalkidou, Aikaterini Chatzikallia, Dimitrios Chionis, Lelia Choremi, Vassiliki Folia, Alexandros Giannakidis, Konstantinos Giannakis, Domniki Karagianni, Anastasia Kolomvatsou, Vasiliki Bochtsiou, Antonios Papageorgiou, Konstantinos Patsakis, Chrysoula Pornari, Chrysa Skaltsogianni, Aris-Dennis Tsaous, and Denis Vevzioglou for their contribution to the data collection.

Appendix A

Word lists for standard form A of the GVLT.

Monday list Tuesday list Recognition (foils) 
Poύχα (clothing) Poύχα (clothing) Same categories as Monday/different words New words/no category 
Παντɛλόνι (Pants) Mπλούζα (Blouse) Πουκάμισο (Shirt) Xαρτομάντηλα (Tissues) 
Zακέτα (Jacket) Κάλτσɛς (Socks) Poδάκινo (Peach) Λάδι (Oil) 
Φούστα (Skirt) Φόρɛμα (Dress) Δυόσμος (Mint) Mπαταρίɛς (Batteries) 
Παλτό (Coat) Σακάκι (Suit jacket) Πριόνι (Saw) Σφουγγάρι (Sponge) 
Φρούτα (fruit) Φρούτα (fruit) New words/words rhyming with Monday words 
Mήλο (Apple) Πɛπόνι (Mellon) Pακέτα (Racket) Πιατέλα (Serving plate) 
Κɛράσι (Cherry) Πορτοκάλι (Orange) Φύλλο (Filo dough) Σχοινί (Rope) 
Αχλάδι (Pear) Mπανάνα (Banana) Φαράσι (Dust pan) Iνδοκάρυδο (Coconut) 
Kαρπούζι (Watermelon) Δαμάσκηνο (Plum) Mπαλόνι (Balloon) Παντɛσπάνι (Sponge cake) 
Eργαλɛία (tools) Λαχανικά (vegetables) Similar (but different) category as Tuesday words  
Σφυρί (Hammer) Kαρότο (Carrot) Kαλαμπόκι (Corn)  
Kατσαβίδι (Screwdriver) Aγγούρι (Cucumber) Mακαρόνια (Spaghetti)  
Tρυπάνι (Drill) Σπανάκι (Spinach) Φασόλια (Beans)  
Φτυάρι (Shovel) Mαρούλι (Lettuce) Pύζι (Rice)  
Mπαχαρικά (spices) Ψάρια (fish)   
Pίγανη (Oregano) Πέστροφα (Trout)   
Πιπέρι (Pepper) Σαρδέλα (Sardine)   
Kανέλα (Cinnamon) Mπαρμπούνι (Red Mullet)   
Mοσχοκάρυδο (Nutmeg) Λαβράκι (Sea bass)   
Monday list Tuesday list Recognition (foils) 
Poύχα (clothing) Poύχα (clothing) Same categories as Monday/different words New words/no category 
Παντɛλόνι (Pants) Mπλούζα (Blouse) Πουκάμισο (Shirt) Xαρτομάντηλα (Tissues) 
Zακέτα (Jacket) Κάλτσɛς (Socks) Poδάκινo (Peach) Λάδι (Oil) 
Φούστα (Skirt) Φόρɛμα (Dress) Δυόσμος (Mint) Mπαταρίɛς (Batteries) 
Παλτό (Coat) Σακάκι (Suit jacket) Πριόνι (Saw) Σφουγγάρι (Sponge) 
Φρούτα (fruit) Φρούτα (fruit) New words/words rhyming with Monday words 
Mήλο (Apple) Πɛπόνι (Mellon) Pακέτα (Racket) Πιατέλα (Serving plate) 
Κɛράσι (Cherry) Πορτοκάλι (Orange) Φύλλο (Filo dough) Σχοινί (Rope) 
Αχλάδι (Pear) Mπανάνα (Banana) Φαράσι (Dust pan) Iνδοκάρυδο (Coconut) 
Kαρπούζι (Watermelon) Δαμάσκηνο (Plum) Mπαλόνι (Balloon) Παντɛσπάνι (Sponge cake) 
Eργαλɛία (tools) Λαχανικά (vegetables) Similar (but different) category as Tuesday words  
Σφυρί (Hammer) Kαρότο (Carrot) Kαλαμπόκι (Corn)  
Kατσαβίδι (Screwdriver) Aγγούρι (Cucumber) Mακαρόνια (Spaghetti)  
Tρυπάνι (Drill) Σπανάκι (Spinach) Φασόλια (Beans)  
Φτυάρι (Shovel) Mαρούλι (Lettuce) Pύζι (Rice)  
Mπαχαρικά (spices) Ψάρια (fish)   
Pίγανη (Oregano) Πέστροφα (Trout)   
Πιπέρι (Pepper) Σαρδέλα (Sardine)   
Kανέλα (Cinnamon) Mπαρμπούνι (Red Mullet)   
Mοσχοκάρυδο (Nutmeg) Λαβράκι (Sea bass)   

Appendix B

Word lists for alternative form B of the GVLT

Monday list Tuesday list Recognition (foils) 
ϒποδήματα (footwear) ϒποδήματα (footwear) Same categories as Monday/different words New words/no category 
Παπούτσι (Shoe) Γόβɛς (Pumps) Σαγιονάρɛς (Flip-flops) Pολόι (Clock) 
Tσόκαρο (Clogs) Πέδιλο (Sandal)a Πανσέτα (Pork side) Κλɛιδιά (Keys) 
Άρβυλα (Military boots) Παντόφλɛς (Slippers) Bιβλίο (Book) Πορτοφόλι (Wallet) 
Σανδάλια (Leather sandal with straps) Mπότɛς (Boots) Aνθότυρο (Anthotyro)b Σκούπα (Broom) 
Kρɛατικά (meats) Kρɛατικά (meats) New words/words rhyming with Monday words 
Mπριζόλα (Steak) Λουκάνικο (Sausage) Φυλλάδιο (Pamphlet) Σβήστρα (Eraser) 
Kοτόπουλο (Chicken) Γαλοπούλα (Turkey) Kɛρήθρα (Bee wax) Γαριδάκια (Cheese puffs) 
Παϊδάκια (Ribs) Ψαρονέφρι (Tenderloin) Φρατζόλα (Loaf) Kουκούτσι (Fruit seed) 
Kιμάς (ground beef) Συκωτάκι (Liver) Mαχαίρι (Knife) Bɛντάλια (Fan) 
Tυριά (cheeses) Θαλασσινά (seafood) Similar (but different) category as Tuesday words  
Kασέρι (Kasseri) Xταπόδι (Octopus) Tσαγιέρα (Teapot)  
Mυζήθρα (Myzithra) Σουπιές (Cuttlefish) Aνοιχτήρι (Can opener)  
Γραβιέρα (Graviera) Kαλαμάρια (Calamari) Kουτάλι (Spoon)  
Φέτα (Feta) Mύδια (Mussels) Πιρούνι (Fork)  
Σχολικά ɛίδη (school materials) Kουζινικά Σκɛύη (cookware)   
Tɛτράδιο (Notepad) Tηγάνι (Pan)   
Στυλό (Pen) Mπρίκι (Coffee pot)c   
Ξύστρα (Pencil Sharpener) Kατσαρόλα (Pot)   
Mολύβι (Pencil) Tσουκάλι (Ceramic pot)d   
Monday list Tuesday list Recognition (foils) 
ϒποδήματα (footwear) ϒποδήματα (footwear) Same categories as Monday/different words New words/no category 
Παπούτσι (Shoe) Γόβɛς (Pumps) Σαγιονάρɛς (Flip-flops) Pολόι (Clock) 
Tσόκαρο (Clogs) Πέδιλο (Sandal)a Πανσέτα (Pork side) Κλɛιδιά (Keys) 
Άρβυλα (Military boots) Παντόφλɛς (Slippers) Bιβλίο (Book) Πορτοφόλι (Wallet) 
Σανδάλια (Leather sandal with straps) Mπότɛς (Boots) Aνθότυρο (Anthotyro)b Σκούπα (Broom) 
Kρɛατικά (meats) Kρɛατικά (meats) New words/words rhyming with Monday words 
Mπριζόλα (Steak) Λουκάνικο (Sausage) Φυλλάδιο (Pamphlet) Σβήστρα (Eraser) 
Kοτόπουλο (Chicken) Γαλοπούλα (Turkey) Kɛρήθρα (Bee wax) Γαριδάκια (Cheese puffs) 
Παϊδάκια (Ribs) Ψαρονέφρι (Tenderloin) Φρατζόλα (Loaf) Kουκούτσι (Fruit seed) 
Kιμάς (ground beef) Συκωτάκι (Liver) Mαχαίρι (Knife) Bɛντάλια (Fan) 
Tυριά (cheeses) Θαλασσινά (seafood) Similar (but different) category as Tuesday words  
Kασέρι (Kasseri) Xταπόδι (Octopus) Tσαγιέρα (Teapot)  
Mυζήθρα (Myzithra) Σουπιές (Cuttlefish) Aνοιχτήρι (Can opener)  
Γραβιέρα (Graviera) Kαλαμάρια (Calamari) Kουτάλι (Spoon)  
Φέτα (Feta) Mύδια (Mussels) Πιρούνι (Fork)  
Σχολικά ɛίδη (school materials) Kουζινικά Σκɛύη (cookware)   
Tɛτράδιο (Notepad) Tηγάνι (Pan)   
Στυλό (Pen) Mπρίκι (Coffee pot)c   
Ξύστρα (Pencil Sharpener) Kατσαρόλα (Pot)   
Mολύβι (Pencil) Tσουκάλι (Ceramic pot)d   

aThe words “Σανδάλι” and “Πέδιλο” in the Greek language are distinguishable describing two distinct types of light or summer shoe.

bType of cheese.

cA small cooking pot traditional of Middle Eastern countries, mainly used for the preparation of Greek coffee.

dA type of ceramic or clay pot typically used to cook food in the oven.

Appendix C

Word lists for alternative form C of the GVLT

Monday list Tuesday list Recognition (foils) 
Λουλούδια (flowers) Λουλούδια (flowers) Same categories as Monday/different words New words/no category 
Mαργαρίτα (Daisy) Γαρδένια (Gardenia) Mαξιλαροθήκη (Pillowcase) Mπλούζα (Blouse) 
Tριαντάφυλλο (Rose) Γαρύφαλλο (Carnation) Pɛτσίνα (Retsina) Tɛτράδιο (Notebook) 
Xρυσάνθɛμο (Chrysanthemum) Kρίνος (Lily) Nυχτολούλουδο (Evening primrose) Δίσκος (Tray) 
Tουλίπα (Tulip) Bιολέτα (Violet) Λάχανο (Cabbage) Λάμπα (Lamp) 
Λαχανικά (vegetables) Λαχανικά (Vegetables) New words/words rhyming with Monday words 
Mɛλιτζάνα (Eggplant) Aρακάς (Pea) Nταμιτζάνα (Carboy) Xάρακας (Ruler) 
Mπάμια (Ocra) Kολοκύθι (Zucchini) Φαλτσέτα (Straight razor) Πίτα (Pie) 
Φασολάκια (String beans) Kρɛμμύδι (Onion) Tαψί (Oven pan) Tουλούμπα (Touloumba)a 
Πιπɛριά (Pepper) Πατάτα (Potato) Kουβɛρτούρα (Dark chocolate) Όζα (Nail polish) 
Oινοπνɛυματώδη ποτά (spirits) Γλυκά (sweets) Similar (but different) category as Tuesday words  
Kρασί (Wine) Mπακλαβάς (Baklava) Kραγιόν (Lipstick)  
Oύζο (Ouzo) Kουραμπιές (Kourampies) Bαμβάκι (Cotton)  
Mπύρα (Beer) Λουκούμι (Greek jelly sweet) Zάχαρη (Sugar)  
Λικέρ (Liqueur) Xαλβάς (Chalvas) Aλɛύρι (Flour)  
Eίδη Προικός (Homeware) Kαλλυντικά (Cosmetics Toiletries)   
Πɛτσέτα (Towel) Σαπούνι (Soap)   
Σɛντόνι (Bed sheets) Aφρόλουτρο (Bubble bath)   
Kουβέρτα (Blanket) Σαμπουάν (Shampoo)   
Mπουρνούζι (Bathrobe) Oδοντόκρɛμα (Toothpaste)   
Monday list Tuesday list Recognition (foils) 
Λουλούδια (flowers) Λουλούδια (flowers) Same categories as Monday/different words New words/no category 
Mαργαρίτα (Daisy) Γαρδένια (Gardenia) Mαξιλαροθήκη (Pillowcase) Mπλούζα (Blouse) 
Tριαντάφυλλο (Rose) Γαρύφαλλο (Carnation) Pɛτσίνα (Retsina) Tɛτράδιο (Notebook) 
Xρυσάνθɛμο (Chrysanthemum) Kρίνος (Lily) Nυχτολούλουδο (Evening primrose) Δίσκος (Tray) 
Tουλίπα (Tulip) Bιολέτα (Violet) Λάχανο (Cabbage) Λάμπα (Lamp) 
Λαχανικά (vegetables) Λαχανικά (Vegetables) New words/words rhyming with Monday words 
Mɛλιτζάνα (Eggplant) Aρακάς (Pea) Nταμιτζάνα (Carboy) Xάρακας (Ruler) 
Mπάμια (Ocra) Kολοκύθι (Zucchini) Φαλτσέτα (Straight razor) Πίτα (Pie) 
Φασολάκια (String beans) Kρɛμμύδι (Onion) Tαψί (Oven pan) Tουλούμπα (Touloumba)a 
Πιπɛριά (Pepper) Πατάτα (Potato) Kουβɛρτούρα (Dark chocolate) Όζα (Nail polish) 
Oινοπνɛυματώδη ποτά (spirits) Γλυκά (sweets) Similar (but different) category as Tuesday words  
Kρασί (Wine) Mπακλαβάς (Baklava) Kραγιόν (Lipstick)  
Oύζο (Ouzo) Kουραμπιές (Kourampies) Bαμβάκι (Cotton)  
Mπύρα (Beer) Λουκούμι (Greek jelly sweet) Zάχαρη (Sugar)  
Λικέρ (Liqueur) Xαλβάς (Chalvas) Aλɛύρι (Flour)  
Eίδη Προικός (Homeware) Kαλλυντικά (Cosmetics Toiletries)   
Πɛτσέτα (Towel) Σαπούνι (Soap)   
Σɛντόνι (Bed sheets) Aφρόλουτρο (Bubble bath)   
Kουβέρτα (Blanket) Σαμπουάν (Shampoo)   
Mπουρνούζι (Bathrobe) Oδοντόκρɛμα (Toothpaste)   

aTraditional syrupy Greek pastry.

References

Barker-Collo
S.
Clarkson
A.
Cribb
A.
Grogan
M.
The impact of American content on California Verbal Learning Test performance: A New Zealand illustration
Clinical Neuropsychologist
 , 
2002
, vol. 
16
 
3
(pg. 
290
-
299
)
Benedict
R. H. B.
Effects of using same- versus alternate-form memory tests during short-interval repeated assessments in multiple sclerosis
Journal of the International Neuropsychological Society
 , 
2005
, vol. 
11
 
6
(pg. 
727
-
736
)
Brandt
J.
Benedict
R.
Hopkins Verbal Learning Test—revised
 , 
2001
Lutz, FL
Psychological Assessment Resources, Inc
Brooks
B. L.
Weaver
L. E.
Scialfa
C. T.
Does impaired executive functioning differentially impact verbal memory measures in older adults with suspected dementia?
Clinical Neuropsychologist
 , 
2006
, vol. 
20
 
2
(pg. 
230
-
242
)
Chelune
G. J.
Evidence-based research and practice in clinical neuropsychology
The Clinical Neuropsychologist
 , 
2010
, vol. 
24
 (pg. 
454
-
467
)
Cherner
M.
Suarez
P.
Lazzaretto
D.
Artiola i Fortuny
L.
Rivera Mindt
M.
Dawes
S.
the HNRC Group
Demographically corrected norms for the Brief Visuospatial Memory Test – revised and Hopkins Verbal Learning Test – revised in monolingual Spanish speakers from the U.S. Mexico border region
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22
 (pg. 
343
-
353
)
Delis
D. C.
Kramer
J. H.
Kaplan
E.
Ober
B. A.
California Verbal Learning Test: Adult version
 , 
1987
San Antonio, TX
The Psychological Corporation
Delis
D. C.
Kramer
J. H.
Kaplan
E.
Ober
B. A.
California Verbal Learning Test—2nd Edition (CVLT-II): Manual
 , 
2000
Toronto
The Psychological Corporation
Field
A.
Discovering statistics using SPSS
 , 
2009
3rd ed.
London
Sage Publications Ltd
Folia
V.
Kosmidis
M. H.
Assessment of memory skills in illiterates: Strategy differences or test artifacts?
The Clinical Neuropsychologist
 , 
2003
, vol. 
17
 (pg. 
143
-
152
)
Fountoulakis
K. N.
Tsolaki
M.
Mohs
R. C.
Kazis
A.
Epidemiological dementia index: a screening instrument for Alzheimer's disease and other types of dementia suitable for use in populations with low education level
Dementia and Geriatric Cognitive Disorders
 , 
1998
, vol. 
9
 
6
(pg. 
329
-
338
)
Giaglis
G.
Giannakou
M.
Fitsioris
X.
Kosmidis
M. H.
Didaskalou
E.
Tsiptsios
I
Impaired cognitive estimation in non-demented patients with Parkinson's disease
2008
Paper presented at the 12th Congress of the European Federation of Neurological Societies
Madrid, Spain
 
August 23–26
Giaglis
G.
Kyriazidou
S.
Paraskevopoulou
E.
Tascos
N.
Kosmidis
M. H.
Evaluating premorbid level: Preliminary findings regarding the vulnerability of scores on cognitive measures in patients with MS
Journal of the International Neuropsychological Society
 , 
2010
, vol. 
16
 
Suppl. 1
pg. 
159
 
Hiou
K.
Vagia
A.
Haritidou
E.
Karakostas
D.
Kosmidis
M. H.
Affect perception as a cognitive function: Validity and clinical application of a neuropsychological test battery in healthy individuals and patients with brain damage
Psychology: The Journal of the Hellenic Psychological Society (in Greek)
 , 
2004
, vol. 
11
 
3
(pg. 
388
-
401
)
Kosmidis
M. H.
Clinical neuropsychological assessment [In Greek]
 , 
2008
Athens
Parisianou Press
Kosmidis
M. H.
Tsapkini
K.
Folia
V.
Lexical processing in illiteracy
Cortex
 , 
2006
, vol. 
42
 (pg. 
1021
-
1027
)
Kosmidis
M. H.
Tsapkini
K.
Folia
V.
Vlahou
C. H.
Kiosseoglou
G.
Semantic and phonological processing in illiteracy
Journal of the International Neuropsychological Society
 , 
2004a
, vol. 
10
 (pg. 
818
-
827
)
Kosmidis
M. H.
Vlahou
C. H.
Panagiotaki
P.
Kiosseoglou
G.
The Verbal Fluency Task in the Greek population: Normative data and clustering and switching strategies
Journal of the International Neuropsychological Society
 , 
2004b
, vol. 
10
 (pg. 
164
-
172
)
Kosmidis
M. H.
Zafiri
M.
Politimou
N.
Literacy vs. formal schooling: Influence on working memory
Archives of Clinical Neuropsychology
 , 
2011
, vol. 
26
 (pg. 
575
-
582
)
Kramer
J. H.
Yaffe
K.
Legenfelder
J.
Delis
D. C.
Age and gender interactions on verbal memory performance
Journal of the International Neuropsychological Society
 , 
2003
, vol. 
9
 
1
(pg. 
97
-
102
)
Lacritz
L. H.
Cullum
C. M.
Weiner
M. F.
Rosenberg
R. M.
Comparison of the Hopkins verbal learning tests-revised to the California verbal learning test in Alzheimer's Disease
Applied Neuropsychology
 , 
2001
, vol. 
8
 
3
(pg. 
180
-
184
)
Malloy-Diniz
L. F.
Lasmar
V. A. P.
Gazinelli
L. D. R.
Fuentes
D.
Salgado
J. V.
The Rey Auditory-Verbal Learning Test: Applicability for the Brazilian elderly population
Revista Brasileira de Psyquiatria
 , 
2007
, vol. 
29
 
4
(pg. 
324
-
329
)
McDowell
B. D.
Bayless
J. D.
Moser
D. J.
Meyers
J. E.
Paulsen
J. S.
Concordance between the CVLT and the WMS-III word lists tests
Archives of Clinical Neuropsychology
 , 
2004
, vol. 
19
 (pg. 
319
-
325
)
Messinis
L.
Tsakona
I.
Malefaki
S.
Papathanasopoulos
P.
Normative data and discriminant validity of Rey's Auditory Verbal Learning Test for the Greek adult population
Archives of Clinical Neuropsychology
 , 
2007
, vol. 
22
 (pg. 
739
-
752
)
Mueller
H.
Hasse-Sander
I.
Horn
R.
Helmsteadter
C.
Elger
C. E.
Rey-Auditory Verbal Learning Test: Structure of a modified German version
Journal of Clinical Psychology
 , 
1997
, vol. 
53
 (pg. 
663
-
671
)
Nell
V.
Cross-cultural neuropsychological assessment: theory and practice
 , 
2000
Mahwah, NJ
Lawrence Erlbaum Associates, Inc
Nolan
P.
Executive memory dysfunctions following mild traumatic brain injury
Journal of Head Trauma Rehabilitation
 , 
2006
, vol. 
21
 
1
(pg. 
68
-
75
)
Parmenter
B. A.
Testa
S. M.
Schretlen
D. J.
Weinstock-Guttman
B.
Benedict
R. H. B.
The utility of regression-based norms in interpreting the minimal assessment of cognitive function in multiple sclerosis (MACFIMS)
Journal of the International Neuropsychological Society
 , 
2010
, vol. 
16
 (pg. 
6
-
16
)
Puente
A. E.
Perez-Garcia
M.
Cuellar
I.
Paniagua
F. A.
Freddy
A.
Neuropsychological assessment of ethnic minorities: Clinical issues
Handbook of Multicultural Mental Health
 , 
2000
San Diego, CA
Academic Press
(pg. 
419
-
435
)
Rey
A.
L ‘examen clinique en psychologie [Clinical tests in psychology]
 , 
1964
Paris
Presses Universitaires de France
Ribeiro
F.
Guerreiro
M.
De Mendonca
A.
Verbal learning and memory deficits in Mild Cognitive Impairment
Journal of Clinical and Experimental Neuropsychology
 , 
2007
, vol. 
29
 
2
(pg. 
187
-
197
)
Rieu
D.
Bachoud-Levi
A. C.
Laurent
A.
Jurion
E.
Barba
G. D.
French adaptation of the Hopkins Verbal Learning Test
Revue Neurologique
 , 
2006
, vol. 
162
 
6–7
(pg. 
721
-
728
)
Rosselli
M.
Ardila
A.
Bateman
J. R.
Guzman
M.
Neuropsychological test scores, academic performance, and developmental disorders in Spanish-speaking children
Developmental Neuropsychology
 , 
2001
, vol. 
20
 
1
(pg. 
355
-
373
)
Smerbeck
A. M.
Parrish
J.
Yeh
E. A.
Hoogs
M.
Krupp
L. B.
Weinstock-Guttman
B.
, et al.  . 
Regression-based pediatric norms for the Brief Visuo-Spatial Memory Test-Revised and the Symbol Digit Modalities Test
The Clinical Neuropsychologist
 , 
2011
, vol. 
25
 
3
(pg. 
402
-
412
)
SPSS Inc
IBM PASW® STATISTICS 18 (SPSS Statistics 18)
 , 
2009
Chicago, IL
IMB SPSS, Inc
Testa
S. M.
Winicki
J. M.
Pearlson
G. D.
Gordon
B.
Schretlen
D. J.
Accounting for estimated IQ in neuropsychological test performance with regression-based techniques
Journal of the International Neuropsychological Society
 , 
2009
, vol. 
15
 (pg. 
1012
-
1022
)
Tremont
G.
Halpert
S.
Javorsky
D. J.
Stern
R. A.
Differential impact of executive dysfunction on verbal list learning and story recall
Clinical Neuropsychologist
 , 
2000
, vol. 
14
 
3
(pg. 
295
-
302
)
Uzzell
B.
Ponton
M.
Ardila
A.
International handbook of cross-cultural neuropsychology
 , 
2007
Mahwah, NJ
Lawrence Erlbaum Associates, Inc
Vlahou
C. H.
Chatzikallia
K.
Kosmidis
M. H.
Cognitive functioning and prospective memory in the elderly
2011
Paper presented at the 3rd Meeting of the Federation of European Societies of Neuropsychology
Basel, Switzerland
Vlahou
C. H.
Kosmidis
M. H.
Dardagani
A.
Tsotsi
S.
Development of a culturally relevant neuropsychological battery for the Greek population
2011
Paper presented at the 3rd Meeting of the Federation of European Societies of Neuropsychology
Basel, Switzerland
Weible
J. A.
Nuest
B. D.
Welty
J.
Pate
W. E.
Turner
M. L.
Demonstrating the effects of presentation rate on aging memory using the California Verbal Learning Test (CVLT)
Aging Neuropsychology and Cognition
 , 
2002
, vol. 
9
 
1
(pg. 
38
-
47
)
Wiens
A. N.
Tindall
A. g.
Crossen
J. R.
California Verbal Learning Test: A normative data study
The Clinical Neuropsychologist
 , 
1994
, vol. 
8
 (pg. 
75
-
90
)
Woods
S. P.
Rippeth
J. D.
Conover
E.
Gonvatana
A.
Gonzalez
R.
Carey
C. L.
the HRNC Group
Deficient strategic control of verbal encoding and retrieval in individuals with methamphetamine dependence
Neuropsychology
 , 
2005
, vol. 
19
 
1
(pg. 
35
-
43
)
Yassuda
M. S.
Diniz
B. S. O.
Flaks
M. K.
Viola
L. F.
Pereira
F. S.
Nunes
P. V.
, et al.  . 
Neuropsychological profile of Brazilian older adults with heterogeneous educational backgrounds
Archives of Clinical Neuropsychology
 , 
2009
, vol. 
24
 (pg. 
71
-
79
)

Author notes

The results were presented at the 39th Annual Meeting of the International Neuropsychological Association in Boston, MA, 2011.