Abstract

Memory deficits are a common feature of neurological and psychiatric disorders. Measures designed to evaluate memory in clinical populations have distinguished between memory for verbal and visual information; however, few tests assess the recall and recognition of emotional information, despite evidence suggesting that brain regions are differentially involved in memory for emotional and neutral stimuli and that affective disturbances are common in psychiatric and neurological disorders. The present study reports the test development and psychometric properties of the Emotional Verbal Learning Test (EVLT), a new neuropsychological measure that allows for the examination of emotional learning and memory. Psychometric analyses indicated that the EVLT has good internal consistency and test–retest reliability, as well as discriminant validity, clinical utility, and sensitivity to mood-congruency effects. This new measure has potential to be a valuable research and a clinical tool in the assessment of emotional memory and learning in healthy individuals and persons with neuropsychiatric disorders.

Introduction

The structure of memory has been a major focus of neuropsychological research. Elegant studies of amnestic and other patient groups have provided significant insights into the component processes underlying normal and abnormal memory function (e.g., Butters & Cermak, 1980; Butters, Delis, & Lucas, 1995) and contributed to memory assessment becoming a critical component of modern-day neuropsychological evaluations. Tests that assess learning and memory for verbal and visual information are now commonly used to assist in diagnosis and rehabilitation planning, to track the course of deterioration or recovery, and to monitor patient response to behavioral and pharmacological interventions. However, available tests do not always accommodate the needs of clinicians and researchers (Lezak, 1995). This is particularly true with regard to the evaluation of emotional memory. The lack of standardized tools designed to assess emotional memory deficits poses a problem for both researchers and professionals, as different neural substrates have been implicated in memory for emotional and neutral stimuli (Blake, Varnhagen, & Parent, 2001; Bradley, Greenwald, Petry, & Lang, 1992; Dolcos, LaBar, & Cabeza, 2004; Hamann, Cahill, & Squire, 1997; Hamann, Ely, Grafton, & Kilts, 1999; LaBar & Phelps, 1998; Lisman & Grace, 2005; Maddock & Frein, 2009; McGaugh & Cahill, 1997; Packard & Cahill, 2001; Wittmann, Schiltz, Boehler, & Duzel, 2008) and impairments in emotional memory may predict pathology and symptoms not associated with impairments in non-emotional memory (for a review, seeDere, Pause, & Pietrowsky, 2010).

In an effort to address this matter, Snyder and Harrison (1997) developed an Affective Auditory Verbal Learning Test (AAVLT), which consisted of two separate lists of 15 positively valenced and 15 negatively valenced words that they presented over five consecutive learning trials to a sample of undergraduate students. Examination of word acquisition across learning trials and primacy and recency effects indicated that the acquisition of these positive and negative word lists was similar to that reported for neutral word lists such as the Rey Auditory Verbal Learning Test (Rey, 1964). Although the approach used in the AAVLT demonstrates the feasibility of applying list-learning procedures to evaluate learning and memory for emotional information, it does not incorporate information from current cognitive and affective neuroscience research to comprehensively evaluate emotional memory functioning.

There are several notable limitations of the AAVLT. First, it uses separate word lists for positive and negative emotional conditions. The use of separate word lists does not allow for the evaluation of preferential encoding and retrieval of certain types of emotional information over others. The evaluation of preferential encoding and retrieval for specific types of emotional stimuli is possible only with the use of a single, mixed word list that has stimuli from multiple discrete affective categories. Given that healthy individuals are known to display better recall for pleasant than unpleasant or neutral stimuli (Amster, 1964; Bradley et al., 1992; Colombel, 2000; Hayward & Strongman, 1987; Libkuman, Stabler, & Otani, 2004; Lishman, 1972; Matlin, Stang, Gawron, Freedman, & Derby, 1979; Phelps, LaBar, & Spencer, 1997; Rychlak & Saluri, 1973) and that such preferential memory processes are predicted by important demographic factors such as age (Mather & Carstensen, 2005), it may be most beneficial to have a test that uses a mixed word list that is capable of assessing preferential memory for discrete emotions (e.g., happiness, sadness, anger). Furthermore, the AAVLT does not include stimuli from multiple discrete emotional conditions that have relevance to psychiatric and neurological disorders—its words are taken from pleasant and unpleasant categories that are broadly defined. This is a limitation given that differences in the recall and recognition of words from discrete categories (e.g., sadness vs. anxiety) have been shown to discriminate between clinical groups such as major depression and posttraumatic stress disorder (Bradley, Mogg, & Williams, 1995; Denny & Hunt, 1992; Dere et al., 2010; Watkins, 2002). Second, the AAVLT does not allow for the evaluation of mood-congruent memory (Bower, 1981, 1991). This is because the AAVLT does not include concurrent measures of state and trait emotional experience. Given the importance of mood-congruency in predicting individual differences in emotional information processing and psychopathology, this too poses a significant limitation of using the AAVLT in clinical populations (Bazin, Perruchet, & Feline, 1996; Blaney, 1986; Bradley et al., 1995; Coles & Heimberg, 2002; Forgas, 1994; Rusting, 1998; Strauss & Allen, 2006, 2009; Strauss, Allen, Jorgensen, & Cramer, 2005; Tarsia, Power, & Sanavio, 2003). Third, the AAVLT does not allow for the evaluation of associative semantic network theories (Bower, 1981, 1991) because semantic and serial clustering cannot be assessed, since words are presented on separate lists, and there are no cued recall sections. This is a limitation given that the ability to assess associative semantic networks in relation to individual emotions may be crucial for evaluating theories of normal and abnormal emotional memory (Doerksen & Shimamura, 2001; Siddiqui & Unsworth, 2011). Fourth, the AAVLT does not include a section for recognition memory, only recall across multiple trials. The ability to assess recall, recognition, and errors such as intrusions and repetitions has proven important for differentiating disorders with different profiles of memory impairment (e.g., cortical vs. subsortical dementia). The inclusion of separate recall and recognition testing sections, as well as procedures for calculating emotional intrusion and repetition scores, may be important for differentiating different forms of emotional memory impairment. Indeed, this has been shown to be the case in major depression (e.g., Howe & Malone, 2011; Jermann, Van der Linden, Laurencon, & Schmitt, 2009). Fifth, the AAVLT does not make use of an Interference List, such as those commonly employed on modern verbal learning tests (e.g., California Verbal Learning Test, 2nd Edition [CVLT-II]: Delis, Kramer, Kaplan, & Ober, 2000). This is a limitation given that the effects of emotional stimuli on proactive and retroactive interference have been shown to predict mood and anxiety symptoms (Ferraro & King, 2004). Finally, the AAVLT positive and negative word lists are not equated on a number of important stimulus domains known to influence emotional memory, such as emotional intensity, word length, and frequency of use in the English language. This poses a significant problem given that differences in stimulus characteristics between word conditions artificially bias memory comparisons across affective categories. Thus, there is need for a new neuropsychological test that addresses these important limitations, while taking into account recent developments and theories in the field of affective neuroscience.

In the current manuscript, we present the development of a new neuropsychological measure of emotional learning and memory, the Emotional Verbal Learning Test (EVLT). The EVLT was designed to take into account recent developments in the field of emotion research, such as those previously reviewed, and allows for an assessment of various learning and memory processes in relation to emotional stimuli from four specific emotion conditions (happiness, sadness, anger, and anxiety). These emotional conditions were selected because they are commonly disrupted in neuropsychiatric disorders, and when coupled with the EVLT's measures of state and trait emotional experience, the various test scores may be useful predictors of neuropsychiatric conditions (e.g., posttraumatic stress disorder (PTSD), major depression, schizophrenia [SZ], dementias) and mood-congruency effects. In addition to describing the EVLT format and test development (seeSupplementary materials for additional details [The Supplementary materials document includes information on test development, including the construction of Target, Interference, and Recognition word lists, analyzes comparing emotion categories within the list on variables known to influence memory, and procedures for the state and trait emotional experience ratings. Additional results are also reported, including differences in proactive/retroactive interference on the EVLT and CVLT, a factor analysis of EVLT and CVLT test scores, and mean and SD data on recall and recognition trials for EVLT discrete emotions to assist in calculating emotion-specific standard scores. Individuals interested in obtaining the EVLT can contact either of the authors to obtain an official version of the test and relevant materials related to administration and scoring.]), we also report data from a series of studies that provide support for the reliability and validity of EVLT scores. Potential clinical utility was also assessed by examining EVLT learning and memory scores in SZ, a neuropsychiatric disorder with known affective abnormalities (e.g., anhedonia, seeStrauss & Gold, 2012; Strauss, Wilbur, Warren, August, & Gold, 2011) and impairments in emotional and non-emotional learning and memory (Gold, Hahn, Strauss, & Waltz, 2009; Gold et al., 2012; Herbener, 2008; Strauss, Frank, et al., 2011).

Overview of Test Format

The EVLT is similar to a number of currently available tests used to assess learning and memory through the presentation of word lists, such as the Rey Auditory Verbal Learning Test (RAVLT; Rey, 1964), Hopkins Verbal Learning Test (HVLT; Brandt, 1991), CVLT (Delis, Kramer, Kaplan, & Ober, 1987), and the Word Lists subtest from the Wechsler Memory Scales (WMSs; Wechsler, 1997). However, EVLT word lists are unique in that they consist of emotional words selected from specific emotional categories. The test first involves the oral presentation of 16 words from the “Target List” over five immediate-recall trials (Trials 1–5). The Target List consists of four words from each of four emotion categories (Happiness, Sadness, Anger, and Anxiety). Following the administration of the five Target List immediate free recall trials, a second “Interference List” is presented for a single trial. The Interference List consists of 16 words, none of which is included on the Target List. Immediately following the Interference List, a short-delay free recall and a short-delay emotion category cued recall testing of the Target List is conducted. Then, after a 20-min delay, long-delay free and cued recall of the Target List is assessed. Finally, the recognition of the Target List is measured using a yes/no recognition format immediately following the administration of the long-delayed emotional cued recall. EVLT word lists are equated on emotional intensity (Strauss & Allen, 2008), representativeness of their discrete emotional categories (Strauss & Allen, 2008), word frequency (Kucera & Francis, 1967), and word length.

Apart from yielding traditional verbal learning and memory scores (e.g., learning slope, primacy, recency), the EVLT allows for an evaluation of the preferential processing of specific emotional content. For example, scores can be obtained for recall of individual emotions on Trial 1, Trials 1–5 total (primary preferential memory measure), short-delay recall, long-delay recall, and recognition. Emotional clustering scores can also be calculated to examine whether the relatedness of emotional words enhances the consecutive recall of words of a specific emotional category. On the EVLT, error scores may also be meaningful predictors of individual differences. In particular, repetitions of words from specific emotional categories on the Target List, as well as intrusions for words not on the Target List, may provide valuable insight into an individual's emotional preoccupations and false memory.

Finally, single-item state and trait emotional experience ratings are also obtained to allow for the assessment of self-reported emotional experience, which can be used to characterize participants and to examine mood-congruent and incongruent memory effects in conjunction with recall and recognition scores. Emotions included in these self-reports include happiness, sadness, anger, anxiety, and disgust. Using a 1–7 Likert scale (1 = not at all and 7 = extremely), subjects are asked to rate their emotional experience “at this moment” (state) and “in general” (trait).

Evaluation of EVLT Psychometric Properties

A series of experiments were conducted to assess the psychometric properties of the EVLT scores as indicated by estimates of reliability/precision (internal consistency, test–retest) and validity (internal structure, convergent, discriminant, and criterion validity). The research was approved by the local institutional review board for the protection of human subjects and all participants provided informed consent.

Reliability/Precision Evidence

Internal consistency

Participants included 329 neurologically and psychiatrically healthy individuals recruited from the community or from a southwestern university in the USA. Individuals were excluded from the study if English was not their first language, or they had a speech impediment, reported a current psychiatric or neurological diagnosis, had experienced a head injury with loss of consciousness, or exhibited severe psychopathology based on routine screening. Individuals were also excluded if they reported history of treatment for a psychiatric disorder. Participants were on average 27.0 years old (SD = 14.1), had 13.7 years of education (SD = 2.1), and were 57% Caucasian, 10% Hispanic, 12% African American, 15% Asian American, 3% bi-racial, 2% other race, and 59% were women. Internal consistency estimates were calculated using this sample.

A research battery was individually administered to the entire sample in a testing session that lasted approximately 1.0–2.5 h. It included a demographic and medical history questionnaire, the EVLT, and the Symptom Checklist-90-Revised (SCL-90-R; Derogatis, 1983) or the Structured Clinical Interview for DSM-IV (SCID; First, Spitzer, Gibbon, & Williams, 2001). A subset of this sample (n = 138), which is described in more detail later, was also administered the CVLT-II (Delis et al., 2000; order of CVLT-II and EVLT administration was counterbalanced). For individuals recruited from the university, the SCL-90-R was used to screen for psychopathology and to exclude individuals scoring in the clinical severity range. Those who were recruited from the community were screened for psychopathology using the SCID.

Results

Descriptive statistics for the EVLT learning and memory conditions are presented in Table 1. We first examined consistency in recall across the five immediate free recall trials of the Target List. This estimate served as the primary measure of internal consistency on the EVLT given that the scores of the five immediate free recall trials are direct indicators of emotional learning and memory and due to issues inherent to healthy individuals having better recall for happiness words, which affect calculations of the other internal consistency estimate that focused on consistency across emotional categories. Similar to other verbal list learning tasks, a split-half correlation was calculated by computing two odd–even correlations between immediate free recall Trials 1 + 3 versus Trials 2 + 4 and Trials 2 + 4 versus Trials 3 + 5. The Spearman–Brown formula was then applied to the average of these correlations with a lengthening factor of 2.0. Reliability for the sample was excellent (r = .96) and comparable with internal consistency estimates using this same method on other tests of verbal learning and memory (e.g., CVLT-II, r = .94). Additionally, there were no differences in internal consistency between male (r = .96) and female (r = .96) participants on the EVLT.

Table 1.

Means and standard deviations for EVLT learning and memory trials (n = 329)

 Men
 
Women
 
Total
 
Mean SD Mean SD Mean SD 
Trial 1 correct 6.68 1.56 6.71 1.80 6.70 1.71 
Trial 2 correct 8.79 1.93 9.35 1.98 9.16 1.97 
Trial 3 correct 9.87 2.29 10.49 2.13 10.27 2.20 
Trial 4 correct 10.45 2.22 11.34 2.17 11.03 2.22 
Trial 5 correct 11.06 2.38 11.82 2.05 11.55 2.20 
Trials 1–5 total correct 46.96 8.38 49.67 8.15 48.73 8.32 
Interference List 5.63 1.80 5.77 1.68 5.72 1.72 
Short-delay free correct 9.17 2.52 10.17 2.62 9.82 2.63 
Short-delay cued correct 8.06 2.54 8.93 2.54 8.62 2.57 
Long-delay free correct 8.91 2.84 10.10 2.60 9.68 2.74 
Long-delay cued correct 8.86 2.86 9.06 2.88 8.99 2.86 
Recognition hits 14.57 1.42 14.95 1.98 14.81 1.29 
Recognition false positives 1.90 2.37 1.42 2.17 1.59 2.25 
Repetitions (Trials 1–5) 4.44 4.61 4.20 4.44 4.28 4.49 
Intrusions (Trials 1–5) 3.53 3.83 2.88 3.41 3.11 3.57 
 Men
 
Women
 
Total
 
Mean SD Mean SD Mean SD 
Trial 1 correct 6.68 1.56 6.71 1.80 6.70 1.71 
Trial 2 correct 8.79 1.93 9.35 1.98 9.16 1.97 
Trial 3 correct 9.87 2.29 10.49 2.13 10.27 2.20 
Trial 4 correct 10.45 2.22 11.34 2.17 11.03 2.22 
Trial 5 correct 11.06 2.38 11.82 2.05 11.55 2.20 
Trials 1–5 total correct 46.96 8.38 49.67 8.15 48.73 8.32 
Interference List 5.63 1.80 5.77 1.68 5.72 1.72 
Short-delay free correct 9.17 2.52 10.17 2.62 9.82 2.63 
Short-delay cued correct 8.06 2.54 8.93 2.54 8.62 2.57 
Long-delay free correct 8.91 2.84 10.10 2.60 9.68 2.74 
Long-delay cued correct 8.86 2.86 9.06 2.88 8.99 2.86 
Recognition hits 14.57 1.42 14.95 1.98 14.81 1.29 
Recognition false positives 1.90 2.37 1.42 2.17 1.59 2.25 
Repetitions (Trials 1–5) 4.44 4.61 4.20 4.44 4.28 4.49 
Intrusions (Trials 1–5) 3.53 3.83 2.88 3.41 3.11 3.57 

The second estimate of internal consistency involved treating the four emotional categories as two halves of the test by combining two of the categories to make one half (happiness + anxiety) and the other two categories to make the other half (sadness + anger). Using this split-half procedure with a Spearman–Brown lengthening factor of 2.0, reliability for the entire sample was r = .84, which is comparable with that of other list learning tasks (Delaney, Prevey, Cramer, & Mattson, 1992; Delis et al., 2000; Geffen, Butterworth, & Geffen, 1994; Woods et al., 2005).

Test–Retest Reliability

To examine test–retest reliability, 32 individuals, whose data were not included in the larger sample used for internal consistency analyses, were administered the EVLT on two occasions with the two testing sessions separated by 7–14 days. These participants were on average 23.0 years of age (SD = 7.8), 72% were women and were 44% Caucasian, 12% African American, 25% Hispanic, 15% Asian American, 3% American Indian/Alaskan Native, and 74% were women. Compared with the reliability/precision sample, the test–retest sample was on average younger and had more female subjects.

Results indicated a test–retest correlation of r = .79 between the Trials 1–5 total scores at the two evaluation sessions. While list learning tests tend to have lower test–retest reliability than other types of neuropsychological tests, the EVLT test–retest reliability coefficient was comparable with that observed for other list learning tasks such as the CVLT-II (Delis et al., 2000), RAVLT (Geffen et al., 1994), HVLT (Benedict, Schretlen, Groninger, & Brandt, 1998), and WMS-III (Wechsler, 1997). Expected practice effects were also present, with an average Trial 1–5 score of 51.41 (SD = 7.38) at the initial evaluation, which increased to 62.21 (SD = 9.47) at the second evaluation (t = 10.83, p < .001). Similar effects were noted for other EVLT scores.

Discussion of Reliability/Precision Studies

Analyses suggested that the primary internal consistency estimate reflecting consistency across immediate free recall trials was excellent and comparable with that of non-emotional list learning tasks. Secondary internal consistency estimates that focused on consistency within emotion categories and test–retest reliability estimates were also comparable with non-emotional learning and memory tests. These results suggest that the EVLT scores demonstrate comparable levels of reliability to other commonly used list learning tasks.

Validity Evidence

Evidence supporting the internal structure of EVLT scores: Factor analysis

Evidence supporting the internal structure of the EVLT scores was initially evaluated through the examination of its factor structure. For the internal structure of the EVLT, we were particularly interested in the structure of the test based on the different emotional categories, and whether there would be evidence supporting the distinction between the four discrete emotional categories based on results of the factor analysis. If the emotional categories were in fact tapping into different discrete emotions, then it would be expected that they would also form different factors. Based on this consideration, we hypothesized that an analysis of the EVLT emotion condition scores would produce four factors that were consistent with the four EVLT emotion categories (Sadness, Anger, Anxiety, and Happiness).

Method and results

In order to test this hypothesis, we examined EVLT performance of the 329 participants used in the reliability/precision studies using Principal Component Analysis (PCA) with Varimax rotation and Kaiser normalization. Factors with eigenvalues >1 were retained, and loadings of 0.40 or higher were considered significant. Scores entered into the analysis included the number of word recalled for anxiety, anger, happiness, sadness for Target List Trial 1, Target List Trial 5, short-delay free recall, long-delay free recall, and recognition hits.

Results of the PCA indicated seven factors with eigenvalues >1.0 and these factors accounted for 64.7% of the total variance. Factor loadings, eigenvalues, and percent variance accounted for are presented in Table 2. Consistent with the hypotheses, four factors emerged that reflected the four emotional categories on the EVLT. These factors were typically composed of the emotion specific scores for Target List Trial 5, short-delay free recall, and long-delay free recall, although for Sadness (Factor 4), Recognition Hits demonstrated a cross loading on Factors 4 and 5. In contrast to the hypothesis, three additional factors emerged. Factor 5 was composed primarily of the Recognition Hit scores for each of the four emotions. Factors 6 and 7 had loadings from Target List Trial 1 loadings, with Factor 6 having a positive loading for happiness and a negative loading for sadness, and Factor 7 demonstrating a positive loading from anger and a negative loading from anxiety. These results suggest that the EVLT consists of four long-term memory factors each of which is emotion-specific for the four target categories, as well as a recognition factor and two short-term memory or attention span factors that are distinguished by the nature of the emotional content of the words. These factor analytic results are consistent with the notion that the EVLT can be used to assess emotional associative semantic networks and may suggest that the EVLT recall scores are more useful than recognition scores at measuring preferential memory for individual emotions.

Table 2.

Factor loadings of 20 EVLT variables (n = 329)

EVLT score Component
 
Anxiety LD 0.81 0.18 0.16 0.17 −0.04 0.11 0.07 
Anxiety SD 0.80 0.22 0.19 0.05 −0.02 −0.03 0.07 
Anxiety T5 0.77 0.14 0.20 0.08 −0.01 −0.14 −0.09 
Anxiety REC 0.63 −0.06 −0.06 0.07 0.34 0.11 −0.03 
Anger SD 0.13 0.85 0.06 0.08 0.06 −0.04 −0.01 
Anger LD 0.13 0.82 0.01 0.16 0.06 0.04 0.08 
Anger T5 0.11 0.71 0.17 0.06 0.03 −0.03 −0.01 
Happiness LD 0.11 0.12 0.82 0.14 0.05 0.08 0.07 
Happiness SD 0.15 0.11 0.80 0.07 0.05 0.14 0.04 
Happiness T5 0.14 0.04 0.73 0.07 0.13 −0.13 −0.08 
Sadness SD 0.16 0.17 0.09 0.81 −0.17 0.09 −0.06 
Sadness LD 0.09 0.20 0.13 0.79 −0.05 0.02 −0.01 
Sadness T5 0.11 0.09 0.03 0.63 0.13 −0.30 0.14 
Sadness REC −0.02 −.19 0.08 0.57 0.41 −0.01 −0.13 
Happiness REC −0.02 .08 0.33 0.01 0.71 −0.14 0.21 
Anger REC 0.18 0.34 0.07 −0.07 0.50 0.35 −0.12 
Sadness T1 0.15 −0.04 0.11 0.26 0.10 0.74 −0.06 
Happiness T1 0.20 −0.11 0.26 0.20 0.13 0.60 0.06 
Anger T1 0.12 0.12 −0.04 0.02 0.16 0.11 0.85 
Anxiety T1 0.21 0.16 −0.14 0.09 0.35 0.05 0.45 
Eigenvalues 4.51 1.86 1.76 1.52 1.16 1.10 1.02 
% Variance 22.56 9.31 8.79 7.62 5.82 5.49 5.12 
EVLT score Component
 
Anxiety LD 0.81 0.18 0.16 0.17 −0.04 0.11 0.07 
Anxiety SD 0.80 0.22 0.19 0.05 −0.02 −0.03 0.07 
Anxiety T5 0.77 0.14 0.20 0.08 −0.01 −0.14 −0.09 
Anxiety REC 0.63 −0.06 −0.06 0.07 0.34 0.11 −0.03 
Anger SD 0.13 0.85 0.06 0.08 0.06 −0.04 −0.01 
Anger LD 0.13 0.82 0.01 0.16 0.06 0.04 0.08 
Anger T5 0.11 0.71 0.17 0.06 0.03 −0.03 −0.01 
Happiness LD 0.11 0.12 0.82 0.14 0.05 0.08 0.07 
Happiness SD 0.15 0.11 0.80 0.07 0.05 0.14 0.04 
Happiness T5 0.14 0.04 0.73 0.07 0.13 −0.13 −0.08 
Sadness SD 0.16 0.17 0.09 0.81 −0.17 0.09 −0.06 
Sadness LD 0.09 0.20 0.13 0.79 −0.05 0.02 −0.01 
Sadness T5 0.11 0.09 0.03 0.63 0.13 −0.30 0.14 
Sadness REC −0.02 −.19 0.08 0.57 0.41 −0.01 −0.13 
Happiness REC −0.02 .08 0.33 0.01 0.71 −0.14 0.21 
Anger REC 0.18 0.34 0.07 −0.07 0.50 0.35 −0.12 
Sadness T1 0.15 −0.04 0.11 0.26 0.10 0.74 −0.06 
Happiness T1 0.20 −0.11 0.26 0.20 0.13 0.60 0.06 
Anger T1 0.12 0.12 −0.04 0.02 0.16 0.11 0.85 
Anxiety T1 0.21 0.16 −0.14 0.09 0.35 0.05 0.45 
Eigenvalues 4.51 1.86 1.76 1.52 1.16 1.10 1.02 
% Variance 22.56 9.31 8.79 7.62 5.82 5.49 5.12 

Notes: EVLT = Emotional Verbal Learning Test; LD = Long-delay free recall; SD = Short-delay free recall; T5 = Target List Trial 5; REC = Recognition hits; T1 = Target List Trial 1.

Differential Item Functioning for Happiness Words and Self-Reported Emotional Experience

Evidence supporting the internal structure of the EVLT scores was also evaluated by examining differential item functioning for happiness words and emotional experience ratings. In this regard, one of the unique features of the EVLT is that it allows for a comparison of recall and recognition across the four discrete emotional categories of happiness, sadness, anger, and anxiety. It also allows for the examination of repetition and intrusion errors for these emotions, as well as self-reported emotional experience ratings for state and trait emotion. Based on previous research, it was hypothesized that normal individuals would (a) recall more happiness words in comparison with the sadness, anger, or anxiety words, (b) demonstrate increased repetitions and intrusions for happiness words, and (c) report greater levels of state and trait happiness than sadness, anger, or anxiety. Such findings would provide evidence supporting the internal structure of the EVLT test scores.

Method and results

In order to test these hypotheses, we examined EVLT performance of the 329 participants used in the reliability/precision studies. Fig. 1A (Standard Order) contains the total recall scores for Target List Trails 1–5 for the four emotion categories. As can be seen from the figure, performance on the EVLT followed the hypothesized pattern, as participants had greater recall of happiness than sadness, anger, or anxiety, and repeated-measures analysis of variance (ANOVA) indicated that these differences in recall were significant, F(3, 328) = 154.78, p < .001, d = 1.4. Simple contrasts further indicated greater recall for happiness than sadness, anger, or anxiety (all p's < .001). Similar patterns of results were found for immediate free recall Trial 1, Trial 5, short-delay free recall, short-delay cued recall, long-delay free recall, and long-delay cued recall. However, participants did not show statistically greater performance for happiness than other conditions on long-delay yes/no recognition memory, potentially due to ceiling effects (seeSupplementary material online).

Fig. 1.

Means and standard errors for EVLT Trials 1–5 Total Recall per emotional condition in standard order (n = 329) and rotated order (n = 48) (A) and means and standard errors for EVLT state and trait emotional experience ratings (n = 329) (B). EVLT = Emotional Verbal Learning Test; H = Happiness; S = Sadness; A = Anger; X = Anxiety; D = Disgust.

Fig. 1.

Means and standard errors for EVLT Trials 1–5 Total Recall per emotional condition in standard order (n = 329) and rotated order (n = 48) (A) and means and standard errors for EVLT state and trait emotional experience ratings (n = 329) (B). EVLT = Emotional Verbal Learning Test; H = Happiness; S = Sadness; A = Anger; X = Anxiety; D = Disgust.

In order to determine whether the patterns of results indicating differences in recall across the four emotions were possibly due to bias introduced by the order of word presentation on the Target List, we conducted a subsequent experiment where we systematically rotated the order of the list so that each of the Target List words appeared at each of the 16 possible positions, resulting in 16 lists. A sample of 48 healthy undergraduate students completed this follow-up study. They were on average 22.8-year old (SD = 5.6), had 13.8 years of education (SD = 3.0), and were 60.0% Caucasian, 17% Hispanic, 15% African American, 8% Asian American, and 75% were women. These individuals were screened for psychopathology using the SCL-90 (Derogatis, 1983), and they were not included in the other analyses reported in this paper. Participants were administered one of the 16 versions of the EVLT. A total of three participants were randomly assigned to receive each version. As in the prior study, it was hypothesized the subjects would show preferential recall for happiness words.

As can be seen in Fig. 1A (Rotated Order), results of the list rotation study collapsing across all list orders were nearly identical to those of the larger sample using the standard Target List order. This is particularly true for recall of the happiness words, which was on average 13.61 words for the entire sample, and 13.57 words for the list rotation sample. Similar to the results for the entire sample, repeated-measures ANOVA indicated a significant within-subjects effect of emotion, F(3, 47) = 7.66, p < .001, d = 0.74. Simple contrasts again indicated greater recall for happiness words than for sadness, anger, or anxiety words (all p's < .001). Thus, findings of the list rotation experiment suggest that the order of word presentation on the Target List was not responsible for greater recall of happiness words and that the preferential recall of happiness words is a robust finding.

We next examined performance on intrusion and repetition errors to determine if there was a memory error bias for happiness words. Repetition errors are thought to reflect perseveration on a particular mood state, as well as source monitoring errors and difficulty with inhibition. Intrusion errors are thought to reflect confabulatory tendencies, disinhibition, or preoccupation with a certain emotional state. Data from the 329 subjects used in the reliability/precision studies were used for these analyses. As can be seen in Table 3, this sample of healthy individuals had a relatively low level of repetitions and intrusions. However, repeated-measures ANOVA indicated that there were significant differences in the number of intrusions made across the emotional categories, F(3, 328) = 29.70, p < .001, d = 0.63, and simple contrasts indicated that there were significantly more intrusions for happiness words than for sadness, anger, or anxiety words (p < .001 for all comparisons). There were also differences in repetitions among the four emotions, F(3, 328) = 7.37, p < .01, d = 0.29, such that participants made the greatest number of repetitions for anxiety words and the fewest for those representing anger. Men and women had similar error patterns (Table 3).

Table 3.

Means and standard deviations for emotional intrusions and repetitions across Trials 1–5 (n = 329)

 Men
 
Women
 
Total
 
Mean SD Mean SD Mean SD 
Repetitions 
 Happiness 1.08 1.38 1.25 1.48 1.19 1.45 
 Sadness 1.06 1.43 1.09 1.25 1.08 1.31 
 Anger 0.94 1.17 1.01 1.39 0.99 1.32 
 Anxiety 1.56 1.85 1.32 2.02 1.41 1.97 
Intrusions 
 Happiness 1.66 2.01 1.25 1.73 1.39 1.84 
 Sadness 0.56 1.07 0.53 1.13 0.54 1.10 
 Anger 0.94 1.77 0.72 1.40 0.80 1.54 
 Anxiety 0.39 1.22 0.42 1.65 0.41 1.51 
 Men
 
Women
 
Total
 
Mean SD Mean SD Mean SD 
Repetitions 
 Happiness 1.08 1.38 1.25 1.48 1.19 1.45 
 Sadness 1.06 1.43 1.09 1.25 1.08 1.31 
 Anger 0.94 1.17 1.01 1.39 0.99 1.32 
 Anxiety 1.56 1.85 1.32 2.02 1.41 1.97 
Intrusions 
 Happiness 1.66 2.01 1.25 1.73 1.39 1.84 
 Sadness 0.56 1.07 0.53 1.13 0.54 1.10 
 Anger 0.94 1.77 0.72 1.40 0.80 1.54 
 Anxiety 0.39 1.22 0.42 1.65 0.41 1.51 

Finally, we examined the validity of the EVLT's self-reported state and trait emotional experience ratings. Descriptive statistics for the state and trait emotional experience self-reports are presented in Fig. 1B. Data from the 329 subjects used in the reliability/precision studies were used for these analyses. Repeated-measures ANOVAs were used to examine differences in state and trait emotional experience, where emotion condition served as the repeated measure. These analyses indicated significant differences for EVLT self-report of both state emotion, F(4, 328) = 500.92, p < .001, d = 2.5, and trait emotional experience, F(4, 328) = 303.72, p < .001, d = 1.96. Simple contrasts indicated that the sample reported greater experience of happiness than sadness, anger, anxiety, or disgust, for both the state and trait scales (p's < .001 for all comparisons). Additionally, subjects reported greater experience of anxiety than sadness, anger, and disgust for both state and trait formats (p's < .001).

Differential Item Functioning Based on Serial Position

Data were collected on 138 participants recruited from the community and university to determine differential item functioning for the EVLT in relation to the CVLT-II (Delis et al., 2000). The sample was on average 22.6-year old (SD = 6.4), had 13.3 years of education (SD = 2.2), and was 58% Caucasian, 13% Hispanic, 12% African American, 17% Asian American, and 64% women. These participants were younger on average than those in the reliability/precision study. Community participants were evaluated for the presence of Axis-I psychopathology using the SCID (First et al., 2001) and undergraduate students were screened for psychopathology using the SCL-90-R (Derogatis, 1983).

The test battery was individually administered to all participants in one sitting that lasted approximately 2.0–2.5 h. Measures included a demographic and medical history questionnaire, the EVLT, CVLT-II, Positive and Negative Affect Scales (PANASs; Watson & Clark, 1992), and the SCL-90-R or SCID. The PANAS is a commonly used emotional self-report scale which asks participants to rate how much they feel a number of specific emotional terms that make up positive (PA) and negative (NA) affect dimensions. On the PANAS, participants provide ratings for both trait emotion (i.e., how do you feel in general) and state emotion (i.e., how do you feel right now). Periods of non-verbal, non-affective neuropsychological testing occurred during the long-delay break sections of the EVLT and the CVLT-II. The order of EVLT and CVLT-II administration was counterbalanced across participants. Analyses indicated no evidence for test order effects on the primary memory and learning measures.

A repeated-measures ANOVA was used to examine differences in serial position effects for the EVLT and the CVLT, where primacy, middle, and recency scores served as the repeated measure, and the EVLT and the CVLT were separate within-subjects factors. Primacy, middle, and recency serial position effects are presented in Fig. 2A. Results of the analysis indicated a significant effect of serial position, F(2, 196) = 86.12, p < .001, d = 1.63, and a significant Test × Serial Position interaction, F(2, 196) = 18.56, p < .001, d = 0.76. However, the main effect of test was non-significant, F(1,196) = 0.98, p = .32. Post hoc pairwise comparisons indicated that the main effect of Serial Position was accounted for by greater recall of primacy and recency words compared with words appearing in the middle of the lists on both tests (all p's < .001). As can be seen in Fig. 2A, the significant Test × Serial Position interaction reflects that healthy individuals displayed greater primacy than recency recall on the CVLT-II (p < .001) and displayed greater recency than primacy recall on the EVLT (p < .01).

Fig. 2.

Serial position effects for the EVLT and the CVLT-II (n = 138). EVLT = Emotional Verbal Learning Test; CVLT-II = California Verbal Learning Test, Second Edition.

Fig. 2.

Serial position effects for the EVLT and the CVLT-II (n = 138). EVLT = Emotional Verbal Learning Test; CVLT-II = California Verbal Learning Test, Second Edition.

To further explore this interaction effect, we plotted the EVLT and CVLT-II serial position curve for Trial 1 free recall (Fig. 2B). Results indicated that participants generally showed the typical serial position curve on the CVLT-II, with better recall on most of the words at the beginning and the end of the list than those in the middle. However, the serial position curve for the EVLT indicated higher scores on several middle position words than the CVLT-II. Pairwise comparisons between words on the CVLT-II and the EVLT indicated greater recall for EVLT words in Positions 6 (glory), 7 (sad), 9 (anxious), and 11 (honor) (p < .01 for all) and greater recall for the CVLT-II word in Position 5 (p < .001). Two of the EVLT words are from the happiness condition (glory and honor), which is consistent with more efficient encoding and retrieval of positive words. However, the other two words, sad and anxious, are from disparate emotional categories and so it was unclear why these words were recalled more frequently.

To further examine this matter, we conducted a subsequent experiment involving 46 participants who were not involved in the other studies reported in this paper. These participants were on average 21.8-year old (SD = 6.3), with 14 years of education (SD = 3.6), and were 65% Caucasian, 26% African American, 9% Hispanic, and 65% women. Compared with the reliability/precision study, these subjects were younger on average; however, they were comparable with the validity sample (n = 138) in demographics. They were administered the EVLT and then asked to rate each EVLT word on the extent to which it represented seven basic emotion categories (happiness, sadness, anger, anxiety, fear, disgust, and surprise) on a 1 (not at all) to 7 (extremely) scale. Word prototypicality scores were calculated by computing means for the extent to which each subject rated each word as representing its designated category, for example, the rating of the word Joy for the category of happiness or the word Rage for the anger category. Correlational analyses indicated that prototypicality ratings were significantly associated with greater recall for words in the middle of the list (r = .43, p < .001), indicating that words that were subjectively rated as more prototypical of their designated emotion categories facilitate encoding and recall, even in the hard-to-remember middle position. Thus, prototypicality may be an important predictor of the disrupted serial position curve on the EVLT.

Sensitivity to Mood-Congruent Memory Effects

Several analyses were conducted to determine whether the EVLT is sensitive to mood-congruency effects. Analyses focused on state happiness and anxiety given that these emotions were the state emotions that participants reported experiencing most intensely.

Anxiety

To examine mood-congruent memory in relation to anxiety, we divided the sample of n = 329 used in the reliability studies into subgroups with high and low state anxiety as measured by participants' EVLT state ratings. Participants with state anxiety self-reports at or below the 25th percentile were considered “low state anxiety” (scores of 1, n = 121), and participants at or above the 75th percentile were considered “high state anxiety” (scores of 4–7, n = 85; scale range = 1 not at all anxious to 7 extremely anxious). One-way ANOVAs were used to compare the high and low state anxiety groups on total recall across immediate free recall Trials 1–5 for happiness, sadness, anger, and anxiety words. Results indicated that the high and low state anxiety groups significantly differed in recall for anxiety, F(1, 205) = 5.86, p < .02, and anger words, F(1, 205) = 4.61, p < .04, but not happiness or sadness (p's > .85). As can be seen in Fig. 3A, participants with high state anxiety recalled significantly more anxiety and anger words than participants reporting low state anxiety. The finding of greater recall for anxiety words by the high state anxiety group suggests that the EVLT is sensitive to detecting mood-congruent memory effects.

Fig. 3.

Mood-congruent memory effects in relation to state anxiety (A) and state happiness (B).

Fig. 3.

Mood-congruent memory effects in relation to state anxiety (A) and state happiness (B).

Happiness

Support for the EVLT's ability to detect mood-congruent memory effects was also supported by analyses related to state happiness. The distribution of state happiness scores was substantially negatively skewed and leptokurtotic, with the majority of scores clustering at the 4–5 scale range. To account for this, we grouped participants into “high state happiness” (scores of 6 and 7, n = 106) and “low state happiness” (scores of 1, 2, or 3, n = 19). One-way ANOVAs indicated that the high state happiness group recalled significantly more happiness word on Trials 1–5 than the low state happiness group, F(1, 124) = 3.91, p < .05; however, groups did not significantly differ in recall of sadness, anger, or anxiety words (p's > .64; Fig. 3B). The finding of greater recall for happiness words by the high state happiness group suggests that the EVLT is sensitive to detecting mood-congruency effects.

Evidence from Convergent and Discriminant Validity

To examine validity of the EVLT scores in relation to the CVLT-II, we examined: (a) the correlation between EVLT scores and demographic factors with known associations with verbal recall, as well as gender differences in recall, (b) correlations between primary recall, recognition, and error scores, (c) differences in performance across memory and learning scores on the two tests, as well as the number of repetitions and intrusions, (d) differences in the use of serial, semantic, and subjective clustering, (e) differences in Trial 1 recall at primacy, middle, and recency locations, (f) differences in proactive and retroactive interference, and (g) factor loadings for key EVLT and CVLT variables entered into a single PCA.

We also examined the construct validity of the EVLT self-reported emotional experience ratings in relation to the PANAS PA and NA dimensions. The trait format of the PANAS was used (i.e., how do you feel in general), and it would be expected that correlations between the PANAS and EVLT trait measures would be somewhat higher than correlations with EVLT state measures since state and trait self-reports are known to rely on different sources of emotion knowledge (Robinson & Clore, 2002).

Method and results

Data used for these analyses were from the 138 participants examined in the serial position effects study. Correlations between the EVLT Trials 1–5 total score and demographic variables were conducted to estimate EVLT validity in relation to variables with known memory associations. Significant correlations were found with age (r = −.38) and education (r = .19); comparable correlations were observed for the CVLT-II: age (r = −.34) and education (r = .24) (p's < .05). Consistent with results reported on many verbal learning tests, which show gender differences in total recall (e.g., CVLT-II), we also found that women have higher total Trials 1–5 recall scores than men on the EVLT, t = −2.77, p < .01, and the CVLT-II, t = −2.98, p < .01. These analyses support the validity of the EVLT scores, suggesting that the test scores are related to demographic variables in the expected way.

As can be seen in Table 4, the EVLT and CVLT-II primary learning and memory conditions were moderately correlated, ranging from r = .27 to .50, with the Target List 1–5 Total score correlating 0.45 with the CVLT-II Trials 1–5 Total score. This magnitude of correlation suggests that the EVLT assesses learning and memory similarly to the CVLT, but that it is also tapping into a different aspect of learning and memory than the CVLT-II, as intended (i.e., emotional memory). A comparison of the means of raw scores provides an estimate of equivalence between the EVLT and the CVLT-II (see Table 4). Paired samples t-tests indicated that participants scored higher on the CVLT-II than the EVLT on the immediate learning Trials 2–5 (p < .001 for all comparisons), but not Trial 1 (p = .16), as well as Interference List recall (p < .01), short-delay free (p < .001) and cued recall (p < .001), and long-delay free (p < .001) and cued recall (p < .001). Long-delay yes/no recognition did not significantly differ between tests (p = .10). There were significantly more repetitions (p < .01) and intrusions (p < .001) on the EVLT than the CVLT-II, and more false positives on the CVLT-II yes/no recognition section than the EVLT (p < .001).

Table 4.

Means, standard deviations, and correlations for EVLT and CVLT-II learning and memory conditions (n = 138)

Score EVLT
 
CVLT-II
 
r 
Mean SD Mean SD 
Trial 1 correct 7.02 1.83 7.30 2.00 .30*** 
Trial 2 correct 9.59 2.03 10.74 2.35 .28*** 
Trial 3 correct 10.68 2.25 11.99 2.32 .40*** 
Trial 4 correct 11.58 2.17 12.89 2.45 .38*** 
Trial 5 correct 11.95 2.28 13.22 2.43 .37*** 
Trials 1–5 total correct 50.93 8.42 56.01 10.28 .45*** 
Interference List/List B correct 5.77 1.73 6.40 1.99 .40*** 
Short-delay free correct 10.25 2.70 12.05 2.61 .37*** 
Short-delay cued correct 9.07 2.62 12.69 2.38 .37*** 
Long-delay free correct 10.23 2.74 12.29 2.60 .43*** 
Long-delay cued correct 8.91 4.75 13.00 2.32 .35*** 
Recognition total correct 14.98 1.07 15.16 1.17 .33*** 
Recognition total false positives 1.21 1.79 1.98 1.75 .27** 
Total repetitions (Trials 1–5) 4.26 3.74 3.33 4.03 .50*** 
Total intrusions (Trials 1–5) 2.52 2.73 0.92 1.92 .34*** 
Score EVLT
 
CVLT-II
 
r 
Mean SD Mean SD 
Trial 1 correct 7.02 1.83 7.30 2.00 .30*** 
Trial 2 correct 9.59 2.03 10.74 2.35 .28*** 
Trial 3 correct 10.68 2.25 11.99 2.32 .40*** 
Trial 4 correct 11.58 2.17 12.89 2.45 .38*** 
Trial 5 correct 11.95 2.28 13.22 2.43 .37*** 
Trials 1–5 total correct 50.93 8.42 56.01 10.28 .45*** 
Interference List/List B correct 5.77 1.73 6.40 1.99 .40*** 
Short-delay free correct 10.25 2.70 12.05 2.61 .37*** 
Short-delay cued correct 9.07 2.62 12.69 2.38 .37*** 
Long-delay free correct 10.23 2.74 12.29 2.60 .43*** 
Long-delay cued correct 8.91 4.75 13.00 2.32 .35*** 
Recognition total correct 14.98 1.07 15.16 1.17 .33*** 
Recognition total false positives 1.21 1.79 1.98 1.75 .27** 
Total repetitions (Trials 1–5) 4.26 3.74 3.33 4.03 .50*** 
Total intrusions (Trials 1–5) 2.52 2.73 0.92 1.92 .34*** 

Notes: EVLT = Emotional Verbal Learning Test; CVLT-II = California Verbal Learning Test, 2nd Edition.

**p < .01.

***p < .001.

Analyses of clustering scores were conducted to examine whether clustering accounted for these recall differences between tests. Results indicated that better recall on the CVLT-II than the EVLT may be due in part to greater reliance on semantic clustering strategies on the CVLT-II and serial clustering strategies on the EVLT. Repeated-measures ANOVA indicated a significant Test × Clustering Strategy interaction, F(2, 137) = 28.22, p < .001 (Semantic Clustering: EVLT M = 0.12, SD = 0.53; CVLT-II M = 1.92, SD = 2.01; Serial Clustering: EVLT M = 0.98, SD = 0.71; CVLT-II M = 0.39, SD = 0.83; Subjective Clustering: EVLT M = −0.28, SD = 0.48; CVLT-II M = −0.74, SD = 0.42). Better total recall on Trials 1–5 was significantly associated with better semantic clustering scores on both tests (CVLT-II r = .52, p < .001; EVLT r = .28, p < .05). Higher serial clustering scores were associated with better total recall on Trials 1–5 for the EVLT (r = .26, p < .05), but worse scores on the CVLT-II (r = −.35, p < .05). Higher subjective clustering scores were associated with poorer immediate recall on both tests, CVLT-II (r = −.43, p < .001) and EVLT (r = −.19, n.s.), although the association was only significant for the CVLT-II. Greater serial clustering may occur on the EVLT because emotional words are predominantly concepts and emotions, and not things like the CVLT words.

Differences in proactive and retroactive interference were examined in relation to the EVLT and the CVLT-II. Results indicated differential effects of interference across tests, such that there was greater proactive interference on the CVLT-II than the EVLT, and greater retroactive interference on the EVLT than the CVLT-II (seeSupplementary material online).

Key EVLT and CVLT-II variables were entered into a single PCA to examine whether the EVLT is unique from other measures of verbal learning and memory. Results supported the uniqueness of the EVLT, as EVLT variables formed a factor that was distinct from CVLT-II variables and there were no high cross loadings (seeSupplementary material online).

To examine the validity of the EVLT self-report emotional experience ratings, correlations were calculated between the EVLT self-report state and trait ratings and state and trait ratings on the PANASs. Correlations indicated that the EVLT self-report scales have good convergent validity, as the PANAS PA was correlated with EVLT happiness and PANAS NA with EVLT sadness, anger, anxiety, and disgust ratings (Table 5). These brief and simple EVLT self-report items can thus have adequate convergent validity with established measures of emotional experience.

Table 5.

Correlations between EVLT state and trait emotional experience ratings and the PANAS (n = 138)

EVLT experience ratings PANAS PA trait PANAS NA trait 
State 
 Happiness .24* .02 
 Sadness −.14 .38** 
 Anger −.09 .40** 
 Anxiety −.03 .37** 
 Disgust .04 .26** 
Trait   
 Happiness .25* −.24* 
 Sadness −.10 .41** 
 Anger −.24* .43** 
 Anxiety −.05 .45** 
 Disgust −.03 .37** 
EVLT experience ratings PANAS PA trait PANAS NA trait 
State 
 Happiness .24* .02 
 Sadness −.14 .38** 
 Anger −.09 .40** 
 Anxiety −.03 .37** 
 Disgust .04 .26** 
Trait   
 Happiness .25* −.24* 
 Sadness −.10 .41** 
 Anger −.24* .43** 
 Anxiety −.05 .45** 
 Disgust −.03 .37** 

Notes: EVLT= Emotional Verbal Learning Test; PANAS = Positive and negative affect scale; PA = PANAS trait positive affect; NA = PANAS trait negative affect.

*p < .05.

**p < .01.

Evidence from Test–Criterion Relationships

To examine potential clinical utility, criterion validity, and incremental validity of the EVLT test scores, we administered the EVLT and CVLT (Delis et al., 1987) to a sample of outpatients with SZ and demographically matched controls (CNs). SZ was selected as the patient group because it is known to be associated with significant memory deficits (for meta-analysis, seeHeinrichs & Zakzanis, 1998), including deficits in verbal memory and emotional memory (for review, seeHerbener, 2008). We hypothesized that patients would show poorer performance than CNs on both the EVLT and the CVLT, but that patients would show a reduced ability to learn new emotional relative to neutral words across the five immediate free recall trials. We also predicted that the EVLT would provide comparable classification statistics as the CVLT. Furthermore, given the known affective symptoms associated with SZ, particularly negative symptoms like anhedonia, we hypothesized that poorer recall on the EVLT would predict greater severity of clinically rated negative symptoms.

Method

Participants included a sample of 28 individuals meeting Diagnostic and Statistical manual for Mental Disorders, 4th edition, text revision (DSM-IV-TR) criteria for SZ and 25 healthy CNs. The SZ group was on average 42.0-year old (SD = 9.3), with 12.7 years of education (SD = 1.8), and were 42.9% Caucasian, 46.4% African American, 3.6% Hispanic, and 53.6% women. The CN group was on average 37.4-year old (SD = 13.2) with 13.8 years of education (SD = 1.6) and were 40.0% Caucasian, 24.0% African American, 4.0% Hispanic, 16.0% Asian, and 68.0% women. The groups did not significantly differ on age, F(1, 52) = 2.1, p = .16, gender, χ2 = 1.15, p = .28, or ethnicity χ2 =7.28, p = .12. Individuals in the SZ group had significantly lower education than the CN group, F(1, 52) = 4.87, p = .03, as is common in studies of SZ. The CN and SZ samples were older, had a larger proportion of African-American subjects, and lower proportion of women than participants in the reliability/precision (n = 329) or validity (n = 138) studies.

Individuals with SZ were recruited from a local community outpatient mental health center and were identified by their treating psychiatrists for inclusion in the study if they were English speaking and carried a primary diagnosis of DSM-IV-TR SZ. This clinical diagnosis was subsequently confirmed using the SCID (First et al., 2001), which was informed by consultation with the treating psychiatrist and other program staff, and thorough the review of medical records. Individuals with SZ were excluded from participation if English was not their first language, they had a history of traumatic brain injury or other medical or neurological conditions that would affect central nervous system function, had a history of substance use disorder within the past 6 months, a diagnosis of mental retardation, or were using prescribed or over-the-counter medications that could produce significant cognitive effects (other than those prescribed to treat SZ).

Healthy CNs were recruited from the community and a local university. In addition to the aforementioned exclusionary criteria for the SZ groups, healthy CNs had no lifetime diagnosis of SZ or bipolar disorder, history of neurological conditions, or current Axis-I psychiatric disorder, as determined by the SCID. Individuals were also excluded if they were currently using any psychotropic medication or had a first- or a second-degree relative diagnosed with or suspected to have a psychotic disorder as determined using a standardized interview.

Participants were administered the EVLT and the CVLT in a counterbalanced order. Periods of non-verbal and non-affective neuropsychological testing occurred during the long-delay periods. Patients also completed a battery of other neuropsychological measures and clinical symptom severity interviews, including the Brief Psychiatric Rating Scale (BPRS; Overall & Gorham, 1962). At the time of the evaluation, symptom measures indicated that patients were experiencing a mild level of general psychiatric symptoms (BPRS total M = 44.9, SD = 12.1),

Results

Separate repeated-measures ANOVAs were conducted to examine group differences in learning rate on the EVLT and the CVLT (Fig. 3A). For the EVLT, repeated-measures ANOVA indicated a significant 2 Group (SZ vs. CN) × 5 Trial (immediate free recall Trials 1–5) interaction, F(4, 51) = 6.35, p < .001. There was also a significant Group × Trial interaction for the CVLT, F(4, 51) = 4.31, p < .01. Interactions remained significant after including education as a covariate. To follow-up these significant interactions, paired-samples t-tests were conducted separately for each group and test to determine whether SZ and CN showed incremental learning from Trial 1 to 2, Trial 2 to 3, Trial 3 to 4, and Trial 4 to 5. For CNs, there was a significant increase in learning on both tests from Trial 1–2, Trial 2–3, and Trial 3–4 (all p's < .03). However, there was no difference in recall on Trials 4–5 on either test (all p's > .66). CNs, therefore, demonstrate robust improvements in learning from Trials 1–4, but show no improvement from Trial 4 to 5 on either test. A different pattern of results was observed in SZ patients. On the CVLT, individuals with SZ showed incremental learning from Trial 1–2 (p < .001), Trial 2–3 (p < .05), and Trial 3–4 (p < .05), but no difference in recall from Trial 4 to 5 (p = .90). In contrast, patients showed improvement from Trial 1 to 2 of the EVLT (p < .001), but no differences in recall between Trial 2–3 (p = .25), Trial 3–4 (p = .41), and Trial 4–5 (p = .26). Thus, individuals with SZ evidenced poorer recall than CNs overall, and they were able to demonstrate incremental gains in learning neutral words, but not emotional words over the five immediate free recall trials (Fig. 4A).

Fig. 4.

Differences in recall across Trials 1–5 on the EVLT and the CVLT in schizophrenia patients (n = 28) and normal controls (n = 25) (A) and correlations between recall across Trials 1–5 and symptom ratings. EVLT = Emotional Verbal Learning Test; CVLT = California Verbal Learning Test, Second Edition; POS = BPRS Positive symptoms; NEG = BPRS Negative Symptoms; DIS = BPRS Disorganization; TOT = BPRS Total score (B).

Fig. 4.

Differences in recall across Trials 1–5 on the EVLT and the CVLT in schizophrenia patients (n = 28) and normal controls (n = 25) (A) and correlations between recall across Trials 1–5 and symptom ratings. EVLT = Emotional Verbal Learning Test; CVLT = California Verbal Learning Test, Second Edition; POS = BPRS Positive symptoms; NEG = BPRS Negative Symptoms; DIS = BPRS Disorganization; TOT = BPRS Total score (B).

CVLT and EVLT Trial 1–5 total scores were subjected to receiver operating characteristic (ROC) analyses in order to determine the sensitivity, specificity, positive predictive value, and negative predictive value of the scores. ROC analyses have been widely applied to determine classification properties of psychological tests (Swets, 1996). The area under the ROC curve (AUC) was used as an estimate of how well EVLT and CVLT scores distinguish between groups. Results indicated that the two tests had comparable classification with the EVLT producing an AUC of 0.910 (SE = 0.041, p < .0001, 95% CI = 0.830–0.990), and the CVLT producing an AUC of 0.894 (SE = 0.045, p < .0001, 95% CI = 0.807–0.982). Comparisons of these AUC's using the method described by Hanley and McNeil (1983) indicated that they were not significantly different (p > .05). Tests with AUCs between 0.80 and 0.90 are considered to have “good” classification accuracy (Hosmer & Lemeshow, 2000). Using a cut score of <37, sensitivity, specificity, positive predictive value, and negative predictive value for the EVLT were 0.92, 0.79, 0.92, and 0.79, respectively. Using a cut score of <45, sensitivity, specificity, positive predictive value, and negative predictive value for the CVLT were 0.82, 0.88, 0.79, and 0.81, respectively.

To evaluate the incremental validity of the EVLT, logistic regression was conducted to determine whether recall on EVLT Trials 1–5 predicted diagnostic group membership more so than recall on CVLT Trials 1–5. The regression equation correctly classified 83% of the cases (SZ = 82%, CN = 84%), model χ2(2) = 37.41, p < .001. EVLT Trials 1–5 recall significantly predicted diagnosis (B = 0.13; odds ratio [OR] 1.13, p < .03), whereas CVLT Trials 1–5 recall did not (B = 0.07; OR 1.07, p = .15). These results support the incremental validity of the EVLT in predicting diagnosis above and beyond that of the CVLT.

To further investigate the incremental validity of the EVLT, the relationships between SZ symptoms and recall performance was examined. SZ symptoms were measured using the BPRS factors of negative symptoms, positive symptoms, and disorganized symptoms, as well as a general indicator of symptomatology, the BPRS total score. These BPRS scores were correlated with EVLT and CVLT Trials 1–5 total scores to examine potential relationships between emotional/non-emotional verbal recall and symptom severity. Results of these correlational analyses are presented in Fig. 4B. There was a significant correlation between the BPRS negative symptom factor and EVLT Trials 1–5 free recall (r = −.41, p < .03), although the comparable correlation with CVLT Trials 1–5 total was non-significant (r = −.26, p = .18). All other correlations were non-significant, although correlations between the BPRS disorganization and total scores were also somewhat higher with the EVLT than for the CVLT-II. Thus, negative symptoms and possibly other symptoms as well are associated with impaired recall of emotional information in SZ patients; the differential pattern of correlations between EVLT and CVLT scores supports the incremental validity of the EVLT.

Discussion of Validity Studies

Overall, findings across these studies provide validity evidence supporting the internal structure of the EVLT recall scores, as the expected pattern of better recall for happiness than sadness, anger, or anxiety was found. Similarly, evidence for errors in emotional intrusions and repetitions were found across tests, providing further evidence supporting the EVLT as a unique measure of emotional information processing. Finally, the sample self-reported a moderate level of happiness for both state and trait emotional experience ratings, and these happiness ratings were significantly higher than their ratings for the other emotions. These findings are similar to the most frequently reported patterns of self-reported emotional experience in healthy individuals who tend to report being in a moderately positive mood and provide support for the validity of the EVLT's self-report emotional experience ratings, which also proved valuable for identifying mood-congruency effects.

As expected, EVLT scores were significantly correlated with the CVLT-II scores across the major learning and memory variables, indicating that the EVLT validly assesses learning and memory. However, the magnitude of these correlations was only moderate, suggesting that the EVLT is not redundant with the CVLT-II and that it measures something different, emotional learning and memory. Scores on the EVLT were generally lower than those of the CVLT-II on portions of the test following immediate free recall Trial 1, suggesting that the EVLT may be a more difficult test. Better scores on the CVLT-II than the EVLT may be attributed in part to a greater reliance on semantic clustering on the CVLT-II and serial clustering on the EVLT. Semantic clustering was possible on both tests, and higher semantic clustering scores were associated with better recall on Trials 1–5 on both tests. Better recall for neutral than emotional stimuli might be expected when the EVLT is given in comparison to a test like the CVLT-II or the HVLT, as it has been reported that neutral stimuli are better recalled than emotional stimuli when the neutral list contains words that are semantically related (e.g., Talmi & Moscovitch, 2004). Studies finding better recall for emotional than neutral words typically do not use neutral words that are semantically related, resulting in potentially artificial differences confounded by semantic relatedness. Thus, future studies should select their neutral comparison measure carefully.

Interestingly, there were differences in primacy and recency recall across the two tests, such that there was better primacy than recency recall on the CVLT-II and the reverse pattern on the EVLT. Unexpected to us, there was also a disruption of the serial position curve on the EVLT, such that there were some words in the middle of the list recalled as accurately as those at the beginning and the end of the list. A follow-up experiment indicated that words rated as more prototypical of their designated emotional categories were more likely to produce increased recall in the middle positions, as were words from the happiness category. Emotional content thus appears to influence encoding in a unique manner, pointing to the validity and distinctiveness of the measure. Disruption of the serial position effect, and recall in the middle of the list in particular, could prove to be a valuable means of measuring psychopathology. For example, it may be that depressed individuals are more likely to show disrupted serial position effects for EVLT middle position words related to sadness, which capture attention and facilitate encoding.

We also administered the EVLT to a sample of individuals with SZ and demographically matched CNs to evaluate incremental and criterion validity. Compared with CNs, SZ patients had lower total performance on both the CVLT and the EVLT, as would be expected of a patient group with episodic memory impairments. However, there were interesting differences between the patient and CN groups with regard to rates of learning new information across the tests. In CNs, there was evidence for incremental gains in recall from Trial 1–2, Trial 2–3, and Trial 3–4 on both the CVLT and the EVLT. However, SZ patients showed incremental improvements across Trial 1–2, Trial 2–3, and Trial 3–4 on the CVLT and only displayed improvement form Trial 1–2 on the EVLT, suggesting a diminished ability to acquire emotional information. Furthermore, lower emotional recall on the EVLT was associated with greater severity of clinically rated negative symptoms, whereas this correlation was non-significant with the CVLT. Thus, findings suggest that a neuropsychiatric patient group with known deficits in emotional memory displayed impairments on the EVLT and that these impairments were associated with greater severity of clinically rated affective symptoms.

Limitations and Future Directions

Limitations

There were several limitations of the current psychometric studies. First, the interval used to evaluate test–retest reliability (∼1 week) was shorter than what might be ideal to evaluate the temporal stability of a verbal learning test. Future studies will want to examine the temporal stability of EVLT scores across longer intervals (e.g., several months). Second, the EVLT's full potential for investigating the associative semantic network and mood-congruent memory theories may not have been realized in the current studies, since we did not assess certain disorders which might be expected to activate nodes for sadness and anxiety words (e.g., PTSD, Generalized Anxiety Disorder, Major Depressive Disorder). Mood congruency and associative network theories could be evaluated more precisely if the EVLT was administered after mood induction (e.g., showing a film clip), and variables such as semantic versus serial clustering or long-delay cued and free recall scores were compared across the four Target List conditions. Third, the recognition test scores were near ceiling for the majority of healthy individuals. It is possible that the high accuracy of recognition memory in healthy individuals prevented us from observing important individual differences related to emotional information processing in the samples obtained; however, clinical populations with deficits in encoding (e.g., Alzheimer's dementia) could be expected to display recognition deficits, and studies in such populations may be valuable for exploring the utility of the EVLT's recognition measures. Fourth, we found consistent evidence for preferential encoding and recall of happiness over sadness, anger, or anxiety words in healthy individuals. A potential alternative explanation for this result is that the EVLT word list contains disproportionately more unpleasant than pleasant words, thereby increasing the salience of the happiness stimuli. Perhaps, recall of the unpleasant categories would have been superior if equivalent numbers of positive and negative stimuli were included in the Target List. We expect that this is unlikely given that the literature on emotional memory indicates that healthy individuals have greater recall for positive than negative words when word lists are balanced for the total number of pleasant and unpleasant words (Matlin et al., 1979).

Future Directions

The EVLT has wide applicability for use in studying the etiology and maintenance of neuropsychiatric conditions. Our data on SZ suggest that the EVLT may be sensitive to the effects of illness on memory, as rate of learning on the EVLT discriminated individuals with SZ from CNs to a greater extent than that on the CVLT. EVLT scores also predicted clinically rated negative symptoms (e.g., anhedonia, restricted affect), while the CVLT did not, indicating that the EVLT is sensitive to detecting affective dysfunction. Although SZ is a disorder characterized by affective dysfunction, it is possible that performance on the individual word conditions (i.e., happiness, sadness, anger, anxiety) may be even more predictive of affective dysfunction in other psychiatric conditions. For example, differences in preferential recall and error rates (i.e., intrusions, repetitions) for unpleasant words have been shown to be valuable in differentiating patients with mood versus anxiety disorders. The EVLT may be even more valuable in this regard, as it includes words categories related to the mood abnormalities characteristic of multiple disorders (e.g., sadness, anger, anxiety). Given the potential contributions of emotional encoding, retrieval, and false memory in the etiology of disorders with affective disturbance, the EVLT may prove particularly useful in studying disorder-specific memory processes in conditions like PTSD, generalized anxiety disorder, major depression, and bipolar disorder.

The EVLT may also be valuable in examining specific neurological disorders. Clinicians could consider using the EVLT in instances of suspected focal lesions to areas impacting affective processes (e.g., amygdala) to determine whether greater impairment to emotional than non-emotional memory has resulted. The EVLT may also be valuable for examining the etiology of disorders with different patterns of neuropathology (e.g., cortical vs. subcortical dementias or left vs. right temporal lobe epilepsy), who could be expected to show different patterns of recall, recognition, and errors between the EVLT and non-emotional verbal learning tests, as well as differences in preferential recall across happiness, sadness, anger, and anxiety. Thus, the EVLT is expected to have both clinical and research applications when used alone or in conjunction with a non-emotional memory tests.

Supplementary Material

Supplementary material is available at Archives of Clinical Neuropsychology online.

Funding

This work was supported by an internal SITE grant from the University of Nevada Las Vegas to Daniel N. Allen.

Conflict of Interest

None declared.

Acknowledgements

The authors wish to acknowledge graduate students and research assistants who assisted in data collection: Shaida Jetha, Matthew Bommarito, Larry Arias, Stacey Cramer, Melinda Jorgensen, Lisa Barnes, Jennifer Goldstein, Amanda Villamar, Nicole Bensky, Carol Randall, Christina Armstrong, Sally Barney, Nick Thaler, Erik Ringdahl, Danielle Bello, Linda Frantom, Josh Caron, Brandon Park, Sylvia Ross, and Lisa Duke.

References

Amster
H.
Evaluative judgment and recall in incidental learning
Journal of Verbal Learning and Verbal Behavior
 , 
1964
, vol. 
3
 (pg. 
466
-
473
)
Bazin
N.
Perruchet
P.
Feline
A.
Mood congruence effect in explicit and implicit memory task: A comparison between depressed patients, schizophrenic patients and controls
European Psychiatry
 , 
1996
, vol. 
11
 (pg. 
390
-
395
)
Benedict
R. H. B.
Schretlen
D.
Groninger
L.
Brandt
J.
Hopkins Verbal Learning Test—Revised: Normative data and analysis of inter-form and test–retest reliability
Clinical Neuropsychologist
 , 
1998
, vol. 
12
 (pg. 
43
-
55
)
Blake
T. M.
Varnhagen
C. K.
Parent
M. B.
Emotionally arousing pictures increase blood glucose levels and enhance recall
Neurobiology of Learning and Memory
 , 
2001
, vol. 
75
 (pg. 
262
-
273
)
Blaney
P. H.
Affect and memory: A review
Psychological Bulletin
 , 
1986
, vol. 
99
 (pg. 
229
-
246
)
Bower
G. H.
Mood and memory
American Psychologist
 , 
1981
, vol. 
36
 (pg. 
129
-
148
)
Bower
G. H.
Christianson
S.
How might emotions affect learning?
The handbook of emotion and memory
 , 
1991
Hillsdale, NJ
Erlbaum
(pg. 
3
-
31
)
Bradley
B. P.
Mogg
K.
Williams
R.
Implicit and explicit memory for emotion congruent information in clinical depression and anxiety
Behaviour Research and Therapy
 , 
1995
, vol. 
33
 (pg. 
755
-
770
)
Bradley
M. M.
Greenwald
M. K.
Petry
M. C.
Lang
P. J.
Remembering pictures: Pleasure and arousal in memory
Journal of Experimental Psychology: Learning, Memory, and Cognition
 , 
1992
, vol. 
18
 (pg. 
379
-
390
)
Brandt
J.
The Hopkins Verbal Learning Test: Development of a new memory test with six equivalent forms
The Clinical Neuropsychologist
 , 
1991
, vol. 
5
 (pg. 
125
-
142
)
Butters
N.
Cermak
L. S.
Alcoholic Korsakoff's syndrome: An information-processing approach to amnesia
 , 
1980
New York
Academic Press
Butters
N.
Delis
D. C.
Lucas
J. A.
Clinical assessment of memory disorders in amnesia and dementia
Annual Review of Psychology
 , 
1995
, vol. 
46
 (pg. 
493
-
523
)
Coles
M. E.
Heimberg
R. G.
Memory biases in the anxiety disorders: Current status
Clinical Psychology Review
 , 
2002
, vol. 
22
 (pg. 
587
-
627
)
Colombel
F.
The processing of emotionally positive words according to the amount of accessible retrieval cues
International Journal of Psychology
 , 
2000
, vol. 
35
 (pg. 
279
-
286
)
Delaney
R. C.
Prevey
M. L.
Cramer
J.
Mattson
R. H.
the VA Epilepsy Cooperative Study #264 Research Group
Test-retest comparability and control subject data for the Rey-Auditory Verbal Learning Test and the Rey-Osterrieth/Taylor Complex Figures
Archives of Clinical Neuropsychology
 , 
1992
, vol. 
7
 (pg. 
523
-
528
)
Delis
D. C.
Kramer
J.
Kaplan
E.
Ober
B. A.
California Verbal Learning Test (CVLT) manual
 , 
1987
San Antonio, TX
Psychological Corporation
Delis
D. C.
Kramer
J. H.
Kaplan
E.
Ober
B. A.
California Verbal Learning Test- Second Edition (CVLT-II)
 , 
2000
San Antonio, TX
The Psychological Corporation
Denny
E. B.
Hunt
R. R.
Affective valence and memory in depression: Dissociation of recall and fragment completion
Journal of Abnormal Psychology
 , 
1992
, vol. 
101
 (pg. 
575
-
580
)
Dere
E.
Pause
B. M.
Pietrowsky
R.
Emotion and episodic memory in neuropsychiatric disorders
Behavioral Brain Research
 , 
2010
, vol. 
215
 (pg. 
162
-
171
)
Derogatis
L. R.
SCL-90-R: Administration, scoring, & procedures manual for the revised version
 , 
1983
Baltimore
Clinical Psychometric Research
Doerksen
S.
Shimamura
A. P.
Source memory enhancement for emotional words
Emotion
 , 
2001
, vol. 
1
 (pg. 
5
-
11
)
Dolcos
F.
LaBar
K.
Cabeza
R.
Dissociable effects of arousal and valence on prefrontal activity indexing emotional evaluation and subsequent memory: An event-related fMRI study
NeuroImage
 , 
2004
, vol. 
23
 (pg. 
64
-
74
)
Ferraro
F. R.
King
B.
Release from proactive interference with positive and negative words
The Psychological Record
 , 
2004
, vol. 
54
 (pg. 
199
-
206
)
First
M. B.
Spitzer
R. L.
Gibbon
M.
Williams
J. B. W.
Structured Clinical Interview for DSM–IV–TR Axis I Disorders—Patient Edition (SCID-I/P 2/2001 Revision)
 , 
2001
New York
Biometrics Research Department, New York State Psychiatric Institute
Forgas
J. P.
The role of emotion in social judgments: An introductory review and an affect infusion model (AIM)
European Journal of Social Psychology
 , 
1994
, vol. 
24
 (pg. 
1
-
24
)
Geffen
G.
Butterworth
P.
Geffen
L. B.
Test-retest reliability of a new form of the Auditory Verbal Learning Test (AVLT)
Archives of Clinical Neuropsychology
 , 
1994
, vol. 
9
 (pg. 
303
-
316
)
Gold
J. M.
Hahn
B.
Strauss
G. P.
Waltz
J. A.
Turning it upside down: Areas of preserved cognitive function in schizophrenia
Neuropsychology Review
 , 
2009
, vol. 
19
 
3
(pg. 
294
-
311
)
Gold
J. M.
Waltz
J. A.
Matveeva
T. M.
Kasanova
Z.
Strauss
G. P.
Herbener
E. S.
, et al.  . 
Negative symptoms in schizophrenia result from a failure to represent the expected value of rewards: Behavioral and computational modeling evidence
Archives of General Psychiatry
 , 
2012
, vol. 
69
 (pg. 
129
-
138
)
Hamann
S. B.
Cahill
L.
Squire
L. R.
Emotional perception and memory in amnesia
Neuropsychology
 , 
1997
, vol. 
11
 (pg. 
104
-
113
)
Hamann
S. B.
Ely
T. D.
Grafton
S. T.
Kilts
C. D.
Amygdala activity related to enhanced memory for pleasant and aversive stimuli
Nature Neuroscience
 , 
1999
, vol. 
2
 (pg. 
289
-
293
)
Hanley
J. A.
McNeil
B. J.
A method of comparing the areas under the receiver operating characteristic curves derived from the same cases
Radiology
 , 
1983
, vol. 
148
 
3
(pg. 
839
-
843
)
Hayward
J. A.
Strongman
K. T.
Emotional words, free recall, and laterality
Bulletin of the Psychonomic Society
 , 
1987
, vol. 
25
 (pg. 
161
-
162
)
Heinrichs
R. W.
Zakzanis
K. K.
Neurocognitive deficit in schizophrenia: A quantitative review of the evidence
Neuropsychology
 , 
1998
, vol. 
12
 (pg. 
426
-
445
)
Herbener
E. S.
Emotional memory in schizophrenia
Schizophrenia Bulletin
 , 
2008
, vol. 
34
 (pg. 
875
-
887
)
Hosmer
D. W.
Lemeshow
S.
Applied logistic regression (2nd ed.)
 , 
2000
New York
John Wiley & Sons
Howe
M.
Malone
C.
Mood-congruent true and false memory: Effects of depression
Memory
 , 
2011
, vol. 
19
 
2
(pg. 
192
-
201
)
Jermann
F.
Van der Linden
M.
Laurencon
M.
Schmitt
B.
Recollective experience during recognition of emotional words in clinical depression
Journal of Psychopathology and Behavioral Assessment
 , 
2009
, vol. 
31
 
1
(pg. 
27
-
35
)
Kucera
H.
Francis
W.
Computational analysis of present-day American English
 , 
1967
Providence, RI
Brown University Press
LaBar
K. S.
Phelps
E. A.
Arousal-mediated memory consolidation: Role of the medial temporal lobe in humans
Psychological Science
 , 
1998
, vol. 
9
 (pg. 
490
-
493
)
Lezak
M.
Neuropsychological Assessment, 3rd ed
 , 
1995
New York
Oxford University Press
Libkumen
T. M.
Stabler
C. L.
Otani
H.
Arousal, valence, and memory for detail
Memory
 , 
2004
, vol. 
12
 (pg. 
237
-
247
)
Lishman
W. A.
Selective factors in memory: II Affective disorder
Psychololgical Medicine
 , 
1972
, vol. 
2
 (pg. 
248
-
253
)
Lisman
J. E.
Grace
A. A.
The hippocampal-VTA loop: Controlling the entry of information into long-term memory
Neuron
 , 
2005
, vol. 
46
 (pg. 
703
-
713
)
Maddock
R. J.
Frein
S. T.
Reduced memory for the spatial and temporal context of unpleasant words
Cognition and Emotion
 , 
2009
, vol. 
23
 (pg. 
96
-
117
)
Mather
M.
Carstensen
L. L.
Aging and motivated cognition: The positivity effect in attention and memory
Trends in Cognitive Sciences
 , 
2005
, vol. 
9
 (pg. 
496
-
502
)
Matlin
M. W.
Stang
D. J.
Gawron
V. J.
Freedman
A.
Derby
P. L.
Evaluative meaning as a determinant of spew position
Journal of General Psychology
 , 
1979
, vol. 
100
 (pg. 
3
-
11
)
McGaugh
J. L.
Cahill
L.
Interaction of neuromodulatory systems in modulating memory storage
Behavior Brain Research
 , 
1997
, vol. 
83
 (pg. 
31
-
38
)
Overall
J. E.
Gorham
D. R.
The Brief Psychiatric Rating Scale
Psychological Reports
 , 
1962
, vol. 
10
 (pg. 
799
-
812
)
Packard
M. G.
Cahill
L.
Affective modulation of multiple memory systems
Current Opinion in Neurobiology
 , 
2001
, vol. 
11
 (pg. 
752
-
756
)
Phelps
E. A.
LaBar
K. S.
Spencer
D. D.
Memory for emotional words following unilateral temporal lobectomy
Brain and Cognition
 , 
1997
, vol. 
35
 (pg. 
85
-
109
)
Rey
A.
L'Examin clinique en psychologie
 , 
1964
Paris
Universitaire de France
Robinson
M. D.
Clore
G. L.
Belief and feeling: Evidence for an accessibility model of emotional self-report
Psychological Bulletin
 , 
2002
, vol. 
128
 (pg. 
934
-
960
)
Rusting
C. L.
Personality, mood, and cognitive processing of emotional information: Three conceptual frameworks
Psychological Bulletin
 , 
1998
, vol. 
124
 (pg. 
165
-
196
)
Rychlak
J. F.
Saluri
R. E.
Affective assessment in the learning of names by fifth- and sixth-grade children
Journal of Genetic Psychology
 , 
1973
, vol. 
123
 (pg. 
251
-
261
)
Siddiqui
A. P.
Unsworth
N.
Investigating the role of emotion during the search process in free recall
Memory and Cognition
 , 
2011
, vol. 
39
 (pg. 
1387
-
1400
)
Snyder
K. A.
Harrison
D. W.
Affective Auditory Verbal Learning Test
Archives of Clinical Neuropsychology
 , 
1997
, vol. 
12
 
5
(pg. 
477
-
482
)
Strauss
G. P.
Allen
D. N.
The experience of positive emotion is associated with the automatic processing of positive emotional words
Journal of Positive Psychology
 , 
2006
, vol. 
1
 
3
(pg. 
150
-
159
)
Strauss
G. P.
Allen
D. N.
Emotional intensity and categorization ratings for emotional and nonemotional words
Cognition and Emotion
 , 
2008
, vol. 
22
 (pg. 
114
-
133
)
Strauss
G. P.
Allen
D. N.
Positive and negative emotions uniquely capture attention
Applied Neuropsychology
 , 
2009
, vol. 
16
 (pg. 
144
-
149
)
Strauss
G. P.
Allen
D. N.
Jorgensen
M. L.
Cramer
S. L.
Test-retest reliability in Stroop and Emotional Stroop Tasks: An investigation of color-word and picture-word versions
Assessment
 , 
2005
, vol. 
12
 
3
(pg. 
330
-
337
)
Strauss
G. P.
Frank
M. F.
Waltz
J. A.
Kasanova
Z.
Herbener
E. S.
Gold
J. M.
Deficits in positive reinforcement learning and uncertainty-driven exploration are associated with distinct aspects of negative symptoms in schizophrenia
Biological Psychiatry
 , 
2011
, vol. 
69
 (pg. 
424
-
431
)
Strauss
G. P.
Gold
J. M.
A new perspective on anhedonia in schizophrenia
American Journal of Psychiatry
 , 
2012
, vol. 
169
 (pg. 
364
-
373
)
Strauss
G. P.
Wilbur
R. C.
Warren
K. R.
August
S. M.
Gold
J. M.
Anticipatory vs. consummatory pleasure: What is the nature of hedonic deficits in schizophrenia?
Psychiatry Research
 , 
2011
, vol. 
187
 (pg. 
36
-
41
)
Swets
J. A.
Signal Detection Theory and ROC Analysis in Psychology and Diagnostics
 , 
1996
 
Mahwah, NJ: Laurence Erlbaum Associates
Talmi
D.
Moscovitch
M.
Can semantic relatedness explain the enhancement of memory for emotional words?
Memory and Cognition
 , 
2004
, vol. 
32
 
5
(pg. 
742
-
751
)
Tarsia
M.
Power
M. J.
Sanavio
E.
Implicit and explicit memory biases in mixed anxiety-depression
Journal of Affective Disorders
 , 
2003
, vol. 
77
 (pg. 
213
-
225
Watkins
P. C.
Implicit memory bias in depression
Cognition and Emotion
 , 
2002
, vol. 
16
 (pg. 
381
-
402
)
Watson
D.
Clark
L. A.
On traits and temperament: General and specific factors of emotional experience and their relation to the five-factor model
Journal of Personality
 , 
1992
, vol. 
60
 (pg. 
441
-
476
)
Wechsler
D.
Weschler Memory Scale for Adults, 3rd ed
 , 
1997
San Antonio, TX
The Psychological Corporation
Wittmann
B. C.
Schiltz
K.
Boehler
C. N.
Duzel
E.
Mesolimbic interaction of emotional valence and reward improves memory formation
Neuropsychologia
 , 
2008
, vol. 
46
 (pg. 
1000
-
1008
)
Woods
S. P.
Scott
J. C.
Conover
E.
Marcotte
T. D.
Heaton
R. K.
Grant
I.
Test–retest reliability of component process variables within the Hopkins Verbal Learning Test–Revised
Assessment
 , 
2005
, vol. 
12
 (pg. 
96
-
100
)