Abstract

Perceptual load of an attended task influences the processing of irrelevant background stimuli. In a series of behavioral, functional magnetic resonance (fMRI) and electroencephalography (EEG) experiments we examined the influence of working memory (WM) load related to a relevant visual stimulus on the processing of irrelevant backgrounds. We further addressed two open questions about the mechanism of load-dependent modulation: (i) is this modulation dependent on regional activity (i.e. phasic)? (ii) At what processing stage does this modulation take place? Load was manipulated by a WM task and concurrently the processing of irrelevant visual objects was assessed with fMRI and EEG. To examine the dependency of this modulation on intrinsic activity, we varied the activity level of visual areas by presenting objects with different levels of degradation. Activity in the lateral occipital complex (LOC) increased with object visibility and was phasically modulated by WM load. Event related potentials revealed that this phasic modulation occurred ∼170 ms after stimulus onset, indicative of an early selection under high load. The results indicate a phasic modulatory effect of WM load on visual object processing in the LOC that is comparable to the effects found for perceptual load manipulations.

Introduction

A common visual scene is a complex arrangement of many different objects. Usually only a few of these objects are relevant, and attentional selection is used to reduce the influence of irrelevant information. For example, a distracting background must be suppressed to extract the desired information of a foreground task.

Attentional load is a top-down factor that is known to modulate visual selection of relevant and irrelevant coexisting stimuli. Load can be categorized as either pure perceptual load — the increase in the amount or complexity of simultaneously presented stimuli — or processing load, where the visual stimulation remains constant but the amount of processing demands increase. For both concepts of load it was demonstrated that high load on the relevant stimulus reduces processing of concomitant irrelevant visual information (Lavie and Tsal, 1994; Lavie, 1995). For example, the effect of distracting irrelevant information on reaction times (RTs) to the relevant stimuli can be reduced either by increasing the display set size of items or by varying the required processing of the relevant stimuli in an identical display.

This was further exemplified in an functional magnetic resonance (fMRI) study in which the processing load on presented letters was modulated and the effect on the blood oxygen level dependent (BOLD) signal to an irrelevant motion stimulus, that was simultaneously presented, was assessed. Under high load on the letter task a reduced BOLD signal in the motion related extrastriate visual cortex was observed, in accord with the assumption that the high load reduces the processing of the irrelevant stimuli (Rees et al., 1997a).

The selective mechanism of attention depends on the load of the relevant task, but also on the cognitive control system which is necessary to select relevant and irrelevant stimuli. Therefore, the cognitive control functions were recently embedded in the load theory (Lavie et al., 2004). This theory predicts impaired selection of relevant and irrelevant stimuli in cases where the function of the cognitive control system is impaired and as a consequence an increase in distractor processing. To test this prediction subsequent studies have employed working memory (WM) tasks to manipulate the cognitive control system (de Fockert et al., 2001; Yi et al., 2004). As predicted, increasing the WM demands resulted in increased distracter processing, and not decreased distractor processing as shown for the processing load manipulations (de Fockert et al., 2001). The results were interpreted as showing that cognitive load manipulations have the opposite effect as perceptual load manipulations, namely increasing cognitive load is assumed to result in increased distractor processing. However, this effect was observed in experimental designs where the WM task was added to a selective attention task and thus it might have interfered with selection rather than directly influencing the processing of the irrelevant stimuli. Therefore, the experiments tested the effect of a third task on the selection between two other tasks, whereas in the experiments on perceptual load, the load was related to the relevant task.

On the contrary, in the present study we tested the hypothesis that load imposed through WM can reduce processing of concomitant irrelevant stimuli in a similar manner as predicted by the theory of perceptual or processing load, if the WM load is imposed on the selected relevant stimuli rather than in addition to the selection.

We manipulated the WM load of the relevant stimuli and assessed the processing of the irrelevant background information that was presented simultaneously. In addition to this conceptual issue, we focused our experiments on two issues that are essential to understand the neural mechanism of attentional modulation and that have remained open in previous experiments.

First, due to the temporal limitations of fMRI the previous studies did not aim at revealing any information about the time-point of selection. This is important, because Lavie's theory of load (Lavie, 1995) and previous imaging studies (Rees et al., 1997a) suggest early selection under high load, but only a technique with high temporal resolution, such as electroencephalography (EEG) can adequately test this hypothesis for WM load manipulations. For pure perceptual load an effect on an early ERP component was reported, indicating that a manipulation of the visual stimuli resulted in a suppression of irrelevant information at an early processing stage (Handy et al., 2001). This has not been examined for WM load. Unfortunately, EEG has a poor spatial resolution and it is thus advantageous to combine EEG with fMRI using identical experimental paradigms.

Second, load-dependent modulation can be additive (e.g. increased background activity) or multiplicative (e.g. a gain effect) (Rees et al., 1997b). A tonic, additive mechanism is independent of local activity and results in a change in the intercept, i.e. there is a general amount of activity that is added. A phasic, multiplicative gain mechanism also depends on (i.e. interacts with) local activity and results in a change of the slope, i.e. the more activity present without attention, the larger the increase evoked by attention. Both mechanisms were assumed to be conceptually distinct and to differ in the influence on sensory processing (Hillyard et al., 1998). The use of a categorical design in the previous fMRI studies did not allow precise assessment of stimulus–response functions, which is helpful in characterizing the mechanism of attentional modulation (i.e. tonic/additive versus phasic/multiplicative).

These questions were addressed in a linked series of behavioral, EEG and fMRI experiments. Load was quantitatively manipulated by varying the WM demand in an n-back task. Participants had to compare a centrally presented letter with the last letter (1-back) or the last but one letter (2-back). Thus, the load was defined as the amount of information that had to be maintained for further processing (Baddeley, 1992). The relevant letters were presented overlaid on irrelevant background images for 1 s and participants had to process and attend only the letters (see Fig. 1). Incidental processing of the irrelevant background images was assessed behaviorally by a surprise recognition task for the images and neurophysiologically by event-related potentials (ERPs) and fMRI.

Figure 1.

Visibility levels of background images and experimental design. Example of visual stimuli used as irrelevant background for the relevant n-back task. Five levels of scrambling were used and are labeled with respect to the amount of original image phase information. Participants had to process the letters according to an n-back rule. For each block (10 stimuli) the n-back instruction (here 2-back) and scrambling of background images remained constant (here 75%). Participants had to indicate the appearance of a target letter by a button press (here the repetition of the ‘A’). Over the experiment n-back and scrambling levels were presented in a pseudorandom order with a block length of 20 s. The actual n-back level was indicated by a number (‘1’ or ‘2’) before each block.

Figure 1.

Visibility levels of background images and experimental design. Example of visual stimuli used as irrelevant background for the relevant n-back task. Five levels of scrambling were used and are labeled with respect to the amount of original image phase information. Participants had to process the letters according to an n-back rule. For each block (10 stimuli) the n-back instruction (here 2-back) and scrambling of background images remained constant (here 75%). Participants had to indicate the appearance of a target letter by a button press (here the repetition of the ‘A’). Over the experiment n-back and scrambling levels were presented in a pseudorandom order with a block length of 20 s. The actual n-back level was indicated by a number (‘1’ or ‘2’) before each block.

To disambiguate tonic from phasic modulation of background processing, we also manipulated the visibility of the irrelevant object stimuli (see Fig. 1) to vary activity of ventral visual stream areas, the hypothesized site of modulation. The use of real-world object images as the irrelevant background stimuli was motivated by their relevance to real-life situations and the well-known systems involved in object processing along the ventral visual pathway (Ungerleider et al., 1994; Ishai et al., 1999) including area V4 (Kastner et al., 1999) and lateral occipital complex (LOC) (Malach et al., 1995; Grill-Spector et al., 2000; Kourtzi and Kanwisher, 2000). LOC, in particular, is sensitive to the degree of image scrambling and activity is related to the amount of object information (Grill-Spector et al., 1998).

In summary, our aim was to probe the processing of the irrelevant background information by the increase of neural activity with increasing object visibility under different levels of WM load to better characterize top-down modulation in the temporal and spatial domain.

We aimed to answer the following research questions: (i) does WM load modulate ventral visual areas? (ii) What is the mechanism? (Phasic or tonic?) (iii) Which areas in the ventral visual stream are modulated? (iv) When does the modulation occur? (Early or late?) (v) What is the behavioral consequence of this manipulation?

In the behavioral experiment we tested the degree of processing of the irrelevant background object images in a surprise recognition task. We hypothesized that processing, and thus recognition, is negatively affected by load of the foreground WM task. In the fMRI experiment we hypothesized that (i) WM activates areas in the frontal and parietal cortex; (ii) there is a positive relationship between object visibility and activation in ventral occipito-temporal cortex (e.g. LOC); and (iii) a phasic (i.e. activity dependent) modulation of these areas leads to a decreased responsiveness (i.e. reduced gain) under high load conditions. In the EEG experiment, designed to answer the question ‘when’ the modulation occurs, we hypothesized that in accord with Lavie's theory of load (Lavie, 1995), a modulation occurs at early processing stages, i.e. not later than 200 ms after stimulus onset.

Materials and Methods

Volunteers

Ten healthy subjects participated in the behavioral study (mean age 31 years, range 26–42 years, 5 females). Fourteen participants performed in the fMRI study (mean age 29 years, range 22–33 years, 4 females) and 10 healthy participants in the EEG study (mean age 26 years, range 23–36 years, 4 females). All subjects were right-handed, and had normal or corrected-to-normal vision. The study was approved by the local ethics committee and subjects gave written informed consent prior to the experiment. Before each experiment participants were trained on the n-back task in the two levels ‘1-back’ and ‘2-back’ that were used in all experiments.

Task and Stimuli

The individual stimuli used in the experiment always consisted of a background image with a superimposed letter. Volunteers had to perform a WM task with the letters, which were always presented centrally (size 0.3° × 0.5° of visual angle). A cue (‘1’ or ‘2’) preceding the active block indicated whether a 1-back or 2-back task should be performed. Only the letters A, B, C, D, E, G were used and only targets had to be indicated by a button press. Background images that contained defined degrees of noise were presented simultaneously with the letters (size 10° × 7.5°).

For the behavioral experiment 200 different images (size 640 × 480 pixels; 24-bit color depth) were selected from a database of natural scenes with animals and plants to ensure that in each trial a different image could be presented. In the MRI and EEG experiment only 20 different images were used because no surprise memory task was performed.

Visibility of images was modulated by scrambling them according to a method described previously (Rainer et al., 2001). In short, each image was transformed into the amplitude and phase component by a Fourier transform for each RGB color channel. Different degrees of scrambling (i.e. noise) patterns were generated by manipulating a fraction of the image phase before transforming the amplitude and phase components back into image space. The phase information was randomized by choosing the phase values at random from the interval (Allison et al., 1994). Distinct levels of scrambling were generated using linear interpolation and the stimulus levels (0, 25, 50, 75 and 100%) reflected the amount of original phase information. Because the manipulation was limited to the phase-space the amplitude spectra remained identical across the scrambling levels (including brightness and luminance), which is not the case if noise is added to the stimuli. Finally, the edges of each image were smoothed using a 28 pixel full-width at half-maximum (FWHM) isotropic Gaussian kernel (Fig. 1). This ensured that the images were smoothly embedded into the background and therefore transient neural activity evoked by the outer edges was minimized.

The task design was identical for all experiments except the recognition task that was included only in the behavioral experiment. In the recognition part all images were presented with full visibility (i.e. 0% scrambling); half of the images had been shown before and the other half were presented for the first time. For comparison the false positive rate was assessed by calculating the percentage of ‘new’ images that were rated as ‘previously seen’. Only values above this baseline were assumed to result from reliable memory formation.

Each participant underwent three successive sessions (each with a duration of 13 min) in the MR scanner, the EEG lab or during the behavioral study. Visibility and n-back levels were presented in a 5 × 2 factorial design (5 visibility levels × 2 n-back levels), resulting in a total of 10 different conditions. The order of conditions was randomized and each condition was presented twice per session. Each block lasted for 20 s and consisted of the presentation of 10 stimuli with identical n-back and visibility level followed by a rest period (20 s) in which the cue (‘1’ or ‘2’) for the next block was presented (Fig. 1). Images and overlaid letters were presented together for 1 s with an interstimulus interval (ISI) of 1 s and participants had to maintain fixation on the letters. Letters were presented in a pseudorandom order to establish the occurrence of always three targets in each block that appeared in pseudorandomized order. Order of background visibility and n-back levels were randomized for each session and images were presented in a pseudorandom order to prevent a repetition of images within and between consecutive blocks.

Stimuli were presented on a gray background controlled by a PC that ensured synchronization with the MR scanner and EEG amplifier using the software ‘Presentation’ (http://www.neurobehavioralsystems.com). An LCD projector projected the stimuli onto a screen positioned on top of the head coil which was viewed by the subjects through a mirror (10° × 15° field of view). In the EEG and the behavioral experiment volunteers sat in a comfortable arm chair in front of a 15 in. LCD monitor. The distance between head and monitor was adjusted to produce identical stimulus sizes as in the fMRI experiment. Participants entered the responses by pressing buttons on a MR-compatible device.

Functional MRI Data Analysis

Functional MRI in the second experiment was performed on a 1.5 T MR system (Siemens Vision) with a gradient–echo EPI T2* sensitive sequence in 32 contiguous axial slices (3 mm thickness with 1 mm gap, TR 2.6 s, TE 40 ms, flip angle 90°, field of view 210 × 210 mm2, matrix 64 × 64). For display purposes, a high-resolution (1 × 1 × 1 mm voxel size) structural MRI was acquired for each participant using a standard three-dimensional T1-weighted FLASH sequence.

Image processing and statistical analysis were carried out using SPM99 (http://www.fil.ion.ucl.ac.uk/spm). All volumes were realigned to the first volume, spatially normalized to a standard EPI template (SPM99), and smoothed using a 10 mm FWHM isotropic Gaussian kernel. The structural volume was coregistered to the functional scans by normalizing it to a T1-weighted template in the same space as the template used to normalize the functional data set.

Data analysis was performed by modeling each block with consistent n-back and visibility level as box-car functions convolved with a hemodynamic response functions (HRF). Regression coefficients for all regressors (10 different regressors from ‘1-back–0% visibility’ to ‘2-back–100% visibility’) were estimated using least squares within SPM99 (Friston et al., 1995). A high-pass filter with a cut-off period of 120 s and a low-pass filter (Gaussian envelope FWHM of 4 s) were used.

For the group analysis, a second level analysis was performed, treating inter-subject variability as a random effect. The threshold for statistical comparisons adopted was P < 0.05 (corrected across the whole brain for multiple comparisons). For regions in which we had an a priori hypothesis a small volume correction (SVC) was performed. The correction was based on a search volume of 900 mm3 in accordance with the expected size of the structures and was used in bilateral LOC, left ventral and medial temporal lobe.

EEG Data Analysis

EEG was recorded from 34 channels referred to the vertex (BrainVision System) and re-referenced offline to the nose. Vertical and horizontal EOG was recorded from above versus below the left eye and from the outer canthi of the eyes, for recognizing eye movements. Electrode resistance was kept below 5 kΩ. EEG and EOG were amplified in the range from 0.03 Hz ( = 5 s time constant) to 120 Hz and A/D converted at 500 Hz sampling rate. Data were segmented from 100 ms before stimulus onset until 900 ms after it. The data were checked for artifacts caused by blinks and eye movements (vertical and horizontal), for zero lines, out-of-range values and fast amplitude shifts >100 μV within 2 ms and slow drifts. Whereas the latter artifacts led to rejection of a trial, EEGs were corrected for blinks and eye-movement artifacts by subtracting both EOG channels weighted by their transmission coefficient (Gratton et al., 1983). The EEG was low-pass filtered offline at 35 Hz.

Only non-target trials were averaged separately for the 10 conditions to exclude influence of target detection and motor response. Mean amplitude measures of the P1 (80–150 ms), N1 (150–200 ms) and P2 (200–300 ms) were obtained and entered into a two-factor analysis of variance (ANOVA) with Huynh–Feldt epsilon correction applied as necessary. For the detection of eye movements, mean activity of unsigned values from both EOG channels was calculated for the whole segment and analyzed separately (three-factor ANOVA with factors channel, load and background visibility).

Source location for the N1 component was estimated using LORETA (Pascual-Marqui et al., 1994). Grand average ERP for low- and high-load conditions were used as input and the three-dimensional map within the time range of the N1 (150–250 ms) was estimated. Comparisons of localization of fMRI effects and N1 source density were possible due to the use of an identical anatomical reference template (Montreal Neurological Institute, MNI305) (Evans et al., 1993).

Behavioral Results

In the behavioral experiment we examined the consequences of load manipulation on the processing of the irrelevant object images using a surprise recognition task after the study phase. During the study phase participants performed the n-back task on centrally presented letters overlaid on the task- irrelevant background images (Fig. 1; see above for details). The letters were overlaid on images that were noisy due to various degree of scrambling (0, 25, 50, 75% and 100 %; see Fig. 1 and above for details of scrambling).

A two-factor repeated-measures ANOVA (factors WM load and visibility) was used to investigate the effect of increasing WM load on performance. In the WM task participants made more errors for the 2-back condition than for the 1-back condition [F(1,9) = 34.0, P < 0.05] and RTs were slower for the 2-back condition [F(1,4) = 9.4, P < 0.05], indicating the successful manipulation of WM load. In the test phase of the experiment participants unexpectedly had to judge images (without WM task) as ‘previously seen’ (i.e. during WM phase) or ‘new’. To assess the degree of object processing independent of physical attributes, we used images without scrambling for the recognition task. As predicted, we observed a main effect of WM load on memory performance [percent correct low load: 31.4 ± 4.2 versus high load: 19.8 ± 3.1; main effect of load: F(1,9) = 29.2, P < 0.05] (Fig. 2). Furthermore, recognition accuracy increased with object image visibility (i.e. less scrambling) [main effect of visibility: F(4,36) = 16.9, P < 0.05]. Most importantly, this increase was steeper under low load conditions [interaction load × visibility F(4,36) = 3.9, P < 0.05].

Figure 2.

Recognition performance for the background images. The degree of object encoding of the irrelevant background images was estimated in a surprise recognition task. Increasing the visibility resulted in increasing memory but later memory was impaired when images were seen under high WM load (2-back, gray). For comparison the false positive rate (i.e. the percentage of ‘new’ images that were rated as ‘previously seen’) is indicated. Error bars represent standard errors.

Figure 2.

Recognition performance for the background images. The degree of object encoding of the irrelevant background images was estimated in a surprise recognition task. Increasing the visibility resulted in increasing memory but later memory was impaired when images were seen under high WM load (2-back, gray). For comparison the false positive rate (i.e. the percentage of ‘new’ images that were rated as ‘previously seen’) is indicated. Error bars represent standard errors.

Functional MRI Results

To identify the site of load-dependent modulation of object processing we investigated 14 additional volunteers in an identical paradigm with fMRI. As in the first experiment the behavioral data demonstrated the effective manipulation of workload with an increase in RT [F(1,13) = 12.99, P < 0.05] and error rate [F(1,13) = 25.79, P < 0.05] for the high WM load conditions.

To evaluate the main effect of the WM manipulation we identified cortical areas in which the BOLD signal increased with WM load. In several areas, which have previously been associated with WM performance, activity was greater during the high load task (2-back) than during the low load task (1-back). These areas included bilateral frontal eye fields (FEF), the superior parietal lobe and prefrontal cortex (Table 1). Furthermore, activity in the anterior part of the left ventral temporal cortex increased with the load of the relevant task, a finding that is in accord with previous studies on letter processing (Allison et al., 1994; Polk et al., 2002).

Table 1

Results from the fMRI random effects group analysis. Main effect of working memory load, i.e. 2-back > 1-back

Region
 
Coordinates (x,y,z)
 
t-score
 
L frontal eye fields −21 −6 51 9.9* 
R frontal eye fields 27 0 57 7.0+ 
R thalamus 9 −3 0 9.6* 
L thalamus −15 0 −3 7.6* 
L superior parietal −6 −63 48 9.0* 
R superior parietal 9 −63 48 8. 6* 
Anterior cingulated 0 21 42 6.6+ 
R cerebellum 36 −60 −33 6.5+ 
L cerebellum −36 −51 −36 7.1+ 
L dorsolateral frontal −45 24 30 6.3+ 
R dorsolateral frontal 45 30 33 5.7+ 
R frontopolar cortex 27 54 −15 8.0* 
L ventral temporal
 
−45 −39 −12
 
6.24*
 
Region
 
Coordinates (x,y,z)
 
t-score
 
L frontal eye fields −21 −6 51 9.9* 
R frontal eye fields 27 0 57 7.0+ 
R thalamus 9 −3 0 9.6* 
L thalamus −15 0 −3 7.6* 
L superior parietal −6 −63 48 9.0* 
R superior parietal 9 −63 48 8. 6* 
Anterior cingulated 0 21 42 6.6+ 
R cerebellum 36 −60 −33 6.5+ 
L cerebellum −36 −51 −36 7.1+ 
L dorsolateral frontal −45 24 30 6.3+ 
R dorsolateral frontal 45 30 33 5.7+ 
R frontopolar cortex 27 54 −15 8.0* 
L ventral temporal
 
−45 −39 −12
 
6.24*
 
*

P < 0.05 corrected for multiple comparisons;

+

P < 0.001 uncorrected

Next, we identified cortical areas involved in the processing of the background images by investigating the correlation of the BOLD signal and image visibility. Several occipital and temporal areas showed a linear BOLD signal increase with increasing visibility (i.e. decreasing scrambling) (Table 2). Within these areas we dissociated tonic and phasic modulation of visual processing. Tonic modulation, i.e. the high load of the foreground task, resulted in a general suppression irrespective of the image visibility, and was found in dorsal occipital cortex and within the anterior fusiform gyrus (Table 3). A prominent effect of increasing image visibility was also found in bilateral LOC. Most importantly, only in LOC was this BOLD signal increase reduced under high load conditions, resulting in a significant interaction of visibility and load. This interaction reflects the phasic modulation of background image processing in the right LOC by load (Fig. 3). A similar interaction effect was also observed within left LOC but failed to reach our predetermined threshold criterion. Another interaction was observed in the right medial temporal lobe (Table 3). In this region the BOLD signal did not increase as a function of image visibility under high load conditions.

Figure 3.

Object-selective regions modulated by working memory load. fMRI effects of the interaction of background visibility with load demonstrating an increasing BOLD signal in bilateral LOC (lower section) that is steeper for the low load conditions (1-back, blue) reflecting a phasic modulation of LOC activity. Tonic modulation was found in the anterior part of the left ventral temporal cortex (upper section). For display purposes activations are thresholded at P < 0.05. The y-axis represents mean beta weights (regression coefficients that corresponds to the BOLD signal) and error bars represent standard errors.

Figure 3.

Object-selective regions modulated by working memory load. fMRI effects of the interaction of background visibility with load demonstrating an increasing BOLD signal in bilateral LOC (lower section) that is steeper for the low load conditions (1-back, blue) reflecting a phasic modulation of LOC activity. Tonic modulation was found in the anterior part of the left ventral temporal cortex (upper section). For display purposes activations are thresholded at P < 0.05. The y-axis represents mean beta weights (regression coefficients that corresponds to the BOLD signal) and error bars represent standard errors.

Table 2

Results from the fMRI random effects group analysis. Main effect of image visibility, i.e. increasing BOLD signal with increasing image visibility

Region
 
Coordinates (x,y,z)
 
t-score (main effect image visibility)
 
L dorsal occipital −36 −87 12 20.9* 
R dorsal occipital 36 −87 12 15.7* 
L LOC −39 −78 −9 15.3* 
R LOC 45 −81 −3 15.6* 
L fusiform gyrus −33 −51 −21 14.4* 
R fusiform gyrus
 
36 −63 −12
 
14.4*
 
Region
 
Coordinates (x,y,z)
 
t-score (main effect image visibility)
 
L dorsal occipital −36 −87 12 20.9* 
R dorsal occipital 36 −87 12 15.7* 
L LOC −39 −78 −9 15.3* 
R LOC 45 −81 −3 15.6* 
L fusiform gyrus −33 −51 −21 14.4* 
R fusiform gyrus
 
36 −63 −12
 
14.4*
 
*

P < 0.05 corrected for multiple comparisons.

Table 3

Results from the fMRI random effects group analysis. Interaction effect of image visibility with WM- load indicating a phasic modulation. Main effect of working memory load, i.e. 1-back > 2-back, indicating a tonic modulation within areas sensitive to background image visibility

Region
 
Coordinates (x,y,z)
 
t-score(interaction image visibility × WM load)
 
Phasic modulation:   
L LOC −42 −75 −12 2.8+ 
R LOC 45 −75 −9 3.1* 
R posterior hippocampus 36 −33 −9 3.6* 
Tonic modulation  t-score (main effect WM load 1-back> 2-back) 
L dorsal occipital −21 −99 12 7.4* 
R dorsal occipital 33 −90 15 5.5* 
L anterior fusiform gyrus −30 −36 −21 4.3* 
R anterior fusiform gyrus
 
33 −33 −21
 
7.4*
 
Region
 
Coordinates (x,y,z)
 
t-score(interaction image visibility × WM load)
 
Phasic modulation:   
L LOC −42 −75 −12 2.8+ 
R LOC 45 −75 −9 3.1* 
R posterior hippocampus 36 −33 −9 3.6* 
Tonic modulation  t-score (main effect WM load 1-back> 2-back) 
L dorsal occipital −21 −99 12 7.4* 
R dorsal occipital 33 −90 15 5.5* 
L anterior fusiform gyrus −30 −36 −21 4.3* 
R anterior fusiform gyrus
 
33 −33 −21
 
7.4*
 
*

P < 0.05 corrected;

+

uncorrected.

The results of the fMRI data indicated that load of the foreground task multiplicatively modulated activity related to irrelevant background information processing (i.e. an interaction) within right LOC. Given that LOC is involved in the early stimulus-driven bottom-up analysis of objects (Kourtzi and Kanwisher, 2000), this result is consistent with an early modulation under high WM load. However, due to the time resolution of fMRI the time point of the load effect cannot be assessed by the fMRI results.

ERP Results

To directly test the hypothesis of an early attentional effect we performed the same experiment with additional 10 volunteers using EEG and examined the ensuing ERP components. As in the previous experiment the behavioral data revealed an increase in RT [F(1,9) = 6.70, P < 0.05] and error rate [F(1,9) = 10.91, P < 0.05] for the high load (2-back) WM condition. The presentation of the visual stimuli with the n-back letter and the background image elicited a P1–N1–P2 complex at occipital and parietal electrodes. P1 (80–150 ms) amplitude was pronounced at occipital sites and showed a main effect of background visibility [electrode O1: F(4,36) = 3.9, P < 0.05]. However, the increase of P1 amplitude across visibility levels was not different between low and high load conditions [F(1,9) = 2.1, NS]. Similarly, the P2 amplitude (200–300 ms), increased with background image visibility [electrode PO8: F(4,36) = 4.51, P < 0.05], but did not show an interaction with load. In contrast, the N1 (150–200 ms) with a peak amplitude over right lateral occipital sites, showed a significantly steeper increase in relation to background image visibility under low load condition than under high load conditions (interaction visibility × load, electrode PO8: F(4,36) = 3.2, P < 0.05].

In addition to the scalp topography of the N1 (Fig. 4a,b), we estimated the current source density of the N1 (Fig. 4d) using a maximum smoothness approach as implemented in low-resolution electro-magnetic tomography (LORETA) (Pascual-Marqui et al., 1994). The maximum of activity for the N1 component was located in the right LOC (46, −67, 1), almost identical to the area showing a significant interaction in fMRI.

Figure 4.

N1 amplitude modulated by working memory load. Grand mean ERPs averaged across low load (1-back; blue) and high load (2-back; red) conditions (a) topographic representation (b) of the differences between high and low load in the time range of the N1 component (150–200 ms). In analogy to the modulation of BOLD signal in LOC, the increase of N1 amplitudes related to image visibility was modulated by load (c). The estimate of the source of the N1 component (d) and fMRI effects (e) overlaid on identical structural T1-weighted template image (z = −9 mm). Error bars represent standard errors.

Figure 4.

N1 amplitude modulated by working memory load. Grand mean ERPs averaged across low load (1-back; blue) and high load (2-back; red) conditions (a) topographic representation (b) of the differences between high and low load in the time range of the N1 component (150–200 ms). In analogy to the modulation of BOLD signal in LOC, the increase of N1 amplitudes related to image visibility was modulated by load (c). The estimate of the source of the N1 component (d) and fMRI effects (e) overlaid on identical structural T1-weighted template image (z = −9 mm). Error bars represent standard errors.

To statistically test for a possible lateralization of this effect, a separate ANOVA was computed [factors: electrodes (O1, O2 and PO7/PO8), hemisphere (left, right), load (1-back, 2-back) and image visibility (five levels)]. The results show no reliable difference between the hemispheres [F(1,9) = 0.1, NS) but the identical interaction effect of visibility × load [F(4,36), = 2.7, P < 0.05].

To rule out the confound that load-dependent modulation was affected by eye movements, we compared vertical and horizontal eye movements precisely measured during EEG recordings between conditions. Mean activities in both EOG channels were not different between conditions [n-back: F(1,9) = 0.25, NS; visibility F(4,36) = 1.7, NS, interaction F(4,36) = 1.7, NS]. Furthermore, we compared mean activity within the baseline (100 ms before stimulus onset). The identical ANOVA design as used for the lateralization test revealed no differences for any factor.

Discussion

All three studies show an influence of WM load on visual processing of concomitant irrelevant stimuli. The fMRI and ERP results indicate a phasic modulatory effect of WM load on visual object processing in the LOC that occurs on average 170 ms after stimulus onset. Behaviorally, this effect is paralleled by greater impairment of recognition memory for the background images that had been presented under high WM load.

In accord with our hypothesis, BOLD signal increase with image visibility in right LOC was shown to be load dependent. Load dependency can either be additive or multiplicative. This distinction is important, because an additive effect (e.g. a tonic increase in background activity) is independent of the intrinsic activity of the area and may reflect a task-dependent, but stimulus-independent mechanism. In terms of attentional modulation, a multiplicative effect, reflecting an activity-dependent mechanism, indicating an interaction of the modulatory influence with the intrinsic activity, is important because it reflects the dynamic adaptation to the ongoing processing. The coexistence of both attentional mechanisms has been demonstrated in several neuroimaging (Rees et al., 1997b) and single-unit experiments (Chelazzi et al., 1993; Miller and Desimone, 1994). In accord with these findings our results show a tonic modulation in several extrastriate visual areas. In the anterior part of the left ventral temporal cortex, the BOLD signal was increased during high WM load irrespective of the visibility of the irrelevant images. The location is in accord with previous studies on letter processing (Allison et al., 1994; Polk et al., 2002), indicating that this effect can be related to an attentional enhancement of the processing of the relevant foreground letters. On the contrary, in the visual system only LOC showed a stimulus-specific, phasic modulation, leading to a reduced increase of BOLD signal under high WM load. In other words, load per se does not change activity in LOC, because in this case the BOLD signal for fully scrambled pictures (0% visibility levels in Fig. 4) should also differ. Thus, the observed effect suggests a gain setting mechanism of top-down afferents (e.g. from frontal or parietal cortex) on projections from earlier visual areas (e.g. V4) to LOC, i.e. not a direct modulation of LOC activity, but a modulation of the input to LOC (Kastner and Ungerleider, 2000).

Attentional modulation of ventral visual stream areas has been demonstrated in single-cell recordings (Moran and Desimone, 1985) and fMRI studies (Kastner et al., 1999). The latter study compellingly demonstrated that this effect is mediated by parietal and/or frontal areas, by showing frontal and temporal activation when expecting a ‘to be attended’ stimulus. In all these studies the modulatory effect leads to attention-dependent increases of activity in ventral visual areas. By occupying parietal and frontal areas with a WM task, as in our experiment, the ability of these structures to exert this positive modulatory influence on ventral visual areas should decrease. This is in accord with our observation that high load resulted in a reduced increase of BOLD signal with image visibility and a reduced N1 amplitude.

The excellent temporal resolution of the EEG data allowed us to show that high WM load resulted in a modulation of the N1 component between 150 and 200 ms after stimulus presentation. Analogous to the modulation of LOC activity in fMRI, the load-dependent modulation of the N1 amplitude was dependent on the level of LOC processing as expressed by the interaction between load and visibility.

Several ERP studies have demonstrated that the occipito-temporal N1 reflects a visual discriminative process related to non-spatial visual object features (Luck and Hillyard, 1995; Hillyard and Anllo-Vento, 1998; Eimer, 2000; Vogel and Luck, 2000; Hopf et al., 2002), which is in accord with the proposed function of LOC (Malach et al., 1995). In accord with our fMRI results and previous ERP findings (Luck et al., 1994; Hillyard et al., 1998), the observed interaction of N1 amplitude between load and visibility indicated a phasic modulation of attention on object processing in LOC, i.e. a gain control rather than a bias signal. The time point of this effect further indicated that the interference from the n-back task originated from the attentional demands rather than from WM or executive processes engaged by the relevant task which were reported to affect latter ERP components (McEvoy et al., 1998; Berti et al., 2000). Previous studies on perceptual load reported effects on the earlier P1 ERP component (Handy et al., 2001). This discrepancy might be related to the difference in the load manipulation of both studies. In the Handy et al., study perceptual load was manipulated, whereas in our study WMy load was manipulated. Interestingly, our data showed a dependency of the P1 amplitude and the visibility of the background images, linking this component to early perceptual processing. However, WM load has no effect on the P1 amplitude but affected the latter N1 component.

The topographical distribution of the estimated three-dimensional electrical activity of the N1 component shows maxima at the occipito-temporal junctions, in accord with a source in LOC. The maximum was fitted within the right hemisphere. However, the statistical comparison showed that the load-dependent modulation of the N1 was not significantly different between the left and right hemisphere (see also Fig. 4b). Therefore, the ERP results showed that the load-dependent effect was present in the LOC of both hemispheres. Thus, our ERP data are in accord with an early attentional suppression of perceptual processing of irrelevant background information at 150–200 ms in LOC.

Early modulation of LOC activity is in agreement with previous studies implicating LOC in the early stimulus-driven, bottom-up analysis of objects rather than in later stages of processing that depend on memory for object identification (Kourtzi and Kanwisher, 2000). Importantly, the early modulation found in the EEG data renders the possibility that the modulation observed in fMRI is linked to a late re-entrant activation of LOC (Martinez et al., 1999; Handy et al., 2001) unlikely.

The measurement of eye movements demonstrated that the phasic modulation of N1 amplitude, and most probably also the related fMRI activity in LOC, was not affected by different fixation accuracies between high and low load conditions.

All our three studies clearly show a modulatory influence of WM on visual processing of irrelevant stimuli. Therefore, the results are in accord with the theory of perceptual or processing load (Lavie, 1995) but are in apparent contrast with previous studies that showed the opposite effect of WM load (de Fockert et al., 2001). In the latter study, an increase in WM load led to an increase in distractor processing, rather than to a decrease as observed in our experiments (de Fockert et al., 2001). However, this apparent controversy can be resolved and, more importantly, it can be shown that a single mechanism can account for these seemingly different effects. In the de Fockert et al. study participants had to maintain digits in WM during a selective attention task. In this setup, high WM load impaired the attentional task, i.e. selection. As a consequence of impaired selection, processing of irrelevant visual stimuli was less attenuated. It is important to note that this is unlikely to be a direct effect of the WM task on the irrelevant visual stimuli. It is more likely that this effect is mediated through the attentional task. In other words, in de Fockert et al.'s paradigm, WM load impairs attentional selection, and thus the suppression of irrelevant background stimuli is less effective. However, in our study, WM load directly suppresses processing of irrelevant background stimuli. The relevant difference of our study is the fact that the WM load is imposed on the relevant stimulus and does not represent an additional task to attentional selection. Thus, both studies show that WM load reduces resources available for another task (i.e. attentional selection in de Fockert et al. and ventral visual processing in our task).

Outside the visual system a phasic modulation by load was found in the right medial temporal lobe (posterior hippocampus). The interaction of load with visibility in this area resulted in an increase of activity with visibility (i.e. decreasing amount of scrambling) only under low load, but not under high load. The known role of the hippocampus in memory formation (Squire and Zola-Morgan, 1991; Eichenbaum, 2001) suggests that the impaired recognition memory performance under high load as observed in our behavioral experiment might be related to this effect. Alternatively, this modulation could reflect a downstream effect from ventral visual areas (e.g. LOC) which are interconnected with the medial temporal lobe (Lavenex and Amaral, 2000).

In conclusion, the identical phasic modulation of LOC activity in fMRI and of the N1 amplitude over occipito-temporal cortex at 150–200 ms in the EEG is indicative of an early attentional filter at LOC for irrelevant visual background information under high processing load which is dependent on the degree of interfering information.

We thank the Physics and Methods group at the Cognitive Neuroscience Laboratory for help with MR scanning. We further thank Andreas Kleinschmidt and Geraint Rees for comments on an earlier draft of this paper. This work was supported by the Volkswagen-Stiftung, DFG and BMBF.

References

Allison T, McCarthy G, Nobre A, Puce A, Belger A (
1994
) Human extrastriate visual cortex and the perception of faces, words, numbers, and colors.
Cereb Cortex
 
4
:
544
–554.
Baddeley A (
1992
) Working memory.
Science
 
255
:
556
–559.
Berti S, Geissler H, Lachmann T, Mecklinger A (
2000
) Event-related brain potentials dissociate visual working memory processes under categorial and identical comparison conditions.
Brain Res Cogn Brain Res
 
9
:
147
–155.
Chelazzi L, Miller EK, Duncan J, Desimone R (
1993
) A neural basis for visual search in inferior temporal cortex.
Nature
 
363
:
345
–347.
de Fockert JW, Rees G, Frith CD, Lavie N (
2001
) The role of working memory in visual selective attention.
Science
 
291
:
1803
–1806.
Eichenbaum H (
2001
) The hippocampus and declarative memory: cognitive mechanisms and neural codes.
Behav Brain Res
 
127
:
199
–207.
Eimer M (
2000
) Effects of face inversion on the structural encoding and recognition of faces. Evidence from event-related brain potentials.
Brain Res Cogn Brain Res
 
10
:
145
–158.
Evans AC, Collins DL, Mills DR, Brown ED, Kelly RL, Peters TM (
1993
) 3D statistical neuroanatomical models from 305 MRI volumes.
Proc IEEE Nuclear Science Symposium and Medical Imaging
 
1–3
:
1813
–1817.
Friston KJ, Holmes AP, Poline JB, Grasby PJ, Williams SC, Frackowiak RS, Turner R (
1995
) Analysis of fMRI time-series revisited.
Neuroimage
 
2
:
45
–53.
Gratton G, Coles MG, Donchin E (
1983
) A new method for off-line removal of ocular artifact.
Electroencephalogr Clin Neurophysiol
 
55
:
468
–484.
Grill-Spector K, Kushnir T, Hendler T, Edelman S, Itzchak Y, Malach R (
1998
) A sequence of object-processing stages revealed by fMRI in the human occipital lobe.
Hum Brain Mapp
 
6
:
316
–328.
Grill-Spector K, Kushnir T, Hendler T, Malach R (
2000
) The dynamics of object-selective activation correlate with recognition performance in humans.
Nat Neurosci
 
3
:
837
–843.
Handy TC, Soltani M, Mangun GR (
2001
) Perceptual load and visuocortical processing: event-related potentials reveal sensory-level selection.
Psychol Sci
 
12
:
213
–218.
Hillyard SA, Anllo-Vento L (
1998
) Event-related brain potentials in the study of visual selective attention.
Proc Natl Acad Sci USA
 
95
:
781
–787.
Hillyard SA, Vogel EK, Luck SJ (
1998
) Sensory gain control (amplification) as a mechanism of selective attention: electrophysiological and neuroimaging evidence.
Phil Trans R Soc Lond B Biol Sci
 
353
:
1257
–1270.
Hopf JM, Vogel E, Woodman G, Heinze HJ, Luck SJ (
2002
) Localizing visual discrimination processes in time and space.
J Neurophysiol
 
88
:
2088
–2095.
Ishai A, Ungerleider LG, Martin A, Schouten JL, Haxby JV (
1999
) Distributed representation of objects in the human ventral visual pathway.
Proc Natl Acad Sci USA
 
96
:
9379
–9384.
Kastner S, Pinsk MA, De Weerd P, Desimone R, Ungerleider LG (
1999
) Increased activity in human visual cortex during directed attention in the absence of visual stimulation.
Neuron
 
22
:
751
–761.
Kastner S, Ungerleider LG (
2000
) Mechanisms of visual attention in the human cortex.
Annu Rev Neurosci
 
23
:
315
–341.
Kourtzi Z, Kanwisher N (
2000
) Cortical regions involved in perceiving object shape.
J Neurosci
 
20
:
3310
–3318.
Lavenex P, Amaral DG (
2000
) Hippocampal-neocortical interaction: a hierarchy of associativity.
Hippocampus
 
10
:
420
–430.
Lavie N (
1995
) Perceptual load as a necessary condition for selective attention.
J Exp Psychol Hum Percept Perform
 
21
:
451
–468.
Lavie N, Tsal Y (
1994
) Perceptual load as a major determinant of the locus of selection in visual attention.
Percept Psychophys
 
56
:
183
–197.
Lavie N, Hirst A, de Fockert JW, Viding E (
2004
) Load theory of selective attention and cognitive control.
J Exp Psychol Gen
 
133
:
339
–354.
Luck SJ, Hillyard SA (
1995
) The role of attention in feature detection and conjunction discrimination: an electrophysiological analysis.
Int J Neurosci
 
80
:
281
–297.
Luck SJ, Hillyard SA, Mouloua M, Woldorff MG, Clark VP, Hawkins HL (
1994
) Effects of spatial cuing on luminance detectability: psychophysical and electrophysiological evidence for early selection.
J Exp Psychol Hum Percept Perform
 
20
:
887
–904.
Malach R, Reppas JB, Benson RR, Kwong KK, Jiang H, Kennedy WA, Ledden PJ, Brady TJ, Rosen BR, Tootell RB (
1995
) Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex.
Proc Natl Acad Sci USA
 
92
:
8135
–8139.
Martinez A, Anllo-Vento L, Sereno MI, Frank LR, Buxton RB, Dubowitz DJ, Wong EC, Hinrichs H, Heinze HJ, Hillyard SA (
1999
) Involvement of striate and extrastriate visual cortical areas in spatial attention.
Nat Neurosci
 
2
:
364
–369.
McEvoy LK, Smith ME, Gevins A (
1998
) Dynamic cortical networks of verbal and spatial working memory: effects of memory load and task practice.
Cereb Cortex
 
8
:
563
–574.
Miller EK, Desimone R (
1994
) Parallel neuronal mechanisms for short-term memory.
Science
 
263
:
520
–522.
Moran J, Desimone R (
1985
) Selective attention gates visual processing in the extrastriate cortex.
Science
 
229
:
782
–784.
Pascual-Marqui RD, Michel CM, Lehmann D (
1994
) Low resolution electromagnetic tomography: a new method for localizing electrical activity in the brain.
Int J Psychophysiol
 
18
:
49
–65.
Polk TA, Stallcup M, Aguirre GK, Alsop DC, D'Esposito M, Detre JA, Farah MJ (
2002
) Neural specialization for letter recognition.
J Cogn Neurosci
 
14
:
145
–159.
Rainer G, Augath M, Trinath T, Logothetis NK (
2001
) Nonmonotonic noise tuning of BOLD fMRI signal to natural images in the visual cortex of the anesthetized monkey.
Curr Biol
 
11
:
846
–854.
Rees G, Frith CD, Lavie N (
1997
a) Modulating irrelevant motion perception by varying attentional load in an unrelated task.
Science
 
278
:
1616
–1619.
Rees G, Frackowiak R, Frith C (
1997
b) Two modulatory effects of attention that mediate object categorization in human cortex.
Science
 
275
:
835
–838.
Squire LR, Zola-Morgan S (
1991
) The medial temporal lobe memory system.
Science
 
253
:
1380
–1386.
Ungerleider LG, Haxby JV (
1994
) ‘What’ and ‘where’ in the human brain.
Curr Opin Neurobiol
 
4
:
157
–165.
Vogel EK, Luck SJ (
2000
) The visual N1 component as an index of a discrimination process.
Psychophysiology
 
37
:
190
–203.
Yi DJ, Woodman GF, Widders D, Marois R, Chun MM (
2004
) Neural fate of ignored stimuli: dissociable effects of perceptual and working memory load.
Nat Neurosci
 
7
:
992
–996.