## Abstract

The spatio-temporal distribution of brain activity as revealed by non-invasive functional imaging helps to elucidate the neuronal encoding and processing strategies required by complex cognitive tasks. We investigated visual short-term memory for objects, places and conjunctions in humans using event-related time-resolved functional magnetic resonance imaging that permitted segregation of encoding, retention and retrieval phases. All conditions were accompanied by the activation of a widespread network of parietal and prefrontal areas during the retention phase, but this retention-related activity showed additional modulations depending on task instructions. These modulations confirmed a posterior — anterior and right — left dissociation for spatial versus non-spatial memory and revealed that conjunction memory does not rely on a linear addition of the component processes.

## Introduction

The neuronal mechanisms subserving the integration of multiple aspects of stimuli in visual STM (see Abbreviations for all definitions) (Fuster and Alexander, 1971; Mishkin and Delacour, 1975; Cohen et al., 1997; Courtney et al., 1997; Fuster, 1998; Prabhakaran et al., 2000), have been investigated in both humans and non-human primates. These studies all suggest a pivotal role of the prefrontal cortex (Miller, 2000), the activation of which has also been shown to reflect individual performance levels and objective memory load in working memory tasks (Callicott et al., 1999; Rypma and D’Esposito, 1999; Prabhakaran et al., 2000). In laboratory settings, STM is frequently studied using DMS tasks (Elliott and Dolan, 1999). Single-unit recordings in behaving monkeys have revealed neurons around the principal sulcus of the lateral prefrontal cortex that increase their firing during the delay between the presentation of sample and test stimuli (Fuster and Alexander, 1971; Funahashi et al., 1989; Miller et al., 1996). fMRI studies in humans have revealed neuronal activation in prefrontal areas during STM for faces (Courtney et al., 1997) and locations (Courtney et al., 1998a). The results of these studies suggested a domain specific dissociation of areas involved in STM: retention of objects engaging more ventrally and retention of spatial relations engaging more dorsally located regions (Courtney et al., 1998b). An alternative interpretation is that the differential engagement of ventral and dorsal subdivisions of lateral prefrontal cortex reflects different processing modes, such as maintenance on the one hand and manipulation of retained information on the other, rather than the nature of the remembered cues (D’Esposito et al., 1998, 1999; Owen et al., 1998, 1999; Nystrom et al., 2000; Postle et al., 2000).

In addition to prefrontal cortex, IT and PP cortex have also been assigned functions in STM. In IT, neurons exhibit delay activity when monkeys perform DMS tasks preferentially for the retention of object-specific features (Miller et al., 1993), while PP neurons seem to be activated more during the retention of spatial relations (Constantinidis and Steinmetz, 1996). Relatively little is known about how these areas cooperate with the prefrontal cortex in STM. In order to address this issue, one requires information about the spatial and temporal distribution of activity associated with encoding, retention and retrieval of information in both domains.

Despite the rather limited temporal resolution of fMRI, evaluation of single trial responses (event-related fMRI) can provide some information about the temporal sequence of processing (Zarahn et al., 1999) and about the coherence of processes occurring simultaneously in different areas (Goebel et al., 1998a). We therefore applied event-related fMRI to investigate visual STM in a design that allowed us to separate in time the encoding, retention, retrieval and response phases. We used a DDT rather than a conventional DMS task, because the latter is not balanced with respect to attention and response preparation for matching and non-matching trials. In experiment 1, subjects performed DDT tasks on series of different objects (Postle and D’Esposito, 1999) or identical objects in different places (‘where’, see Fig. 1A). Functional images were acquired at high rate (TR = 1 s) in order to allow for a separation of activity that is evoked by the presentation of the stimuli from the sustained activity that is related to retention. In experiment 2, visual sample stimuli consisted of four natural objects that were sequentially presented in an imaginary two-dimensional grid (Fig. 2A). After the delay period, subjects had to decide whether one object presented as test stimulus at one of the positions of the imaginary grid matched one of the objects (Postle and D’Esposito, 1999), locations (Postle and D’Esposito, 1999), or both (‘what & where’) of the preceding sample stimulus. This design permitted the comparison of cortical activation patterns associated with retention of conjunctions and single features (Rypma and D’Esposito, 1999), respectively. As most human subjects attempt to use verbal descriptions in order to retain information about natural objects, we added a control experiment using abstract stimuli.

## Materials and Methods

### Subjects, Stimulation and Behavioral Task

We recruited five right-handed healthy volunteers (four male, one female; mean age 30.8 years, range 27–36 years) for experiment 1, 10 (eight male, two female; mean age 29.2 years, range 24–39 years) for experiment 2 and eight (six male, two female; mean age 27.2 years, range 21–35 years) for the non-verbal control experiment, who gave their informed consent to participate in the study. The reported experiments were undertaken with the understanding and written consent of each subject and in accordance with the Declaration of Helsinki. Three volunteers participated in all experiments. Experiment 1 was preceded by a training session which allowed subjects to undertake as many trials as necessary to familiarize themselves with the structure and timing of the task. Visual stimuli (for details about stimulus content and sequence see legends to Figs 1 and 2) were delivered under PC control to an LCD projector (EIKI LC-6000). The image was back-projected onto a frosted screen positioned at the foot end of the scanner.

In experiment 2, three different instructions (‘what’, ‘where’ and ‘what and where’) were presented at the beginning of each trial in a pseudo-randomized sequence. The structure of all trials was identical. Four out of 24 different fruit drawings were presented in 4 s (1 s per item) in 1 out of 8, 12 or 16 spatial positions, depending on subjective performance. After a delay of 12 s, a test stimulus was presented for 4 s, after which the subjects had to respond as above. Prior to experiment 2, each subject had to perform a training session in front of a PC screen with at least 240 trials, for which reaction times were recorded. The training sessions were balanced for match and non-match trials for each task type. Depending on the task instruction, which was randomized for each trial, the task-relevant and the task-irrelevant (spatial versus object) information was also balanced. The scanner sessions were then designed for each individual subject by selecting 3 × 12 trials from the training session that had yielded reaction times within one standard deviation of the individual mean, again balanced for task-relevant and task-irrelevant information.

### fMRI Measurements and Analysis

fMRI data were acquired with a 1.5 T Magnetom Vision MRI scanner (Siemens, Erlangen, Germany) using a gradient echo EPI sequence [1 volume = 6 (experiment 1)/16 (experiment 2, control) axial slices; TR = 1000 ms (experiment 1)/2000 ms (experiment 2, control); TE = 60 (experiment 2, control)/69 ms (experiment 1); FA = 90°; FOV = 210 × 210 mm2; voxel size = 1.6 × 1.6 × 5.0 (experiment 1) or 3.2 × 3.2 × 5 (experiment 2, control) mm3] for fMRI. Each scan comprised the acquisition of 128 (experiment 1) or 256 (experiment 2, control) volumes. In experiment 1, the slices covered large parts of the occipital, temporal and frontal lobes (z-coordinate range from –5 to 25 at y = –50 and from 15 to 45 at y = 20, Talairach coordinates), whereas in experiment 2 and the control they covered the whole cerebrum. A T1-weighted 3-D MP RAGE scan was recorded in each session (magnetization-prepared rapid acquisition gradient echo, TR = 9.7 ms, TE = 4 ms, FA = 12°, matrix = 256 × 256, voxel size 1.0 × 1.0 × 1.0 mm3).

In experiment 1, subjects underwent four scans of each condition, yielding an overall of 16 ‘what’ and 16 ‘where’ trials. Experiment 2 consisted of three functional scans with a pseudo-random sequence of task types, yielding 12 trials of every task type (‘what’, ‘where’, ‘what and where’).

The statistical analysis was based on the application of the multiple regression analysis to time-series of task-related functional activation (Friston et al., 1995). These analytical tools were implemented in BrainVoyager 4.4 (Goebel et al., 1998a,b; Dierks et al., 1999).

Talairach transformation (Trojano et al., 2000) was performed for the complete set of functional data of each subject, yielding a 4-D data representation (volume time-course: 3 × space, 1 × time). Prior to statistical analysis, the time-series of functional images was aligned in order to minimize the effects of head movements. The central volume of the time-series was used as a reference volume to which all other volumes were registered, using a 3-D motion correction that estimates the three translation and three rotation parameters of rigid body transformation. Data pre-processing furthermore comprised spatial smoothing with a Gaussian kernel (FWHM = 8 mm), the removal of linear trends and (in experiment 2 and the control) temporal lowpass filtering (lowpass: 48 per functional run of 256 volumes).

The high resolution T1-weighted anatomical 3-D data set of a template brain (courtesy of the Montreal Neurological Institute) was used for the surface reconstruction and flatmap representation of both hemispheres.

The GLMs of the ‘what’ and ‘where’ sessions of experiment 1 were computed from the 20 (five subjects, four scans per subject) z-normalized volume time-courses. The signal values during the encoding, delay and retrieval phases were considered effects of interest. The GLMs of experiment 2 were computed from 30 volume time-courses (10 subjects, three scans per subject). GLMs were computed for the encoding (two volumes), early delay (two volumes), delay (six volumes) and retrieval (two volumes) phases, and the task type (‘what’, ‘where’, ‘what and where’) was considered the effect of interest. The corresponding predictors, obtained by convolution of an ideal box-car response (assuming a value of 1 for the volumes of task presentation and a value of 0 for the remaining time points) with a linear model of the hemodynamic response (Boynton et al., 1996), were used to build the design matrix of the experiment. The global level of the signal time-courses in each session was considered to be a confounding effect and a fixed effects analysis was employed. To analyze the effects of conditions compared to baseline and contrasts between conditions, 3-D individual and group statistical maps were generated by associating each voxel with the F-value corresponding to the specified set of predictors and calculated on the basis of the least mean squares solution of the GLM. Statistical results were then visualized through projecting 3-D statistical maps on the flattened surface reconstruction of the MNI template. Effects were only shown if, considering an F distribution with n1 and n2 degrees of freedom (n1 = number of orthogonal predictors and n2 = number of time samples – n1 –1), the associated P-value yielded P′ < 10−2, corrected for multiple comparisons (RC maps), or P < 10−3, uncorrected (superposition maps) and if a minimum cluster size of 100 mm3 was reached.

### RC Maps

For significantly activated voxels, the relative contributions, RC, between two selected sets of conditions in explaining the variance of a voxel time-course were computed as

$\mathit{RC}\ {=}\ (\mathit{b}_{1}\ {\mbox{--}}\ \mathit{b}_{2})/(\mathit{b}_{1}\ {+}\ \mathit{b}_{2})$
where bi is the sum of the estimates of the standardized regression coefficients of all conditions included in set i (Trojano et al., 2000). The RC index was visualized with the pseudo-color scales shown in the respective figures. In experiment 1, the first four time points (convolved with the hemodynamic function) were taken to represent the encoding predictor. This was contrasted with the late delay predictor (time points 9–12 convolved with the hemodynamic function) in order to minimize the influence of encoding-related signal modulation on the delaypredictor (Fig. 1). In experiment 2, the encoding, early delay, delay and retrieval predictors were defined as outlined above and separated by condition, yielding 12 different predictors. The RCs of the encoding and delay predictors of each condition were presented in six separate maps (Fig. 3). All predictors were used for the statistical analysis of significant differences between conditions in the four phases, based on the t-test of differences between the individual (for each subject) beta weights of the three respective predictors (‘what’, ‘where’, ‘what and where’) after removal of serial correlation (Bullmore et al., 1996) (Table 2).

### Superposition Maps

In experiment 2, each of the effects of interest (the three task types) was given a color of the RGB system. In order to visualize all three effects on a single flatmap, colors were superimposed and areas of overlap (cortical regions showing an activation during more than one condition) received the appropriate mixed color (superposition maps). Time-courses of experiment 1 and experiment 2 were computed by event-related averaging of the mean time-courses of indicated clusters over all 20 volume time-courses, using the same voxels (in Talairach space) for all subjects and all repetitions.

## Results

### Behavioral Data

Subjects performed at high accuracy (>85%) in all experiments. In experiment 1, reaction times of correct responses did not differ significantly between the ‘what’ and ‘where’ conditions (P = 0.68, Mann–Whitney U-test). In experiment 2, where the response had to be delayed until the disappearance of the stimulus, accuracy rather than reaction time was used as measure of task performance. Accuracy rates (‘what and where’, 86%; ‘where’, 87%; ‘what’, 89%) did not differ significantly between conditions (χ2, P = 0.78).

### Experiment 2

#### RC Maps

The RC maps for the delay period tended to show a higher degree of separation of the predictors. The ‘what’ versus ‘where’ contrast yielded distinct clusters of ‘where’-related activation in the parietal lobes and DLPFC bilaterally and of ‘what’-related activation in left VLPFC and INS. The ‘what and where’ versus ‘where’ map showed preponderance of ‘where’ activation in the right SPL and IPL and of ‘what and where’ activation in the left VLPFC and bilateral INS. The ‘what and where’ versus ‘what’ map showed that bilateral superior parietal and frontal midline activity had a higher contribution from ‘what and where’ trials.

## Discussion

The time-courses and maps presented in Figures 1 and 2 confirm that the design of the study permitted a separation of encoding-, retention- and retrieval-related brain activity. Experiment 1 revealed distinct time-courses in IT and prefrontal cortex during visual STM. While activity in IT peaked at ~5 s — the commonly assumed time-to-peak of the BOLD signal (Boynton et al., 1996) — after the onset of a visual stimulus and returned to baseline immediately afterwards, activity in prefrontal cortex rose more slowly after the presentation of the target stimulus, remained high during retention and peaked after the test stimulus. Experiment 2, which used a lower sampling rate than experiment 1 (2 instead of 1 s) but covered the entire brain, revealed a similar pattern of time-courses (Fig. 4) and additional retention-related activation in the parietal lobe, predominantly for the ‘where’ and conjunction conditions. Retention-related activity can thus be regarded as being largely confined to the frontal and parietal lobes (Fig. 2D), while the first response to a new target stimulus, which can be seen as the neural correlate of stimulus encoding, was observed in IT (Figs 1C,D and 2C). Retrieval-related activity was also present in the temporal, frontal and parietal lobes (Fig. 4).

### Comparison of Spatial, Non-spatial and Conjunction Memory

#### Frontal activation

Beyond the segregation of the phases of a typical STM task, our design permitted the comparison of cortical activation patterns associated with spatial, non-spatial and conjunction STM. Particularly during delay (and most clearly in the left hemisphere), the RC maps (Fig. 3) show a separation of more ventral prefrontal areas (anterior IFG and MFG) involved in the ‘what’ and more dorsal prefrontal areas (posterior MFG and SFG) involved in the ‘where’ condition, while ‘what and where’ recruits parts of both regions. From this perspective, our results might seem to confirm a clear-cut segregation of dorsal ‘where’ and ventral ‘what’ areas in prefrontal cortex. Yet the superposition maps (Fig. 2) and the time-courses of the prefrontal areas (Fig. 4) even more so, reveal that the issue is more complex than this. Even areas with high RC values in favor of one condition still show a considerable and very stable departure from baseline during the other conditions. For example, the left INS, albeit displaying a significantly higher activation for ‘what’ and ‘what and where’ compared to ‘where’ during delay, still shows a clear difference from baseline for the ‘where’ condition. Conversely, the ‘what’ condition was accompanied by a consistent activation of right SFG in all phases, although this area clearly showed a more prominent modulation for the spatial conditions. This shows that if only activations that survive a very stringent threshold are considered, some aspects of the distributed cortical activity subserving complex cognitive processes might be lost, as has also been observed for categorical visual processing (Ishai et al., 1999). It thus seems that a wide range of prefrontal areas is recruited during visual STM, regardless of the characteristics to be remembered and that the additional processing required by the precise nature of the task leads to the differential modulation of subsets of this network. Moreover, the role of the prefrontal cortex in STM is clearly not only confined to functions during the delay period. Most areas of PFC that showed task-related activity in the present study responded even more strongly in the encoding phase (although they differed from pure encoding areas in that their activity remained significantly higher than baseline during the entire delay period) and showed a second peak for retrieval (Fig. 4). Our data show that PFC is active during all phases of visual STM. Furthermore they confirm a non-exclusive DLPFC/ VLPFC dissociation for spatial and non-spatial memory. This is consistent with the claim that while there is considerable overlap of delay activity in lateral prefrontal areas, the level of participation is generally higher for the SFS region bilaterally in spatial and for left inferior and mid-frontal cortex in non-spatial tasks (Courtney et al., 1998a; Haxby et al., 2000).

Most identified frontal areas were also active during the different phases of the conjunction task (‘what and where’). Yet conjunction-related activation was clearly not an addition of the activations related to the component processes. Some ventro-lateral prefrontal areas showed higher activity for ‘what’ than conjunction and some dorsolateral areas for ‘where’ than conjunction. However, in areas where one of the component tasks evoked the highest activation, conjunction always took the second place. This would be compatible with a theory that regards not the addition, but the recruitment of parts of the networks for the components as the likely neuronal mechanism for the solution of conjunction tasks. The only area that consistently displayed the highest BOLD signal change for conjunction versus the component tasks was found in the mesial superior frontal cortex bilaterally (extending from the SMA to the anterior cingulate). This region has been identified as being a central element of the network for feature integration in working memory in a number of previous studies (Mitchell et al., 2000; Prabhakaran et al., 2000).

The superposition map of experiment 2 (Fig. 2) shows that the dissociation of lateral PFC into more dorsal areas that participated more in the spatial conditions and more ventral areas that participated more in the non-spatial conditions tended to be present in both hemispheres. Yet significantly higher activation for ‘what’ versus ‘where’ during delay was only found in the left inferior and mid-frontal cortex, whereas the SFG and parietal activation was significantly higher for ‘where’ than ‘what’ in both hemispheres (Fig. 3A and Table 2). Predominantly left hemispheric ‘what’ activation during maintenance has recently been described by Postle and D’Esposito (Postle and D’Esposito, 2000) who proposed that the difference between maintaining spatial and non-spatial information might be hemispheric. However, of the previous studies that included a direct comparison between spatial and non-spatial working memory, only some have reported a left lateralized prefrontal activation for the non-spatial task (Courtney et al., 1998a), while a number of studies have found bilateral activation in mid-frontal cortex (McCarthy et al., 1996; Belger et al., 1998). Prefrontal activation for the spatial task was either bilateral (Courtney et al., 1998a) or predominantly on the right (McCarthy et al., 1996; Belger et al., 1998). In terms of lateralization, the most consistent finding of both the previous and the present studies seems to be the predominantly left-hemispheric IFG activation for the non-spatial task. The fact that the activation of left IFG could be confirmed in our control experiment suggests that it is not exclusively associated with the verbal components of working memory.

#### Parietal Activation

The parietal retention activation seems to be linked to the spatial component of STM, because it was mainly observed in the ‘where’ and conjunction conditions of experiment 2 and much less prominent in the ‘what’ task that was based on the same stimulus material. Thus, our results confirm the view that spatial STM involves coactivation of PP and prefrontal cortical areas (Chafee and Goldman-Rakic, 1998). Primate PP is known to play a key role in visuomotor integration (Sakata et al., 1997; Goodale, 1998; Quintana and Fuster, 1999), the spatial analysis of the visual scene (Colby and Goldberg, 1999) and the integration of spatial information from different sensory modalities (Andersen, 1997). Posterior parietal areas LIP, 7a and 7ip of non-human primates have been shown to be active during delayed saccade tasks (Andersen et al., 1990; Chafee and Goldman-Rakic, 1998) and DMS paradigms (Constantinidis and Steinmetz, 1996). A preponderance of parietal over DLPFC activity during visuospatial STM in humans has recently been described by Pochon et al. (Pochon et al., 2001) who found a prominent DLPFC activation only when the preparation of a sequential movement was required. While our data suggest that the STM-related DLPFC activation also occurs in the delay phase of simple response tasks, we can confirm their finding of the important role of the parietal-premotor network in visuospatial STM. The observation of a hemispheric difference of parietal activation is consistent with most of the imaging and neuro-psychological literature on the spatial functions of the parietal lobe. However, the finding that the right SPL showed a higher response for the ‘where’ than the conjunction condition might seem surprising, because the spatial attention load and need to rehearse the positions mentally would have been the same in both conditions. Yet, in the present experiment, the ‘where’ condition actually involved a higher demand on visuospatial attention because the number of possible locations was higher (based on individual performance in the test trials) in order to match the two conditions for difficulty. Furthermore, there is evidence that the presence of a second feature on which the match–non-match judgement can be based (in this case the identity of the object) leads to a reduced recruitment of the parietal lobes in conjunction as opposed to pure visuospatial tasks (Sack et al., 2002).

#### Infero-temporal Activation

The typical time-course of the BOLD signal in IT cortex showed a prompt response to sample stimuli, returned to baseline during the delay and peaked again in response to the probe stimulus. Thus we could observe the expected stimulus responses, but not the delay activity described for IT in a number of studies (Fuster and Jervey, 1981; Miller et al., 1993, 1996). A possible explanation might be provided by the particular nature of our DDT task. Our sample stimulus always consisted of four sequentially presented items. Based on the finding that intervening stimuli cancel out delay activity in IT but not in prefrontal neurons of macaque monkeys (Miller et al., 1996), we would expect delay activity only in IT neurons responding to the fourth item of the samples. Considerably fewer neurons in IT cortex than PFC would thus be active during the delay phase of our task and the population of active IT neurons might have been too small to evoke a BOLD response.

## Conclusion

In conclusion, our data suggest that retention of different aspects of visual stimuli (‘what’, ‘where’ and conjunctions) depends on processes that recruit, in a task-specific manner, partly overlapping combinations of prefrontal and parietal areas.

## Abbreviations

BOLD blood oxygen level-dependent

CaS calcarine sulcus

CiS cingulate sulcus

CoS collateral sulcus

CU cuneus

DLPFC dorsolateral prefrontal cortex

FA flip angle

fMRI functional magnetic resonance imaging

FOV field of view

GF gyrus fusiformis

GL gyrus lingualis

GLM general linear model

GPrC precentral gyrus

IFG/IFS inferior frontal gyrus/sulcus

INS insula

IPL inferior parietal lobule

IPS intraparietal sulcus

IS insular sulcus (sulcus circularis insulae)

IT inferior temporal cortex

LS lateral sulcus

MFG/MFS middle frontal gyrus/sulcus

MNI Montreal Neurological Institute

MTS middle temporal sulcus

OF orbito-frontal sulci

OTS occipito-temporal sulcus

PCS postcentral sulcus

PFC prefrontal cortex

POS parieto-occipital sulcus

PP posterior parietal cortex

RC relative contribution

RGB red–green–blue

RS Rolandic (central) sulcus

SFG superior frontal gyrus

SFS superior frontal sulcus

SMA supplementary motor area

SPL superior parietal lobule

STM short-term memory

STS superior temporal sulcus

TE echo time

TR repetition time

VLPFC ventrolateral prefrontal cortex

Table 1

Talairach coordinates for centers of mass of activation clusters shown in Figure 1

‘What’ ‘Where’
x y z Voxels x y z Voxels
Temporal lobe, left hemisphere –37 –75 –9 2621 –50 –62 –5 1194
Temporal lobe, right hemisphere 31 –67 –9 4088 43 –55 –2 1587
Frontal lobe, left hemisphere –44 29 7860 –46 29 2270
Frontal lobe, right hemisphere 35 38 30 380 45 31 1857
‘What’ ‘Where’
x y z Voxels x y z Voxels
Temporal lobe, left hemisphere –37 –75 –9 2621 –50 –62 –5 1194
Temporal lobe, right hemisphere 31 –67 –9 4088 43 –55 –2 1587
Frontal lobe, left hemisphere –44 29 7860 –46 29 2270
Frontal lobe, right hemisphere 35 38 30 380 45 31 1857
Table 2

Centers of mass and significance levels of contrasts between conditions (for each task phase) for regions of interest in relative contribution maps

Area Hem. RC > 0.2 x y z No. of voxels Encoding Delay Delay 1 Retrieval
See list of abbreviations for areas. Hem., hemisphere (LH, left; RH, right); w&w, ‘what and where’; x, y, z, Talairach coordinates. Conditions listed in column ‘RC > 0.2’ denote the condition with the stronger BOLD response in the respective map (see headlines in table). Cluster size (No. of voxels) is provided in mm3. Most footnotes refer to additional contrasts, all remaining contrasts did not reach significance.
aFor contrast w&w versus where significant at P < 10−6. bHere, RC for where > what. cFor w&w versus where significant at P < 10−3. dFor w&w versus what significant at P < 10−4. eFor w&w versus what significant at P < 0.05. fFor w&w versus what significant at P < 10−2. gFor where versus w&w significant at P < 0.05. hFor where versus w&w significant at P < 10−5. iFor where versus w&w significant at P < 0.05. j For what versus where significant at P < 10−4. kFor what versus where significant at P < 2 × 10−4. lFor what versus where significant at P < 0.01. mFor where versus what significant at P < 10−6. nFor where versus what (where > what) significant at P < 2 × 10−5. oFor w&w versus what significant at P < 10−3. pFor what versus where significant at P < 0.02. qFor where versus what (where > what) significant at P < 10−6. rFor where versus what significant at P < 2 × 10−4. sFor w&w versus what P < 0.06. tHere, RC for where > w&w.
Encoding, ‘where’ versus ‘what’
MFG/SFG LH where –26  –7 46 144  0.6  2 × 10−4 10−5  0.4
GL LH what –14 –62 –3 404 10−6a  0.3  0.4 0.01b
CU RH what –85 21 2128 10−6c  0.06  0.02  0.04b
SPL RH where 32 –49 43 1999  0.6 10−6d  10−4 >0.9
SFG RH where 20 –7 62 974  0.3  0.05 e  0.05f  0.5
GPrC RH where 46 23 1197  0.05g 10−4  10−6h  0.05i
Encoding, ‘w&w’ versus ‘where’
GF LH w&w –31 –39 –14 1204 10−6j  0.004 >0.9  0.7
POS LH w&w –5 –78 20 850  0.01k  0.3  0.5  0.004
INS LH w&w –32 10 10 354  0.2  0.05l  0.2  0.07
IPL RH where 35 –34 43 994  0.6  0.01m 10−4m  0.9
GL RH w&w 15 –39 986 10−4n  0.2  0.8  0.07n
Delay, ‘where’ versus ‘what’
MFG/IFG LH what –43 24 25 6434  0.006  0.003  0.3  0.002
IFG/INS LH what –38 14 2742  0.9  0.001  0.5  0.03
SPL LH where –26 –60 47 10907  0.2  2 × 10−6  0.0002  0.2
MFG/SFG LH where –27 –7 47 391  0.5 10−3 10−6  0.3
Delay, ‘w&w’ versus ‘what’
SFG RH w&w 27 –9 59 1768  0.4  0.01  0.01  0.6
IFG LH what –49 11 12 299  0.4  0.7  0.5  0.1
SPL LH w&w –22 –66 46 8977  0.8  3 × 10−4  0.01  0.6
SPL RH w&w 12 –72 46 2897  0.8 10−6  0.05  0.6
SMA LH w&w –4 43 383  0.2  0.009  0.3  0.5
SMA RH w&w 49 774  0.6  0.02  0.6  0.8
Delay, ‘w&w’ versus ‘where’
IPL LH where –47 –41 36 569  0.4  0.03o  0.0007 >0.9
IFG LH w&w –44 20 23 4723 10−3  0.003  0.6  0.005
INS LH w&w –36 13 2255  0.2  0.005  0.2  0.004
MFG LH w&w –43 36 2584  0.002  0.05p  0.4 >0.9
SMA LH w&w –4 11 45 835  0.4  0.02  0.7  0.01
SPL RH where 25 –57 49 7022  0.3 10−4q 10−3q  0.2
SFG RH where 25 –8 47 462  0.2  0.01r  0.01q >0.9
SMA RH w&w 51 263  0.8  0.01s >0.9  0.01t
INS RH w&w 32 14 1585  0.6  0.05  0.2 >0.9
Area Hem. RC > 0.2 x y z No. of voxels Encoding Delay Delay 1 Retrieval
See list of abbreviations for areas. Hem., hemisphere (LH, left; RH, right); w&w, ‘what and where’; x, y, z, Talairach coordinates. Conditions listed in column ‘RC > 0.2’ denote the condition with the stronger BOLD response in the respective map (see headlines in table). Cluster size (No. of voxels) is provided in mm3. Most footnotes refer to additional contrasts, all remaining contrasts did not reach significance.
aFor contrast w&w versus where significant at P < 10−6. bHere, RC for where > what. cFor w&w versus where significant at P < 10−3. dFor w&w versus what significant at P < 10−4. eFor w&w versus what significant at P < 0.05. fFor w&w versus what significant at P < 10−2. gFor where versus w&w significant at P < 0.05. hFor where versus w&w significant at P < 10−5. iFor where versus w&w significant at P < 0.05. j For what versus where significant at P < 10−4. kFor what versus where significant at P < 2 × 10−4. lFor what versus where significant at P < 0.01. mFor where versus what significant at P < 10−6. nFor where versus what (where > what) significant at P < 2 × 10−5. oFor w&w versus what significant at P < 10−3. pFor what versus where significant at P < 0.02. qFor where versus what (where > what) significant at P < 10−6. rFor where versus what significant at P < 2 × 10−4. sFor w&w versus what P < 0.06. tHere, RC for where > w&w.
Encoding, ‘where’ versus ‘what’
MFG/SFG LH where –26  –7 46 144  0.6  2 × 10−4 10−5  0.4
GL LH what –14 –62 –3 404 10−6a  0.3  0.4 0.01b
CU RH what –85 21 2128 10−6c  0.06  0.02  0.04b
SPL RH where 32 –49 43 1999  0.6 10−6d  10−4 >0.9
SFG RH where 20 –7 62 974  0.3  0.05 e  0.05f  0.5
GPrC RH where 46 23 1197  0.05g 10−4  10−6h  0.05i
Encoding, ‘w&w’ versus ‘where’
GF LH w&w –31 –39 –14 1204 10−6j  0.004 >0.9  0.7
POS LH w&w –5 –78 20 850  0.01k  0.3  0.5  0.004
INS LH w&w –32 10 10 354  0.2  0.05l  0.2  0.07
IPL RH where 35 –34 43 994  0.6  0.01m 10−4m  0.9
GL RH w&w 15 –39 986 10−4n  0.2  0.8  0.07n
Delay, ‘where’ versus ‘what’
MFG/IFG LH what –43 24 25 6434  0.006  0.003  0.3  0.002
IFG/INS LH what –38 14 2742  0.9  0.001  0.5  0.03
SPL LH where –26 –60 47 10907  0.2  2 × 10−6  0.0002  0.2
MFG/SFG LH where –27 –7 47 391  0.5 10−3 10−6  0.3
Delay, ‘w&w’ versus ‘what’
SFG RH w&w 27 –9 59 1768  0.4  0.01  0.01  0.6
IFG LH what –49 11 12 299  0.4  0.7  0.5  0.1
SPL LH w&w –22 –66 46 8977  0.8  3 × 10−4  0.01  0.6
SPL RH w&w 12 –72 46 2897  0.8 10−6  0.05  0.6
SMA LH w&w –4 43 383  0.2  0.009  0.3  0.5
SMA RH w&w 49 774  0.6  0.02  0.6  0.8
Delay, ‘w&w’ versus ‘where’
IPL LH where –47 –41 36 569  0.4  0.03o  0.0007 >0.9
IFG LH w&w –44 20 23 4723 10−3  0.003  0.6  0.005
INS LH w&w –36 13 2255  0.2  0.005  0.2  0.004
MFG LH w&w –43 36 2584  0.002  0.05p  0.4 >0.9
SMA LH w&w –4 11 45 835  0.4  0.02  0.7  0.01
SPL RH where 25 –57 49 7022  0.3 10−4q 10−3q  0.2
SFG RH where 25 –8 47 462  0.2  0.01r  0.01q >0.9
SMA RH w&w 51 263  0.8  0.01s >0.9  0.01t
INS RH w&w 32 14 1585  0.6  0.05  0.2 >0.9
Figure 1.

Experiment 1. (A) Structure and timing of the STM task (for details see Materials and Methods). The lines labeled ‘L’ and ‘R’ schematically represent the voltage of the two response buttons, which increases after the reaction time (RT) to signal a correct ‘match’-response and which was fed back to the subject by a green fixation cross at the end of the trial. (B) Color maps for the major sulci of the MNI template brain flattened with BrainVoyager software, see list of abbreviations. (C, D) Averaged BOLD time-courses (percentage signal change) and activation maps for the left and right hemispheres of five subjects. Error bars denote mean ± SEM. The onset of the target stimulus was at 4 s. Note that sampling of functional data was restricted to six axial slices covering most of the frontal, temporal and occipital cortex and excluding the entire parietal lobe. Green lines and green/blue clusters represent activity in areas that respond to visual stimulation during sample and test presentation, whereas yellow lines and yellow/blue clusters represent significant activity during the delay. (C) ‘What’ task. (D) ‘Where’ task.

Figure 1.

Experiment 1. (A) Structure and timing of the STM task (for details see Materials and Methods). The lines labeled ‘L’ and ‘R’ schematically represent the voltage of the two response buttons, which increases after the reaction time (RT) to signal a correct ‘match’-response and which was fed back to the subject by a green fixation cross at the end of the trial. (B) Color maps for the major sulci of the MNI template brain flattened with BrainVoyager software, see list of abbreviations. (C, D) Averaged BOLD time-courses (percentage signal change) and activation maps for the left and right hemispheres of five subjects. Error bars denote mean ± SEM. The onset of the target stimulus was at 4 s. Note that sampling of functional data was restricted to six axial slices covering most of the frontal, temporal and occipital cortex and excluding the entire parietal lobe. Green lines and green/blue clusters represent activity in areas that respond to visual stimulation during sample and test presentation, whereas yellow lines and yellow/blue clusters represent significant activity during the delay. (C) ‘What’ task. (D) ‘Where’ task.

Figure 2.

Experiment 2. (A) Structure and timing of the task (for details see Materials and Methods). (B) Color legend for activation maps in (C, D). Mixing of the three basic colors is performed in RGB-space. Intersections of the maps computed for the different tasks appear as natural color combinations, e.g. red overlapping with green appears as yellow, overlap of all three colors appears as white. (C, D) Superimposed semi-transparent activation maps (superposition maps) for the three task types during encoding (C) and delay (D) epochs.

Figure 2.

Experiment 2. (A) Structure and timing of the task (for details see Materials and Methods). (B) Color legend for activation maps in (C, D). Mixing of the three basic colors is performed in RGB-space. Intersections of the maps computed for the different tasks appear as natural color combinations, e.g. red overlapping with green appears as yellow, overlap of all three colors appears as white. (C, D) Superimposed semi-transparent activation maps (superposition maps) for the three task types during encoding (C) and delay (D) epochs.

Figure 3.

Relative contribution maps for the three pairs of conditions. Left and right pairs of maps in each row represent activation during encoding and delay, respectively. Conventions as in Figure 1C,D. (A) ‘What’ versus ‘where’. (B) Conjunction versus ‘what’. (C) Conjunction versus ‘where’.

Relative contribution maps for the three pairs of conditions. Left and right pairs of maps in each row represent activation during encoding and delay, respectively. Conventions as in Figure 1C,D. (A) ‘What’ versus ‘where’. (B) Conjunction versus ‘what’. (C) Conjunction versus ‘where’.

We thank N. Kriegeskorte for designing the stimuli of the control experiment, S. Kemble for her contribution to data analysis, J. A. Waltz for comments on the manuscript, A. Wiatrowski for help with the training sessions and R. Ruhl for help with the figures.

## References

Andersen RA (
1997
) Multimodal integration for the representation of space in the posterior parietal cortex.
Phil Trans R Soc Lond B Biol Sci

352
:
1421
–1428.
Andersen RA, Bracewell RM, Barash S, Gnadt JW, Fogassi L (
1990
) Eye position effects on visual, memory, and saccade-related activity in areas LIP and 7a of macaque.
J Neurosci

10
:
1176
–1196.
Belger A, Puce A, Krystal JH, Gore JC, Goldman-Rakic P, McCarthy G (
1998
) Dissociation of mnemonic and perceptual processes during spatial and nonspatial working memory using fMRI.
Hum Brain Mapp

6
:
14
–32.
Boynton GM, Engel SA, Glover GH, Heeger DJ (
1996
) Linear systems analysis of functional magnetic resonance imaging in human V1.
J Neurosci

16
:
4207
–4211.
Bullmore E, Brammer M, Williams SC, Rabe-Hesketh S, Janot N, David A, Mellers J, Howard R, Sham P (
1996
) Statistical methods of estimation and inference for functional MR image analysis.
Magn Reson Med

35
:
261
–277.
Callicott JH, Mattay VS, Bertolino A, Finn K, Coppola R, Frank JA, Goldberg TE, Weinberger DR (
1999
) Physiological characteristics of capacity constraints in working memory as revealed by functional MRI.
Cereb Cortex

9
:
20
–26.
Chafee MV, Goldman-Rakic PS (
1998
) Matching patterns of activity in primate prefrontal area 8a and parietal area 7ip neurons during a spatial working memory task.
J Neurophysiol

79
:
2919
–2940.
Cohen JD, Perlstein WM, Braver TS, Nystrom LE, Noll DC, Jonides J, Smith EE (
1997
) Temporal dynamics of brain activation during a working memory task.
Nature

386
:
604
–608.
Colby CL, Goldberg ME (
1999
) Space and attention in parietal cortex.
Annu Rev Neurosci

22
:
319
–349.
Constantinidis C, Steinmetz MA (
1996
) Neuronal activity in posterior parietal area 7a during the delay periods of a spatial memory task.
J Neurophysiol

76
:
1352
–1355.
Courtney SM, Ungerleider LG, Keil K, Haxby JV (
1997
) Transient and sustained activity in a distributed neural system for human working memory.
Nature

386
:
608
–611.
Courtney SM, Petit L, Maisog JM, Ungerleider LG, Haxby, JV (
1998
) An area specialized for spatial working memory in human frontal cortex.
Science

279
:
1347
–1351.
Courtney SM, Petit L, Haxby JV, Ungerleider LG (
1998
) The role of prefrontal cortex in working memory: examining the contents of consciousness.
Philos Trans R Soc Lond B Biol Sci

353
:
1819
–1828.
D’Esposito M, Aguirre GK, Zarahn E, Ballard D, Shin RK, Lease J (
1998
) Functional MRI studies of spatial and nonspatial working memory.
Brain Res Cogn Brain Res

7
:
1
–13.
D’Esposito M, Postle BR, Ballard D, Lease J (
1999
) Maintenance versus manipulation of information held in working memory: an event-related fMRI study.
Brain Cogn

41
:
66
–86.
Dierks T, Linden DEJ, Jandl M, Formisano E, Goebel R, Lanfermann H, Singer W (
1999
) Activation of Heschl’s gyrus during auditory hallucinations.
Neuron

22
:
615
–621.
Elliott R, Dolan RJ (
1999
) Differential neural responses during performance of matching and nonmatching to sample tasks at two delay intervals.
J Neurosci

19
:
5066
–5073.
Friston KJ, Frith CD, Frackowiak RS, Turner R (
1995
) Characterizing dynamic brain responses with fMRI: a multivariate approach.
Neuroimage

2
:
166
–172.
Funahashi S, Bruce CJ, Goldman-Rakic PS (
1989
) Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortex.
J Neurophysiol

61
:
331
–349.
Fuster JM (
1998
) Cellular dynamics of network memory.
Z Naturforsch [C]

53
:
670
–676.
Fuster JM, Alexander GE (
1971
) Neuron activity related to short-term memory.
Science

173
:
652
–654.
Fuster JM, Jervey JP (
1981
) Inferotemporal neurons distinguish and retain behaviorally relevant features of visual stimuli.
Science

212
:
952
–955.
Goebel R, Linden DEJ, Lanfermann H, Zanella FE, Singer W (
1998
) Functional imaging of mirror and inverse reading reveals separate coactivated networks for oculomotion and spatial transformations.
Neuroreport

9
:
713
–719.
Goebel R, Khorram-Sefat D, Muckli L, Hacker H, Singer W (
1998
) The constructive nature of vision: direct evidence from functional magnetic resonance imaging studies of apparent motion and motion imagery.
Eur J Neurosci

10
:
1563
–1573.
Goodale MA (
1998
) Vision for perception and vision for action in the primate brain.
Novartis Found Symp

218
:
21
–34.
Haxby JV, Petit L, Ungerleider LG, Courtney SM (
2000
) Distinguishing the functional roles of multiple regions in distributed neural systems for visual working memory.
Neuroimage

11
:
380
–391.
Ishai A, Ungerleider LG, Martin A, Schouten JL, Haxby JV (
1999
) Distributed representation of objects in the human ventral visual pathway.

96
:
9379
–9384.
McCarthy G, Puce A, Constable RT, Krystal JH, Gore JC, Goldman-Rakic P (
1996
) Activation of human prefrontal cortex during spatial and nonspatial working memory tasks measured by functional MRI.
Cereb Cortex

6
:
600
–611.
Miller EK (
2000
) The prefrontal cortex: no simple matter.
Neuroimage

11
:
447
–450.
Miller EK, Li L, Desimone R (
1993
) Activity of neurons in anterior inferior temporal cortex during a short-term memory task.
J Neurosci

13
:
1460
–1478.
Miller EK, Erickson CA, Desimone R (
1996
) Neural mechanisms of visual working memory in prefrontal cortex of the macaque.
J Neurosci

16
:
5154
–5167.
Mishkin M, Delacour J (
1975
) An analysis of short-term visual memory in the monkey.
J Exp Psychol: Anim Behav Process

1
:
326
–334.
Mitchell KJ, Johnson MK, Raye CL, D’Esposito M (
2000
) fMRI evidence of age-related hippocampal dysfunction in feature binding in working memory.
Brain Res Cogn Brain Res

10
:
197
–206.
Nystrom LE, Braver TS, Sabb FW, Delgado MR, Noll DC, Cohen JD (
2000
) Working memory for letters, shapes, and locations: fMRI evidence against stimulus-based regional organization in human prefrontal cortex.
Neuroimage

11
:
424
–446.
Owen AM, Stern CE, Look RB, Tracey I, Rosen BR, Petrides M (
1998
) Functional organization of spatial and nonspatial working memory processing within the human lateral frontal cortex.

95
:
7721
–7726.
Owen AM, Herrod NJ, Menon DK, Clark JC, Downey SP, Carpenter TA, Minhas PS, Turkheimer FE, Williams EJ, Robbins TW, Sahakian BJ, Petrides M, Pickard JD (
1999
) Redefining the functional organization of working memory processes within human lateral prefrontal cortex.
Eur J Neurosci

11
:
567
–574.
Pochon JB, Levy R, Poline JB, Crozier S, Lehericy S, Pillon B, Deweer B, Le Bihan D, Dubois B (
2001
) The role of dorsolateral prefrontal cortex in the preparation of forthcoming actions: an fMRI study.
Cereb Cortex

11
:
260
–266.
Postle BR, D’Esposito M (
1999
) ‘What’–then–‘where’ in visual working memory: an event-related fMRI study.
J Cogn Neurosci

11
:
585
–597.
Postle BR, D’Esposito M (
2000
) Evaluating models of the topographical organization of working memory function in frontal cortex with event-related fMRI.
Psychobiology

28
:
132
–145.
Postle BR, Berger JS, Taich AM, D’Esposito M (
2000
) Activity in human frontal cortex associated with spatial working memory and saccadic behavior.
J Cogn Neurosci

12
(Suppl. 2):
2
–14.
Prabhakaran V, Narayanan K, Zhao Z, Gabrieli JD (
2000
) Integration of diverse information in working memory within the frontal lobe.
Nat Neurosci

3
:
85
–90.
Quintana J, Fuster JM (
1999
) From perception to action: temporal integrative functions of prefrontal and parietal neurons.
Cereb Cortex

9
:
213
–221.
Rypma B, D’Esposito M (
1999
) The roles of prefrontal brain regions in components of working memory: effects of memory load and individual differences.

96
:
6558
–6563.
Sack AT, Hubl D, Prvulovic D, Formisano E, Jandl M, Zanella FE, Maurer K, Goebel R, Dierks T, Linden DEJ (
2002
) The experimental combination of rTMS and fMRI reveals the functional relevance of parietal cortex for visuospatial functions.
Cogn Brain Res

13
:
85
–93.
Sakata H, Taira M, Kusunoki M, Murata A, Tanaka Y (
1997
) The TINS Lecture. The parietal association cortex in depth perception and visual control of hand action.
Trends Neurosci

20
:
350
–357.
Trojano L, Grossi D, Linden DEJ, Formisano E, Hacker H, Zanella FE, Goebel R, Di Salle F (
2000
) Matching two imagined clocks: the functional anatomy of spatial analysis in the absence of visual stimulation.
Cereb Cortex

10
:
473
–481.
Zarahn E, Aguirre GK, D’Esposito M (
1999
) Temporal isolation of the neural correlates of spatial mnemonic processing with fMRI.
Brain Res

7
:
255
–268.