We recorded the activity of parietal area 7a neurons in monkeys performing an object construction task. In each trial, a model object consisting of a variable arrangement of squares was presented, followed after a delay by a copy of the model object that was missing a single square. Monkeys replaced the missing square to reconstruct the model configuration. Activity of many 7a neurons varied systematically with the position of the missing square and predicted where monkeys were going to add parts to the object they were building. The location of the missing square was a computed spatial datum important to object construction which did not correlate with the retinal location of a visual stimulus or the direction of the required motor response. The population of cells coding this coordinate was generally inactive when the same spatial locations were made relevant by visual targets to which monkeys either planned saccades or directed attention in other behavioral contexts. The data suggest that some parietal neurons participate in neural representations of space that reflect spatial cognitive as opposed to sensorimotor processing, coding the results of spatial computations performed on visual stimuli to meet cognitive objectives.
The spatial functions of posterior parietal neurons in monkeys have often been investigated by recording neural activity while monkeys viewed, attended to or moved toward peripheral visual stimuli (Mountcastle et al., 1975; Robinson et al., 1978; Hyvarinen, 1981). A principle of parietal neurophysiology deriving from this work has been that many posterior parietal neurons have visual receptive or movement fields — regions of space where visual sensory input or motor output is associated with neural activation. Thus the spatial coordinates coded by parietal neurons have most often represented the location of an attended stimulus or the direction of a planned movement. This has been confirmed in recent experiments examining the relation between neural activity in posterior parietal cortex and various forms of spatial cognition, including spatial attention (Mountcastle et al., 1981; Steinmetz et al., 1994; Gottlieb et al., 1998; Bisley and Goldberg, 2003), spatial working memory (Gnadt and Andersen, 1988; Chafee and Goldman-Rakic, 1998), motor intention (Bracewell et al., 1996; Mazzoni et al., 1996; Snyder et al., 1997, 1998a) and decision processing (Platt and Glimcher, 1999; Shadlen and Newsome, 2001; Sugrue et al., 2004). Typically in these studies, cognitive processing was seen to modulate either the intensity of neural discharge (as in the case of attention or decision processing) or the duration of neural discharge (as in the case of working memory or motor intention) that represented the location of a stimulus or the direction of a movement. Thus cognitive processing has been correlated with the modulation of either the timing or the intensity of sensorimotor representations in parietal cortex. It is an interesting question whether all forms of spatial representation encode the locations of stimuli or movements, or whether the brain can also sustain abstract spatial representations that do not. In order to solve a spatial problem, for example, it may be necessary for cortical neurons to compute and represent abstract spatial parameters that do not correlate with the locations of their visual receptive or movement fields and do not represent the locations of stimuli or the directions of movement. The question seems an important one in assessing the potential variety of neural processes engaged in spatial cognition, and whether and to what degree the neural representation of space can be dissociated from sensorimotor function.
In the present experiment, we investigated neural activity in parietal cortex during object construction. Our choice of this behavior for study was motivated by evidence that parietal damage can interfere with a person's ability to assemble or construct objects, while sparing basic motor function. This defect in the spatial organization of movement is known as constructional apraxia (Kleist, 1934), and follows damage to various cortical areas (Benson and Barton, 1970; Arena and Gainotti, 1978; Carlesimo et al., 1993), but is most common and severe following lesions of the right posterior parietal cortex (Black and Strub, 1976; Villa et al., 1986; Ruessmann et al., 1988). Consequently object construction is likely to depend at least in part on spatial processing normally supported by the intact posterior parietal cortex.
Objects are typically constructed in order to match a desired configuration or ‘blueprint’, and consequently the act of construction involves largely directing the addition of new parts. As the constructive process progresses, a builder has to compare the object that they have constructed so far to a central representation of the object they ultimately want to construct. The difference between these spatial representations determines where new parts should go. This comparative process is typically formalized in clinical tests of constructional ability in which a model object is presented to patients, who are then instructed to produce a copy of it. The task facing the patient in these cases is to draw or assemble the parts of their copy together in such a way that the resulting structure matches the model object. The construction task we developed for monkeys incorporated this feature. Monkeys were trained to construct a copy of a model object, and to do this, they were required to compare the copy to the model and form a spatial judgement regarding the difference between them. If a comparative spatial operation such as this was performed during object construction, it seemed reasonable that this process would be reflected in (correlated with) neural activity in posterior parietal cortex, given the impact of parietal damage on constructional ability.
We found that during object construction, parietal neurons coded a spatial coordinate that correlated with the location of parts that were missing from copy objects in comparison to model objects monkeys were attempting to reproduce. These data related to one of the overall objectives of the work, which was to determine whether spatial information represented in the activity of parietal neurons could be identified as the product of spatial cognitive as opposed to sensorimotor processing. At the center of this distinction was the question of whether or not neural activity in parietal cortex could code spatial information that did not correspond to the location of a visual stimulus or the direction of a pending movement. The present results suggest that area 7a neurons, when confronted with complex and cognitively demanding spatial problems such as object construction, can represent spatial information that does not correlate with sensorimotor variables, but instead reflects the product of a computation performed on visual stimuli to meet cognitive objectives.
Materials and Methods
Two male rhesus macaque monkeys (weighing 6 and 4 kg) were prepared for single neuron recording using standard aseptic surgical technique under Isoflurane (1–2%) gas anesthesia. In each animal, two recording chambers (7 mm i.d.) were implanted bilaterally, one each above a craniotomy overlying area 7a in the left and right cerebral hemisphere. Four titanium posts were attached to the skull with titanium screws, and a halo was attached to provide an anchor point to stabilize head position during neural recording experiments. Analgesia was provided for a period of several days following surgery (Buprenex, 0.05 mg/kg BID, i.m.). Care and treatment of the animals conformed to the Principles of Laboratory Animal Care of the NIH (NIH publication no. 86-23, revised in 1995). The Internal Animal Care and Use Committees of the University of Minnesota and the Minneapolis Veterans Affairs Medical Center approved all experimental protocols.
Monkeys viewed visual stimuli appearing on a 19″ color computer monitor (Gateway EV900, 60 Hz refresh) centered at eye level, located at a distance of 42 cm. The left foot of each monkey rested on a single response pedal. Extension of the ankle produced pressure on the pedal closing a switch, and the time of switch closure relative to the stimulus sequence in the behavioral task determined whether or not the trial was correct (see below). Correct trials were rewarded with a drop (0.1 ml) of juice. Monkeys worked for either three or four trial sets each day. Each set contained 200–300 trials, requiring 45 min to 1 h to complete.
Eye position was measured using an infrared video eye tracking system sampling eye position at 60 Hz (ISCAN Inc., Burlington, MA). Single neuron activity was recorded with a 16-microelectrode matrix (Eckhorn system, Thomas Recording, GMbH, Giessen, Germany). The electrode signals were amplified (at a gain of 20 000), filtered (bandpass between 0.5 and 5 kHz), and action potentials were discriminated in each channel using a time-amplitude window discriminator (DDIS-1, Bak Electronics, Mount Airy, MD) or waveform discriminator (Multi Spike Detector, Alpha Omega Engineering, Nazareth, Israel). Spike timing was sampled with 40 μs resolution (DAP 5200a data acquisition processor, Microstar Laboratories, Bellevue, WA). Computer files containing the timing of action potentials relative to stimulus and behavioral events were saved for all neurons encountered. Data were obtained in groups of simultaneously recorded neurons (typically 25–30), and neurons whose average discharge rate exceeded 0.5 Hz during the model, delay and copy periods of the construction task were included in subsequent analyses. In both animals, ∼15% of isolated neurons failed to meet this spontaneous activity criterion; the remaining 85% of all cells isolated comprise the present database.
We conducted two series of experiments. In experimental series 1, we recorded the activity of 1614 area 7a neurons in monkeys M1 and M2 as they performed the visual object construction task and also the delayed saccade task. In experimental series 2, we recorded the activity of 367 area 7a neurons in monkey M2 performing the visual object construction task, the mixed-choice construction task and the peripheral attention task (see below). The locations of recording sites were estimated from the magnetic resonance (MR) image of a metal microelectrode placed at the center of each chamber entering the inferior parietal gyrus (Fig. 2c). Microelectrode penetrations were restricted to a circular region 4.8 mm in diameter centered on this point and were confined to area 7a in the inferior parietal lobule. A small minority of penetrations encroached on the bounding intraparietal and superior temporal sulci, although neurons within the banks of these sulci were not sampled. MR localization of recording location was confirmed in the first monkey at sacrifice (Fig. 2d). MR and histological localization were in close agreement (compare Fig. 2c,d, ‘Monkey 1’).
The Visual Object Construction Task
In the construction task, monkeys reconstructed the configuration of a remembered model object by adding elements to an incomplete copy object. Each model and incomplete copy object was made up of a group of square elements in various configurations. (Each square element was blue in color and subtended 1.4° of visual angle. Adjacent squares were separated by a 0.25° gap.) These stimuli were adapted from those used by Driver and colleagues to demonstrate object-based neglect after parietal lobe damage (Driver et al., 1994). The squares making up each object were placed on a regular five by five grid (8.3° in both width and height). Each model and incomplete copy included an object frame (Fig. 1a), consisting of a base row and central column of square elements forming an inverted ‘T’. The frame provided a base and central axis to the object. Model objects varied in the number and position of square elements added to this frame. Models contained either one (Fig. 1b) or two (Fig. 1c) additional squares on either the left or right sides of the object at various vertical positions, producing a set of 36 different configurations. Monkeys were trained on the task using the complete set of model objects, but a subset was used to record neural activity in order to reduce the number of trials that had to be performed for each set of neurons. We employed two basic variants of the construction task. In one of these, the model and copy objects appeared at the same location in the visual display, both centered on the central fixation target (Fig. 2a, ‘Model’ and ‘Incomplete copy’ panels of ‘Trial 1’). In this case, monkeys performed the task using the subset of eight model configurations shown within the solid square outlines in Figure 1b,c. In the second variant, the model and copy objects appeared at different locations in the visual display. The model object was offset a fixed distance to one side of the fixation target or the other at random, such that the model appeared entirely within either the left or right visual hemifields (Fig. 2a, ‘Model’ panel of ‘Trial 2’). The incomplete copy appeared as before centered on the fixation target (Fig. 2a, ‘Incomplete copy’ panel of ‘Trial 2’). In this case, monkeys performed the task using the subset of 14 model configurations shown within both the solid and dashed square outlines in Figure 1b,c.
Monkeys initiated the construction trial sequence by directing their gaze to a small fixation target (a red spot of light) presented at the center of the display. The monkey was required to maintain its gaze position within 1.3–1.7° of the fixation target until the trial was complete (Fig. 2a; the approximate size of the fixation window relative to the model object is indicated by the dotted circle in the ‘Model’ panel of ‘Trial 1’). If at any time before the end of the trial gaze position deviated by more than this distance from the fixation target the trial was terminated. After 500 ms of fixation, a model object was presented for 750 ms (Fig. 2a, ‘Model’) and then disappeared. The disappearance of the model marked the beginning of a 750 ms delay period in which only the fixation target was visible (Fig. 2a, ‘Delay’). At the end of the delay period an incomplete copy object appeared (Fig. 2a, ‘Incomplete copy’). The copy object remained visible for 750 ms, before an array of choice squares appeared (Fig. 2a, ‘Choice array’). Neural activity during the 750 ms period beginning with the appearance of the incomplete copy object and ending with the appearance of the choice array is the focus of this report. Each incomplete copy object was identical to the model object preceding it on a given trial, except that one square element had been removed, referred to as the ‘missing square’ (Fig. 2b). The missing square was always one of the additional squares placed adjacent to the object frame in the model and not one of the squares making up the frame itself. In the case of model objects having two squares on the same side of the object and at the same vertical position (second row of objects in Fig. 1c), the missing square was always the outermost of the two squares added to the frame. In order to successfully complete the trial, monkeys were required to add a new square to the incomplete copy object in order to replace the missing square and so reconstruct the model configuration. This was done by selecting one of the two choice squares in the choice array. One choice square was the correct choice, and was located at the same vertical position and on the same side of the incomplete copy object as the missing square. (In Fig. 2a, ‘Trial 1’, the correct choice is the leftmost of the two choice squares. In Fig. 2a, ‘Trial 2’, the correct choice is the lower of the two choice squares.) The incorrect choice square was a distracter. The array of choice squares randomly appeared in one of two orientations, forming either a horizontal array (Fig. 2a, ‘Choice array’ panel of ‘Trial 1’) or a vertical array (Fig. 2a, ‘Choice array’ panel of ‘Trial 2’). In horizontal choice arrays, correct and incorrect choice squares were located on opposite sides of the copy object. In vertical choice arrays, correct and incorrect choice squares were located on the same side of the copy object. After a variable interval (300–600 ms), one of the choice squares selected at random increased in luminance for a period of 700–1000 ms (Fig. 2a, ‘1st choice’). If the monkey pressed the response key while the first choice square was bright (as in Fig. 2a, ‘1st choice’ panel of ‘Trial 2’), the first choice was selected for addition to the incomplete copy object. If the monkey did not press the response key in this interval, the first choice square dimmed to its original luminance at the same time that the second choice square brightened. The second choice square then remained bright for a period of 700–1000 ms (Fig. 2a, ‘2nd choice’ panel of ‘Trial 1’). If the monkey pressed the response key during this interval, the second choice square in the sequence was selected for addition to the incomplete copy object. In either case, at the time that the response key was pressed, the dim choice square disappeared and the bright hoice square translated smoothly inward toward the model in a horizontal direction, coming to rest adjacent to the outermost square in the copy object to produce a new configuration (Fig. 2a, ‘Complete copy’). The complete copy remained visible for a period of 300 ms before disappearing at the end of the trial. If the monkey selected the correct choice square, its addition produced a complete copy object whose configuration matched the preceding model, and the trial was rewarded after the complete copy period (Fig. 2a; compare ‘Model’ and ‘Complete copy’ panels of ‘Trial 1’). If the monkey selected the incorrect choice square, an erroneous configuration resulted. The complete copy object did not match the preceding model, and the trial was not rewarded (Fig. 2a; compare ‘Model’ and ‘Complete copy’ panels of ‘Trial 2’).
Delayed Saccade Control Task
We recorded the activity of area 7a neurons while monkeys performed a delayed saccade task. After a period of central fixation (500 ms), a saccade target was presented at one of eight peripheral locations (Fig. 5a). Fixation and saccade targets were both red spots. Saccade target locations corresponded to the centers of missing squares and choice squares in the construction task (object squares were not visible in the saccade task but are shown as dotted outlines to illustrate their locations relative to saccade targets). After a fixed delay of 750 ms, the central fixation target disappeared, and the monkey made a saccade toward the peripheral target. The trial was rewarded if that saccade ended within 1.3–1.7° of the peripheral target location and gaze was maintained at that location for an additional 500 ms.
Dual Attention–Construction Task
We used a dual task paradigm in order to probe the allocation of visual attention during object construction. In this dual task, all model objects consisted of the frame plus a single additional square (Fig. 1b). Most trials (85 or 75%) were standard construction trials as described above. A minority of trials (15 or 25%) were attention probe trials. The monkey was not cued as to which trial type was in progress — construction and probe trials progressed through model, delay and copy periods presenting identical visual stimuli. On probe trials, 150–750 ms after the incomplete copy object appeared, a visual probe stimulus was flashed for 50 ms and the choice square array did not appear. The monkey's task was to depress the response key as soon as it detected the probe (the key press had to occur between 50 and 600 ms after probe onset). The probe stimulus was a single red square whose contrast was greater than threshold (the monkey successfully detected 93% of probe stimuli). The horizontal position of the probe stimulus was varied relative to the position of the missing square (Fig. 9), and was expressed in degrees offset from that position. The probe was presented either at the position of the missing square (position 0°), at the mirror opposite location within the copy object (position −3.3°), at the location of the correct choice square (position 3.3°) or at the location of the incorrect choice square (position −6.6°).
Mixed-choice Construction Task
The horizontal position of the missing square and the correct choice square were dissociated in a mixed choice task. Choice squares were arranged in a vertical array and presented in one of two locations relative to the missing square. On standard trials, the choice array appeared on the same side of the copy object as the missing square, and so the correct choice and the missing square shared the same horizontal position relative to the central axis of the copy object (Fig. 10a, left). On crossed trials, the choice array appeared on the opposite side of the copy object as the missing square, and so the correct choice and missing square had different horizontal positions relative to the central axis of the copy object (Fig. 10a, right). In either case, the correct choice was located at the same vertical position as the missing square. The monkey was cued as to whether a standard or crossed trial was in progress by the color of the fixation target (red indicated a standard trial, green indicated a crossed trial). On crossed trials, the correct choice square translated across the central axis of the copy object and came to rest at the missing square location on the opposite side of the object.
Peripheral Attention Task
We recorded the activity of 7a neurons while monkeys directed attention toward a peripheral visual stimulus in order to detect its dimming (Bushnell et al., 1981; Colby et al., 1996). Trials began when monkeys fixated a central fixation target (a blue spot of light). After 500 ms a second blue spot target identical to the fixation target appeared in a peripheral location (Fig. 11a, left). Monkeys were required to maintain fixation of the central fixation target while directing attention to the peripheral target in order to detect a subtle decrease in its luminance that occurred at an unpredictable time. The peripheral target was presented in one of four locations selected at random. Peripheral target locations corresponded to the centers of the four possible locations of the missing square relative to the fixation target in the construction task (the missing square locations are indicated by the dotted square outlines; Fig. 11a, left). After a random interval of between 400 and 1400 ms, the peripheral target dimmed slightly (Fig. 11a, right). The monkey was rewarded for pressing the response key between 50 and 500 ms following the dimming of the peripheral target.
All data analyses were implemented in Matlab (The MathWorks, Inc., Natick, MA). We employed an analysis of covariance (ANCOVA) to assess the influence of task variables on neural activity. The ANCOVA was implemented as a general linear model (Kleinbaum et al., 1988; Howell, 1997) estimated by the REGRESS function in the Matlab Statistical Toolbox. All ANCOVA analyses included two covariates. The first covariate was the baseline firing rate during a 500 ms period while the monkey fixated the central target before the appearance of the model object at the beginning of each trial. The second covariate was the time elapsed since the beginning of data collection for a given set of isolated neurons and the start of each trial in the set. The influence of the independent variables on neural activity was therefore assessed independently of either fluctuation in basal activity across trials or linear trends in activity across the recording session. An α level of 0.05 was used unless otherwise noted.
In experimental series 1, we analyzed neural activity on the construction and delayed saccade tasks. In the construction task, the dependent variable in the analysis was the firing rate of a given neuron on each trial during the incomplete copy period. The copy period began with the appearance of the incomplete copy object, and ended with the appearance of the choice array 750 ms later. The two fixed factors in the analysis were the horizontal position (left/right) and the vertical position (high/low) of the missing square in the incomplete copy object coded as categorical factors (Fig. 2b). These factors together defined the location of the missing square relative to the central axis (horizontal position) and base (vertical position) of the incomplete copy object. In the delayed saccade task, the dependent variable was the firing rate on each trial during the delay period. This period began with the appearance of the saccade target in peripheral vision, and ended with the offset of the fixation target, which was the go signal to make a saccade to the target. The two fixed factors were the horizontal position (left/right) and vertical position (high/low) of the saccade target, again coded as categorical factors. Trials with near targets (at the locations of removed squares in the incomplete copy object) and far targets (at the locations of choice squares) were pooled (Fig. 5a).
In experimental series 2, we employed an ANCOVA to analyze the activity of neurons in a mixed-choice construction task. The dependent variable in this analysis was neural activity during the copy period. The independent variables were the horizontal position of the missing square (left/right), and the horizontal position of the choice square (left/right), with respect to the central axis of the copy object. A relation between neural activity and the position of the choice square was anticipatory in nature because the neural activity analyzed was measured during the copy period, and the choice array had not yet appeared at this point in the trial (Fig. 2a). Horizontal positions of missing and choice squares were included as two independent factors in this design because these positions were uncorrelated in the mixed-choice task. In experimental series 2, we also analyzed the activity of a group of neurons that were recorded on both the standard construction and the peripheral attention tasks. In this group of neurons, we simplified the ANCOVA by collapsing the coding of position into a single factor with four levels (to code the location both of the missing square and also the peripheral target). This simplified the comparison between tasks to the pattern of significance obtained across two (instead of four) factors, as was the case in analyzing the mixed-choice task data.
We constructed spike density functions (SDF) from the pooled activity of neural populations. First we identified (partially overlapping) populations of neurons whose activity varied significantly either with the horizontal or the vertical position of the missing square as assessed by the above ANCOVA. For the purposes of constructing population spike density functions in Figures 4 and 5, we used a stringent statistical criterion to select cells with the strongest signals in the population (including only those neurons whose firing rates were affected by the relevant spatial variable at P < 0.001). Trials were segregated into preferred and nonpreferred groups as a function of the location of the missing square along the horizontal or vertical axis that was associated with the greatest activity in each neuron. Each trial, represented by a sequence of spike times, was then converted into a SDF evaluated at 1 ms intervals using the KSDENSITY kernel smoothing function in the Matlab Statistical Toolbox. A Gaussian kernel of σ = 15 was used to construct population spike density functions (in the case of the spike density functions constructed from the activity of the single neuron in Figure 3, a kernel of σ = 20 ms was used). Individual trial SDF were then averaged across the trials in preferred and nonpreferred groups separately for each neuron and normalized (by subtracting the minimum at each point and dividing by the range). These average normalized SDF for each neuron were again averaged across all the neurons in each population to yield the population SDF in Figures 4, 5, 10 and 11. In Figures 4 and 5, the standard error of the mean of the population SDF was computed across neurons and is represented by the region of lighter shading surrounding each SDF. When neural activity was contrasted across tasks, the preferred horizontal or vertical position of each neuron was defined by the activity of that neuron on the standard construction task, and this preferred position was then used to segregate trials in either the delayed saccade task (Fig. 5d), the mixed-choice construction task (Fig. 10d,f) or the peripheral attention task (Fig. 11e) into preferred and nonpreferred groups. The difference between population activity on preferred and nonpreferred trials in the various control tasks therefore illustrates the degree to which neurons in the population maintained a consistent spatial preference across the tasks.
We recovered the location of the missing square by decoding neural population activity using linear discriminant analysis (Johnson and Wichern, 1998; Averbeck et al., 2003). Linear discriminant analysis (LDA) is a multivariate statistical technique that classifies observations to categories on the basis of a group of discriminating variables. In the present analysis, observations were trials of the construction task, and each trial was represented by a population activity vector consisting of the mean firing rate of each neuron in a population during the copy period of the task. The population activity vector therefore represented the distribution of firing rates across the neurons in the population on a given trial while the copy object was visible. There were four categories in the analysis, corresponding to the four possible locations of the missing square in the copy object. Classification of trials to categories was carried out using the CLASSIFY function of the Matlab statistical toolbox. A discriminant function for each category was derived by computing the parameters of a multivariate normal probability density distribution whose mean was equal to the mean population activity vector averaged across the trials in a given category and whose covariance was estimated by pooling trials across categories (see equation 3 of Averbeck et al., 2003). Equal prior probabilities were assumed (as the square could be removed from any one of the four potential locations with equal probability). The analysis proceeded by using the discriminant functions to compute a set of scores for each trial associated with its membership in each of the categories. The trial was then classified to the category with the highest score. This was carried out using fivefold cross-validation — the mean and covariance of the discriminant functions for the categories were defined on the basis of four-fifths of the trials and the remaining fifth was treated as being of unknown category and was classified using LDA. The procedure was replicated for successive fifths of the data until all trials were classified. Generally, LDA performance scales with the number of neurons included in the analysis (Averbeck et al., 2003). The present analysis was based on a population of 7a neurons whose firing rate during the copy period varied significantly with the location of the missing square. This population included neurons that belonged to different ensembles recorded at different times (that is not all of the neurons in the population were recorded simultaneously). Neurons recorded at different times were aggregated into this larger population by considering a single trial in the analysis to be a particular repetition of a given pair of model and copy objects. The firing rates of all neurons in the population on a given trial defined in this way were then grouped into a single population activity vector.
In a separate analysis, we recovered the location of the missing square when monkeys chose the incorrect choice square and built the incorrect object. We identified error trials and divided them into four groups. These four groups included trials in which the missing square was located on the left side of the copy but the monkey added a square on the right, and vice versa; and trials in which the missing square was located at the upper position within the copy but the monkey added a square at the lower position, and vice versa. We constructed neural populations for each error type. The neurons included for each error type were those in which (i) activity on correct trials varied significantly with the position of the missing square along the spatial axis (horizontal or vertical) associated with that type of error (ANCOVA; P ≤ 0.05); and (ii) a neuron was recorded while the monkey committed at least 10 errors of that type. Once the populations had been identified, categories for LDA classification were constructed on the basis of the activity in each population on correct trials. In the case of each population and error type, the classification included two categories. In populations recorded during horizontal errors, the two categories corresponded to the left and right side of the copy object. In populations recorded during vertical errors, the two categories corresponded to the high and low positions in the copy object. We then classified error trials to these categories using LDA as above. On error trials, one position in each classification corresponded to the location of the missing square whereas the other corresponded to the location of the square that the monkey erroneously added to the object (on error trials these locations were different).
We recorded the activity of 1981 neurons in posterior parietal area 7a (Fig. 2c,d) of two male rhesus macaque monkeys (M1 and M2) as they performed the visual object construction task (1092 of these neurons were recorded within the first monkey, 889 in the second). Of these cells, 1614 were recorded from both monkeys in experimental series 1 (standard construction task plus delayed saccade task). An additional 367 neurons were recorded from M2 in experimental series 2 (standard construction task plus mixed-choice and peripheral attention tasks, see Materials and Methods). Monkey M1 performed 83% of construction trials correctly in which the model object appeared randomly in left or right visual hemifields, and 94% of trials correctly in which model and probes both appeared centered in the display. Overall performance in series 1 was 87% correct. Monkey M2 performed 90% of construction trials correctly in which the model object appeared randomly in left or right visual hemifields, and 91% of trials correctly in which model and probes both appeared centered in the display. In series 2, Monkey M2 performed 97% of mixed-choice construction trials correctly, with comparable performance on standard (99% correct) and crossed (97% correct) trials. Performance was more accurate in monkeys M1 and M2 on trials with vertical choice arrays (88% and 91% respectively) than with horizontal choice arrays (86% and 89% correct respectively), although the difference was slight (only 3% difference in both animals).
Neural Activity in 7a Correlates with Spatial Judgement during Object Construction
In the construction task, monkeys viewed two objects in sequence — first the model object and then, after a delay, the incomplete copy object. Both objects consisted of varying arrangements of identical squares presented on a video monitor. The incomplete copy was identical to the preceding model each trial, except that a single square had been removed (Fig. 2).
We found that after the incomplete copy appeared, 7a neurons became active as a function of the location of the square that had been removed from the incomplete copy object relative to the preceding model. An example of this pattern of activity in a single 7a neuron is shown in Figure 3. During the incomplete copy period (indicated by the vertical gray bar of shading through each raster), neural activity increased on trials in which model and incomplete copy objects together connoted a square removed from the lower left position in the incomplete copy object (Fig. 3b,f,j). This increase in activity was not a function of the pattern of visual input at the time it occurred. On trials in which the model consisted of the object frame plus one additional square, the incomplete copy object consisted of the frame only. In spite of the fact that the incomplete copy object was physically identical across these trials, its appearance evoked markedly different levels of neural activity depending on where the square had been removed from the copy object (Fig. 3a–d). On trials in which the model consisted of the object frame plus two additional squares, the neuron exhibited the same preference, and incomplete copy objects in which squares had been removed from the lower left position excited the neuron (Fig. 3f,j); physically identical copy objects from which squares had been removed at other locations did not (compare copy objects in Fig. 3f and l, and those in Fig. 3j and h). Activity during the incomplete copy period varied across trials with the same model object (compare model objects in Fig. 3e and f, and those in Fig. 3j and k), and so was not likely to represent either a working memory of the model configuration or the reactivation of a visual receptive field stimulated by the model. Activity during the incomplete copy period did not reflect an association between a particular pair of model and incomplete copy objects because it generalized across all object pairs in the set that together defined a missing square at the preferred position (Fig. 3b,f,j).
We assessed the influence of the location of the missing square in the copy object on neural activity during the copy period using a two-way ANCOVA. The two orthogonal fixed factors in the analysis (horizontal and vertical position) jointly determined the location of the missing square (Fig. 2b). The activity of approximately one in four area 7a neurons during the incomplete copy period that met a minimal activity criterion (see Materials and Methods) varied significantly as a function either of the horizontal position (27%) or the vertical position (24%) of the square removed from the incomplete copy object relative to the preceding model (Table 1, ‘Combined’). The proportions of neurons exhibiting these effects were comparable in the two monkeys considered individually (Table 1, ‘Monkey 1’ and ‘Monkey 2’). The group of neurons whose activity was significantly related to either the horizontal or vertical position of the missing square (or both spatial factors) included 43% of all spontaneously active area 7a neurons isolated for study (700 of 1614 cells).
|Main effects in ANCOVA (position of missing square)||Monkey 1||Monkey 2||Combined|
|Horizontal position||27% (298/1092)||28% (146/522)||27% (444/1614)|
|Vertical position||27% (296/1092)||18% (97/522)||24% (393/1614)|
|Interaction||10% (111/1092)||8% (43/522)||9% (154/1614)|
|Horizontal or vertical||45% (489/1092)||40% (211/522)||43% (700/1614)|
|Main effects in ANCOVA (position of missing square)||Monkey 1||Monkey 2||Combined|
|Horizontal position||27% (298/1092)||28% (146/522)||27% (444/1614)|
|Vertical position||27% (296/1092)||18% (97/522)||24% (393/1614)|
|Interaction||10% (111/1092)||8% (43/522)||9% (154/1614)|
|Horizontal or vertical||45% (489/1092)||40% (211/522)||43% (700/1614)|
The dependent variable in the analysis was neural activity during the copy period. Trials in which the model and copy object appeared at different locations, and trials in which the objects appeared at the same location, were pooled for this analysis. The row labeled ‘Horizontal position’ tallies the percentages of neurons (and proportion of the total sample in parenthesis) for which the horizontal position of the missing square significantly affected firing rate, regardless of the effect of vertical position. Likewise, the row labeled ‘Vertical position’ tallies the percentages and proportion of neurons for which the vertical position of the missing square significantly affected firing rate, regardless of the effect of horizontal position. The row labeled ‘Interaction’ tallies the percentages and numbers of neurons for which the interaction between these two spatial factors significantly influenced neural activity. Finally, the row labeled ‘Horizontal or vertical’ tallies the percentages and proportion of neurons in which activity related either to horizontal position alone, to vertical position alone, or to both spatial factors.
Trials in which model and incomplete copy objects appeared at the same or different locations in the display are pooled in Table 1. Trials in Table 2 are restricted to those in which model and incomplete copy objects appeared at different locations (the model object was offset into the left or right visual hemifield while the copy object was centered on the fixation target). The effects of the position of the square removed from the incomplete copy were equally prevalent in this subset of the data. This is evidence that neural activity in 7a coding where the square had been removed from the incomplete copy object did not require that the corresponding square in the model object appear at the same retinotopic location.
|Main effects in ANCOVA (position of missing square)||Combined (Monkeys 1 and 2)|
|Horizontal position||27% (275/1016)|
|Vertical position||25% (251/1016)|
|Horizontal or vertical||44% (445/1016)|
|Main effects in ANCOVA (position of missing square)||Combined (Monkeys 1 and 2)|
|Horizontal position||27% (275/1016)|
|Vertical position||25% (251/1016)|
|Horizontal or vertical||44% (445/1016)|
The dependent variable in the analysis was neural activity during the copy period. Data included in this analysis were restricted to trials in which the model and copy appeared at different locations (other conventions as in Table 1). Neurons whose activity related to the horizontal or vertical position of the missing square were approximately as common in this sub-sample as when all available trials were combined (Table 1). The relationship between neural activity and the location of the missing square did not require that the model and copy objects appear the same retinotopic location.
The set of model objects included some in which two squares were stacked together on the same side and at the same height in the model object (second row of model objects in Fig. 1c). In these objects, the missing square (always the outermost of the two squares added to the frame) was located at a farther eccentricity from the central axis of the object than in other model configurations. We tested the effect of the eccentricity of the missing square on the activity of those neurons coding the horizontal position of the missing square. We found that the activity of approximately one in five of these neurons (62 of 275 neurons coding horizontal position in Table 2) was significantly influenced not only by the side but also by the eccentricity of the missing square relative to the central axis of the object (the activity of these neurons was significantly affected by both the side and eccentricity of the missing square at P ≤ 0.05 in a two-way ANCOVA employing side and eccentricity as factors).
We selected neurons in which activity during the copy period varied significantly (P ≤ 0.001) with either the horizontal position (Fig. 4a,c; 152 neurons) or the vertical position (Fig. 4b,d; 108 neurons) of the missing square. We then constructed average normalized population spike density functions (SDF) illustrating the time course of activity in these two populations (see Materials and Methods). In each panel (Fig. 4a–d), separate SDF illustrate population activity on trials in which the missing square was in the preferred position (blue SDF), or the nonpreferred position (red SDF) of each neuron included in the population. (The preferred position of each neuron was defined as the position of the missing square associated with the greatest activity during the copy period.) On the subset of trials in which the model object consisted of the frame plus one additional square (Fig. 1b), the copy object consisted of the object frame only (Fig. 1a), and so the visual form of the copy object did not vary. Population activity on these trials clearly differentiated the horizontal (Fig. 4a) or vertical (Fig. 4b) position of the missing square during the copy period (note the separation between blue and red SDF), in spite of the fact the same copy object was presented on every trial. This confirmed at the population level what was seen at the single cell level (Fig. 3) — that neural activity during the copy period was not a function of the form of the copy object. On these trials, we also noted that the population signal coding the position of the missing square emerged early in the trial, shortly after the onset of the model object (Fig. 4a,b; note black arrows pointing at the point of divergence between the blue and red SDF during the model period). We wondered whether the early emergence of this population signal might be due to the fact that on this subset of trials monkeys could anticipate the location of the missing square with certainty on the basis of the model object alone. (Models with one additional square determined the position of the missing square unambiguously because only the one additional square could be removed to produce the copy object, and consequently the missing square could be located at only one position). To examine this possibility, we constructed neural population activity functions (Fig. 4c,d) using trials in which model objects included two squares in addition to the object frame (Fig. 1c). In this case, monkeys could not anticipate the location of the missing square on the basis of the model object alone, because either of the two additional squares present in the model could be removed to produce the copy. On these trials, we found that the emergence of the population signal coding the horizontal position (Fig. 4c) and the vertical position (Fig. 4d) of the missing square was delayed until after the copy object appeared (Fig. 4c,d; note black arrows pointing at the divergence between blue and red SDF during the copy period). Neural population activity therefore varied with the missing square both in space and in time — coding the horizontal and vertical position of the missing square at the point in time that the position of the missing square was fixed by model and copy objects.
Potential Relation of Neural Activity to Oculomotor Factors
It was not likely that neural activity during the incomplete copy period reflected motor planning for the upcoming response. The monkey pressed a single response key with its foot every trial and movement direction did not vary. To examine whether neural activity might reflect oculomotor intention, we tested the activity of 7a neurons on both the construction task and a delayed saccade task. Saccade targets were placed at locations that included those of the missing squares, and also the squares in the choice array (Fig. 5a). Under the hypothesis that neural activity during object construction reflected the oculocentric direction of a planned saccade, the same population of neurons should be active in the two tasks and exhibit the same spatial preference for missing squares and saccade targets. Instead we found that the object construction and delayed saccade tasks recruited largely distinct neural populations. Of the 444 neurons in the sample whose activity varied with the horizontal position of the missing square during object construction (Table 1, ‘Combined’), activity in only 83 neurons (19%) varied significantly with the horizontal position of the saccade target in the delayed saccade task. Likewise, of the 393 neurons in the sample whose activity varied with the vertical position of the missing square during object construction (Table 1, ‘Combined’), activity in only 56 neurons (14%) varied with the vertical position of the saccade target in the delayed saccade task. Therefore ∼84% of the population whose activity coded the location of the missing square along one or the other spatial axis during object construction failed to code the location of the saccade target along that axis in the context of the delayed saccade task. In the 139 neurons in which activity varied significantly with both the position of the missing square and the saccade target, we calculated two positional preference indices (see Materials and Methods) for each neuron recorded in the two tasks. The index quantified the degree to which each neuron preferred left versus right or high versus low positions. The position preferences of single neurons were weakly correlated between the saccade and construction tasks (r = 0.246, P = 0.003); but there was considerable scatter and the indices were not tightly aligned. Many neurons preferred saccade targets and missing squares at opposite horizontal or vertical positions (Fig. 5b; points in upper left and lower right quadrants).
We selected the subpopulation of neurons whose activity was most strongly related to the horizontal position of the missing square during object construction (Fig. 5c; the neurons included were those in which activity significantly related to the main effect of horizontal position at P ≤ 0.001). We then examined the activity of these neurons during the delayed saccade task, segregating trials into preferred and nonpreferred groups according to whether the saccade target was located at the same horizontal position each neuron preferred in the construction task or not. We found that whereas neural activity in this population clearly varied with the horizontal position of the missing square during the construction task (Fig. 5c), population activity did not vary with the horizontal position of the target in the delayed saccade task (Fig. 5d). The directional signal evident in the activity of this neural population when monkeys were building objects was largely absent when monkeys were planning saccades.
Whereas few neurons active during object construction were active during the saccade task, the saccade task in itself was effective in driving a largely distinct population of neurons in area 7a. For example, the activity of 7a neurons varied significantly (P ≤ 0.05) as a function of the horizontal position of the saccade target both during the delay period (266 neurons) and the saccade period (271 neurons) of the delayed saccade task. Population activity clearly differentiated the position of the saccade target during both task epochs (Fig. 5e,f; neurons included were those in which neural activity related significantly to the horizontal position of the target at P ≤ 0.001; 61 and 48 neurons during delay and saccade periods respectively). Neural activity during the delay period of the saccade task may have been related to the visual stimulation provided by the continuing presence of the saccade target, or to the monkey's directing attention toward the target. Neural activation during the saccade period was largely postsaccadic. Population activity peaked approximately 50 ms after the eyes entered the response window surrounding the saccade target.
During the construction task, neural activity may have alternatively related to a variation in eye position within the fixation window. To test this possibility, we assessed the distribution of sampled eye position within the fixation window as a function of the location of the missing square. We found that the average eye position within the fixation window varied by only 0.1° across trials with different missing square locations (Fig. 6). A negligible variation in firing rate would be predicted by a variation in eye position this small (Andersen et al., 1990b).
Neural Activity Predicts Performance in the Construction Task
We decoded neural population activity using linear discriminant analysis (LDA; see Materials and Methods) with fivefold cross-validation (Johnson and Wichern, 1998) to provide a measure of how well population activity represented the location of the missing square. The analysis classified trials to one of four categories corresponding to the four possible missing square locations in the experiment. The classification was based on the pattern of activity observed in a population of 7a neurons. The frequency with which the classification was correct indicated the strength of the prediction about the location of the missing square we could make on the basis of the firing rates of 7a neurons. The results of the trial classification procedure are shown in Figure 7. Trials were first segregated into four groups based on the known position of the missing square. Trials in each group were then classified to four categories, corresponding to the potential locations of the missing square. The heights of the four vertical columns in each copy object (Fig. 7a–d) are proportional to the frequency that trials of each type were classified to each of the four corresponding categories in the analysis. In each case, the tallest column representing the category to which the majority of trials were classified corresponds to the true location of the missing square on each trial. Overall, 85% of trials were correctly classified on the basis of the pattern of population activity.
On error trials, the square missing from the copy object and the square added to the copy object were located at different positions. We identified four types of error trials on this basis: trials in which the missing square was on the left side of the object and the added square was on the right side (Fig. 8a), or vice versa (Fig. 8b), and trials in which the missing square was at the high position and the added square was at the low position (Fig. 8c), or vice versa (Fig. 8d). We classified each of these error trials to one of two categories corresponding to the position of the missing square or the added square on each error trial (see Materials and Methods). LDA classified most error trials to the category corresponding to the position of the added square and not the missing square (Fig. 8a–d). This indicated that the population activity pattern predicted where the monkey was going to add a square when it committed an error in construction, establishing an association between neural activity and the animals' behavioral performance. The overall accuracy of classification performance on error trials (Fig. 8) was lower than that on correct trials (Fig. 7). This was likely to reflect the availability of fewer neurons contributing to the population on which the error trial classification was based.
Deployment of Covert Attention during Object Construction
To measure the deployment of covert spatial attention during the construction task, we employed a dual-task paradigm. The monkey performed a set of trials including two trial types. Most trials in the set were standard construction trials. A minority of trials in the set were probe trials (probe trials comprised either 15 or 25% of trials in each set). Probe trials, like standard construction trials, began with the sequential presentation of a model and copy object (the monkey was not cued as to which type, standard construction or probe trial, was in progress). Then, on probe trials, after a random interval following the appearance of the copy object, a probe stimulus (red square) was flashed for 50 ms at one of four locations next to the copy object. Probe positions varied relative to the location of the missing square as defined by the model and copy objects on that trial (probe position was expressed as degrees horizontal offset from the location of the missing square). Probe stimuli (Fig. 9) appeared either at the location of the missing square (position 0°), at the mirror opposite location in the copy object (position −3.3°), at the location of the correct choice square (position +3.3°) or at the position of the incorrect choice square (position −6.6°) on a given trial. Because probe trials were infrequent, the expectation was that spatial attention would be distributed as it was normally during construction task performance. The RT to detect the probe would then provide a means to measure this distribution of attention.
Mean reaction time to detect the probe varied significantly across the four probe locations [Fig. 9; one-way ANOVA; F(3,1163) = 3.03, P = 0.008]. RT was at a minimum when the probe appeared at the location of the missing square (position 0°), and was significantly less in this case than when the probe appeared at the mirror opposite location on the other side of the copy object [planned comparison between mean RT at positions 0° and −3.3°; asterisks in Fig. 9; t(1163) = −2.071, P = 0.039]. Probes at the missing square and mirror opposite locations were the same distance from the fixation target and shared the same retinal eccentricity. Consequently an influence of stimulus eccentricity on reaction time (Carrasco et al., 1995) could not account for this effect.
In Figure 9 trial sets in which probe trials were comparatively frequent (25% of trials) or infrequent (15% of trials) were pooled. The difference in RT to detect probes at positions 0° and −3.3° positions achieved significance only when all data were included. The size of the RT effect, particularly between adjacent probe positions, was small in relation to the variance in RT, perhaps because probes were closely spaced (∼3° apart), and all probes were within 6° of the location of the missing square.
We compared sets of trials with infrequent and frequent probe trials separately. The minimum in the RT function was located at the missing square in either case. We found that increasing the frequency of probe trials from 15% to 25% reduced the impact of the missing square location on RT [mean RT no longer varied significantly across probe location; F(3,506) = 1.50, P = 0.215]. Increasing probe trial frequency also reduced percent correct performance on the construction task slightly from 95% to 90% correct [t(1250) = −3.56, P < 0.001].
Representing Multiple Locations Simultaneously: Distinct Neural Populations Code Missing and Choice Squares during Construction
On standard construction trials, neural activity related to the location of the missing square may in fact have coded the location of the correct choice square the monkey was about to select, as these locations were never varied with respect to one another in the standard task. To resolve this ambiguity, we recorded the activity of 367 neurons during a mixed-choice construction task dissociating the location of the missing and choice squares (Fig. 10a; see Materials and Methods). On standard trials, the correct choice square and the missing square were both located on the same side of the copy object. On crossed trials, missing and choice squares were located on opposite sides of the copy object. The monkey was cued as to whether a standard or crossed trial was in progress by the color of the fixation target. We analyzed neural activity during the copy period of the mixed-choice task using a two-way ANCOVA. The two factors in this analysis were the horizontal position of the missing square, and the horizontal position of the correct choice square. Neural activity during the copy period preceded the appearance of the choice square array, so any relation between neural activity and the location of the choice square was anticipatory in nature. In this neural sample, the activity of 57 neurons was significantly related to the location of the missing square, whereas the activity of 30 neurons was significantly related to the location of the choice square (Fig. 10b; P ≤ 0.05). Thus neurons coding the missing square outnumbered those coding the choice square by ∼2 to 1. The activity of only three neurons was statistically related to both spatial factors in the analysis.
We segregated standard and crossed trials according to the position of the missing square, and defined the preferred position of each neuron on the basis of its activity on standard trials. The 57 neurons in which activity related significantly to the position of the missing square exhibited a consistent spatial preference across standard and crossed trials (Fig. 10c,d). Activity in these neurons continued to code the location of the missing square in spite of the fact that the choice array had switched from the same side of the copy object (Fig. 10c) to the opposite side of the copy object (Fig. 10d). Conversely, the 30 neurons in which activity related significantly to the position of the choice square switched their spatial preference from standard (Fig. 10e) to crossed trials (Fig. 10f; note that red and blue SDF are inverted). This was consistent with neural activity in this group of cells coding the anticipated position of the choice array as it switched from one side of the copy object to the other. Thus two physiologically distinct groups of 7a neurons were concurrently active during the copy period, one which coded the position of the missing square and one which coded the anticipated position of the choice square before the choice array appeared. Both locations were relevant to construction task performance.
Largely Different Populations of area 7a Neurons Code the Location of the Missing Square during Construction, and the Location of Attended Visual Stimuli
One important question in the present experiments is whether neurons in 7a are generically active whenever a particular spatial locus becomes behaviorally relevant and draws attention. We compared the activity of 7a neurons in monkey M2 on the construction task and on a peripheral attention task (Fig. 11a; see Materials and Methods). This task was similar to a classical attention task that has been previously used to study the influence of attention on neural activity in posterior parietal cortex (Bushnell et al., 1981; Colby et al., 1996). In the peripheral attention task, monkeys fixated a central spot target while a peripheral spot target was presented at one of four locations (Fig. 11a). Peripheral targets were located at positions relative to the fixation target that corresponded to the centers of the missing squares in the construction task (Fig. 11a, left). The monkey was required to press the response key when the peripheral target dimmed slightly (Fig. 11a, right). The time between target onset and target dimming was randomly selected between six intervals (400–1400 ms in 200 ms increments).
We examined the distribution of reaction times to depress the response key relative to the dimming of the peripheral target. If the monkey attended to the target, its motor response ought to be tightly coupled to the time that the target changed luminance. The frequency distribution of reaction time calculated relative to the time that the target dimmed was unimodal, with a relatively sharp peak at 325 ms (Fig. 11b). We then examined the frequency distribution of reaction time calculated relative to the onset of the peripheral target. This distribution of reaction time exhibited six peaks each corresponding to one of the 6 random intervals between target onset and target dimming (Fig. 11c; the times of target dimming are indicated by vertical arrows, peaks in the RT distribution are delayed by ∼350 ms). The coupling of the motor response to the dimming of the target is evidence that the monkey attended to the target in order to detect a change in its luminance.
Of the 246 neurons recorded during the performance of both the construction and peripheral attention tasks, the activity of 60 neurons was significantly related to the location of the missing square in the construction task, and the activity of 20 neurons was related to the location of the peripheral target in the attention task (Fig. 11d). These were largely distinct populations, and the activity of few neurons (six cells) related to the locations of both the missing square and the peripheral target. Thus three times as many neurons in the sample were engaged to code a spatial location relevant to object construction than to code the location of an attended visual stimulus.
We computed the normalized activity of these two populations of neurons during the performance of the peripheral attention task. The first population consisted of the 60 neurons in the sample whose activity coded the location of the missing square during object construction (Fig. 11d). In this population of cells, we segregated peripheral attention trials into preferred and nonpreferred groups according to whether or not the attention target was located at the position of the missing square preferred by each neuron on the construction task. This population did not exhibit a consistent positional preference for missing squares and attended visual stimuli — as indicated by the fact that population activity functions on preferred (blue SDF) and nonpreferred (red SDF) trials of the attention task largely overlap (Fig. 11e). This pattern is consistent with the result of the ANCOVA, indicating that the activity of few neurons (six) in this group coded the location of the attended stimulus. The second population of neurons we examined included the 20 neurons in which activity varied significantly with the position of the attended target in the attention task (Fig. 11d). In this population of cells, we segregated peripheral attention trials into preferred and nonpreferred groups on the basis of the location of the peripheral target associated with greatest activity on the attention task. The relation between activity in these cells and the location of the attended visual stimulus is indicated by the difference between population activity functions on preferred and nonpreferred trials (Fig. 11f).
Damage to human parietal cortex can impair spatial cognitive function. For example, constructional apraxia (Kleist, 1934; Black and Strub, 1976; Villa et al., 1986; Ruessmann et al., 1988) is a syndrome in which patients lose the ability to accurately process and reproduce object structure — they lose, for example, the cognitive ability to accurately represent the spatial arrangement of the parts of an object. This is often tested clinically by presenting a model object to a patient and asking them to produce a copy of it, either by drawing the model or arranging a duplicate set of its parts. Patients produce copies that are internally disorganized — the parts are placed in disarray relative to one another. The implication is that loss of normal parietal function has impaired the brain's ability to analyze and reproduce object structure, although little is known regarding what spatial analytic operations may be supported by parietal cortex or how these operations may be mediated by parietal neurons. Our objective in the present experiment was to examine these questions by developing an object construction task that monkeys could perform, to enable concurrent recording of neural activity in posterior parietal cortex. By probing spatial representations at the neural level during object construction, we sought to gain information regarding the nature of spatial analytic operations that were taking place during object construction and how parietal neurons may be involved.
The object construction task we employed incorporated the basic model–copy design used in clinical tests of human constructional ability. Monkeys viewed a model object, consisting of a contiguous grouping of identical square elements. After a delay, a copy object appeared. The copy was an incomplete version of the model, identical except that it lacked a single square element, which we refer to as the missing square. Monkeys were required to replace this missing square by adding a new square at the same position to the incomplete copy object. If this was accurately done, the addition of the square produced a new object (the complete copy object) whose structure matched the structure of the preceding model. This defined successful construction.
Construction is a process of accruing parts. To add a new part to an object, at the simplest level the brain has to figure out where to place that part. From this perspective, object construction hinges on a spatial judgement, that is, where each new part should be placed. When trying to copy an object, this judgement takes a particular form. The brain must determine, or judge, exactly how an object under construction is incomplete, or in other words, where parts are missing within the object being built relative to the model object that one is trying to copy. At each stage in the process, as each part is added, this amounts to finding where the emerging copy object and the model configuration differ. Often, one configuration is visible (the copy that is being built) and one is stored in working memory (the model object). Even in the case where a model object is continuously available to view (as in most clinical tests of constructional ability), the model is typically out of view or in peripheral vision when a patient is examining their own workspace and deciding where to place the next part of the copy. Consequently, working memory may be integral to copy tasks in which subjects view models briefly to acquire spatial information, and then saccade to the workspace in order produce the copy. The difference between the visible copy and the remembered model defines where parts are missing in one relative to the other and where new parts should be added. Comparing the structure of the object that one is attempting to build to a central representation of a desired configuration, or blueprint, therefore appears to be a fundamental component of object construction. In the case where objects are physically assembled, the spatial comparison defined above may provide the target that the motor system is subsequently engaged to reach, namely the position in the object where the next part should be placed. The two processes, a targeting step dependent on a spatial analytic process assessing object structure and a motor implementation step involving physical movement, may be dissociable. The construction task we designed minimized motor planning and execution aspects of object construction, and was based on this premise. We hoped in this way to isolate patterns of neural activity related to a spatial analysis of object structure from neural activity coding the metrics of the movements that would normally implement the constructive operation.
By recording neural activity in parietal cortex during object construction, we found that parietal neurons represented the void within the copy object under construction, the place where the next part should go. Specifically, during the time that the copy object was visible, neural activity coded the location of the square that was missing from the copy relative to the immediately preceding model. This neural signal could provide the analysis-based targeting function necessary to direct construction. We found in fact that this neural signal predicted where monkeys were going to add squares to the copy object on both correct and error trials (Figs 7 and 8). The task was designed in such a way that the location of the missing square did not correlate with the direction of the movement required in order to add the square and complete the constructive operation.
From the perspective of prior parietal research, one interesting aspect of the neural signal we found during object construction was that neurons represented a position in empty space — a position that was devoid of a visual stimulus and was made significant by a cognitive operation. Prior work in area 7a in primates has characterized the involvement of this cortical area in spatial vision (Robinson et al., 1978; Motter and Mountcastle, 1981; Andersen and Mountcastle, 1983; Andersen et al., 1985, 1987, 1990b; Motter et al., 1987; Steinmetz et al., 1987; Snyder et al., 1998b) and spatial attention (Mountcastle et al., 1981, 1987; Steinmetz et al., 1994; Robinson et al., 1995; Steinmetz and Constantinidis, 1995; Constantinidis and Steinmetz, 2001a,b). In both roles, the activity of 7a neurons signaled the presence of a visual stimulus within their retinotopic visual receptive fields. In both roles, the intensity of neural activity evoked by the stimulus was found to be modulated by extraretinal factors, such as the position of the eyes or head (Andersen and Mountcastle, 1983; Andersen et al., 1985, 1990b; Snyder et al., 1998b) or the location or state of covert attention (Mountcastle et al., 1981, 1987; Steinmetz et al., 1994; Robinson et al., 1995; Steinmetz and Constantinidis, 1995; Constantinidis and Steinmetz, 2001a,b). In this prior work, the presence of a visual stimulus within the visual receptive field typically initiated neural activity in area 7a, and the location of the stimulus relative to a neurons retinotopic receptive field was a key spatial variable coded in neural activity. During object construction, the activity of 7a neurons was not initiated by the appearance of a visual stimulus within their receptive fields. There was no stimulus at the position that neurons were engaged to represent. Therefore neurons did not code the position of a visual stimulus. Instead, 7a neurons coded locations that were computed. The sensory independence demonstrated by this finding may have important implications for the involvement of parietal cortex in higher order spatial cognitive processing that is also likely to depend on the brain's ability to represent and manipulate spatial information apart from the particulars of concurrent sensory input.
Neurons Coded the Location of the Missing Square in the Construction Task
We found that the activity of a population of 7a neurons during the period that the copy object was visible varied systematically as a function of the location of the missing square and not as a function of the form of the copy object. This was evidence that neural activity coded a computed spatial datum and not the location of a visual stimulus. For example, neural activity during the time the copy object was visible clearly related to the location of the missing square across trials in which the physically identical copy object was presented (Figs 3 and 4a,b). The visual stimulation provided by the copy object was constant. The fact that neural activity continued to vary with the missing square in this case indicated that neural activity was not a function strictly of where light fell on the retina relative to the visual receptive fields of 7a cells. Further evidence that this was the case was provided by the fact that neural activation generalized across diverse pairs of model and copy objects that together connoted a common missing square location (Fig. 3).
One alternative interpretation of these data is that neural activity was due to the modulation of visual receptive field sensitivity by eye position, as shown in prior work (Andersen et al., 1985). However, gaze fixation was required during the construction task and mean eye position within the fixation window varied by a small fraction of a degree as a function of the location of the missing square (Fig. 6). Another alternative interpretation of these data might be that neural activity was related to the reactivation of a visual receptive field stimulated earlier in the trial, or to spatial working memory of the retinal location of a previously visible stimulus. We found, however, that the neural population continued to code the location of the missing square on trials in which the retinal position of the model was shifted relative to the copy object (compare Table 2 to Table 1). On these trials, no prior stimulus appeared at the retinal location of the missing square in the copy object, thus reactivation of a visual receptive field or spatial working memory were unlikely to account for the activity we observed.
Neurons whose activity varied with the location of the missing square were prevalent in area 7a. The activity of 43% of all area 7a neurons encountered that met a minimal spontaneous activity requirement (see Materials and Methods) varied significantly with either the horizontal or vertical position of the missing square (Table 1). One possible account for the prevalence of this signal in parietal cortex is that spatial representations in area 7a conform to the processing demands of tasks monkeys have been trained to perform. This possibility is generally consistent with the finding that parietal neurons represent different types of information as a function of task context. For example, when the color of a target is made significant in a saccade task, neurons in LIP acquire color sensitivity which they do not otherwise exhibit (Toth and Assad, 2002).
Potential Relation of Neural Activity to the Oculocentric Direction of a Planned Saccade
Other interpretations of the present data more closely related to the known sensory and motor properties of parietal neurons should be considered. One of these is that neural activity during the object construction task reflected motor planning or intention (Bracewell et al., 1996; Mazzoni et al., 1996; Snyder et al., 1997, 1998a). Specifically, neural activity may have reflected a motor plan that was never physically executed to move in a direction that was coupled to the location of the missing square. It is important to consider the potential relation to motor planning, particularly with regard to the question framed above as to whether cortical systems always represent spatial information that is defined in relation to the receptive fields or movement fields of the participating neurons. By design, movement direction did not vary in the construction task — monkeys controlled the constructive operation by varying the timing and not the direction of motor output. For that reason, neural activity coding the location of the missing square did not reflect a neural representation of the direction of the movement which the monkey planned and executed for reward while performing the task.
The monkey may have planned a movement it never made, however, and neural activity in area 7a may have related to this motor plan. We examined this question in the context of saccade planning, as various subdivisions of the inferior parietal cortex have been implicated in the cortical control of eye movements (Gnadt and Andersen, 1988; Andersen, 1989; Barash et al., 1991; Bracewell et al., 1996). We compared the activity of the same individual neurons on the construction task and on a delayed saccade task. In the saccade task, monkeys planned saccades toward spot targets that were placed at the same spatial positions where missing squares were located during the construction task (we also placed saccade targets at the positions of the choice squares). We focused our analyses on the subpopulation of neurons in which activity varied significantly with the location of the missing square during construction, and found that in the large majority of these neurons, activity did not vary significantly with the location of the target in the delayed saccade task. Populations of neurons in area 7a were indeed active during the delay and saccade epochs of the saccade task (Fig. 5e,f); however, these populations were largely distinct from the population of 7a neurons that coded spatial information relevant to the performance of the object construction task. We found that in a minority of neurons, activity varied significantly with the location of the missing square and also the location of the saccade target. However, the spatial preferences of these neurons in the two tasks were not tightly aligned (Fig. 5b–d). In a different behavioral context, we have similarly observed that parietal neurons coding task-relevant spatial information in spatial cognitive and saccade tasks do not necessarily code the same spatial coordinate in each of them (Crowe et al., 2004). Together, these data suggest that individual parietal neurons can participate in diverse spatial representations and that the spatial information they code in each may be to a degree independent. The failure of 7a neurons coding the location of the missing square to activate during the saccade task is evidence their activity did not represent an oculocentric saccade plan.
This interpretation is generally consistent with prior evidence that area 7a does not play a primary role in saccade planning and control. The activity of some area 7a neurons varies with the direction of saccades, but in most of these cases the onset of neural activity follows the initiation of the saccade (Barash et al., 1991), and so does not participate in planning the eye movement. Additional evidence that 7a does not play a direct role in saccade control has been provided by the results of microstimulation experiments; saccades are not evoked by the injection of electrical currents into area 7a that readily evoke saccades in neighboring area LIP (Thier and Andersen, 1998). Finally, the anatomical connectivity of area 7a does not suggest an immediate role in oculomotor control, at least in contrast to other cortical eye fields. For example, projections between area 7a and the superior colliculus or the frontal eye fields are weak or absent (Lynch et al., 1985; Andersen et al., 1990a).
The above findings suggest that neural activity in area 7a during the construction task did not represent the oculocentric direction of a saccade that monkeys planned but never executed (we consider the possibility that neurons were engaged to plan a saccade in object-centered coordinates below). Nonetheless, it seems likely that neural activity in 7a during object construction played a role in response planning, at least in a more abstract sense. For example, the construction task required monkeys to make a forced choice between alternative choice squares in order to determine the final configuration of the constructed object. This choice was based on the spatial relation between the alternative choice squares and the missing square. Consequently a neural representation of the location of the missing square may have been an important component of the mechanism by which that selection among alternative choice squares was made. Which choice square was selected in turn determined when the motor response occurred in relation to the choice illumination sequence, and ultimately the form of the object that was built. In this sense, neural activity may have served to guide behavior without coding the endpoint of a movement. If so, the early emergence of this signal (Fig. 4a,b), long in advance of the motor response, may be thought to reflect a form of response planning, but not necessarily in the simplest sense of coding a movement vector.
Potential Relation of Neural Activity to Object-centered Representations of Space
The primary finding in this study was that parietal neurons represented a spatial locus defined by a cognitive process during object construction. Neurons generally were not active to code the same spatial locus when defined by direct visual input, namely by placing a stimulus at the preferred location to serve either as the endpoint of a saccade (Fig. 5) or as the focus of attention (Fig. 11). This demonstrates that neural activity during construction was not related to the retinotopic or oculocentric coding of a spatial location when attention was directed to a visual target or a saccade was planned to a visual target. However, the differential activity of neurons between these contexts, object construction versus the control tasks, is an observation with at least two alternative interpretations because the construction and control tasks differed along at least two dimensions simultaneously. First, spatial cognitive processes involved in building objects were required in the construction task and were not required in the control tasks. Second, visual objects were displayed in the construction task and were not displayed in the control tasks.
The physical presence of a visual object may have influenced neural activity independently of cognitive operations involved in building objects. For example, Olson and colleagues have shown that neurons in the supplementary eye fields (SEF) code location in object-centered coordinates (Olson and Gettner, 1995, 1999; Olson and Tremblay, 2000; Tremblay et al., 2002). Importantly, SEF neurons exhibit object-centered spatial selectivity as long as movements are directed to the part of an object, even when object-centered spatial judgements are not required (Olson and Gettner, 1995; Tremblay et al., 2002). By analogy, 7a neurons in the present study may have failed to consistently activate on the control tasks because the control tasks involved behaviors directed toward spot targets (Figs 5 and 11). Visual objects were not present in the control tasks and consequently monkeys could not form object-centered spatial representations. The population of 7a neurons we studied may therefore have been engaged if the control tasks had required planning saccades or directing attention to parts of objects, in which case their activity would be more generally related to object-centered spatial coding, engaged for multiple behaviors in which this form of spatial representation was required.
It remains an open possibility that the neurons we studied would be active in tasks other than object construction, that support the formation of object-centered spatial representations. Determining the degree to which 7a neurons are exclusively specialized for construction versus other object-based tasks is an important question that will require further study. However, the present data establish the involvement of area 7a neurons in a specific spatial judgement relevant to object construction in a relatively simple and straightforward sense. The location of the missing square itself constituted a constructional judgement. The neural representation of this particular spatial datum establishes a link between a spatial analytic process assessing the structure of the objects presented in the construction task and neural activity in parietal cortex. This link is further substantiated by the finding that the location coded by neural activity predicts behavior. Neurons code the position where monkeys will place the new square within the object before the fact on correct and error trials (Figs 7 and 8). In this sense, the spatial representation of 7a neurons predicts the form of the object that is built. Thus neurons appear to provide the analysis-driven targeting function that might normally inform the motor system as to where to place the next part in an object during assembly. This targeting mechanism may be engaged in other contexts, such as when directing a saccade or attention to a part of an object. In representing specifically the location of the missing square in the context of object construction, however, the spatial targeting mechanism evident in the activity of 7a neurons displays a form of spatial intelligence suited to solving the spatial problem of building objects. The fact that neurons represented the location of the missing square also demonstrates that 7a neurons can code space independently of the retinal locations of visual targets, a fact that was further supported by the finding that the same neurons were not activated when a visual target was placed at spatially congruent locations.
Available evidence suggests that the influence of constructional judgements and object-centered spatial coding are not mutually exclusive, and that the two factors jointly determine spatial representation in parietal cortex. For example, preliminary evidence indicates that 7a neurons code locations in an object-centered framework during construction (Chafee et al., 2003). Consequently, the presence of a visual object influences the spatial representation of neurons in area 7a, as it does in the SEF. These data favor the view that constructional judgements and object-centered spatial coding are simultaneously influential factors whose intersection shapes neural activity in parietal cortex during object construction.
Relation of Neural Activity to Spatial Attention
The area of greatest overlap between the present data and the results of previous research in parietal cortex appears to relate to the established correlation between parietal function and spatial attention. We found evidence that spatial attention was covertly directed toward the location of the missing square during object construction (Fig. 9), the same location represented by the majority of 7a neurons that were active during the copy period (Fig. 10c,d). This deployment of attention was top-down in the construction task, attention moved to a point in empty space defined by a cognitive operation.
A correlation between the state or location of attention and neural activity in parietal cortex has been extensively documented (Mountcastle et al., 1981, 1987; Steinmetz et al., 1994; Robinson et al., 1995; Gottlieb et al., 1998; Kusunoki et al., 2000; Constantinidis and Steinmetz, 2001a,b; Bisley and Goldberg, 2003). For example, in area LIP, neurons preferentially code the locations of salient stimuli (Gottlieb et al., 1998; Kusunoki et al., 2000), and covert attention is found at the location represented by neural activity elicited by saccade targets or visual distractors during a delayed saccade task (Bisley and Goldberg, 2003). Likewise, prior studies in area 7a have shown that, as in area LIP, neurons preferentially represent the locations of salient visual stimuli (Constantinidis and Steinmetz, 2001b). These studies have also shown that attention has a suppressive effect on the magnitude of neural activity evoked by the appearance of a visual stimulus within the retinotopic receptive fields of 7a neurons (Steinmetz et al., 1994; Robinson et al., 1995; Constantinidis and Steinmetz, 2001a). This finding has been interpreted as indicating that 7a neurons are active in relation to the movement of attention (Steinmetz and Constantinidis, 1995), a hypothesis that is generally consistent with the positive correlation between response magnitude and stimulus salience (as attention often moves toward salient stimuli).
The suppressive effect of attention on visual sensitivity, which has been well documented in area 7a (Steinmetz et al., 1994; Robinson et al., 1995; Constantinidis and Steinmetz, 2001a), appears to be a correlate of attention that is distinct form the one we have found in the same cortical area during object construction. For example, during construction, attention was generally correlated with an increase and not a decrease in the firing rate of neurons coding the attended locus (Figs 3, 4 and 10). An additional distinction was that attention during construction did not manifest as a variable gain on the visual sensitivity of parietal neurons, because parietal neurons became active without a visual stimulus appearing in their receptive fields. It may therefore be that the neural activity we observed during object construction was more closely related to the previously reported correlation between neural activity in area 7a and behavioral salience (Constantinidis and Steinmetz, 2001b), except that in the present case, neural activity correlated with the behavioral salience of an abstract spatial coordinate and not the salience of a visual stimulus.
Parietal Activity and Abstract Spatial Processing
The present data indicate that abstract spatial information logically related to the solution of a spatial problem is present in the activity of neurons in area 7a. In this regard, our results are generally consistent with prior evidence that parietal neurons participate in neural representations of abstract information. For example, LIP neurons continue to code the inferred direction of motion of a moving stimulus while the stimulus itself is occluded from view (Eskandar and Assad, 1999). This effect cannot be mediated by the stimulation of visual receptive fields during the period that the coded stimulus is invisible. Other studies have shown that the activity of neurons in the intraparietal sulcus systematically relates to the number of items in a visual display regardless of their location (Nieder and Miller, 2004), and that neural activity in LIP represents elapsed time in a manner that parallels time perception (Leon and Shadlen, 2003). It is difficult to account for these findings solely in terms of the activation of the visual receptive or movement fields of parietal neurons. The present data suggest that, like the neural representation of numerosity, visual motion, or time, the neural representation of space in parietal cortex can likewise be abstracted from sensory or motor processing. In this regard the present results share some features with other neural data we have recently reported in area 7a during the performance of a visual maze task (Crowe et al., 2004). In that study, neural activity was systematically tuned to the direction of a path through a maze that a monkey was required to solve in the absence of eye movements. In both the maze and construction tasks we found evidence that the activity of area 7a neurons coded an abstract spatial parameter that was relevant to the spatial problem at hand.
In the present experiment, we have observed this form of task-tailored, abstract spatial representation in area 7a in a behavioral context involving the top-down control of spatial attention. Top-down attention is strategically deployed by humans to satisfy cognitive task constraints in natural visuomotor tasks (Shinoda et al., 2001; Triesch et al., 2003), including a block copy task (Hayhoe et al., 1998) that is directly analogous to the object construction task we have presently employed. The present data indicate that attention is strategically deployed during object construction in the nonhuman primate as well, and that this deployment is correlated with the activity of neurons in parietal area 7a. These findings may provide some insight into why damage to parietal cortex in humans can produce constructional apraxia (Kleist, 1934; Black and Strub, 1976; Villa et al., 1986; Ruessmann et al., 1988). Without the benefit of normal parietal function patients may lose the ability to deploy top-down attention strategically and this may interfere with constructional performance. We found in the present experiments that attention and constructional performance were both correlates of neural activity in parietal cortex.
Correlation versus Causality in Object Construction, the Scope and Limit of Interpretation Supported by the Neural Data
Though nearly ubiquitous in studies of this type, the distinction between correlation and causality places important limits on how the current results should be interpreted. Specifically, the present data do not establish that 7a mediates the spatial cognitive processing required to build objects. Instead, they document a correlation between neural activity and spatial cognitive function during object construction. Establishing a causal link, as opposed to a correlation, between the function of neurons and a behavioral or cognitive act requires the convergence of data from neural recording, lesion and microstimulation experiments at a minimum (the last two of which we have not conducted). Given a simplifying assumption (namely that neural firing rate is acceptable shorthand for information processing in the brain), the difference between correlation and causality is one essentially of anatomical scope. If the pattern and evolution of neural activity throughout the brain as a whole is considered causal with respect to the spatial cognitive processing that enables object construction, the question then reduces to what part of this functionally integrated but anatomically distributed processing takes place within area 7a.
It is nearly certain that the neural and cognitive operations required to construct objects are mediated by a distributed cortical system (Mountcastle, 1978, 1995) of which parietal cortex is only a part. This distributed system includes projections linking posterior parietal and dorsolateral prefrontal cortex directly (Cavada and Goldman-Rakic, 1989; Chafee and Goldman-Rakic, 1998, 2000), but also involves a parallel network of projections linking parietal and prefrontal cortex to numerous other cortical and sub-cortical targets in common (Selemon and Goldman-Rakic, 1988). The division of computational load among the nodes of this integrated system during complex spatial behaviors such as object construction is not yet known. Neurons in area 7a may serve as a passive spatial register storing the results of spatial computations performed by other nodes in this more broadly distributed cortical system. Alternatively, neurons in area 7a may participate in spatial computation directly, which in this case amounts to controlling the evolution of neural activity in the system in such a manner that the coordinate of the missing square is reliably computed and encoded. Differentiating between these possibilities is generally beyond the scope of single neuron recording experiments.
Granting the above limit, the present results illustrate that the representational capability of parietal neurons extends to abstract spatial variables computed by cognitive operations. This capability was not immediately apparent from previous work in which parietal neurons coded a spatial coordinate that most often coincided with the position of a visual stimulus. The present results also show that during object construction, parietal neurons represent a spatial datum that captures the difference between a model object and its copy, and predicts the position where parts are placed as the copy is built. Importantly, the coded spatial datum does not correlate with the direction of the motor response planned and executed during the task. This suggests that parietal cortex is part of a network providing a spatial targeting function during object construction that is based on a spatial analysis of object structure, and that selects positions within objects. Although the targeting mechanism may normally instruct the motor system, it appears to be at least partially independent in the present context, as points in space are selected during construction that do not coincide with the direction of the forthcoming movement.
We thank D. Lee, H. Merchant, T. Naselaris, G. Pellizzer and L. Romanski for providing constructive criticism and comments regarding this work. We thank D. Boeff for electrical engineering and computer systems assistance, and D. Evans for surgical and animal care assistance. Supported by USPHS grant NS17413, the United States Department of Veterans Affairs, and the American Legion Brain Sciences Chair.
1Brain Sciences Center, Veterans Affairs Medical Center, Minneapolis, MN 55417, USA, 2Department of Neuroscience, University of Minnesota, Minneapolis, MN 55455, USA, 3Department of Neurology, University of Minnesota, Minneapolis, MN 55455, USA and 4Center for Cognitive Sciences, University of Minnesota, Minneapolis, MN 55455, USA