There is growing evidence that in primate cerebral cortex the areas along the ‘dorsal pathway’ are involved in the transformation of visual motion information towards a motor command. To pursue this cortical flow of information from visual motion areas to the motor cortex, single-cell activity was recorded from visual areas MT/MST (middle temporal area/medial superior temporal area) and from primary motor cortex (M1) while monkeys tracked moving targets with their right hand. Spike activity of 353 directionally tuned motor cortex cells was combined to a time-varying population vector, and similarly a time-resolved visual population vector was calculated from 252 MT/MST cells. Both population vectors code faithfully for the direction of the collinear motion of target and hand. For a given direction, the length of the population vectors varied over time during the performance of the task. The temporal evolution of both population responses reflects the different relationship between the early visual responses to the moving target and the directional motor command controlling the hand movement. The results indicate that during the visual tracking task visual and motor populations which code for similar directions of movement are co-activated with considerable temporal overlap. Despite this co-activation in both modalities, we failed to observe any significant synchronization between areas MT/MST and M1.
When we perform reaching movements to moving targets, our visual system has to process the information about target motion, and this information is finally used to update the motor system during the control of hand movements. The cortical mechanisms involved in such a continuous transformation of visual information towards a motor action are not well understood. The cortical areas along the ‘dorsal stream’ are candidates for such a transformation, as they combine the processing of visual information with neuronal activity related to eye and limb movements (Ungerleider and Mishkin, 1982; Maunsell and Newsome, 1987; Van Essen et al., 1992). In recent years there have been several studies which aimed to analyze the different levels of visuo-motor transformation along the areas of the dorsal stream (Battaglia-Mayer et al., 1998, 2000; Goodale, 1998; Burnod et al., 1999). From this work one can conclude that the control of visually guided reaching requires a coordinated activation of multiple areas located in the medio-temporal and parietal cortex. To our knowledge, there has been no approach in primates to compare spike activity recorded simultaneously from different areas along this pathway during a functionally demanding visuo-motor task. In our experiments, we therefore set out to combine recordings from two positions along the dorsal ‘vision-for-action’ pathway while a monkey performed visually guided tracking movements. We selected visual motion areas MT/MST (middle temporal area/medial superior temporal area) and the primary motor cortex (M1) for simultaneous recordings with two multi-electrode systems to compare cortical activity on a visual ‘input level’ with neuronal activity from a motor cortical ‘output level’. In the current study, we searched for possible neuronal interaction on the basis of population responses calculated separately for directionally tuned visual and motor cells. We also analyzed all pairs of cells recorded simultaneously from visual and motor areas to test for possible synchronization of neuronal activity between these areas.
The areas MT and MST, located at the superior temporal sulcus, are specialized for the coding of the direction and velocity of moving stimuli. These areas can be regarded as an entry point of visual motion processing to the dorsal pathway. Cells in areas MT and MST are most sensitive to moving stimuli (Dubner and Zeki, 1971; Zeki, 1974; van Essen et al., 1981; Maunsell and van Essen, 1983a,b; Tanaka et al., 1986). They project to cortical area 7a and to other areas within the intraparietal sulcus: namely, the lateral intraparietal area (LIP) the ventral intraparietal area (VIP) (Boussaoud et al., 1990) and to the parieto-occipital area (Colby et al., 1988). Posterior parietal areas themselves are linked with premotor cortex, the superior colliculus and pontine nuclei (Wise et al., 1997; Lacquaniti and Caminiti, 1998). All these areas are known to influence various aspects of the visual control of eye, limb and body movements. In summary, one can conclude that the dorsal stream has the functional properties and interconnections that are needed for the moment-to-moment control of visually guided actions.
The main motor output of the cerebral cortex is routed through M1, which has a prominent role in the specification, initiation and execution of motor acts. The discharge of neurons in M1 relates to muscular activity, output force and torque (Georgopoulos et al., 1992; Wise, 1993) and there is now strong evidence that it also relates to the kinematics of movement, i.e. parameters such as direction and velocity, and to higher-order processing of sensorimotor information (Georgopoulos et al., 1986; Carpenter et al., 1999; Port et al., 2001; Lee et al., 2001). The majority of cells in the arm area of M1 is directionally tuned and most cells show a single preferred direction. Based on the ubiquitous presence of direction selectivity in the activity of motor cortical cells, the directional signal from a large number of cells can be read out as a population vector (Georgopoulos et al., 1986; Schwartz, 1994). Such a weighted vector sum of the population activity gave access to the temporal dynamics of the cortical control during ongoing hand movements (Schwartz, 1993) and it has been used successfully to monitor covert operations during cognitive tasks (Carpenter et al., 1999).
With regard to directional coding, the functional properties of direction selective cells in the visual motion areas MT and MST show similarities to cells in M1. Cells in MT and MST typically respond also with a high rate for a particular direction of stimulus movement and progressively less for directions further away from the preferred direction. Accordingly, we extended the population vector analyses to the data obtained from the visual motion areas. By processing the data from both modalities with an identical analysis, we are able to directly compare visual and motor activity on the population level. The population vectors from visual and motor areas code accurately for the direction of movement in retinal and in hand-centered coordinates, respectively. In a second step of analysis, we were able to elucidate the temporal relationship between the activity on the population level. With this approach, we here show that during visually guided tracking movements, the visual motion areas MT and MST mainly follow the time course of stimulus movement with 80 ms latency, whereas, at the same time, the motor cortex prepares for the upcoming linear tracking movement with a lead time of up to 300 ms. The motor activity was closely related to the kinematics of the hand, and the visual population response was dominated by the movement of the visual target. Additional to these well-known relationships, we could show that the motor population was partially related to parameters of the visual stimulus guiding the movement. However, the visual response did show only a weak relation to hand velocity which appeared too late in time to have influence on the control of movement.
The current paper proves that on the population level, simultaneous activity in both areas is present during visually guided tracking. The prerequisite for the cross-correlation study — a simultaneous activation in different modalities during a single behavioral task — was fulfilled. According to the so-called ‘binding hypothesis’ [for review, see (von der Malsburg, 1999; Singer, 1999)] one might expect that the distributed activity in spatially separated cortical areas show some amount of temporal synchronization, which could be interpreted as a sign of a dynamical cooperation between these areas. It is obvious that the visual areas MT/MST and M1 have quite different functional roles in an (hypothetical) ‘vision to action’ pathway. On the other hand, the idea seemed tempting to us that the behavioral demands of the tracking task should be sufficient to generate task related cortical synchronization and we therefore analyzed the simultaneously recorded cell activity from both areas for any sign of synchronization. However, with cross-correlation analysis of single cell spike trains we have not detected any sign of interaction between pairs of cells from M1 and visual areas MT and MST.
Materials and Methods
Single-cell recordings were conducted in awake, behaving monkeys performing visually guided manual tracking movements. Recordings were made from two hemispheres of two monkeys [two male monkeys (Macaca mulatta, 5.5 and 6.1 kg)]. All procedures were in accordance with published guidelines on the use of animals in research (European Communities Council Directive 86/609/ECC).
All monkeys were surgically prepared for chronic neurophysiological recordings. Monkeys were pretreated with atropine and sedated with ketamine hydrochloride. Under general anesthesia [pentobarbital sodium, 10 mg/kg i.v.] and sterile surgical conditions each animal was implanted with a device for holding the head. A scleral search coil was implanted to monitor eye position according to the method published by Judge et al. (Judge et al., 1980) and was connected to a plug on top of the scull. In both animals, two recording chambers were implanted over two separate craniotomies to allow recordings with two independent multi-electrode systems. One chamber was placed over the occipital cortex in a parasagittal stereotaxic plane tilted back 60 from the vertical. A second chamber was implanted over the frontal cortex centered just anterior to the central sulcus. The placement of both chambers was guided by magnetic resonance images, which were taken from each animal before the first surgery. Recording chambers, eye coil plug, and head holder were all embedded in dental acrylic that itself was anchored to the skull by self-tapping screws. Analgesics were applied postoperatively and recordings started no sooner than 1 week after surgery.
Experimental Set-up for Control of Behavioral Data
During the experiments, the animals were seated comfortably in a primate chair allowing them to move both hands freely. Both animals were trained to use their right hand only for moving a two-joint manipulandum placed in front of them in a horizontal plane above the level of their hip. The sensor of a digitizing table was attached to the handle of the manipulandum and was moving almost frictionless above the digitizing table. This allowed to measure the position of the hand with a spatial resolution of 0.1 mm. The hand position data were sampled with 75 Hz by a PC and were displayed in real time as a feedback cursor (red dot, radius 1.4° visual angle). All visual stimuli were presented on a translucent vertical screen placed 114 cm in front of the animal which subtended a viewing angle 57° wide and 43° high. Moving the handle of the manipulandum 10 mm towards the screen caused an upward movement of the feedback cursor by 1.17° visual angle. During the experiment, black curtains darkened the area surrounding the animal.
The feedback cursor and all additional visual stimuli were generated by a single PC which controlled the hand data as well. A high performance graphic board (ELSA Winner, 2000 Pro/X, Aachen, Germany) served to generate the real time video output, which was back-projected to the translucent screen with a video projection system (Electrohome ECP 4100, Kitchener, Ontario, Canada, 75 Hz frame rate, 800 × 600 pixels video resolution). Eye position was measured with the scleral search coil technique and the analog output of the eye monitor system (Primelec, Regensdorf, Switzerland) was fed to the same control PC. By this setup, hand and eye position were continuously controlled with a single PC and the information about the hand position was used in real time to update the video display of the feedback cursor. The same computer was used to store all neuronal data collected during the experiments. The software for the generation of the visual stimuli, the online control of the animals behavior and for the collection of all data was developed by one of the authors (W.K.).
Visually Guided Tracking Task
During all phases of the task, the monkey had to fixate a stationary, central green spot which was always drawn on top of all other stimuli on the vertical screen. The fixation window had a radius 2.1° in visual space and was not visible to the animal. A trial was aborted immediately if the animal broke fixation. This requirement of central and immobile fixation was included to avoid any possible influence of eye position on cell activity.
The visually guided tracking task consisted of four phases (Fig. 1). A tracking trial was started when the animal placed its hand in a central position in the horizontal workspace by moving the feedback cursor in a central start window indicated by a white circle drawn on the vertical screen (radius 2.1° visual angle, equals 18 mm radius in manual workspace). This start position (Fig. 1A), which was always the same during all conditions and all recordings, had to be held for a variable time period ranging between 800 and 1500 ms. After the initial center hold phase, a bright bar appeared in one of four peripheral positions and started immediately to move towards the center of the screen with a constant velocity of 7°/s. The bar moved in one of the four cardinal directions (up, down, left, right), and the orientation of the bar was always perpendicular to the direction of movement (Fig. 1B). During this pre-track phase, the monkey had to maintain the position of the eyes and of the hand in the central window. Accordingly, the feedback cursor stayed in the central window during this time. After 1250 ms, the bar passed the midpoint of the central window and moved uninterrupted through the center of the screen. At the same time, the circle marking the size of the central window was extinguished and the monkey had to initiate a manual tracking movement, resulting in an simultaneous movement of the stimulus and the feedback cursor. The visual fixation on the immobile, central green dot had to be maintained the whole time. During the movement of the hand, the maximum distance between the center of the bar and the feedback cursor had to be <2.1° (Fig. 1C). The corresponding tracking window was not visible to the monkey, and a trial was aborted immediately when the monkey failed to fulfill this spatio-temporal condition. Accordingly, the animal had to initiate the movement in a temporal window of ±300 ms relative to the time where the target crossed the fixation point. To facilitate the continuous tracking of the center of the target, the moving bar contained a central dot, which was slightly brighter than the rest of the bar. In case that during the manual tracking movement the feedback cursor and moving target overlapped in space, the red dot representing the feedback cursor was always displayed on top of the white bar. The tracking phase lasted again 1250 ms, causing a symmetrical movement of the bar around the central start window of the hand. Then the bar stopped and stayed visible for an additional 1000 ms (Fig. 1D). In this target hold phase, the animal had to keep the hand position in a target hold window (2.1°) centered on the middle of the bar. After successful completion of a trial, the animal was given a liquid reward.
To obtain the directional tuning of cells from M1, the animals were trained to make straight movements from a central start position to one of eight equally distributed peripheral targets (70 mm in hand space). This center→out task has been used in a similar form by several other investigators before [e.g. (Georgopoulos et al., 1982; Fu et al., 1993]. In contrast to previous studies, we added the requirement for the central fixation during this task to avoid switching between conditions with constrained eye movements (as during the tracking task) and conditions with free eye movements. The eight center→out conditions were presented to the animals intermingled with the four tracking conditions in a randomized block design. We recorded a minimum of five successfully completed repetitions from four tracking conditions and eight center→out conditions.
Visual Control Task
The preferred directions of the visual cells were measured by having the monkeys fixate a central spot and moving a whole field random dot pattern (57° × 43° visual angle) on a circular pathway (7° radius) in clockwise or counterclockwise direction. The computer-generated pattern was always covering the full screen, causing the impression as it moves behind a large aperture. The translatory motion of the random pattern along the circular path produced at a fixed position on the screen (for example, at a receptive field of a neuron) a movement in continuously changing directions (Schoppmann and Hoffmann, 1976; Hoffmann and Distler, 1989). Activity of all direction sensitive cells was modulated according to the continuous change of stimulus direction in its receptive field. During completion of a full circle, all possible directions were covered. In a single trial, the pattern moved for 3.5 s, and needed 3.33 s to complete the circular path. For each cell, a minimum of five repetitions with translations in clockwise and counterclockwise directions was measured. Additionally, the receptive fields of most visual cells were determined qualitatively with handheld stimuli while the animals were fixating.
Extracellular Recording and Data Acquisition
Neuronal activity was recorded from M1 and from areas MT and MST at the superior temporal sulcus using two separate multi-electrode manipulators (Thomas Recording, Giessen, Germany). Activity of single cells was detected in real-time by means of a computer controlled multi-channel spike sorter (Plexon Inc., Dallas, TX). Time stamps for detected spikes were stored with 10 μs resolution by the same PC controlling the behavior of the animal.
Calculation of Preferred Directions of Motor Cells
The directional tuning of the motor cells was calculated from the discharge rates during five repetitions of the center→out movements in eight directions. The mean spike discharge rate for each direction of movement was calculated for the time from the appearance of the peripheral target until the feedback cursor reached the target. The preferred direction was calculated using standard directional statistics [‘mean direction’ (Mardia, 1972)]. The statistical significance of the directional tuning was tested with a non-parametric, statistical bootstrapping technique, similar to a method used by Lurito (Lurito et al., 1991). For this test, the length of the mean resultant, R, was calculated (Mardia, 1972) using the discharge rates during all 40 movements to weight the corresponding movement directions. A new sample was generated by assigning randomly the observed discharge rates to movement directions, and the mean resultant was calculated. This procedure was repeated 100 times, and the lengths of 100 resultants were rank ordered. If the length of the observed mean resultant, R, was greater than the 95th percentile in the distribution of the mean resultants, the cell was considered to be directionally tuned. If the cell was directionally tuned, the direction of the mean resultant was taken as the cell's preferred direction. This test was chosen to allow a similar approach for the data from motor and visual cortex without making assumptions about a particular shape of the directional tuning (e.g. a cosine model).
Calculation of Preferred Directions of Visual Cells
The spike activity during the visual control task was analyzed to obtain the directional tuning of the visual cells. To avoid an influence of transient responses to motion onset, the activity during the first 150 ms after the start of the pattern motion was excluded. Each spike measured during the following 3.333 ms (time for the completion of pattern motion on a full circular path), was transformed in a unit vector pointing in the direction of stimulus movement at the time of spike occurrence. For the pattern movements in clockwise and counterclockwise direction, the sum of all unit vectors was calculated separately. The preferred direction of a cell was taken as the mean vector of both directions. A possible influence of the response latency on the preferred direction of a cell was cancelled out by averaging data from clockwise and counterclockwise translation.
For testing the statistical significance of the preferred direction of the visual cells, we again used the non-parametric, statistical bootstrapping technique described above. For this test, the spike trains of an individual trial were rearranged by shuffling the sequence of spike intervals, and each spike was transformed in a unit vector pointing in the direction of stimulus motion at the corresponding time. The resultant vector constructed from the shuffled data was noted. The procedure was repeated 100 times for each trial recorded during clockwise and counterclockwise stimulus movement, and the length of the resulting vectors were rank ordered. If the length of the observed mean, R, was greater than the 95th percentile in the distribution of the shuffled mean resultants, the directional tuning for the given condition (clockwise or counter-clockwise) was considered to be significant. Only when this 95th percentile was reached for both stimulus directions, the average direction from both stimulus directions (clockwise and counterclockwise) was taken as the preferred direction of the cell.
Calculation of Neuronal Population Vectors
The neuronal population vector is the weighted sum of vectorial contributions of individual cells (Georgopoulos et al., 1988). For the calculation of the population vector, peristimulus time histograms (13.3 ms bin-width, 75 Hz) were computed for each cell which proved to have a statistically significant preferred direction. This vector sum was calculated in an ongoing, time-varying fashion for all four conditions, and we used counts of fractional intervals as a measure of the intensity of cell discharge. For a given time bin, each cell made a vectorial contribution in the direction of its preferred direction and of magnitude equal to the change in cell activity compared to the rate observed during the last 0.5 s preceding the onset of the moving bar in the periphery (‘control rate’, that is, while the monkey was fixating the center and while holding the handle at the center of the plane). The population vector P for the jth stimulus condition (i.e. tracking direction) and the kth time bin is
The data from M1 and from visual areas MT/MST were combined separately in two different population vectors which evolved over time during the performance of the tracking task.
Multiple Linear Regression of Population Vector
To quantify the relation of the population vectors to the kinematics of the target and the hand, we performed a multiple linear regression of the population response with the averaged time course of hand and target kinematics. In this regression, the length of the population vector was expressed as a function of hand position, hand velocity, target position and target velocity. To analyze the temporal relationship between these parameters and the population activity, we shifted the data of the hand and the target independently relative to the population data. Accordingly, we calculated a single regression for each combination of time shifts:
To analyze the temporal structure in the spike trains of simultaneously recorded pairs of cells from areas MT/MST and M1, we calculated cross-coincidence histograms (CCHs) from the corresponding spike trains. The CCH comprises all intervals between the spikes from both cells within the time window under study, up to a maximal delay of typically ±128 ms. All CCHs were computed with a bin width of 1 ms. Only cells with a total of more than 500 spikes during the time course of the recording were included in this analysis. Each CCH was tested for any prominent modulation (oscillatory or non-oscillatory) indicating synchronized activity in both cells. The particular steps of the synchronization analysis was guided by a previous study of our group (Cardoso de Oliveira et al., 1997), in which non-oscillatory synchronization in area MT has been observed especially during an expectation period while the monkey expected a low contrast pattern in a direction discrimination task. Accordingly to the previous study, we analyzed only CCHs which contained more than 1000 entries with a maximum delay ofless than 100 ms. Briefly, the subsequent steps were as follows: the shift-predictor was low-pass filtered and subtracted from the raw CCH and the resulting difference correlogram was expressed in Z scores, i.e. in units of standard deviation. Only peaks in this difference correlograms which exceeded a Z score of 3.0 were taken as significant. To avoid false positives, two additional criterion had to be fulfilled: (i) the peak had to be significant after filtering the correlogram by a three-point averaging filter which assured that the peak was not caused by a single bin exceeding the significance level, (ii) we tested whether the correlation was consistently detectable when the trials were divided in two sub-groups. Only if in both groups a significant correlation occurred, the cell pair was scored as significantly correlated.
Histology and Reconstruction
During the last days of recording we made electrolytic microlesions (10 μA for 12 s) in the motor cortex and in the recording area located at the superior temporal sulcus of both monkeys. Standard histological procedures were used to identify the location of the electrolytic lesions and to reconstruct the relative spatial position of the electrode tracks during the recording sessions from the first monkey. All recording positions located at the superior temporal sulcus which yielded cells with significant directional bias were compatible with a location in area MT or area MST. Based on this preliminary separation between MT and MST cells, we calculated two distinct population vectors for MT and MST cells from the first animal. We could not find any significant difference between results obtained from these sub-populations and therefore combined the data from all directionally tuned visual cells to a single population vector. The recording positions from the precentral cortex proved to be located in the area rostral of the anterior bank of the central sulcus, which is in correspondence to the functional properties of the cells showing a clear relation to movements of the proximal arm.
Directional Tuning of Cells from Visual and Motor Areas
We recorded the activity of 605 arm-related cells from M1 from two animals. Of those, 294 and 311 cells were recorded in the first and second monkey, respectively. Most cells changed activity in relation to proximal movements of the contralateral arm as judged by examination of the animal outside the behavioral task. A fraction of 353 motor cells (58.3%) showed a significant directional tuning in the center→out task. For the individual animals, this ratio was 189 out of 294 (= 64.3%) for the first monkey and 164 out of 311 (= 52.7%) for the second monkey.
From 426 cells located in visual areas MT and MST we recorded the spike activity during the visually guided tracking task, whereof 174 cells were from the first and 252 cells were from the second monkey. Two hundred and fifty-two cells (59.3%) showed a statistically significant preferred direction when tested in the visual control task (64.4 and 55.6% of the cells in the first and second monkey, respectively). The distribution of preferred directions of the directionally tuned cells from both the visual and the motor cells ranged throughout the directional continuum (Fig. 2). None of the visual and motor sub-populations from either animal showed any statistically significant directional bias (Rayleigh test for uniformity, P > 0.1 in both cases).
Motor Population Vector
We included all directional motor and visual cells in the calculation of a motor and a visual population vector, respectively. To construct the population vector over time, all trials were aligned to the onset of target movement. The temporal evolution of the motor population vector is shown in Figure 3A. The four rows correspond to the tracking movements in four different directions. In each row, the temporal evolution of the population vector is plotted together with the temporal profile of hand and target kinematics. It can be seen that the motor population vector predicts the direction of an upcoming tracking movement as the vectors starts to point constantly in the direction of the target movement several hundred milliseconds before the hand starts to move. Even during the late center hold period in which the moving stimulus is approaching the start position, the motor vector already points in the direction of the upcoming movement. This continuous directional signal which is visualized here as the motor population vector can be read by other structures of the CNS to control the upcoming movement of the hand.
To make the variability of the vector length in Figure 3 more obvious, the time-varying length of the population vector is included as a gray line. In all conditions, the motor population vector starts to lengthen early before the onset of the movement. In general, the temporal profile of the length of the population vector resembles the profile of the hand velocity, with the population vector leading the hand velocity by ~300 ms.
The mean velocity profile of the hand does not match the constant velocity of the target very well. This is probably related to the relative high velocity of the tracking movement the monkey had to perform. To ensure that the moving target causes an appropriate activation in areas MT and MST, a visual velocity of the target of 7°/s was selected, which is already at the lower end of velocities preferred by these visual motion areas. A higher speed of the moving target would have increased the difficulties for the monkey to make proper tracking movements.
Visual Population Vector
One goal of the study was to elucidate whether a similar directional signal can be read out form the population signal of visual cells coding for visual motion. The corresponding vectors calculated from the cell activity of the visual population are shown in Figure 3B. The data are arranged in the same way as the motor data. Again, the direction of the population vector corresponds well to the direction of stimulus movement. For the upward movement (shown in the second row in Fig. 3), the length of the population vector corresponds best to the speed profile of the stimulus. During horizontal movements, the time course of the visual population vector shows a remarkable asymmetry: the population response becomes stronger when the stimulus moved through the right visual hemifield. In the opposite direction, there is a reduced response when the stimulus entered the left hemifield. We assumed that this asymmetry was not related directly to the particular directions, but can be accounted for by the fact that we restricted our recordings in both animals to the left cortical hemisphere. As a consequence, most of the visual cells had receptive fields in the right hemifield. This is documented in Figure 4, where we plotted the density of receptive fields recorded from the second monkey as a function of the horizontal distance from the fixation spot. For this analysis, we counted the receptive fields which covered visual space at a given horizontal eccentricity. From the distribution of receptive fields it becomes obvious that a visual stimulus was represented by many more neurons when it was located in the right hemifield. Visual stimuli in the left visual field were only represented poorly in the population response due to this bias in the receptive field locations. Accordingly, the overall neuronal activity in our population of visual cells increased when the stimulus entered the right hemifield (first row in Fig. 3B, movement to the right) and decreased when the stimulus moved from the right to the left hemifield (third row in Fig. 3B). Of course, with recordings from the right hemisphere it would have been the reverse, as demonstrated by the mirror image of the distribution given in the left curve in Figure 4. If we construct a distribution representative of recordings from both hemispheres by mirroring the receptive field locations obtained from the right hemisphere and subsequent summation with the measured distribution (bold curve in Fig. 4), the receptive fields would have covered the central visual space evenly.
To overcome the resulting asymmetry in the population vectors, we mirror imaged the x-component of each vector and of the hand and target data during the movement to the left, and subsequently averaged the population vectors from both horizontal movements, as well as the kinematic data. The corresponding y-components were averaged without being mirrored before. The mirror-imaged data emulate recordings from area MT and MST in the right hemisphere during movements in a single horizontal direction, i.e. to the right. This procedure of mirroring and subsequent summation of measured and mirror imaged data has a similar effect as observed for the receptive field density function shown in Figure 4. When non-mirrored data from the left hemisphere and mirrored data were added, the lengths and the direction of the averaged vectors for horizontal movements (now pointing to the right) are almost constant during a movement across the visual field.
Taking the data from one hemisphere only, the gradients in the population response during the horizontal movements could be interpreted as a code for the instantaneous distance of the target from the horizontal meridian. However, the positional information of the visual target is encoded also through the retinotopic organization of area MT, and this coding is not well captured by the vector interpretation of the cell response [in contrast to a positional population code, for example used by Jancke et al. (Jancke et al., 1999)]. With the mirroring of the vector data we stayed in the conceptual frame of the vector model, which favors the correspondence between the magnitude of the visual population vector and target speed.
The same transformation of mirroring the horizontal component and subsequent averaging was performed for the motor data, and the results for both populations during the horizontal movements are shown in Figure 5. For better visualization of the data, vectors were rotated 90° counterclockwise so that vectors coding for rightward movement are now pointing upwards. This rotation avoids that most vectors fall in line with the temporal axis and allows a better estimation of the vector's length. This rotation is purely for visualization and does not affect the data or has any implications for further steps of the analysis.
Average Across Conditions
The average across both horizontal directions was introduced to eliminate the effect of the strong bias in the responses of the visual areas to the stimulation in the contra-lateral visual field. We adopted the mirroring also on the data obtained during the vertical components to treat both subsets of data similarly and to eliminate a possible bias in the spatial distribution of the receptive fields along the vertical axis. This vector average gave us two sets of time-evolving population vectors, one for the horizontal movements (Fig. 5) and a similar set for vertical movements (data not shown), together with kinematic data of hand and target movement. For horizontal and for vertical movements, the population vectors code faithfully for the direction of the moving target and for the direction of the upcoming hand movement, respectively. For the motor population, this directional coding manifests more than 300 ms before the onset of the hand movement, whereas for the visual population vector, the vectors start to indicate the direction of target movement with a latency of ~80 ms after movement onset. At this point, the population data from vertical and horizontal movement do not show remarkable differences. As the direction of target and hand movement was constant during a particular condition, we disregard the directional information of the population vectors for the analysis of the temporal modulation of the population activity and base our subsequent analysis on the time-varying magnitude of the vectors. The scalar average of the length of the population vectors from horizontal and vertical movements is shown in Figure 6, together with the kinematic data of hand and target describing the movement co-linear with the target direction. This grand average displays the temporal modulation of the population vector length and the relative timing of the directional population response relative to the onset of target and hand motion. At this point, several aspects of the population response can be summarized: the visually guided tracking task employed in this study proved to be well suited to activate motor and visual populations at the same time. During the task, the activity of both cell populations constitute a continuous directional signal in M1 and in visual motion areas MT and MST. The temporal evolution of both signals shows that the directional signal is coexistent in both populations for an extended time span. The length of the motor population vector was modulated in time according to the speed profile of the hand, with a remarkable early activation up to 300 ms before onset of hand movement. The visual population vector resembled most closely the speed profile of the visual stimulus, following the onset of stimulus movement with a latency of 80 ms.
From the time course of both population vectors shown in Figures 3, 5 and 6, it is not clear whether position or velocity of target and hand motion is coded most faithfully by the two populations. Furthermore, the temporal relationships between the population vectors and the kinematics of hand and target can be obtained only qualitatively from Figure 6. For further analysis of the population activity, we therefore performed a multiple linear regression of the population data, where we tried to explain the variability of the population vector length by the time course of hand position and velocity as well as target position and velocity.
Results from Multiple Linear Regression
In the multiple linear regression, the average of time-varying length of the population vector across all four conditions (as shown in Fig. 6) was taken as the dependent variable. The corresponding traces of the averaged components of hand and target data were taken as the independent variables. Calculation of the regression yielded the relative contribution of these components to the motor and to the visual population, respectively. This analysis was not broken down into separate correlations on each parameter as the multiple linear regression allows to estimate the relative contribution of each parameter when all parameters contribute to the model.
The resulting coefficients of determination (R2) can be improved by introducing time-shifts between the population data and the traces of hand and target motion. These time-shifts compensate for the temporal latencies of the visual response and the temporal lead of the motor activity, respectively. The multiple regression was recalculated therefore for multiple combinations of time shifts (see Materials and Methods). Each combination of time shifts for the hand and target data (in steps of the sampling interval = 13.3 ms) yielded a different result for the regression and a different R2 value. We searched for the combination of time shifts where the highest R2 value was observed by varying the time shifts over larger periods, up to ±400 ms. The results for time shifts up to ±133 ms relative to the combination yielding the maximum regression were plotted in Figure 7.
For the regression of the motor population vector, the highest R2 = 0.943 was obtained for a shift of the hand data by +293 ms and a shift of the target data by –240 ms. In other words, the regression was best when the motor population vector was fitted with the data of hand movement (position and velocity) from 293 ms later, and with the data of target motion 240 ms before. These shifts for the maximum regression correspond to a prediction of the upcoming hand movement by the motor cortical activity, and a reaction of the motor cells to the target movement 240 ms before. The regression equation for this combination of shifts was:
For the visual population vector, the similar analysis yielded a maximum R2 = 0.953 for a shift of hand data by –260 ms and a shift of target data by –80 ms. The maximum value for regression was obtained with hand data from 260 ms before and the response to the visual stimulus movement 80 ms before. The corresponding regression equation was:
For the visual population vector (Fig. 7B), the most prominent feature in the plot is the pronounced rim along a temporal shift of the target data by –80 ms, which is accompanied by the steep decreases of the R2 values for neighboring shifts. In fact, a convolution of the square wave velocity profile with the square wave shaped length of the population vector would give a triangle-shaped form for the plane of the regression results. Accordingly, the clear maximum along the shift of target data by –80 ms, together with the steep gradients on both sides of the maximum, can be taken as an indication that the visual population vector is mainly influenced by the speed of the visual target. The maximum at –80 ms reflects the temporal latency of the visual population response relative to the motion of visual stimulus. For the shift of the hand data, the location of the maximum in the regression plane occurred at a shift of approximately –260 ms, but the absence of a clear gradient indicates that the hand data have only a weak influence on the regression. In Figure 6 it can be recognized that the visual vectors respond with 80 ms latency to the start of target motion. An additional peak in the length of the vector appears after the hand started to track the target. This additional activation — probably caused by the visual movement of the feedback cursor — seems to occur with a longer latency relative to hand movement than the initial activation at the start of target motion. According to the relative timing of this peak, the highest regression was obtained when the hand data were shifted by –260 ms. However, as mentioned above, the only small variation in the regression coefficient for a shift of the hand data (Fig. 7B) suggests that there exist only a weak, but nevertheless significant, interaction between hand data and population activity in visual areas MT/MST.
In contrast, the equivalent plot resulting from the regression of the motor population data (Fig. 7A) shows a stronger gradient along the shift of the hand data and a weaker, albeit pronounced, gradient along the shift of the target data. The location of the maximum implies that the temporal evolution of the motor population generates the upcoming hand movement with a lead time of 296 ms. Along the same lines of interpretation, the position of the maximum R2 value along the shift of the target data indicates that the modulation of the motor population reflects the parameters of target motion from 240 ms before. From comparison of the regression of both population vectors, one can summarize that the population vector in M1 is related to hand and target parameters, as the regression is influenced by both temporal shifts of hand and target data. In contrast, the population vector obtained from activity in MT and MST shows a strong relation only to the parameters of the visual target and only a weak influence by the parameters of the hand movement. Furthermore, there is no indication for a predictive component, neither for target nor for hand motion, in the population vector of visual areas MT and MST during this visually guided manual tracking task. We point out that the time-shifts yielding the highest regression results do not imply a strict serial processing in which the hand and target data are coded by the population vector subsequently with fixed relative latencies. Instead, we assume that the latencies gained from the regression model indicate a parallel processing of multiple parameters, including hand and target kinematics. Also, it should be noted at this point that these temporal relations strongly depend on the particular temporal layout of the tracking task, where the information about the direction of movement is given early in the task and the hand movement is held back until the second half of the target movement was reached.
In addition to the temporal relationship between the kinematic parameters and the length of the population vector, we wanted to evaluate the relative importance of all parameters entered in the multiple linear regression. To compare the relative importance of a given variable in the regression equation, we calculated the standardized regression coefficients, which were obtained by expressing the observations as Z scores (i.e. in standard deviation units). This facilitates a comparison among variables with different units (i. e. position versus speed). The standardized coefficients were calculated for the regressions yielding the highest R2 values. Rank ordering the standardized coefficients showed that for the motor population vector, the speed of the hand is the most important parameter, followed by target speed, hand position and target position. To test whether the length of the population vector was related to one or more of the parameters tested, the t statistic and its probability level were calculated for each coefficient. For the motor vector, all four parameters tested contributed significantly at the 5% level (after Bonferroni correction for multiple comparisons) to the variation of the population vector. For the visual population vector, target speed ranked highest, followed by hand speed. Both hand and target position did not contribute significantly to the regression model of the visual population vector.
Results from Cross-correlation Analysis
In part of the experiments, we recorded extracellular spike activity with two multi-electrode systems simultaneously from visual areas MT/MST and from motor cortex. In 44 recordings with different electrode constellations, we collected activity from both structures simultaneously. These recordings yielded a total of 744 inter-areal cell pairs. To avoid unreliable results from recordings with low numbers of spikes, we restricted our analysis to cells from which we recorded a minimum of 500 spikes (427 pairs). The quantification of the resulting CCHs was performed when the CCH holds more than 1000 entries. This set of requirements was fulfilled by 382 cell pairs. From these 382 CCHs, not a single correlogram showed a significant peak according to our criteria described in the Materials and Methods section. When we analyzed intra-areal CCHs with both cells recorded from visual areas MT/MST, 22 out of 276 cell pairs (8.0%) were classified as synchronized. In M1, 4.5% (8 CCHs out of 178) showed a significant peak (Fig. 8). Whereas we found a moderate incidence of synchronized activity in areas MT/MST during the visually guided movements and a lower fraction of synchronized pairs in recordings from motor cortex, we have not found any sign of significant synchronization between M1 and visual areas MT/MST during visually guided tracking movements.
This study compares population activity in M1 and in visual motion areas MT and MST during visually guided tracking movements. A behavioral task was designed in such a way that the visual information about the moving target was essential to fulfill the spatial and temporal requirement of the motor response. We found that both areas were active over an extended period of time during the manual tracking, and that the direction of movement is coded faithfully and co-linear in both areas on the population level. The temporal evolution of activation in the motor and in the visual areas was dominated by the velocity profile of the corresponding hand and target movement, respectively. These results emphasize the notion that a continuous processing across multiple areas takes place during visually guided hand movements.
Based on the co-activation of both populations coding for similar movement directions, we set out to gain insight about the temporal aspects of the transformation of visual motion information towards a motor command and the possible interaction between these areas. Since movement direction was constant during the task, the parameter that varied in time was speed. We therefore collapsed the data obtained during movements in different tracking directions to a single set of data. In the subsequently obtained ‘generalized’ population activity, the temporal variability of the population response was most closely related to the velocity profiles of hand and target.
The temporal relationship between the kinematics of target and hand movement and the length of the population vector was further analyzed by multiple linear regression. The regression analysis revealed that during the visual tracking task the motor cortex codes for the kinematics of upcoming movement with a lead time of ~300 ms. The visual motion areas MT and MST follow the onset of stimulus motion by ~80 ms.
In previous studies that analyzed the motor population vector, a time lag of 120 ms between motor cortical activity and the limb movement has been described for continuous drawing movements (Schwartz, 1993). A study by Ashe and Georgopoulos (Ashe and Georgopoulos, 1994) compared the coding of movement parameters in M1 and area 5 during center→out movements. They used a comparable multiple linear regression model to analyze single-cell activity from both areas and obtained a median shift for the highest R2 of –90 and +30 ms for motor cortex and area 5, respectively. The remarkably extended lead time of the motor population during the tracking task in our study might be accounted for by the fact that the direction of the movement was indicated quite early during the task. As the target started to move 1250 ms before the animal had to start the hand movement, the information about the movement reached the motor cortex early before movement onset. The latency of the visual response to the onset of the target is in similar range as latencies obtained from single-cell responses during visual stimulation with flashed stimuli (Schmolesky et al., 1998).
Motor cortex activity as well as visual driven activity in MT and MST is most tightly coupled to the velocity of the hand and the target, respectively. A similar ranking was observed by Ashe and Georgopoulos (Ashe and Georgopoulos, 1994) in M1. In their study, they found a strong influence by the direction of movement in 55% of the cells. Hand velocity influenced most strongly the activity of 27% of the cells, followed by hand position and hand acceleration. In our population data, the influence of motion direction is implicitly included in the direction of the population vector. As we averaged the population data from different directions, the direction of movement is not included as a parameter in the regression. In our data, the regression of the visual population vector underlines the functional specialization of areas MT and MST for the processing of motion information. The results from our regression analysis emphasizes the tight coupling to the velocity of the visual stimulus.
Possible Influence of Set-related Activity on Preferred Direction of Motor Cells
A comparable visually guided tracking task has been used in a study that focused on the activity in M1 and in the cerebellum (Johnson et al., 1999). In a fraction of cells from M1, the authors described a significant change in the preferred direction during the time course of the tracking. In contrast to their study, we tested the preferred directions of the motor cells with a separate center→out task. We used the preferred directions derived from this control task as constant values for the calculation of the population vector. The change in preferred direction observed by Johnson et al. (Johnson et al., 1999) was equally distributed without any bias towards a particular direction. We therefore do not expect that such a possible modulation of direction preference had a systematic influence on the resulting population vector in our analysis.
Simultaneous Recordings Do Not Reveal Inter-modal Synchronization
The negative result from the synchronization analysis might not be too surprising as the visual and motor cortical areas under study are separated by several cortical processing levels or even cerebro-cerebellar loops. On the other hand, a main proposal of the binding hypothesis gives rise to the expectation that cortical activity could be dynamically synchronized, even when the corresponding areas are separated by large distances (Eckhorn et al., 1988; Engel et al., 1991) [for a review see (Singer, 1999)]. Given that the cortical activity is related to a single external object — or to a single behavioral action — the neuronal coding of such an entity might be particularly supported by internally generated synchronization. In the conceptual frame of the binding hypothesis, such synchronization should even occur across different modalities, for example, to support the linking of the dispersed neuronal representation of a single object (Roelfsema et al., 1997). The behavioral visuo-motor task used in our study was designed partly with respect to these requirements, as real-time processing of visual information was mandatory to control the sustained motor response. We failed to elicit inter-modal neuronal synchronization on the level of paired spike trains recorded simultaneously during several repetitions of the behavioral task. However, in ~20% of the neurons recorded from MT and MST, we observed pronounced oscillatory modulation of spike activity in the gamma range (>40 Hz), which was synchronized without phase lag between spatially distant visual cells. Such stimulus dependent oscillatory modulation and synchronization in areas MT/MST was observed only during a purely visual task which did not require a manual motor response (for example, our visual control task described in the Materials and Methods section). These stimulus-dependent oscillations and their synchronization in areas MT/MST will be described in a separate paper. During the manual tracking task, we did not find any such oscillatory activity in areas MT/MST nor any synchronization between neuronal activity in MT/MST and M1.
We thank Dr C. Distler for performing surgery on the monkeys and for the histological processing. This work was supported the Deutsche Forschungsgemeinschaft, SFB 509 ‘Neurovision’ and by a Helmholtz stipend to W. Kruse from the Bundestministerium für Bildung, Wissen-schaft, Forschung und Technologie (BMBF).