As our eyes move, we have a strong percept that the world is stable in space and time; however, the signals in cortex coming from the retina change with each eye movement. It is not known how this changing input produces the visual percept we experience, although the predictive remapping of receptive fields has been described as a likely candidate. To explain how remapping accounts for perceptual stability, we examined responses of neurons in the lateral intraparietal area while animals performed a visual foraging task. When a stimulus was brought into the response field of a neuron that exhibited remapping, the onset of the postsaccadic representation occurred shortly after the saccade ends. Whenever a stimulus was taken out of the response field, the presaccadic representation abruptly ended shortly after the eyes stopped moving. In the 38% (20/52) of neurons that exhibited remapping, there was no more than 30 ms between the end of the presaccadic representation and the start of the postsaccadic representation and, in some neurons, and the population as a whole, it was continuous. We conclude by describing how this seamless shift from a presaccadic to postsaccadic representation could contribute to spatial stability and temporal continuity.
Every time we move our eyes, a shifted view of the visual scene lands on the retina. Because neurons in visual cortex have retinocentric receptive fields, cortical representations of the visual scene change, sometimes dramatically, from fixation to fixation (see Fig. 1 in Wurtz 2008). However, our percept is that the visual scene is stable in space and time: It does not appear to smear or jump around as our eyes move; instead, it appears as a stable environment, even though the retinal image is abruptly changed by each saccade. We refer to this perceptual stability in time as temporal continuity. For >150 years (Bridgeman 2007), mechanisms have been proposed that could generate this stable percept (Thier et al. 2001; Wurtz 2008; Hall and Colby 2011; Mathot and Theeuwes 2011). Many have incorporated the idea that brain areas involved in generating eye movements can send signals to other brain areas telling them when and to where the eyes are going to move. This would allow the visual system to keep track of where objects are in space by accounting for the eye movements. Strong evidence for such a theory has accumulated in the last 20 years or so. In 1992, Duhamel and colleagues showed that a subset of neurons in the lateral intraparietal area (LIP) of parietal cortex update their response fields such that a stimulus that is going to appear in the response field after an eye movement can elicit a response before the eyes move (Duhamel et al. 1992). This mechanism is commonly referred to as remapping or predictive remapping and is predominantly found in brain areas that are thought to guide covert attention (Bisley and Goldberg 2010; Krauzlis et al. 2013; Squire et al. 2013). More recently, Sommer and Wurtz (2002, 2006) provided evidence that remapping relied, at least in part, on a signal being sent from the superior colliculus, an area which drives eye movements, to the frontal eye field via the medio-dorsal nucleus of the thalamus. This corollary discharge (or efference copy) acted in much the same way that had been predicted earlier (Bridgeman 2007): An area controlling the movement tells relevant cortical areas when and to where the movement will be made.
While these data provide evidence that a corollary discharge signal is present and that it contributes to remapping, it is unclear how remapping itself may contribute to our perceived stability in space and time. A primary issue has been that remapping in parietal cortex (and in the frontal eye field of prefrontal cortex) is not temporally consistent. When examining onset latencies, remapping can occur any time from 150 ms before the eyes move to 100 ms after the eyes move (Umeno and Goldberg 1997; Kusunoki and Goldberg 2003), so it is hard to see how a temporally inconsistent response could contribute to a temporally stable percept.
Several years ago, we found that when presented with an array of stimuli, a robust informative signal is remapped in LIP across the entire visual field just after eyes stop moving (Mirpour and Bisley 2012a). In this study, we further analyze these data to examine the responses before and after the saccade to identify when activity in LIP shifts from the presaccadic to the postsaccadic representation of stimuli in visual space. We find that the remapping mechanism coupled with a sudden shutoff of the presaccadic representation produces a quick and precise shift from the presaccadic to postsaccadic representation. In our discussion of the data, we suggest a way in which these responses could explain the percept of stability.
Materials and Methods
All experiments were approved by the Chancellor's Animal Research Committee at UCLA as complying with the guidelines established in the Public Health Service Guide for the Care and Use of Laboratory Animals. Two male rhesus monkeys (8–10 kg) were implanted with head posts, scleral coils and recording cylinders during sterile surgery under general anesthesia (Bisley and Goldberg 2006; Mirpour et al. 2009). Animals were initially anesthetized with Ketamine and Xylazine and maintained with isofluorane. The animals were trained on a memory-guided saccade task and on the foraging task (Fig. 1). Experiments were run using REX (Hays et al. 1982). Visual stimuli were presented on a CRT (running at 100 Hz, 57 cm in front of the animal) using the associated VEX software. Eye position signals were sampled using a magnetic search coil system (DNI) at 2 kHz and recorded for analysis at 1 kHz.
Neuronal Recording and Behavioral Tasks
Single unit activity was recorded from 2 monkeys using tungsten microelectrodes (Alpha Omega, Israel). The location of LIP was determined using MRI images, and neurons were only included if they or their immediate neighbors showed typical visual, delay and/or peri-saccadic activity in a memory-guided saccade task (Barash et al. 1991). The size and position of the response field of each neuron was estimated using the visual responses from an automated memory-guided saccade task (Mirpour et al. 2010). In this task, a target was presented for 200 ms after a fixation period of 300–500 ms. This was followed by a 600-ms delay after which the fixation point was extinguished and the animal had 450 ms to make a saccade to the remembered location of the target. After a correct saccade, the animal was rewarded and the target reappeared. Targets were presented in 9 and then 25 locations in a square grid, the size of which depended on the estimated size of the response field. In our analyses, we use the abbreviation CRF (current response field) to refer to the response field before the saccade and FRF (future response field) to refer to the response field after the saccade. Note that after each saccade, the FRF becomes the neuron's response field after the visual latency. In the text, we refer to the FRF as the postsaccadic response field when describing responses well after the saccade. Neurons that did not allow us to have only a single stimulus in the receptive field were rejected, so the CRF and FRF never overlapped.
To examine responses across multiple eye movements within a trial, we trained the animals on a variation of a visual search task in which they must forage among many stimuli for one loaded with a reward. This task allows us to gather data across multiple saccades within each trial and to examine the responses to different stimuli as they are brought into and out of the response field. Each trial of the foraging task (Fig. 1) started with a fixation point appearing on the left, right, or the center of the monitor. The monkeys had to fixate on the fixation point for 450 to 700 ms to start the task, after which an array of 5 potential targets (T) and 5 distractors (+) appeared on the screen. Each stimulus was 1.2° × 0.8° of visual angle. One of the potential targets was loaded with reward. The monkeys had 8 s after the start of the trial to fixate the reward loaded target (within a 2° window) for 500 ms to receive the reward. Usually, the animals looked from T to T, waiting at each for ∼650 ms (Mirpour et al. 2009). The stimuli were arranged such that when the monkey looked at one stimulus, the neuron's response field often encompassed another stimulus (large ovals, Fig. 1). On every trial within a session, stimuli were presented in the same spatial locations, but the positions of the potential targets and distractors within the array were randomly assigned.
Saccades were detected using velocity and amplitude threshold criteria and were visually verified by the investigator. To be included in the analyses presented here, the saccade had to take gaze from within 2.5–4° of one stimulus to within 2.5–4° of another stimulus; the size of these windows depended on the eccentricity of the response field. We use the term fixation to refer to the period between the detection of the end of a saccade and the detection of the start of the next saccade. We use the term stable fixation to refer to a period starting ∼80–90 ms after the detection of the end of the saccade, at which time discriminative information reaches LIP (Ipata et al. 2006; Buschman and Miller 2007; Mirpour and Bisley 2012a) and ending ∼20 ms before saccade onset is detected.
Neuronal Data Analysis
The data here are of a further analysis of those presented previously (Mirpour et al. 2009; Mirpour and Bisley 2012a). Data were recorded from 52 LIP neurons (29 from Monkey E and 23 from Monkey C). Action potentials were discriminated online using the MEX pattern spike sorter, and sorted spikes were time stamped and stored at 1 kHz in REX. We analyzed neuronal activity around saccades in which there was 1 object or no objects inside the response field before the saccade and 1 object or no objects inside the response field after the saccade. For most analyses, data were aligned by the end of the saccade (i.e. when the eye stopped moving and fixation began), but when examining the postsaccadic burst, we also align the data by the start of the saccade. All other results were similar, but temporally less constrained, when the data were aligned by the start of the saccade (see also Mirpour and Bisley 2012a). We only included data from saccades that were not directed toward the response field of the neuron to avoid the responses being affected by the movement activity.
For all analyses, we counted the number of spikes within 25-ms windows, shifted by 1-ms steps. When plotted over time, the data for the window are plotted at the mid-point of the window, so the response at time 0 is taken from −12.5 to 12.5 ms. For single neuron responses, the mean in any single condition represents the average activity from all the saccades in that condition and the mean across conditions is the average of the means for each condition. For the population responses, the mean in each condition is the average of the means for that condition from each neuron. When a subset of neurons is examined, the mean in each condition is the average of means for that condition from each of the neurons in the subset. For all statistical tests, we used P < 0.01 to indicate significance. When comparing significance over time, it is important to remember that isolated significant events may be due to chance, given the use of multiple comparisons. We do not infer anything from these points but present all the data so the reader can get a feel for the probability of type I errors (false positives).
To determine whether the responses to targets and distractors were different, we performed ANOVAs using 25-ms bins, shifted by 1- or 5-ms steps. Under conditions in which there was a stimulus in both the CRF and FRF, we used a two-way ANOVA in which the dependent variable was raw firing rate in the 25-ms window from each saccade event and the 2 independent, main factors were stimulus identity (target or distractor) in the CRF and stimulus identity in the FRF. For this analysis, a significant interaction would indicate that the stimulus identity in one condition affected the responses in the other condition. Under conditions in which there was a stimulus in only the CRF or the FRF, we performed 2 ANOVAs: one for each condition. In each case, the dependent variable was raw firing rate in the 25-ms window from each saccade event and the independent, main factor was stimulus identity (target or distractor) in the CRF or in the FRF, depending on which data set was being analyzed. In this case, there was no interaction analyzed. When pooling all conditions, we also performed 2 ANOVAs: one to examine the effect of stimulus identity in the CRF and one to examine the effect of stimulus identity in the FRF. Because we only wanted to see a main effect due to responses to targets and distractors, we only used saccades in which either a target or a distractor was in the response field being tested (CRF or FRF). However, we used saccades in which a target, distractor, or nothing could be present in the other response field. Because we ran one ANOVA to test for each condition, we obtained 2 sets of interaction effects: one when examining the CRF and one when examining the FRF. For a given neuron at any epoch, we defined it as having a significant interaction if the interaction was significant in either ANOVA.
Neuronal data were collected from 2 monkeys performing a foraging version of a visual search task. The animals were trained to search through an array of 10 objects (Fig. 1) to find the stimulus linked with a reward. The reward was only ever linked to 1 class of stimulus (T-shaped potential targets), and the LIP responses were greater for these potential targets than for the unrewarding, +-shaped, distractors (Mirpour et al. 2009; Mirpour and Bisley 2012a). Within a session, we arranged the stimuli such that when the animal was looking at 1 item (small circles, Fig. 1), another was usually in its response field (ovals, Fig. 1). Here, we examine the responses of 52 LIP neurons around the time of a saccade. We refer to a stimulus in the response field before and after the saccade as the presaccadic and postsaccadic stimulus respectively and refer to the response fields as the CRF (current response field) and FRF (future response field). For example, RF0 in Figure 1 is the CRF for Saccade 1 (S1) and RF1 is the FRF for Saccade 1. In this case, there is no presaccadic stimulus in RF0 and the postsaccadic stimulus in RF1 is a distractor. Similarly, RF1 is the CRF for Saccade 2 (S2) and RF2 is the FRF for Saccade 2. In this case, the presaccadic stimulus in RF1 is a distractor, and the postsaccadic stimulus in RF2 is a potential target. When describing responses during stable fixation after the end of the saccade, we refer to the FRF as the postsaccadic response field.
Many neurons had responses that discriminated between targets and distractors during stable fixation. In these neurons, whenever a stimulus was in the CRF before the saccade onset, the response during stable fixation was greater if the stimulus was a target rather than a distractor. Likewise, whenever a stimulus was in the postsaccadic response field after the saccade, the response during stable fixation was greater than if that stimulus was a target rather than a distractor (Mirpour and Bisley 2012a). This can be seen in Figure 2, which shows the response profiles under 8 stimulus conditions of 2 neurons in Figure 2A–D and the pooled population response of all 52 neurons in Figure 2E,F. The left column shows the responses when a stimulus was present in both the CRF and FRF, and the right column shows the responses when a stimulus was present in either the CRF or the FRF and the other was empty. The legends beneath each column show the 4 classes of trials in each condition. These cover all the classes of trials recorded when a single stimulus was in the response field.
Whenever a stimulus was in the CRF before the saccade onset, the response during stable fixation was greater if the stimulus was a target (black and green traces in the left column and black traces in the right column) than if it was a distractor (red and blue traces in the left column and blue traces in the right column). Likewise, whenever a stimulus was in the postsaccadic response field after a saccade, the response during stable fixation was greater if the stimulus was a target (blue and green traces in the left column and green traces in the right column) than if the stimulus was a distractor (red and black traces in the left column and red traces in the right column). There are several additional points to take from this figure. First, the presaccadic responses to stimuli in the CRF appeared to be biased by what would appear in the FRF and the postsaccadic responses to stimuli in the postsaccadic response field appeared to be biased by what had appeared in the CRF before the saccade. For example, in the neuron in Figure 2A, the presaccadic response to a distractor in the CRF was biased by whether a target would follow in the FRF (blue trace) or whether a distractor would follow in the FRF (red trace). In the same neuron, the response during stable fixation, starting 100 ms after the saccade, to a target in the postsaccadic response field was biased by whether it was preceded by a target in the CRF (green trace) or by a distractor in the CRF (blue trace). Similar biases are apparent in the population response (Fig. 2E); in a following analysis, we will illustrate the number of neurons that show interactions in their responses during these periods. Second, within 50–150 ms after the end of the saccade, the responses to both targets and distractors in the postsaccadic response field were slightly higher when nothing had been in the CRF (right column) than when a stimulus had been in the CRF (left column). The lower responses in the latter are probably due to neuronal adaptation from elevated activity during the presaccadic fixation period. Finally, many cells appeared to have a brief burst after the saccade. This can be seen as the upward blip in the traces in Figure 2C,D, which are absent in the traces in Figure 2A,B. Given that many cells exhibited this burst, it is also clear in the population response (Fig. 2E,F). We will examine this response property below, after analyzing the timing of the remapped signal.
To examine the transformation from the presaccadic to postsaccadic representation of the array, we pooled responses when like stimuli were in the CRF or FRF. Figure 3 shows the responses of the single neuron illustrated in Figure 2AB in these conditions. The left column shows the pre- and post-saccadic responses to targets (black traces) and distractors (gray traces) when a stimulus was in both the CRF and FRF. Each trace is the average of 2 traces in Figure 2A. For example, the black trace in Figure 3A (T in CRF) is the average of the black (T to D) and green (T to T) traces in Figure 2A. The middle column shows the pre- and post-saccadic responses to targets (black traces) and distractors (gray traces) when a stimulus was in either the CRF (Fig. 3B) or the FRF (Fig. 3E). These traces are those from Figure 2B, replotted to declutter the panels. The right column shows the pre- and post-saccadic responses to targets (black traces) and distractors (gray traces) averaged across all conditions. So, for example, the black trace in Figure 3C (T in CRF) is the average of the black (T to D) and green (T to T) traces from Figure 2A and the black (T to O) trace from Figure 2B. The proportions of saccades in each condition were fairly constant across sessions because of the geometry of the array. Using the nomenclature from Figure 2: T to T, T to D and D to T each made up ∼10% of saccades, D to D, T to O and D to O each made up ∼15% of saccades and O to T and O to D each made up ∼13% of saccades (standard deviations for each condition was 3–4%). Given that we had an average of 1175 ± 665 (mean ± SD, range 141–2723) saccades per session, each condition had enough trials that the mean responses are a good representation of each neuron's average activity and, thus, the combined responses are a good representation of each neuron's average activity.
To determine whether the black and gray traces were significantly different, we performed ANOVAs using a sliding 25-ms bin, shifted in 1-ms intervals (see Materials and Methods for details). Note that the use of the 25-ms bins limits the accuracy with which we can show changes in response rate but gives us an estimate for when these occur. The solid black bars on the x-axis in Figure 3A show the times in which there was a main effect (P < 0.01) of stimulus identity in the CRF, and the black bars on the x-axis in Figure 3D show the times in which there was a main effect of stimulus identity in the FRF. The responses of this neuron consistently and significantly discriminated target from distractor in the CRF until ∼18 ms after the end of the saccade (Fig. 3A) and consistently and significantly discriminated target from distractor in the FRF starting ∼8 ms after the end of the saccade (Fig 3D). Thus, in this neuron, both the pre- and post-saccadic stimulus were represented in the response, as measured in 25-ms bins, ∼8–18 ms after the end of the saccade. So when a stimulus was present in the response field both before and after the saccade, the neuron appeared to represent either the presaccadic or postsaccadic stimulus at all times.
When the saccade took the stimulus out of the CRF, but no stimulus entered the FRF (Fig. 3B), the timing of the neuronal response was similar to when the stimulus leaving the CRF was followed by a stimulus being brought into the FRF (Fig. 3A): The neuron consistently and significantly discriminated target from distractor in the CRF until 20 ms after the end of the saccade. There were sporadic periods after the activity had gone down when the response following a target in the CRF was lower than the response following a distractor in the CRF (∼75–175 ms in Fig. 3B). This was somewhat common in neurons that discriminated between targets and distractors and may be a result of adaptation following the higher response to the target. When the saccade brought a stimulus into the FRF, and no stimulus had been in the CRF, the neuron did not discriminate between a target and distractor in the FRF until ∼54 ms after the saccade ended (Fig. 3E). This is ∼45 ms after the discrimination occurred when a stimulus had been present in the CRF (Fig. 3D).
When we pooled all conditions and examined the responses, there was even more time during which the neuron discriminated between a target and distractor in both the CRF (Fig. 3C) and FRF (Fig. 3F). The neuronal activity significantly (P < 0.01, ANOVA) discriminated between a target and distractor in the CRF up until 50 ms after the end of the saccade (Fig. 3C). However, significance was not found in every window after 22 ms. Given the many multiple comparisons we perform and the use of a 25-ms window, we take the time until which all windows were significant (22 ms) as a better indication of when the presaccadic representation ended in the neuronal response, but our interpretation remains the same if the later time is used. In this neuron, the onset of the postsaccadic representation was relatively clear (Fig. 3F): Approximately 6 ms after the saccade ended, the response to targets or distractors in the FRF became consistently and significantly different. So when we include all possible conditions in which a target or a distractor is in the CRF or FRF, the neuronal response represented the presaccadic stimulus before the saccade, the postsaccadic stimulus well after the saccade, and both for ∼15 ms starting shortly after the end of the saccade.
The population responses (Fig. 4) were similar to the single neuron example (Fig. 3). When a stimulus was in both the CRF and FRF, the population significantly (P < 0.01, Wilcoxon Sign Rank tests) discriminated between a target and distractor in the CRF until ∼17 ms after the end of the saccade (Fig. 4A) and significantly and consistently discriminated between a target and distractor in the FRF starting ∼7 ms after the end of the saccade (Fig. 4D). This shows that in the population, as in the single neuron example, there is a rapid (∼10 ms) transition from the presaccadic representation to the postsaccadic representation, with a brief period of overlap. Even considering that we are using 25-ms bins, it is unlikely that the population does not represent one or the other at any time. There are a few points before the saccade where the population activity significantly discriminated between stimuli in the FRF (Fig. 4D), which could be interpreted as indicating that the postsaccadic representation appears before the saccade ends. We are hesitant to make this conclusion for 2 reasons. First, because we have not controlled for multiple comparisons, it is likely that we may have false-positive errors. When we have hundreds of consecutive bins that are significant at the P < 0.01 level, we feel confident that this is unlikely to occur due to chance, but when we see significance for only 8 ms using 25-ms bins, it is possible that this is a false positive. Second, as we will show later, no single neuron showed a consistent significant difference during this time (see Fig. 5D) and when we examined the difference between the response to targets and distractors in neurons that discriminate between the 2 during stable fixation, we found no difference in the response at this time (Fig. 6D). So while there is a hint of an overall response bias before the saccade ends, it does not appear to be robust in single neurons or in the population.
As in the single neuron example, the response profile when the response field was empty before or after the saccade was a little different (Fig. 4B,E). When the saccade took a stimulus out of the CRF and there was nothing in the FRF, the neuronal response significantly discriminated between a target and distractor until ∼22 ms after the end of the saccade (Fig. 4B). As in the single neuron example, there was a residual difference that appeared 120–200 ms after the saccade ended. We suggest that this difference, in which there is sporadic but robust significance, may be a result of adaptation: After responding more to the target, the activity drops to a lower level. A more important difference is that while the response to a stimulus in the FRF following nothing in the CRF begins around the time the saccade ends (Fig. 4E), the response does not discriminate between a target in the FRF or distractor in the FRF until ∼43 ms. This is >20 ms later than in the condition in which a stimulus was present in the CRF. On its face, our data may appear to suggest that remapping does not occur in this situation, but there are 2 arguments against this interpretation. First, we found that the response began to increase just as the saccade ended. This is too early to be driven by the retinal input. Second, the time of discrimination (∼43 ms) is much earlier than can be explained by the feedforward inputs from the retina, which occurs, on average, around 80–90 ms (Ipata et al. 2006; Buschman and Miller 2007; Mirpour and Bisley 2012a).
When we take all the conditions in which a target or distractor is in the CRF (Fig. 4C), we find that the population response distinguishes between the stimuli until ∼20 ms after fixation onset and when we take all the conditions in which a target or distractor is in the FRF (Fig. 4F), we find that the population response begins to distinguish between the stimuli about 12 ms after the end of the saccade. Because we are pooling neurons with different response fields and saccades from all over the array, we can interpret the population response as giving us a good representation of what neurons across LIP would be doing in different locations after any given saccade. Thus, we can conclude that even when we include all conditions, there is clean transition from one representation to the next: LIP appears to represent the array at all times during search.
The transition from the presaccadic to postsaccadic representation was due to consistent remapping within a subset of neurons. To examine when single neuronal responses could discriminate between a target and distractor, we used running ANOVAs on data from 25-ms bins, every 5 ms, with pre- and post-saccadic object identity as main factors. We illustrated the results of this type of analysis in Figure 3 by the black bars on the x-axes. Figure 5 shows the results of these analyses for all neurons. The top row illustrates the percentage of neurons that showed significant effects as a function of time (P < 0.01, ANOVA). The red traces show the percentage of neurons that show a main effect of presaccadic stimulus identity, the blue traces show the percentage of neurons that show a main effect of postsaccadic stimulus identity, and the green traces show the percentage of neurons that show a significant interaction. The bottom row shows the blocks of time (25-ms windows every 10 ms) in which each neuron had a significant main effect of presaccadic stimulus identity (red), postsaccadic stimulus identity (blue), or both main effects were significant (yellow). The arrows indicate the cell illustrated in Figure 3.
When a stimulus was present in both the CRF and FRF (Fig. 5A,D), ∼40–50% of the neurons significantly discriminated targets from distractors in the CRF before the saccade (red) and in the FRF after the saccade (blue). No more than 20% of neurons showed significant interactions at any time; the majority of these interactions occurred before the saccade or in the 50 ms after the saccade ended (green trace, Fig. 5A). This confirms that, in some neurons, the responses before the saccade were significantly biased by the identity of the stimulus after the saccade. This may be a result of the early presaccadic remapping seen in previous studies. There was a brief period of time from ∼10 to 25 ms after the end of the saccade (gray column, Fig. 5A), in which <20% of the neurons represented the pre- or post-saccadic stimulus identity, after which a population of ∼20% of neurons showed a significant main effect of postsaccadic identity. The percentage of neurons then increased starting about 50 ms after the end of the saccade as more and more neurons began to show significant differences in their responses.
When a stimulus was present in only the CRF or FRF (Fig. 5B,E), the same neurons significantly discriminated targets from distractors in the CRF (red). As in the population response, we found that it took longer for individual neuronal responses to discriminate between target and distractor in the FRF after the saccade (blue). Some neurons, which had consistent significant differences when a stimulus had been in the CRF, had sporadic significance to stimuli in the FRF when they had not been preceded by a stimulus in the CRF (compare blue data in Fig. 5D,E), again suggesting that the presence or absence of a stimulus in the CRF can affect responses after a saccade quite dramatically.
When we pooled all the conditions (Fig. 5C,F), we found that a robust 50% of neurons showed a significant difference in response between target and distractor in the CRF up until ∼5 ms before the end of the saccade. This dropped to chance levels (∼5%) within 30 ms. Starting about 15 ms after the end of the saccade, the proportion of neurons that showed a significant difference in response between target and distractor in the FRF increased until ∼20% of neurons showed significance: ∼30 ms after the end of the saccade. The gray column shows a 10-ms window, centered at 25 ms, during which <20% of the neurons represented the pre- or post-saccadic stimulus identity. Even during this time, there are ∼10% of neurons that show a significant main effect of presaccadic stimulus identity, postsaccadic stimulus identity, or significant main effects of both pre- and post-saccadic identity, suggesting that in this small population the transition from presaccadic to postsaccadic representation appears to be continuous.
Thus far, we have analyzed the times that single neurons and the population can significantly discriminate between a target and distractor; however, we have not looked at the magnitude of the responses. For our claim that there is a sudden transition from the presaccadic to the postsaccadic representation of the array to be meaningful, the remapped activity should be robust and remain relatively constant until the visual signal from the eyes can reach LIP. We have previously shown that from the subset of 29 neurons from which we could examine the responses to array onset in this experiment, it took ∼90 ms for the response in LIP to discriminate between a target and distractor (see Fig. 2D in Mirpour and Bisley 2012a), so we predict that the difference in response between targets and distractors should remain constant from the onset of the postsaccadic representation to ∼90 ms. An examination of the data in Figure 4D,F confirms this intuition, the traces appear to separate quickly and consistently and then widen ∼90 ms. To examine this in more detail, we plotted the difference between these traces for our single example neuron (Fig. 6, top row) and for the 19 neurons in which we saw a significant main effect of presaccadic stimulus identity in the CRF 150 ms before the end of the saccade and a significant main effect of postsaccadic stimulus identity in the postsaccadic response field 150 ms after the end of the saccade (Fig. 6, bottom row). We chose to use only neurons that show a significant difference during stable fixation because these neurons have the most reliable differences and, thus, should give the most accurate representations of the differences as a function of time. We get qualitatively similar, albeit noisier, results if we pool all the neurons.
When a stimulus was present in both the CRF and the FRF, the response difference in the single neuron between the target and distractor in the FRF (black trace, Fig. 6A) took ∼100 ms after the end of the saccade to match the difference in response in the CRF seen before the saccade (gray trace). Importantly, the response difference in the FRF remained somewhat constant from when it started around 10 ms until ∼70 ms, after which it began to increase again. We saw a similar trend in the population response: The response difference to the stimuli in the FRF began to match the response difference in the CRF ∼130 ms after the end of the saccade (Fig. 6D), and the initial response difference in the FRF stayed constant until ∼80 ms after fixation onset, after which it slowly increased. Given that not all of these neurons displayed remapping, the increase beginning at ∼80 ms is probably a combination of the incoming visual information from neurons that remap and the addition of responses from neurons that do not exhibit remapping. Note that before the saccade ends, the response difference is close to zero. This suggests that the small difference seen in the full population (Fig. 4D) is not due to neurons that encode both pre- and post-saccadic stimulus identity. Importantly, in this population, there was only ∼20 ms between when the full presaccadic representation began to disappear and when the remapped postsaccadic representation plateaued out and the traces crossed at ∼15 ms. Because we are using 25-ms bins, it is possible that there is an even sharper transition than we are showing here.
In both the single neuron (Fig. 6B) and the subpopulation of neurons that encode both pre- and post-saccadic stimulus identity (Fig. 6E), we saw evidence of remapping prior to 90 ms when a stimulus was only in the FRF (black traces). In both cases, the response difference in the FRF rose to an initial plateau which then increased again once the afferent response from the retina reached LIP. As we saw when examining the statistics of the differences, this began later than when there was a stimulus in both the CRF and FRF, particularly in the subpopulation of neurons that encode both pre- and post-saccadic stimulus identity (compare the onset of the black trace in Fig. 6D,E). In this case, the remapped response reached its plateau around 50 ms. However, the second increase in response difference occurred around the same time under both sets of conditions.
When we pool all our data to examine the response differences to targets and distractors in the CRF or FRF (Fig. 6C,F), the pattern of responses is a little less clear. We continue to see the difference in response to stimuli in the CRF decrease around the end of the saccade (gray traces), but there is less of a stable plateau of response difference in the FRF early after the saccade ends. This is because we are averaging traces that show an earlier remapping effect (Fig. 6A,D) and a later remapping effect (Fig. 6B,E), so when combined they made a shallower slope. Nonetheless, we find that there is only about a 40-ms period during which there is neither a full presaccadic representation nor a stable postsaccadic representation. Moreover, we believe that this illustrates how robust the timing is of the remapped onsets in the 2 cases with stimuli in both response fields (Fig. 6D) and stimuli in one or the other response field (Fig. 6E): Both show very sharp increases in response difference even when averaging across 19 neurons. We interpret these data to mean that there is a rapid transition from the presaccadic representation to the postsaccadic representation when the saccade takes one stimulus out of the response field and brings another in and that the stabilization of the postsaccadic representation when nothing had appeared in the CRF appears within 40 ms. Most importantly, even before stabilization, the population activity in LIP always represented either the presaccadic or postsaccadic stimulus array.
To see whether the postsaccadic burst of activity, which was obvious in some neurons, was present in all neurons exhibiting remapping and, thus, may be related to the remapping mechanism, we devised a way of quantifying whether a neuron displayed the burst and then asked whether it was present in neurons that exhibited remapping or not. Given that the amplitudes and durations of the saccades in this free viewing task vary (see Fig. 2E,F), we are able to see whether the burst of activity is better aligned by saccade onset or by the end of the saccade, when the postsaccadic stimulus enters the FRF. The underlying idea is that if the activity is triggered by an event (saccade onset or stimulus entering the response field), then the onset in the mean response, which averages across trials, should be more sudden when aligned by the event that triggers it. Figure 7 shows the mean normalized population response in a 75-ms window around the time of burst onset when aligned by saccade onset (left panels) or by the end of the saccade (right panels) when a stimulus was in both the CRF and FRF (top panels) and when a stimulus was in only the CRF or FRF (bottom panels; see Figure 2 for more detailed legend). Because the burst in response rides on top of the response to the presaccadic stimulus, we normalized the responses by the level of activity just prior to the onset of the burst. We find similar results when we subtract the response at this time but use normalization here because detection of bursts was more robust when using a percentage increase than an absolute response increase.
It is clear that the onset of the burst of activity is more tightly linked to the start of the saccade than to the end of the saccade (Fig. 7). Burst onset occurred more rapidly in all conditions when a stimulus was in both the CRF and FRF, and the data were aligned by saccade onset (Fig. 7A) than when the data were aligned by saccade offset (Fig. 7B). In fact, when aligned by the end of the saccade, the burst was almost unnoticeable when nothing was in the CRF (red and green traces in Fig. 7D, see also Fig. 2F) but is clearly present, albeit small, when aligned by saccade onset (Fig. 7C). The relative burst height appears lowest when a target had been in the CRF (black and green traces in Fig. 7A, black traces in Fig. 7C), but because the burst is riding on a strong presaccadic response, the absolute responses are relatively strong (see Fig. 2).
Approximately two-thirds of the neurons (33/52) exhibited the small burst after the saccade. Given that the burst is best seen when aligned by the start of the saccade, we used this alignment to define whether neurons had the burst or not. To identify bursts, we asked whether the activity 35 ms after the start of the saccade was >10% higher than (1.1 times) the activity found 15 ms after the start of the saccade in both the target in CRF to distractor in FRF condition (black trace in Fig. 7A) and the target in CRF to nothing in FRF condition (black trace in Fig. 7C) or whether the activity 35 ms after the start of the saccade was >25% higher than (1.25 times) the activity 15 ms after the start of the saccade in 1 of those 2 conditions. We chose to use traces in which a target was in the CRF followed by a lower response postsaccadically to minimize our chances of false positives given that the activity at that time would otherwise be declining. We also attempted to minimize our chances of false positives when using the lower threshold (1.1× activity), by making sure that both traces displayed the burst. For the higher threshold, 1.25 times the activity was high enough that noise fluctuations did not impact the selections. We confirmed that our criteria were appropriate by examining each by eye and finding no obvious spurious categorizations. In all, 63% (33/52) of neurons had a postsaccadic burst defined this way.
The exhibition of a postsaccadic burst and the presence of remapping were not correlated. We defined neurons as remapping responses if they showed a significant and sustained main effect of stimulus identity in the FRF prior to 90 ms after the end of the saccade in both the stimulus-to-stimulus condition (blue squares in Fig. 5D) and the pooled data (blue squares in Fig. 5F). We considered the effect sustained if it was present for >30 ms starting before 90 ms after the end of the saccade. Using this definition, 38% (20/52) of the neurons exhibited remapping. Of these, 70% (14/20) had a postsaccadic burst. Thus, the percentages of burst containing neurons were 70% in neurons that display remapping and 59% (19/32) in neurons that do not display remapping. These percentages were not significantly different (P = 0.55, chi-squared test). Thus, we suggest that the small burst may be a brief period of postsaccadic excitability, similar to that seen in V1 (Rajkai et al. 2008).
We have presented data showing that shortly after a saccade ended, the activity in LIP shifted continuously from representing the presaccadic stimulus to representing the postsaccadic stimulus. This occurred earlier when a stimulus was in the response field before the saccade and was seen both in terms of statistical differences and in terms of absolute response differences in ∼20% of neurons. Finally, we also showed that the small postsaccadic burst of activity is unrelated to remapping, suggesting that it is just a period of postsaccadic excitability.
We have previously shown that when a stimulus was in both the CRF and FRF, a subset of ∼20–30% of LIP neurons differentiated between targets and distractors shortly after a saccade (Mirpour and Bisley 2012a). Our data build on these results in 3 ways. First, we showed the difference in response onset when the CRF was empty before the saccade. Second, we compared each of these with the disappearance of the presaccadic representation. Third, we show that the remapped response is stable in terms of spike rate until the discriminable signal from the retina reaches LIP. While the second and third of these are necessary to understand how remapping could contribute to perceptual stability, the first is perhaps the most interesting in terms of its implications on the remapping field.
We found that when the CRF was empty, neurons began responding at the end of the saccade, but the signal did not differentiate between target and distractor until ∼45 ms later. This result could mean that the remapping mechanism may shift activity in different ways depending on whether a stimulus had been present in the CRF or not, but an alternative possibility is that for a remapped signal to distinguish between stimuli, it is necessary to have an elevated response. When a stimulus is in the CRF, then the activity is already elevated, but when no stimulus is in the CRF, it may be that the remapping needs to start by elevating the response before it separates based on stimulus identity. In any case, these data show that any interpretation of data from a task in which a stimulus enters a response field that had previously been empty must be tempered by the knowledge that the responses appear to be different than when a stimulus was in the CRF. Given the richness of most natural scenes, our results urge that caution be used when interpreting the remapping responses to a single stimulus as it relates to perceptual stability.
How this Remapping Could Contribute to Perceptual Stability
To understand how remapping may relate to perceptual stability, it is first important to understand where this remapping is occurring in the brain. One of the roles LIP is thought to have involves the guidance of visual attention (Bisley and Goldberg 2003, 2006; Herrington and Assad 2010) using an attentional priority system (Gottlieb et al. 2009; Bisley and Goldberg 2010; Bisley 2011; Zelinsky and Bisley 2015). The idea is that the activity in some neurons in this area, and the network it is part of, tells the rest of the brain which spatial locations represented in earlier visual areas should be attended and, thus, available for further processing. The higher the activity in LIP, the higher the priority and the more likely that the location or object represented by that activity will be attended. Without attention, we become blind to that we think we see, which means that most of what we think we see outside of the locus of attention is an illusory construct (Rensink 2000). The important converse of this is that the things we are aware of are usually the things we are attending. Thus, if LIP guides attention and attention generally leads to awareness, then it is easy to see how continuity of the presaccadic to postsaccadic representation in LIP can lead to perceptual continuity. Specifically, we suggest that the rapid switch from the presaccadic representation of visual space to the initial postsaccadic representation of visual space (see Fig 6D) occurs so that the attentional priority at each location in space remains stable across saccades. This means that covert attention, which continues to shift, can continue to be guided by the activity in LIP, which provides the spatiotemporal continuity that makes saccade-related interruptions and displacements unnoticeable. So the theory for perceptual stability is simple: As the visual scene jumps from location to location on the retina and in visual cortex, the attentional guidance system predictively remaps before the visual information reaches cortex so that the attended locations remain stable in space and time (Cavanagh et al. 2010). We then suggest that the rest of the scene is filled in as it is during stable fixation to create a percept that the rest of the world is present.
Such a mechanism explains why information about most visual features cannot be integrated across saccades (Irwin et al. 1983), although motion information, which is thought to be accumulated in LIP (Roitman and Shadlen 2002), can (Melcher and Morrone 2003). It also leads to an interesting question about saccadic omission. Saccadic omission (Campbell and Wurtz 1978) is the term given to describe our lack of awareness of motion due to saccades or reduced visual input during saccades, due to saccadic suppression (Burr et al. 1982). Does remapping occur during the period of saccadic omission so that the shifting of attention is masked or is this shift part of the mechanism that contributes to saccadic omission? Based on earlier work (Burr et al. 1994; Thilo et al. 2004), we would have suggested the former. However, more recent work has suggested that saccadic omission may occur because information available in visual areas does not reach awareness (Watson and Krekelberg 2009). If LIP acts to guide attention, and attention is driving awareness, then the smooth transition in LIP could be the mechanism that drives saccadic omission by keeping the movement from reaching awareness. Thus, predictive remapping may provide a way to bridge suppression in both space and time (Higgins and Rayner 2015).
Questioning Whether Remapping Plays a Role in Spatial Stability
Remapping is not the only neuronal mechanism that has been suggested to explain how the percept of spatial stability could be garnered from retinotopic inputs (Wurtz 2008). Neurons in many cortical areas show response modulations based on the direction of gaze—termed gain fields (Andersen et al. 1990; Bremmer et al. 1997, 1999; Bremmer 2000). Multiple studies have shown that the responses from neurons with gain fields can be decoded to identify where objects are in space (Zipser and Andersen 1988; Pouget and Sejnowski 1997; Morris et al. 2013). Indeed, it is likely that this mechanism is involved in this process during stable fixation; however, the gain field responses appear to be too slow to explain spatial stability around the time of a saccade (Xu et al. 2012). Thus, we suggest that remapping is important for the temporal and spatial continuity observed across saccades.
Although sparse in the brain, some neurons have receptive fields that appear to be stable in space (Galletti et al. 1993; Duhamel et al. 1997; Dean and Platt 2006), at least in animals that are stationary. These neurons, with putative allocentric receptive fields, are found in several parietal areas, including area V6, the ventral intraparietal area (VIP), and posterior cingulate cortex. These allocentric receptive fields must be created from retinocentic inputs, which requires a process for stabilizing the visual inputs. We suggest that these could come from remapping or gain fields, or a combination of the 2.
Prior to this study, it had often been suggested that remapping may play a role in maintaining spatial stability (Duhamel et al. 1992; Wurtz 2008; Hall and Colby 2011; Higgins and Rayner 2015), but a recent study brought this idea into question (Zirnsak et al. 2014). In that study, the authors showed that when a single isolated stimulus was placed at different locations across the visual field, the response of the postsaccadic receptive field was influenced by the stimulus, even when it was placed far away, suggesting a compression of space. However, the limited stimulus conditions used in that study can lead to perceptual compression (Ross et al. 1997; Kaiser and Lappe 2004) or perceptual mislocalization (Honda 1989; Schlag and Schlag-Rey 1995; Jeffries et al. 2007), percepts that are, by definition, unstable. In addition, here we showed that responses to a stimulus brought into a response field that had been empty have different dynamics to the responses when a stimulus is present in both the CRF and FRF. As such, we believe Zirnsak's results cannot address the question of visual stability when there is a full scene in front of the viewer. Indeed, our data imply that compression is not even occurring in LIP: If it was, we would not be able to differentiate among stimulus classes in the response fields, which we clearly can. A second, more important, difference is that we tested whether the remapped response represented the stimulus that was being remapped. With the exception of 3 studies that have examined changes in responses over a saccade (Crapse and Sommer 2009; Churan et al. 2011; Subramanian and Colby 2014), previous studies on remapping in cortex (Duhamel et al. 1992; Umeno and Goldberg 1997; Kusunoki and Goldberg 2003; Heiser and Colby 2006; Sommer and Wurtz 2006), including that of Zirnsak et al. (2014), only examined whether a response occurred. In doing so, they miss the fact that response levels in these areas are behaviorally important, a fact we previously relied on to illustrate the extent of remapping in LIP (Mirpour and Bisley 2012a) and that allows us, here, to understand how this activity can lead to the percept of spatial stability.
Questions Raised by Our Results and Perceptual Framework
Given that, at the time of a saccade, attention is pinned to the saccade goal (Shepherd et al. 1986; Hoffman and Subramaniam 1995; Deubel and Schneider 1996), one may ask why activity is remapped across the entire priority map. We already know that the response in LIP is always greatest at the goal of the saccade just before the saccade begins, even in our foraging task (Mirpour et al. 2009), so if spatial stability is entirely due to attention being focused at the saccade goal, then remapping of the rest of the visual field may not be necessary. We suggest that remapping of the whole visual field occurs so that the attentional priority at each location remains stable across saccades, so that covert attention, which continues to shift, can continue to be guided by the activity in LIP. Note that attention is allocated based on the relative responses in LIP, not the absolute response (Bisley and Goldberg 2003; Mirpour and Bisley 2012b), so although we found that the presaccadic responses in LIP sometimes varied as a function of what will appear in the response field (e.g. 50–100 ms before the saccade in Fig. 2E), the relative response was always greater for targets than for distractors: a similar pattern to that seen after the saccade. Thus, remapping the entire visual field, even with biases in presaccadic responses, allows for the continued guidance of covert attention.
The interpretation of our data rests on the measured differences in responses between targets and distractors. We have used these because they are simple stimuli that produce different attentional priorities and, thus, different levels of responses in LIP. Previous studies have found that LIP activity can be affected by shape (Sereno and Maunsell 1998; Janssen et al. 2008; Subramanian and Colby 2014), color (Toth and Assad 2002; Ogawa and Komatsu 2009), and even categorization (Freedman and Assad 2006; Swaminathan and Freedman 2012). We have recently described how many of these responses can be viewed in the context of a priority map (Zelinsky and Bisley 2015): A major function of a priority map is to match stimuli in the visual world with features (or combinations of features) of interest. Elevated activity identifies stimuli with features that are relevant for the current behavior and will be used to guide covert and, ultimately, overt attention. This suggests that stimuli that induce strong response differences in LIP should allow us to see these differences in remapped responses with high fidelity. Note that individual neurons may have different inherent biases for particular stimuli, but this is a minor response (compare the magnitude of the nonspatial with the spatial responses in Freedman and Assad 2009). We believe this explains the weak correspondence of responses in the pre- and post-saccadic representations of task irrelevant shapes in LIP (Subramanian and Colby 2014). Indeed, we predict that if an experiment was done using the same shape stimuli they used, but with some behavioral relevance for the animal, then a strong correspondence would be seen in the responses in the pre- and post-saccadic representations.
This work was supported by a McKnight Scholar Award and the National Eye Institute (R01 EY019273-01).
We thank the members of the UCLA DLAM for their superb animal care, fellows at the ZiF for useful discussions about the data, and an anonymous reviewer who seeded our focus on temporal continuity. Conflict of Interest: None declared.