Perception is an active inferential process in which prior knowledge is combined with sensory input, the result of which determines the contents of awareness. Accordingly, previous experience is known to help the brain “decide” what to perceive. However, a critical aspect that has not been addressed is that previous experience can exert 2 opposing effects on perception: An attractive effect, sensitizing the brain to perceive the same again (hysteresis), or a repulsive effect, making it more likely to perceive something else (adaptation). We used functional magnetic resonance imaging and modeling to elucidate how the brain entertains these 2 opposing processes, and what determines the direction of such experience-dependent perceptual effects. We found that although affecting our perception concurrently, hysteresis and adaptation map into distinct cortical networks: a widespread network of higher-order visual and fronto-parietal areas was involved in perceptual stabilization, while adaptation was confined to early visual areas. This areal and hierarchical segregation may explain how the brain maintains the balance between exploiting redundancies and staying sensitive to new information. We provide a Bayesian model that accounts for the coexistence of hysteresis and adaptation by separating their causes into 2 distinct terms: Hysteresis alters the prior, whereas adaptation changes the sensory evidence (the likelihood function).
Perception not only depends on currently available sensory information, but also strongly on previous experience (Fecteau and Munoz 2003; Schwartz et al. 2007). For example, in the waterfall illusion, prolonged exposure to one direction of motion causes subsequently viewed stimuli to appear moving in the opposite direction (Purkinje 1820). Such repulsive after effects in perception are commonly attributed to neuronal adaptation (Anstis et al. 1998). However, previous experience can also have the opposite effect: After brief exposure to a moving stimulus, the subsequent ambiguous probe appears to move in the same direction (Kanai and Verstraten 2005), a phenomenon known as “hysteresis” or “priming”. It has been hypothesized that each effect serves a different computational purpose: While adaptation could prepare the brain for the uptake of new information (Barlow 1990), hysteresis might stabilize percepts in the face of constantly changing low-level stimulus features (Kleinschmidt et al. 2002). However, it is currently unknown what determines the direction of such experience-dependent perceptual effects and how the brain entertains these 2 opposing processes.
Here, we used multistable stimuli to address those questions as the effects of previous experience are most obvious when stimulus evidence by itself is insufficient to determine perception. It is known that when multistable stimuli are presented intermittently, perception on any given trial depends on hysteresis and adaptation (Pearson and Brascamp 2008). We combined functional magnetic resonance imaging (fMRI) with a paradigm in which we could investigate hysteresis and adaptation concurrently, but dissociate them experimentally, enabling us to probe whether both effects are also separable in the human brain. We directly tested the predictions of 2 competing classes of models that explain hysteresis and adaptation either with a single mechanism (Gepshtein and Kubovy 2005), for example, neuronal adaptation (Orbach et al. 1963; Blake et al. 2003; Chen and He 2004), or with 2 separate mechanisms that, however, co-localize to the same early sensory areas (Noest et al. 2007; Wilson 2007; Brascamp et al. 2009). We also focused on the question at which stage of neural processing the 2 effects might occur, as hysteresis and adaptation are either ascribed exclusively to early sensory processing (Noest et al. 2007; Wilson 2007; Brascamp et al. 2009), or to interactions between early and higher-order processing stages (Sterzer and Rees 2008; Gigante et al. 2009).
Our results refute models that define hysteresis and adaptation as resulting from the same neural process, as we found that the 2 effects map into distinct, anatomically and hierarchically segregated cortical networks. Activity in higher-order fronto-parietal areas and higher-order visual areas was related to perceptual stabilization, with a central node in the dorsomedial prefrontal cortex (dmPFC). In contrast to the widespread network subserving hysteresis, adaptation was confined to a local node in early visual areas. We propose a Bayesian model that accounts for the coexistence of hysteresis and adaptation by separating their causes into 2 distinct terms: Hysteresis alters the prior, whereas adaptation changes the currently available sensory information (the likelihood function). Keeping the effects of previous experience separated in this way may allow the brain to maintain the balance between exploiting redundancies and staying sensitive to new information.
Materials and Methods
Thirty healthy human subjects (mean age 25.1, range 21–38, 11 males) participated in the psychophysical part of this study. Twenty-nine were right-handed, as assessed with the Edinburgh Inventory (Oldfield 1971). All subjects had normal or corrected-to-normal vision, reported no history of neurological or psychiatric disease, and gave written informed consent before participation, in accordance with the Declaration of Helsinki. Twenty of these subjects (mean age 24.6, range 21–29, 7 males, all right-handed) also participated in the fMRI study. Two further subjects participated in the fMRI study, but their data were excluded due to technical artifact. Subjects received monetary compensation for their participation.
Procedure and Stimuli
For the psychophysical experiments, stimuli were displayed on a cathode ray tube monitor (HP p1230, resolution 1024 × 768, visible screen size 30° × 22.9° at 75 cm distance, refresh rate 150 Hz). Inside the scanner, a video goggle system was employed (Resonance Technology MR Vision 2000, resolution 800 × 600, visible screen size 30° × 22.5° at 1.2 m virtual distance, refresh rate 60 Hz). Stimuli were generated using Matlab (R2007a, The MathWorks). Stimulus presentation and response collection were controlled by Presentation software (v13.1, Neurobehavioral Systems). All stimuli were presented on a gray background (7.62 cd/m2); a red fixation circle was continuously present in the center of the screen (6.72 cd/m2).
To investigate perceptual hysteresis and adaptation, we used a modified version of a paradigm introduced by Gepshtein and Kubovy (2005) that allows the manipulation of both phenomena concurrently. We presented sequences of stimuli with different degrees of multistability: First, a bistable, rectangular dot lattice (800 ms), followed by a tristable, hexagonal dot lattice (300 ms; Fig. 1). Both lattices were aligned to the same orientation (henceforth referred to as 0°), but this orientation was randomly varied from trial to trial, covering 90° in 1° steps. The perceived orientation of these stimuli alternates spontaneously. For the first bistable rectangular dot lattice, participants are likely to perceive the lattice as a collection of illusory lines parallel to 0° or 90°. The likelihood to perceive a particular orientation can be controlled by manipulating the evidence for that orientation, that is, the aspect ratio (AR) of the dots (AR = Δ0/Δ90, where Δ0 and Δ90 correspond to the interdot distances along 0° and 90°). Due to grouping by proximity, perception of an orientation is more likely to occur along the shortest interdot distance (Kubovy and Wagemans 1995). Hence, when the interdot distance along the 0° orientation is shorter, perception favors 0°, and subjects tend to perceive the dot lattice as tilted to the right (AR <1, Fig. 1A). When the interdot distance along the 90° orientation is shorter, perception favors 90°, and the dot lattice is perceived as tilted to the left (AR >1). At AR = 1, P(0°) = P(90°), and with interdot distance of 1°, the stimulus is fully bistable (Kubovy and Wagemans 1995; Gepshtein and Kubovy 2005; Nikolaev et al. 2010). We varied the likelihood to perceive 0° or 90° in the first dot lattice by manipulating the AR of the interdot distance in 7 steps (1.3−1, 1.2−1, 1.1−1, 1, 1.1, 1.2, and 1.3). After the rectangular dot lattice, we presented a tristable hexagonal dot lattice in which participants are equally likely to perceive 0°, 60°, and 120°. Hexagonal dot lattices have the advantage of being highly instable (Kubovy and Wagemans 1995), thus providing the highest sensitivity to the effects of previous experience. Importantly, while varying the parameters of the first dot lattice, we kept the second lattice constant and maximally instable. Critically, keeping the second stimulus constant and instable throughout the trials enabled us to unravel how previous experience determines perception of the current stimulus. Hysteresis can be investigated by assessing how perceiving an orientation in the second stimulus depends on whether the same orientation was previously perceived. Adaptation can be investigated by assessing how perceiving an orientation in the second stimulus depends on the amount of evidence provided for that orientation in the first stimulus. These effects are similar to the direct (repulsive) and indirect (attractive) tilt after effect originally described by Gibson and Radner (1937). We exploited these sequential dependencies to independently define brain areas reflecting the opposing effects of hysteresis and adaptation. One major advantage of this paradigm over the paradigms used in previous studies (Kleinschmidt et al. 2002; Sterzer and Rees 2008; Raemaekers et al. 2009) is that it allows the separation of hysteresis from adaptation while measuring both phenomena concurrently.
Dot lattices (11.5° diameter) were made up from ordered spatial arrangements of white Gaussian blobs (“dots”, 0.25° diameter), presented at the center of the screen. To ensure roughly equal dot density for all ARs, the product of the interdot distances for 0° and 90° was kept invariant (∼1). To avoid interactions of the oriented lines with the edges of the stimuli, we smoothed the total area of dot lattices with a Gaussian (standard deviation [SD] 0.15). This gives the impression that the lattice continues behind a virtual aperture (Nikolaev et al. 2010). We also pseudorandomly jittered the exact position of the dot lattice within the aperture (0–1.15°) to prevent that dots of subsequent displays occupy systematically related portions of space.
After each presentation of a dot lattice, subjects chose the orientation they had perceived from a screen displaying 4 alternative orientations (0°, 90°, and the 2 diagonal orientations for the rectangular dot lattices; 0°, 60°, 120°, and the unlikely 90° orientation for the hexagonal dot lattices). The position at which the alternatives appeared (and thus the corresponding response button) was fully randomized within and between trials. Subjects were instructed to be accurate, to fixate on a central fixation dot (as ascertained by eye tracking), and to report the first perceived orientation in case the percept switched during the presentation of an individual dot lattice. Note that subjects were never required to assess whether they had seen the same orientation in the 2 stimuli. To avoid after images and between-trial carry-over effects, we presented a dynamic random dot mask (updated at 25 Hz) for 550 ms after the response to the hexagonal dot lattice.
Subjects completed 9 blocks of 70 trials. Pseudorandomization assured that each AR occurred equally often during a block. The experiments were conducted in a darkened, sound-attenuating chamber. Constant head position was assured by the use of a chinrest with forehead support. Subjects received 30 practice trials.
To ensure that none of the behavioral effects were ascribable to eye movements, we monitored eye position (n = 26) using a binocular infrared eye tracker in all psychophysical experiments (SR Research Eyelink 1000, sampling rate 1000 Hz). The eye tracker was recalibrated using a standard 9-point calibration procedure at the beginning of each block.
Twenty of the subjects subsequently participated in the fMRI experiment. To keep scanning within reasonable time limits, we reduced the number of ARs to 3 (AR = 1, 1.1, and 1.2). Thus, rectangular dot lattices would be either bistable or biased toward 90°. To specifically investigate brain activity related to perception of the constant, instable hexagonal dot lattice, we introduced trials in which the second stimulus was omitted. This “partial trial design” makes it possible to separate the hemodynamic response to the second stimulus from the convolved responses to both stimuli (Ollinger, Corbetta, et al. 2001; Ollinger, Shulman, et al. 2001). This enabled us to rule out that differences observed for the contrast hysteresis versus no hysteresis were merely due to differences arising for the first stimulus (see below).
Subjects completed 10 blocks of 72 trials in 2 sessions. Each block consisted of 30 trials with 2 stimuli, 30 trials with 1 stimulus, and 10 “null” baseline trials (during which only the fixation cross was visible on the display). To avoid long sequences of identical trial types, they were presented in pseudorandom order, achieved by shuffling conditions in sequences of 14 stimuli, 5 times per block. Furthermore, the first trial was randomly chosen from all available conditions and later discarded from all analyses to eliminate T1 saturation artifacts in the fMRI data. We added a baseline trial at the end of each block to avoid curtailing the hemodynamic response for the last event in a block. Subjects were instructed to respond accurately within 1.7 s. This was practiced inside the scanner before the experiment started.
MRI data were acquired on a 3-T scanner (Siemens Allegra), using a 4-channel head coil. We acquired 216 volumes of 29 transversally oriented slices per run of functional (T2* weighted) echoplanar imaging data (repetition time [TR] = 2.5 s, echo time [TE] = 30 ms, voxel size 3 × 3 × 3 mm, gap thickness 0.3 mm). Anatomical images were acquired using a T1-weighted magnetization-prepared rapid gradient echo (MPRAGE) sequence (160 slices, TR = 2.25 s, TE = 4.38 ms, voxel size 1 × 1 × 1 mm). All sequences covered the whole brain.
Behavioral data were analyzed with a logistic regression using Generalized Estimating Equations (GEE, Liang and Zeger 1986) in SPSS (v17 and v18, SPSS, Inc.). GEE is an extension of the generalized linear model (GLM) developed for the analysis of repeated-measures designs (for an introduction see Hanley et al. 2003). Following Gepshtein and Kubovy (2005), we restricted our analyses to the response alternatives with equal likelihood at AR = 1, that is, cases were subjects responded either 0° or 90° to the rectangular dot lattices and 0°, 60°, or 120° to the hexagonal dot lattices. Analyses using all response alternatives yielded comparable results. Data were sorted by the subject, block, and trial number. To account for the correlations between successive trials, we used a working correlation matrix with a first order autoregressive relationship (Liang and Zeger 1986). Two logistic regressions were computed. In the first, we modeled the responses to the first stimulus using AR as a predictor. In the second, we modeled the responses to the second stimulus using the AR of the first stimulus and responses given to the first stimulus (R1) as predictors. We excluded the interaction term since a previous model revealed no significant interaction between AR and R1 (Wald's χ2(1) = 0.251, P = 0.617).
Data Analysis—Eye Tracking
For offline analyses of the eye tracking data, temporal windows were defined from 200 ms before the stimulus onset until stimulus offset (800 ms for stimulus and 1300 ms for stimulus 2). A trial was excluded from the analyses if a blink occurred within 100 ms either before stimulus onset or after stimulus offset. We then calculated the percentage of eye position samples falling into a square window around the fixation dot for 2 levels of precision (window size 1.5° × 1.5° or 2° × 2°). Repeated-measures analyses of variance were run separately at each level of precision for stimulus 1 and stimulus 2, with factors eye (left and right) and hysteresis (0–0° and 90–0°) to control for the effect of fixation stability on the hysteresis effect, and with factors eye (left and right) and AR (1.3−1, 1.2−1, 1.1−1, 1, 1.1, 1.2, and 1.3) to control for the effect of fixation stability on the adaptation effect.
MRI data were analyzed in Brain Voyager QX (v2.1, Brain Innovation), SPSS, and Matlab using both the Brain Voyager Toolbox and custom code. The first 3 volumes of each functional run were excluded to prevent T1 saturation effects. Preprocessing included slice scan time correction with cubic spline interpolation, 3-dimensional motion correction using trilinear/sinc interpolation, and high-pass filtering (0.01 Hz). Functional and anatomical data were brought into anterior commissure posterior commissure space using cubic spline interpolation and then transformed into standard Talairach space (Talairach and Tournoux 1988) using trilinear interpolation. For the whole-brain analyses, data were spatially smoothed with a Gaussian kernel (8-mm full-width at half-maximum). To create inflated surface reconstructions, the gray–white matter boundary in the structural scans was segmented, reconstructed, smoothed and inflated (Kriegeskorte and Goebel 2001) separately for each hemisphere.
Blood oxygenation level-dependent (BOLD) responses were estimated using a deconvolution model in a random effects (RFX) GLM. For each condition, we defined appropriately placed series of 8 finite impulse response (FIR) predictors (one per volume) to model the 20-s BOLD response following the onset of each trial. Contrasts were run over the third and fourth predictors (corresponding to 5- and 7.5-s poststimulus), thus covering the peak of the BOLD response while accounting for variability in the peak time in different subjects and brain regions (Handwerker et al. 2004). To reduce the number of voxel-by-voxel comparisons, we restricted the analyses by the use of a cortex mask based on the individual gray–white matter boundary. This reduced the number of voxels to 47 405. For follow-up region-of-interest (ROI) analyses, we extracted the mean deconvolved time courses of the clusters identified in the whole-brain analysis per subject from the unsmoothed data.
To identify cortical regions involved in perceptual hysteresis, we defined a GLM with hysteresis (2 stimuli trials where subjects responded 0° to both stimuli) and no hysteresis (2 stimuli trials where subjects responded 90° to the first stimulus and 0° to the second) as independent predictors. Both predictors were modeled with a FIR set comprising 8 poststimulus periods. To discard spurious carry-over effects from perceiving 0° or 90° in the first stimulus, we defined 2 additional predictors for trials with only 1 stimulus: 1 for trials in which subjects responded 0°, and 1 for trials in which subjects responded 90° (see below). Finally, we added 3 predictors of no interest to account for variance from trials with 2 stimuli where subjects chose one of the remaining orientations on the first, second stimulus, or both stimuli; trials with 1 stimulus where subjects chose a diagonal; and trials where subjects failed to respond.
We created beta maps per subject for 2 contrasts: (hysteresis vs. no hysteresis) and (0° vs. 90°). To reveal brain areas with significant activity changes related to hysteresis, we then contrasted (hysteresis vs. no hysteresis) > (0° vs. 90°) in a second-level analysis. The comparison of (hysteresis vs. no hysteresis) with (0° vs. 90°) was essential in this contrast as it guards the results against any BOLD differences that are merely due to participants perceiving 0° versus 90° in the first stimulus without any hysteresis (however, virtually identical results were observed for the simple contrast [hysteresis vs. no hysteresis]). The contrast map was initially thresholded at a voxel level t(19) = 2.860 and subsequently submitted to statistical inference with a cluster size threshold at P < 0.05 (5000 iterations), corrected for multiple comparisons. The resulting cluster size threshold was 263 mm3/297 voxels. Since perceptual memory can only build up with the presentation of the first stimulus, solely areas showing a significant BOLD response to the first stimulus (post hoc t-test on the mean of 0° and 90° one stimulus trials, P < 0.05, uncorrected) were considered to be directly involved in perceptual hysteresis.
To identify regions expressing adaptation, we defined a new GLM for the same set of data (and thus the same baseline) as above (A single GLM incorporating both hysteresis and adaptation would have been optimal. However, it was not possible to build a single RFX GLM for both effects while maintaining the same statistical power, as some subjects showed strong hysteresis effects on the high aspect ratios, and thus, no predictor for some combinations of aspect ratio and no hysteresis perception could be built for these subjects.). We regrouped the trials into 6 new independent predictors (each modeled with the FIR set comprising 8 poststimulus periods). Since adaptation depends on the AR, we constructed one predictor per AR for 1 and 2 stimuli trials, respectively. We also included a predictor of no interest for trials where subjects failed to respond. We then created beta maps per subject for the contrasts (AR = 1 vs. baseline), (AR = 1.1 vs. baseline), and (AR = 1.2 vs. baseline) for trials with 2 stimuli and ran an F-test to compare between the 3 conditions. The resulting map was initially thresholded at a voxel level F2,38 = 3.244 and submitted to statistical inference with the same statistical threshold as for the hysteresis contrast (cluster size thresholded at P < 0.05 [5000 iterations], corrected for multiple comparisons). The resulting cluster size threshold was 749 mm3/783 voxels. For each of the clusters, we extracted the deconvolved time courses per subject. Areas showing a significant positive BOLD response to the first stimulus (post hoc t-test on the mean of 0° and 90° one stimulus trials, P < 0.05, uncorrected) and a linear trend (as assessed by linear trend analysis, Howell 2002, p. 408f) of the peak BOLD response over the 3 ARs for the second stimulus were considered to be involved in perceptual adaptation. Note that a trend analysis requires that the factor for which the trend is investigated is significant in the preceding omnibus F-test, but that the presence or absence of a linear trend is statistically independent of this preceding test. Thus, ROI analyses were statistically independent of the contrasts performed for the whole brain.
Eye tracking was not available inside the scanner. However, to assess whether eye movements could account for the observed effects, we used a method introduced by Beauchamp (2003). This method exploits the fact that major signal changes can be observed in the eyeballs themselves during eye movements. Thus, differential eye movements in different conditions can be assessed by comparing signal changes in the eyeballs. We first identified for each functional run voxels with time course jumps ≥4 SD. The resulting maps were averaged per subject. Based on the average maps, we then defined ROIs for each eyeball in each subject (mean number of voxels left eye: 108.2, standard error (SE) 3.93; mean number of voxels right eye: 111.86, SE 2.79), and joined these ROIs into one ROI per subject. If ROIs could not be identified this way, we defined a sphere (radius 3 voxels, size 123 voxels) around the group-average peak voxel of the respective eyeball. This was done for 6 eyeballs. ROIs could be defined for 19 subjects. These ROIs were used as control regions for all other contrasts.
Perception of the first, rectangular dot lattice should follow the Gestalt law of proximity such that the likelihood to perceive an orientation depends on the shortest interdot distance. We confirmed this prediction in a logistic regression using the AR as a predictor of the responses given to the first lattice. As shown in Figure 2A, the probability of responding 0° decreased as AR increased (Wald's χ2(1) = 111.880, P < 0.01, Table 1). We also confirmed bistability of the rectangular dot lattice at AR = 1, as the probability to respond 0° or 90° was close to the chance.
|Response||Predictor||β (log odds)||SE β||Wald's χ2||df||P-value||eβ (odds ratio)|
|R1||Constant aspect ratio||5.091||0.4966||105.087||1||<0.01||162.524|
|R2||Constant aspect ratio R1||−4.399||0.4228||108.232||1||<0.01||0.012|
|R1 (fMRI)||Constant aspect ratio||7.293||0.9470||59.308||1||<0.01||1469.697|
|R2 (fMRI)||Constant aspect ratio R1||−6.923||.9151||57.230||1||<0.01||0.001|
|R1 (one stimulus)||Constant aspect ratio||6.550||0.7739||71.630||1||<0.01||698.995|
|Response||Predictor||β (log odds)||SE β||Wald's χ2||df||P-value||eβ (odds ratio)|
|R1||Constant aspect ratio||5.091||0.4966||105.087||1||<0.01||162.524|
|R2||Constant aspect ratio R1||−4.399||0.4228||108.232||1||<0.01||0.012|
|R1 (fMRI)||Constant aspect ratio||7.293||0.9470||59.308||1||<0.01||1469.697|
|R2 (fMRI)||Constant aspect ratio R1||−6.923||.9151||57.230||1||<0.01||0.001|
|R1 (one stimulus)||Constant aspect ratio||6.550||0.7739||71.630||1||<0.01||698.995|
We then turned to the second stimulus, which was kept constant and instable throughout the trials, enabling us to unravel how previous experience determines perception. In particular, to investigate the effects of hysteresis and adaptation on perception of the second dot lattice, we modeled the responses to the second stimulus (R2) as a function of the AR of the first stimulus and the response given to the first stimulus (R1) in a logistic regression. As the AR of the second stimulus does not favor a particular orientation, one would not expect any differences in the likelihood to perceive a particular orientation, unless the first stimulus exerts an effect onto the second one. We confirmed this later scenario as both AR and R1 significantly predicted R2 (AR: Wald's χ2(1) = 58.797, R1: Wald's χ2(1) = 146.186, both P < 0.01, Table 1). Figure 2B shows the carry-over effect from R1 to R2: The black line in the figure depicts the likelihood that subjects perceived 0° in the first dot lattice and continued perceiving 0° in the second dot lattice, thus exhibiting perceptual hysteresis. The gray line shows the likelihood that subjects first perceived 90° and then switched to perceiving 0° in the second stimulus. Importantly, the 2 lines run in parallel, showing that perceptual hysteresis does not interact with, or depend on, the AR of the first stimulus. The vertical separation of the 2 lines reflects the size of the perceptual hysteresis effect, indicating that subjects were likely to perceive the same orientation in the first and second lattice. The effect of adaptation becomes apparent when comparing the slope in Figure 2A with B: While the likelihood to respond 0° decreases with AR for the first stimulus, it increases with AR for the second stimulus. Thus, the more likely subjects were to perceive 0° in the first dot lattice, the less likely subjects were to perceive 0° in the second lattice. This inverse effect is characteristic of adaptation. Interestingly, adaptation was even present for the orientations that were not explicitly reported. For instance, in the black line in Figure 2B, adaptation to 90° is evident by the positive slope (i.e. the inverse slope of Fig. 2A). However, on these trials, subjects reported perceiving 0° in both stimuli. This indicates that both 90° and 0° orientations are always processed, regardless of whether they are consciously perceived or not (Blake and Fox 1974; Hock et al. 1996; Gepshtein and Kubovy 2005). All behavioral results could be reproduced in the scanner (Table 1). Here, AR also significantly predicted whether subjects responded 0° or 90° on trials with only one stimulus (Wald's χ2(1) = 78.286, P < 0.01, Table 1). None of these effects were attributable to differences in fixation stability, as there was neither a significant effect of hysteresis nor AR in the eye tracking data (all P > 0.05, Greenhouse–Geisser corrected).
Having established that perceptual hysteresis and adaptation are independently expressed in the behavioral response patterns, we now turned to the brain imaging data to investigate the cortical areas underlying each effect. To this end, we employed a logic that closely followed the analyses of the behavioral data. To identify regions involved in hysteresis, we contrasted trials in which subjects expressed hysteresis (subjects reported twice the same percept, i.e. 0°) with trials in which they did not express hysteresis (subjects reported a perceptual switch, i.e. 90° followed by 0°). By analyzing only trials in which perception of the second stimulus was 0°, these conditions only differ in their history. However, due to the sluggishness of the BOLD hemodynamic response, in trials with 2 stimuli responses related to the first one are convolved with responses to the second one, introducing the potential confound that the differences between hysteresis and no hysteresis merely arise from perceiving 0° or 90° in the first stimulus without any true difference at the level of the second stimulus. In order to rule out this potential concern, we contrasted 0° and 90° percepts on trials with only 1 stimulus and subtracted this difference from the difference between hysteresis and no hysteresis trials in 2 stimulus trials. Thus, the result of the contrast (hysteresis vs. no hysteresis) > (0° vs. 90°) was matched for the percept and the response to the second stimulus and was not confounded by potential differences in perceiving or reporting 0° or 90° in the first stimulus. Figure 3 shows that hysteresis is expressed in a distributed network of brain areas spanning ventral visual areas (including bilateral fusiform gyrus), superior parietal (bilateral intraparietal sulcus [IPS]), and frontal cortices (right anterior insula, right presupplementary motor area [pre-SMA], and right dmPFC). The time course of BOLD signal change in these regions is displayed in Figure 3 and illustrates that, in all regions, the absolute response on hysteresis trials was smaller than on no hysteresis trials. This resembles repetition suppression (Grill-Spector et al. 2006), which has been proposed to reflect an effect of top-down perceptual expectations (Friston 2005; Summerfield et al. 2008; Todorovic et al. 2011).
While the previous results suggest a network of brain areas involved in perceptual hysteresis, in the next analysis, we attempted to tie the contribution of those regions more directly to percept maintenance. To that end, we took advantage of the interindividual variability in the size of the behavioral hysteresis effect (log odds) and correlated this with the difference in activation between hysteresis and no hysteresis trials for each brain region. Regions showing a difference between conditions and a correlation with interindividual differences in behavior are likely to be centrally involved in percept maintenance (Yarkoni and Braver 2010). From all the areas identified in the hysteresis contrast, only the right dmPFC positively correlated with the propensity to maintain perception across successive stimuli (Pearson's r = 0.59, P = 0.0058, Fig. 3E; all other P > 0.1).
To identify regions expressing adaptation, we searched for areas that showed a linear effect of AR, as observed in the behavioral data. To that end, we sorted all trials with 2 stimuli according to the AR of the first stimulus. We identified voxels showing a significant difference between ARs and then tested for a linear increase of the peak BOLD response as a function of AR. This procedure was motivated by the following rationale: For the first stimulus, we parametrically modulated the stimulus evidence for 0°, such that as AR increases from 1 to 1.2, evidence for 0° decreases. Assuming that neural adaptation is proportional to the amount of evidence for 0° in the first stimulus and that the response to the second stimulus reflects the degree of adaptation, adaptation to 0° should decrease as AR increases. Thus, as 0° is the only orientation that repeats in the second stimulus, the BOLD response to the second stimulus should be lowest for AR = 1 (reflecting strong adaptation to 0°) and highest for AR = 1.2 (reflecting weaker adaptation to 0°). The only region exhibiting this profile was found in the left occipital cortex (Fig. 4, F1,19 = 12.672, P = 0.002, η2 = 0.400), slightly dorsal from the occipital pole, whose location (−17 −100 5) is in good agreement with the localization of human V2/V3 by Shipp et al. (1995) and Wohlschläger et al. (2005). This result was also obtained when the ROI analyses were based on the same data as for the hysteresis contrast, that is, only trials in which subjects responded 0° or 90° to the first and 0° to the second stimulus (F1,19 = 10.587, P = 0.004, η2 = 0.358), and when we performed the same analyses in left dorsal V2 and V3 of n = 17 subjects in which early visual areas were independently defined based on retinotopic mapping (left dorsal V2 effect of AR: F1.880,30.083 = 3.798, P = 0.036, η2 = 0.192, Greenhouse–Geisser corrected; linear trend: F1,16 = 8.484, P = 0.010, η2 = 0.347; left dorsal V3 effect of AR: F1.823,29.164 = 3.453, P = 0.049, η2 = 0.177, Greenhouse–Geisser corrected; linear trend: F1,16 = 8.591, P = 0.010, η2 = 0.349). The effect was specific to the second stimulus, as on trials with only one stimulus, the area neither showed a significant effect of AR (Fig. 4B, F1.742,33.098 = 1.788, P = 0.186, η2 = 0.086) nor a linear trend (F1,19 = 3.104, P = 0.094, η2 = 0.140). The location of this area coincides with the representation of the lower right quadrant of the visual field, which exhibits the highest switch rate in binocular rivalry (Chen and He 2003). The switch rate in such stimuli is also thought to be due to neuronal adaptation.
We finally investigated whether areas exhibiting hysteresis and adaptation would also express the respective other effect. To this end, we tested whether the areas expressing adaptation showed a difference between hysteresis and no hysteresis trials, and whether the areas expressing hysteresis showed a linear effect of AR. To assure that our conclusions were based on the same set of data, we used only the trials in which subjects responded 0° or 90° to the first and 0° to the second stimulus for this analysis. None of the effects were significant, mirroring the behavioral results in which hysteresis and adaptation also did not interact (no significant linear trend in hysteresis areas, all P > 0.2, Supplementary Fig. S2; no significant difference between hysteresis and no hysteresis in V2/V3, P > 0.3, Supplementary Fig. S3). In line with the eye tracking data from the psychophysical experiments, we found that none of the effects could be accounted for by eye movements, as we did not observe any significant effect at P < 0.05 in the control ROIs. Taken together, the areal and hierarchical segregation of hysteresis from adaptation might explain how the brain entertains 2 distinct processing modes without mutual interference. This set of results fits well with the hypothesized functions of hysteresis and adaptation, as stabilization can be more easily achieved in higher processing stages in which neurons exhibit invariance to low-level features, while the extraction of new information may require the detailed and sensitive representation in early sensory processing stages.
Based on our behavioral and neuroimaging results, we developed a simple mathematical model that can account for the independence of hysteresis from adaptation. We chose the Bayesian framework as it explicitly takes into account the influence of previous experience on what is currently perceived. Here, perception is the result of an inference in which the available sensory information (the likelihood) is combined with previously acquired knowledge (the prior; Yuille and Kersten 2006). Our model accounts for perception of both the first and the second dot lattice in Bayesian terms. For both stimuli, grouping into one of the dominant orientations (θ) depends on the evidence (i.e. AR) in support of 0° and 90° for the rectangular dot lattice, and 0°, 60°, and 120° for the hexagonal lattice. According to Bayes' rule we can write:
The probability of perceiving θ given AR is the product of the likelihood of θ for a particular AR and the a priori knowledge about the estimated variable θ. The maximum a posteriori estimate (MAP) is given by:
The perceived orientation θ corresponds to the mode of the posterior distribution (Eq. 2). If θ does not exactly match one of the dominant orientations present in the lattice, the subject chooses the response alternative that is closest to the perceived orientation, since we are using a forced choice task.
We base our model on minimal assumptions about the distributions underling the likelihood and the prior probability of each of the 2 stimuli. For the rectangular lattice, we assume that the prior distribution is uniform across θ (Fig. 5A), reflecting that each orientation between 1° and 180° appears with equal likelihood throughout the experiment. Alternatively, a Gaussian distribution can be used without qualitatively changing the results. The likelihood of θ is modeled as a mixture of Gaussian distributions centered at 0° and 90° for the first stimulus and at 0°, 60°, and 120° for the second stimulus. At AR = 1, the SDs are equal and set to 15°, concurring with previous psychophysical studies showing that adaptation is still present for subsequently presented dot lattices with orientations falling 15° apart (Gepshtein and Kubovy 2005). When the evidence in favor of one of the dominant orientations is strong (i.e., the interdot distance is short), the corresponding Gaussian has a low SD and high amplitude, while weaker evidence leads to a corresponding high SD (low amplitude). For the first stimulus, SD 0°/SD 90° = AR. As a consequence, AR determines the relative difference between the peaks of the likelihood distribution (Fig. 5B). Both the likelihood and prior distributions are wrapped around the circumference of a circle of unit radius to give a proper representation of orientation.
Following our results showing that hysteresis and adaptation concurrently, but independently, determine perception of the second stimulus both in behavior and in the brain, we model these 2 processes independently. We implement hysteresis by increasing the prior distribution around the perceived orientation θp of the first stimulus (Fig. 5A). The new prior is a mixture of a uniform distribution and a Gaussian centered at θp:
Our results reveal that perceptual hysteresis and adaptation are functionally dissociated in the human brain. Whereas a widespread network of frontal, parietal, and ventral occipital brain areas is involved in hysteresis, BOLD responses following the behavioral profile of adaptation are only evident in extrastriate cortex. Previous psychophysical and modeling work have explained hysteresis and adaptation as resulting from the same mechanism, either neuronal adaptation (Orbach et al. 1963; Blake et al. 2003; Chen and He 2004), or a “persistent bias” (Gepshtein and Kubovy 2005). The functional division observed in our data refutes these interpretations. Recent models that propose separate mechanisms for hysteresis and adaptation seem to account better for our results. However, several of these models colocalize the mechanisms for hysteresis and adaptation to the same early sensory area (Noest et al. 2007; Wilson 2007; Brascamp et al. 2009), an idea that also seems at odds with our data. We provide a parsimonious alternative explanation of the coexistence of hysteresis and adaptation within a Bayesian framework of perception.
A Bayesian Account of Hysteresis and Adaptation
In the Bayesian framework, perception depends on the probability distribution of the available evidence (the likelihood function) and on knowledge about the world (the prior). For multistable stimuli, the likelihood function is multimodal, reflecting the different possible interpretations (Sundareswara and Schrater 2008). We perceive the posterior distribution, which is again multimodal. Given the nonflat prior, one of its peaks is maximal, and this perceptual interpretation gains dominance. In our model, perception of one interpretation adjusts the prior toward this interpretation, inducing hysteresis. We implement adaptation as a change in the likelihood function (i.e., the sensory evidence), reflecting that neurons show orientation specific reductions in responsiveness to repeated stimuli. Thus, the Bayesian framework is compatible with a dissociation between hysteresis and adaptation in neural space: One network computes the prior, whereas another, sensory network changes the likelihood function.
The Cortical Network Expressing Hysteresis
Many current Bayesian models of brain function envision perception as Bayesian inference and propose that this involves higher-level brain areas generating predictions (priors), which are then tested in lower, sensory areas (Mumford 1992; Rao and Ballard 1999; Friston 2005; cf. Summerfield and Koechlin 2008). Accordingly, our contrast (hysteresis vs. no hysteresis) should reveal higher-level brain areas involved in the generation and sensory areas involved in testing of a prediction against the incoming evidence. In our paradigm, the sensory feature for which a prior was generated and which was perceptually stabilized was the orientation of the dot lattices. Several of the areas we have identified are indeed processing orientation, in particular, the fusiform gyrus (Orban et al. 1997) and the IPS (Faillenot et al. 1999, 2001; Shikata et al. 2001). Furthermore, neurons in the IPS participate in grouping dots into oriented lines (Yokoi and Komatsu 2009), which is another basic operation required to perceive orientation in our stimuli. These areas are thus prime candidates for testing predictions against incoming evidence, showing BOLD repetition suppression for repeating percepts (the functional interpretation of BOLD repetition suppression is still under investigation, as it can represent top-down facilitation (resulting, e.g., in a reduced prediction error), sharpening of representations, and/or neural fatigue (Grill-Spector et al. 2006)). Regarding the higher-order areas, it has been proposed that perceptual hysteresis might be closely linked to working memory (Maier et al. 2003; Sterzer and Rees 2008), which would not be predicted by a Bayesian account. The pre-SMA (Petit et al. 1998) and also the IPS (Todd and Marois 2004) have been shown to be active in working memory tasks. However, the remaining areas do not show strong overlap with regions that are specifically involved in working memory for orientations, in particular because of the relative lack of lateral frontal activity in our study (Cornette et al. 2001, 2002). This might be explained by the fact that hysteresis is a form of implicit memory, whereas working memory is explicit. Importantly, Bayesian inference in perception is implicit. The IPS (Corbetta and Shulman 2002), pre-SMA (Fox et al. 2006), and the right anterior insula (Sridharan et al. 2008; Eckert et al. 2009) are also implicated in the control of top-down attention. Kanai and Verstraten (2006) have shown in a perceptual memory paradigm employing ambiguous motion that if attention is distracted during the presentation of the first stimulus or even in the blank period before the presentation of the second stimulus, perceptual stabilization fails. Thus, top-down attention seems to be essential to instantiate perceptual memory. In a Bayesian framework, spatial attention can be understood as a prediction of where a particular stimulus (or percept) will appear (Chikkerur et al. 2010). A test or update of the prior would then evoke transient activity in these areas, as has been found for perceptual switches in investigating binocular rivalry (Lumer et al. 1998) and apparent motion (Muckli et al. 2002; Sterzer et al. 2002; Müller et al. 2005). Finally, the only region in the network expressing perceptual hysteresis showing a correlation with the individual size of the hysteresis effect was the right dmPFC. This highlights its central involvement in perceptual hysteresis. It is likely that this area generates predictions about the upcoming stimulus, which are then tested in the remaining areas, an interpretation that concurs with previous studies showing that the dmPFC is involved in generating predictions under uncertainty (Volz et al. 2003).
Hysteresis trials entailed repetitions of both percepts and choices. Thus, it could be argued that the pattern of BOLD repetition suppression we observed for hysteresis trials reflects choice repetition. Previous studies implicate areas within the dorso- (Heekeren et al. 2008) and ventrolateral prefrontal cortices (Race et al. 2008) in abstract perceptual decision making. However, we did not observe differences between hysteresis and no hysteresis trials in these regions in our whole-brain analyses or in further ROI analyses (Supplementary Material). The lack of activation in areas systematically implicated in abstract perceptual decision making appears incompatible with the idea that the repetition suppression observed for hysteresis versus no hysteresis trials reflects repetition suppression for choices in a decision-making network.
Further analyses of the reaction times using a diffusion model of decision making (Ratcliff 1978) confirmed that the hysteresis effect in our study was most probably perceptual and not due to a decisional bias (Supplementary Material), as hysteresis was best explained by a change in the drift rate, that is, the speed of evidence accumulation, which is thought to capture perceptual effects (Voss et al. 2004). Interestingly, a higher drift rate may also explain the lower BOLD responses in hysteresis than in no hysteresis trials. In particular, the earlier crossing of the decision bound in the hysteresis when compared with the no hysteresis condition may be reflected in an earlier peak in neural activity, and thus a lower BOLD response, as the BOLD signal reflects the integral of neural activity over time (James and Gauthier 2006). However, the difference in the drift rate cannot explain our full set of results, because we also find areas that directly code for orientation and do not represent the drift rate itself (Gold and Shadlen 2007). Thus, the change in the drift rate may be a component of perceptual hysteresis, but cannot explain it alone. To further elucidate the relationship of percepts and choices, future studies will be needed that directly contrast the brain areas involved in perceptual hysteresis with those involved in choice repetition.
A Local Network Expressing Adaptation
In contrast to the widespread activation observed for hysteresis, adaptation was only evident in a portion of early extrastriate cortex, corresponding to dorsal areas V2/V3. Areas V2/V3 contain orientation-tuned neurons (Felleman and Van Essen 1987; Levitt et al. 1994). Importantly, these neurons also respond to dots that are perceptually bound into oriented lines (Peterhans et al. 2005) and more generally to illusory contours (von der Heydt et al. 1984). Such binding is critical for dot lattices to appear as having an orientation. The involvement of an early sensory area fits well with our Bayesian model, in which adaptation does not change the prior, but rather the representation of the available evidence. Moreover, our finding that adaptation was evident at an earlier level of the visual hierarchy than hysteresis meshes well with the observation that adaptation in multistable dot lattices is orientation specific, that is, the effect of AR on the second lattice vanishes when the orientation of the main axis differs by more than 15°. In contrast, hysteresis persists up to offsets of 30°, implying an involvement of regions with less-precise orientation tuning (Gepshtein and Kubovy 2005). The differences in orientation specificity, and the fact that hysteresis and adaptation display markedly different time constants (Kanai and Verstraten 2005), further speak for a dissociation of their underlying mechanisms.
The absence of a hysteresis effect in areas V2/V3 indicates that, in our paradigm, predictions may not be tested against sensory evidence in these early areas, but only at a later stage. In fact, Figure 2B shows that hysteresis can overrule the effects of adaptation. For example, it is possible to stabilize a 0° percept even at AR = 1.3−1, at which strong adaptation to 0° should lead to a repulsive after effect. Thus, it is conceivable that the output of areas V2/V3 serves as input for subsequent processing stages, for example, the fusiform gyrus or IPS where the hypothesis test is performed. In fact, if hysteresis serves to stabilize percepts against constantly changing low-level stimulus features (Kleinschmidt et al. 2002), larger receptive fields of higher visual areas are advantageous when compared with small receptive fields in earlier visual areas because they allow for a certain degree of invariance. Psychophysical evidence suggests that visual areas with larger receptive fields may be involved in hysteresis by showing that hysteresis is retinotopically organized, but extends beyond the exact location where the first stimulus was presented (Knapen et al. 2009). However, current Bayesian theories of brain function predict that the testing of predictions against incoming evidence occurs throughout the visual hierarchy (e.g., Friston 2005) and should thus also be evident in V2/V3. Although signatures of prediction errors have been observed in early visual areas, many of these studies have not reported prediction errors on every level of the hierarchy (Fang et al. 2008; den Ouden et al. 2009; Alink et al. 2010). We thus suggest that the signatures of a perceptual hypothesis test may be found in the brain areas that provide the information that is most relevant for the task at hand, and not at every stage of processing. Future studies could test this proposal explicitly, by varying the visual features that are relevant for perceptual decisions about identical stimuli from trial-to-trial.
Interestingly, our psychophysical data indicate a further dissociation: Adaptation is present even if the adapting orientation is not consciously perceived, whereas hysteresis does depend on what was perceived in the previous stimulus. This dichotomy might be explained by the fact that hysteresis involves distributed processing, whereas adaptation is a purely local phenomenon: Several current theories of conscious perception predict that the spatial scale at which neuronal processing occurs determines whether content enters awareness or not (Dehaene et al. 2006; Melloni and Singer 2010; Edelman et al. 2011).
Hysteresis and adaptation in multistable stimuli illustrate how previous experience is used to guide perception: The brain needs to cope with constantly changing, unreliable, or ambiguous input and exploiting the regularities of the world is an efficient strategy to achieve this goal. Recurring patterns are informative, because they allow predicting what will happen next, helping to stabilize perception in face of constantly changing low-level stimulus properties. This is reflected by the hysteresis effect. On the other hand, recurring patterns are redundant and could be discarded from further processing, thus emphasizing the uptake of new information. This is evident in the repulsive after effects resulting from adaptation. The brain is optimized for both strategies. Our study shows that hysteresis and adaptation map into 2 distinct, anatomically and hierarchically segregated cortical networks: While higher-order fronto-parietal areas and higher-order visual areas are involved in perceptual stabilization, adaptation is confined to a local node in early visual areas (V2/V3). This result fits well with the proposed functions of hysteresis and adaptation, as stabilization can be more easily achieved at higher processing stages where neurons exhibit invariance to low-level features, while the extraction of new information requires detailed and sensitive representations at earlier sensory processing stages. This separation possibly endows the brain with the flexibility to switch between 2 modes, one that emphasizes the new, and another that exploits already available information. Such flexibility is crucial in a nonstationary environment where one should always be prepared to predict and to react to the unexpected.
This work was supported by the Max Planck Society, and partially funded by LOEWE Neuronale Koordination Forschungsschwerpunkt Frankfurt (NeFF). C.M.S. is currently funded by a Human Frontier Science Program Long-Term Fellowship. Funding to pay the Open Access publication charges for this article was provided by LOEWE Neuronale Koordination Forschungsschwerpunkt Frankfurt (NeFF).
We thank R. Deichmann, U. Nöth, and S. Anti for their help with the sequences, G.-F. Paasch for his help with data acquisition, and J. Aru, M. Schmid, and 2 anonymous reviewers for their comments on this paper. We are indebted to A. Kohler for his input and invaluable help with data analysis. Conflict of Interest: None declared.