Neurons, even in the earliest sensory areas of cortex, are subject to a great deal of contextual influence from both within and across modality connections. In the present work, we investigated whether the earliest regions of somatosensory cortex (S1 and S2) would contain content-specific information about visual object categories. We reasoned that this might be possible due to the associations formed through experience that link different sensory aspects of a given object. Participants were presented with visual images of different object categories in 2 fMRI experiments. Multivariate pattern analysis revealed reliable decoding of familiar visual object category in bilateral S1 (i.e., postcentral gyri) and right S2. We further show that this decoding is observed for familiar but not unfamiliar visual objects in S1. In addition, whole-brain searchlight decoding analyses revealed several areas in the parietal lobe that could mediate the observed context effects between vision and somatosensation. These results demonstrate that even the first cortical stages of somatosensory processing carry information about the category of visually presented familiar objects.
It is well known that most input to neurons even in early sensory areas of cortex (i.e., primary sensory areas) comes from other cortical neurons (i.e., local or long-range connections). These connections provide a way for prior experience and context to shape responses of early sensory neurons. We recently showed (Smith and Muckli 2010) that such connections transmit content-specific information about natural visual scenes to nonstimulated subregions of primary visual cortex. Early sensory neurons, however, are also subjected to contextual influences from other sensory modalities via feedback from multisensory areas and direct links between early sensory cortices of different modalities (see Driver and Noesselt 2008). Here, we investigated whether different visual images of common object categories would be reliably discriminated in early somatosensory cortex despite participants having no haptic interaction whatsoever with those visual stimuli during the experiment. We expected that this might be possible due to associative links that are formed through experience between different sensory aspects of specific objects (e.g., the sight and haptic experiences associated with wine glasses; see Damasio 1989; Meyer and Damasio 2009).
Support for the above hypothesis is provided by several lines of evidence: 1) early sensory areas are subject to modulatory influences from other modalities (e.g., Calvert et al. 1997; McIntosh et al. 1998; Driver and Noesselt 2008), 2) silent but sound-evoking movies (e.g., a movie of a piano key being struck) can be discriminated in early auditory cortex (Meyer et al. 2010), 3) natural auditory scenes (e.g., traffic vs. people talking) presented to blindfolded participants can be discriminated in early visual cortex (Vetter et al. 2011), and 4) visual stimuli presented in isolation lead to selective firing in primary somatosensory cortex after previous association with tactile stimulation (Zhou and Fuster 2000). Thus, there is some evidence for the hypothesis that cross-modal context can trigger content-specific representation in early sensory areas. In the present experiment, we ask whether the same is true with respect to object categories in the domain of vision and touch. Meyer et al. (2011) addressed a related point. These authors presented visual movies of hands exploring real objects and successfully discriminated such movies in primary somatosensory cortex. In contrast, here we only ever presented visual images of object categories, and no haptic interactions were ever depicted in our visual stimuli. Hence, the present experiments test whether the observation of haptic interaction is required in order to observe such cross-modal context effects from vision to touch. In addition, in the present work, we explicitly test whether such effects are present only for visual object categories for which we have a significant degree of visuo-haptic experience.
In Experiment 1, we presented participants with visual images of object categories for which they would tend to have rich visuo-haptic experience (wine glasses, mobile phones, and apples; see Fig. 1A and Supp. Fig. 1A) in a block design fMRI experiment. We used common object categories since we expected cross-modal connections to transmit contextual information maximally about specific object categories when participants have rich multisensory experience with those objects. Participants fixated and performed an orthogonal color-change detection task at fixation. We used a linear classifier to investigate whether or not the 3 categories of visual object could be discriminated from the fMRI activity in primary somatosensory cortex (see, e.g., Kamitani and Tong 2005; Smith and Muckli 2010). If the decoding accuracy were significantly greater than chance then we could conclude that cross-modal connections from vision to the earliest cortical stages of somatosensory processing transmit content-specific information about object categories, even in the absence of any haptic interaction with those object categories during the experiment. In Experiment 2, we replicated Experiment 1, but also included a set of unfamiliar objects (modified from Op De Beeck et al. 2008) to explicitly test whether any observed cross-modal context effects were due to prior visuo-haptic experience with familiar objects. In addition, in Experiment 2, we also eliminated any need for a button-press response from the participant to eliminate any potential interactions with our effects of interest.
Materials and Methods
A total of 8 right-handed (via self-report) participants took part in Experiment 1 (6 females) and a new set of 10 right-handed (via self-report) participants took part in Experiment 2 (6 females). All participants gave written, informed consent in accordance with procedures approved by the ethics committee of the Department of Psychology at the University of Western Ontario.
Stimuli and Design
Participants were presented with full color images of visual objects in a block design protocol. There were 9 images in Experiment 1, comprising 3 different exemplars of three different categories of visual object (wine glasses, mobile phones, and apples; see Fig. 1A and Supplementary Fig. 1A). We chose these visual objects categories since most participants would tend to have a rich haptic experience with such objects. Each stimulus was superimposed upon a white background of 14 by 14° of visual angle. The maximum horizontal and vertical extent of any stimulus was ∼10.5° and 12°, respectively.
Each run comprised 18 stimulation blocks (12 s) preceded and followed by blocks of fixation (12 s). Thus, each visual image was shown twice within 1 run, with the sequence being randomly determined. Each run lasted hence 444 s, allowing for an initial 12 s of fixation. Within each 12 s block of stimulus presentation, the stimulus was flashed on and off repeatedly (i.e., on 200 ms, off 200 ms, repeated 30 times; we used this presentation cycle to maximize signal to noise, see Smith and Muckli 2010). Participants were instructed to maintain fixation throughout each run on a small central fixation checkerboard (0.4°). The task of the participant was to monitor the stream of frames for a change in color of the fixation marker (randomly chosen frames were changed from black and white to red and green). Participants responded to such a change with a button press with their left (i.e., nondominant) hand. Note that this task is orthogonal to the category of visual object presented. Participants completed between 4 and 6 runs of the main experiment.
In an independent finger-mapping experiment, we localized the cortical representation of the first 4 digits of the right (dominant) hand in the primary and secondary somatosensory cortices (see, e.g., Ruben et al. 2001; Van Westen et al. 2004; Huang and Sereno 2007; Nelson and Chen 2008). Note that no mention of this part of the experiment, which was run at the very end of the session, was ever made to participants until all the visual object runs had been completed. We stimulated the pads of the first 4 digits of the right hand in a random sequence as in Huang and Sereno (2007) in a traditional block design (alternating blocks of 12 s stimulation vs. baseline). We chose this localizer design after pilot-testing revealed that it reliably activated both primary and secondary somatosensory cortices. There were 18 stimulation blocks in each run, each preceded and followed by 12 s of baseline, thus giving a total run length of 444 s. Small plastic clamps were placed around the pad of each of the first four fingers that could be driven by air pressure to displace the skin surface (somatosensory stimulus system, 4D neuroimaging). The amplitude and timing of stimulus delivery was controlled by a pneumatic delivery system (300 ms off, 100 ms on, within each 12-s stimulation block), itself controlled via in-house software written in Matlab. Participants were required to close their eyes for the duration of each finger mapping run. Each participant completed either 2 or 3 runs of this localizer.
In Experiment 2, participants were presented with the same objects as those in Experiment 1 (i.e., familiar objects: wine glasses, mobile phones, and apples) as well as a matched set of unfamiliar objects (Fig. 1B and Supplementary Fig. 1B). The unfamiliar objects were included to explicitly test to what extent prior visuo-haptic experience with the object categories might mediate decoding from visual images in somatosensory cortex. There were 3 categories of unfamiliar object (modified from Op De Beeck et al. 2008): cubies, smoothies and spikies (see Fig. 1B and Supplementary Fig. 1B) and 3 instances of each category to match the design with the familiar objects. In a block design, each object (9 familiar plus 9 unfamiliar) was presented for 12 s, being preceded and followed by 12 s fixation. Order of object presentation was random and participants each completed 10 runs. The task in this experiment was altered in order to remove any requirement for a finger-based button response—hence, participants again fixated centrally but here simply had to count the number of fixation color changes present in each run (randomly sampled from 7 to 17). In addition, we again performed a finger mapping experiment as in Experiment 1 but, this time, we mapped the digits of both hands independently (left and right). The remaining details were the same as Experiment 1.
MRI Data Acquisition
Visual images were rear projected via an LCD projector (Silent Vision 6011, screen resolution of 1024 × 768, Avotec) onto a screen mounted behind the participant as he or she lay in the bore of the magnet. MRI data were collected with a 3T Siemens Tim Trio System with a 32-channel head coil and parallel imaging techniques (IPAT factor: 2) at the Robarts Research Institute, London, Ontario. Blood oxygen level–dependent (BOLD) signals were measured with an echo-planar imaging sequence (echo time: 30 ms, repetition time: 2000 ms, field of view: 200, flip angle: 75, 34 oblique slices, resolution 2.5 × 2.5 mm, slice thickness of 2.8 mm, no gap). Slices were positioned to cover the entire brain except anterior frontal and temporal regions (see Fig. 5). A high-resolution 3D anatomical scan (3D MPRAGE, 1 × 1 × 1 mm3 resolution) was recorded in the same session as the functional scans. All experimental runs (either 4 or 5) plus 2 or 3 runs of the finger mapping localizer were completed in the same imaging session in Experiment 1. In Experiment 2, experimental runs (10) were completed in a first session followed by finger mapping runs in a second session.
MRI Data Processing
Functional data for each main experimental run were slice time corrected, corrected for 3D motion, linearly de-trended and spatially normalized into the Talaraich space with Brain Voyager QX (Brain Innovation, Masstricht, The Netherlands). Note, we did not apply temporal filtering to the data used for pattern classification (see, e.g., Kamitani and Tong 2005; Walther et al. 2009). We then normalized the time series of each voxel by its mean value, independently per run. Activity patterns were obtained by averaging the fMRI BOLD activity during each 12-s stimulation block (shifted for 4 s or 2 TRs in order to account for the hemodynamic lag) independently for each voxel. These are the estimates that we use as the input to the pattern classification algorithm described below. The functional localizer data were subject to the standard processing stream above including temporal filtering (2 cycles) and subsequently analyzed by using a GLM approach with 1 stimulus predictor coding stimulus onset convolved with a double gamma model of the hemodynamic response. We used t-values defined from the contrast of stimulation minus fixation to determine “finger-pad sensitive” voxels in each of our predefined anatomical masks of S1 and S2. For the mapping data of Experiment 1, we selected the top 100 voxels to contralateral stimulation (left PCG) or when pooled across hemispheres, and the top 50 voxels responding to ipsilateral stimulation (right PCG), since we would expect only smaller sub-regions to be active in this case. For the mapping data of Experiment 2, we selected the top 100 voxels within each region corresponding to either A) contralateral stimulation or B) both contra- and ipsilateral stimulation (top 50 to each).
Anatomical Mask of S1
The primary somatosensory cortex, also known as S1 or the post-central gyrus (PCG) is a complex structure, consisting of four separate areas (see, e.g., Keysers et al. 2010): area 3a, 3b, 1, and 2. Area 3a deals primarily with proprioceptive information, area 3b with tactile information, area 1 with a second level of tactile analysis, and finally area 2 combines information from each other area (3a, 3b, and 1) thus being the first level at which tactile and proprioceptive information is combined. Thus, defining an anatomical mask allows us to provide our pattern classification algorithms with the entire repertoire of information that is processed in primary somatosensory cortex (see Meyer et al. 2011). It further allows us to directly compare our results to those of Meyer et al. (2011) who likewise defined an anatomical mask of S1. Anatomical masks of both the left and the right PCG were defined for each participant on their anatomical MRI in Talairach space (we used MRIcron for this purpose, Rorden et al. 2007). We defined the masks in Talairach space so they would be directly comparable across participants and in the same space as our S2 masks (see below). In defining our masks, we used the same criteria as Meyer et al. (2011): the lateroinferior border was taken to be the last axial slice upon which the corpus callosum was not visible. From anterior to posterior the masks were defined by the floors of the central and postcentral sulci and they did not extent to the medial wall. Example masks are shown for a representative participant in Figure 2. The size of each mask is reported in Supplementary Tables 1 and 2.
Probabilistic Maps of S2
The human secondary somatosensory cortices (S2) are located along the parietal operculae and consist of at least four cytoarchitectonic sub-regions (OP1-4; Eickhoff et al. 2006A, 2006B). In order to define anatomical masks of S2 we summed the probabilistic maps of OP1–OP4 given in the Juelich Anatomy toolbox (Eickhoff et al. 2005). We used the 30% probability cutoff for each map since when summed together across maps this gave comparable numbers of voxels as in our anatomical masks of S1. The probability maps are defined in the MNI single subject space (COLIN27) with an affine correction in the y and z axes to make the Anterior Commissure the exact center of the coordinate system (so-called MNI Anatomical Space; see Eickhoff et al. 2005). In order to transform the coordinates from MNI space to TAL space (the reference space used in Brain Voyager and hence the present work), we applied the MNI to TAL transformation given in (Lancaster et al. (2007); icbm_fsl2tal.m downloaded from http://brainmap.org/icbm2tal/) and shown to generate relatively good agreement between MNI and TAL space across the brain volume. Thus, we obtained one left S2 and one right S2 anatomical mask from these procedures that were subsequently used for each individual participant.
We used the same procedure, as outlined above for S2, to define probabilistic anatomical masks of V1, premotor, and motor cortex in each hemisphere (again from the Juelich Anatomy toolbox, Eickhoff et al. 2005; see Amunts et al. 2000; Geyer et al. 1996; and Geyer 2003 for the cytoarchitectonic maps corresponding to V1, primary motor cortex, and premotor cortex, respectively).
Multivariate Pattern Classification Analysis
We trained a linear classifier (linear support vector machine, SVM) to learn the mapping between a set of multivariate observations of brain activity and the particular visual object category that had been presented (3 categories). We then tested the classifier on an “independent” set of test data. We trained the classifiers with a set of single-block brain activity patterns. We tested the classifiers on single block activity patterns for each stimulus class in the independent set of test data. To assess the performance of our classifiers in Experiment 1, we used a leave one-example out cross-validation procedure: thus our models were trained from all examples minus one for each class, and then tested on the remaining example in each class (see, e.g., Duda et al. 2001; Ethofer et al. 2009; Harrison and Tong 2009). As we used a traditional block design (12 s ON 12 s OFF), this ensures that there is no significant influence of one trial to the next in terms of BOLD signal estimates, unlike in a rapid event-related design (see Misaki et al. 2010). Thus, a leave-one example out procedure is justified in the present case (see also Pereira and Botvinick 2011). In order to specify the sequence of cross-validation cycles (there are many possible here), we sorted the trials from each class (24 or 30, respectively) in terms of temporal occurrence and then left out the nth trial from each class (see also Etzel et al. 2008; Harrison and Tong 2009). This ensures that the test data were independent across cross-validation cycles, as would be the case in leave-one run out cross-validation schemes (see, e.g., Kamitani and Tong 2005; Smith and Muckli 2010). We tested whether group level decoding accuracy was above chance by performing 1-sample t-tests against the chance level of 1/3 (see, e.g., Walther et al. 2009, 2011; Chen et al. 2011). In Experiment 2, we report our results with an even stronger test of the classifier's ability to generalize well, leave-one run out cross-validation performance (see Coutanche and Thompson-Schill 2012; for an advantage of having more runs to cross-validate over even with the same number of events).
The linear SVM algorithm was implemented using the LIBSVM toolbox (Chang and Lin 2011), with default parameters (notably C = 1). Note that the activity of each voxel in the training data was normalized within a range of −1 to 1 prior to input to the SVM. The test data were normalized using the same parameters (min, max) as obtained from the training set normalization in order to optimize the classification performance (see Chang and Lin 2011).
We additionally performed a whole-brain searchlight analysis (Kriegeskorte et al. 2006; Walther et al. 2009; Nestor et al. 2011; Pereira and Botvinick 2011). We used the SearchMight toolbox of (Pereira and Botvinick 2011), implemented in Matlab, to perform whole-brain decoding with a linear SVM (C = 1, as above). This involved selecting a specific searchlight size, here a cube of 343 2 mm3 voxels (7 × 7 × 7), and systematically shifting this cube throughout the whole-brain volume and performing the decoding analysis independently at each different center voxel position. These analyses were performed independently for each participant, using a common group mask, and then the resulting decoding accuracy maps were averaged across participants. Statistical significance was assessed by testing whether the mean accuracy across participants was significantly higher than chance (1/3) at each voxel (see also Walther et al. 2009, 2011; Chen et al. 2011). Correction for multiple comparisons was ensured by use of a cluster threshold estimated by the program AlphaSim (B.D. Ward, http://afni.nimh.nih.gov/afni/docpdf/AlphaSim.pdf) Experiment 1 (t > 3.5, voxelwise P < 0.01, cluster P < 0.05, equal to 35 voxels), Experiment 2 (t > 3.25, voxelwise P < 0.01; cluster P < 0.05; equal to 37 voxels and 33 voxels for familiar and unfamiliar maps, respectively), Experiments 1 and 2 pooled (t > 2.9, voxelwise P < 0.01; cluster P < 0.05, equal to 64 voxels). Results are shown projected onto the cortical surface reconstruction of a reference brain (COLIN27) available in the NeuroElf package (www.neuroelf.net).
We defined anatomical masks of the bilateral postcentral gyrus (PCG) in each individual participant (see Fig. 2 and Materials and Methods). Figure 3A shows the cross-validated performance of the linear classifier (linear SVM) in decoding visual object category (3AFC) from the PCG. Remarkably, decoding of visual object category significantly exceeded chance in each case: right PCG mean decoding accuracy 40%, t(7) = 3.68, P = 0.004 (chance 1/3); left PCG 36.8%, t(7) = 2.18, P = 0.03 (chance 1/3); and pooled 37.5%, t(7) = 2.71, P = 0.015 (chance 1/3). Decoding accuracy was significantly higher in the right as opposed to the left PCG (t(7) = 2.61, P = 0.035). Thus, the earliest stages of somatosensory cortex, that is, the PCG, carry information about different categories of visual object presented to participants. The full confusion matrices underlying this performance can be seen in Supplementary Figure 2. Crucially, univariate signal changes did not differ across the 3 object categories in the PCG (Supplementary Fig. 3 and Supplementary Text).
We additionally performed an independent tactile finger mapping experiment where we localized the cortical representation of the first 4 digits (thumb to ring finger) of the right hand (see Supplementary Fig. 4). This allowed us to define voxels that were maximally sensitive to tactile stimulation of the fingertips in the right hand in each hemisphere (note that we would of course expect a smaller number in the right hemisphere—ipsilateral to tactile stimulation). We used these data to select a subset of those voxels present in the anatomical PCG masks that showed high sensitivity to such tactile stimulation (see Materials and Methods). Cross-validated mean decoding accuracy was again above chance for each hemisphere independently and pooled across hemispheres (see Fig. 3B: left PCG 37%, t(7) = 2.01, P = 0.042 (chance 1/3); right PCG 39%, t(7) = 3.36, P = 0.006; pooled 39%, t(7) = 2.02, P = 0.042), although there was no difference in decoding across hemispheres (P = 0.26). Thus, visual object category could still be decoded reliably above chance even when restricting the classifier only to voxels that show high responses to tactile stimulation of the right-hand finger pads.
What about higher order somatosensory brain regions? We defined anatomical masks of S2 on the basis of probabilistic maps (Eickhoff et al. 2005). Figure 3C shows mean decoding accuracy independently in left and right S2, and pooled across hemispheres. Decoding accuracy was reliably above chance in right S2: mean Decoding accuracy 37% (t(7) = 3.18, P = 0.008, chance = 1/3), and when pooled 38% (t(7) = 2.84, P = 0.0125, chance = 1/3) but not in left S2 35% (t(7) = 0.76, P = 0.235, chance = 1/3, NS). There was no difference in decoding performance across hemispheres (P = 0.54). Thus, in agreement with our results in S1, S2 contains information that discriminates between categories of visual objects. We also repeated the analysis in S2 when preselecting voxels that had high responses to tactile stimulation of the right-hand finger tips (see Materials and methods) and found reliable decoding only when pooled across hemispheres (mean 39.8%, t(7) = 2.28, P = 0.028). Thus, in agreement with our analyses on S1, S2 also contains information about visual object categories.
We also defined anatomical masks of the left and right motor and premotor cortices. Decoding of visual objects never reached significance in any of these ROIs (Fig. 3E,F, all P's > 0.05). In V1, however, as expected decoding accuracy was very high and robustly greater than chance (Fig. 3D: right V1: 97%, t(7) = 41.03, P < 0.0001; left V1: 98%, t(7) = 80.74, P < 0.0001; pooled: 98%, t(7) = 70.33, P < 0.0001). Furthermore, we ran a 2-way RM ANOVA (ROI: PCG, S2, V1, premotor, motor; side: right, left, pooled) to test whether decoding performance differed across different regions of interest. This analysis revealed only a main effect of ROI, F4,28 = 379.02, P < 0.0001. Follow-up tests demonstrated that decoding performance was significantly higher in V1 than in each other region (all t's > 22.2, all P's < 0.0001). No other effects reached significance although there was a trend for higher decoding in PCG than in premotor cortex (P = 0.0785).
Within-Category Decoding in V1 and PCG
Finally, in order to further probe the nature of the representations present in the PCG we investigated within-category decoding within the PCG and V1 (i.e., discriminating between different images of the same object category—e.g., the 3 different images of apples). If the decoding within the PCG is due to differences in grasping/motor affordances for each object category then we would not expect to find any within-category decoding in this region, since such affordances would be equal within a category. Within V1, in contrast, we would expect to be able to decode the different images within a category (as well as between categories). We found, as expected, that within-category decoding was highly significant within each subregion of V1 (right: 59%, t(7) = 4.55, P = .0013; left: 57%, t(7) = 5.23, P = 0.0006; pooled: 60%, t(7) = 5.07, P = 0.0007; chance = 33%). Crucially, however, such decoding was never significant within the PCG (all P's > 0.05, all means ≤ 33%) and moreover, between-category decoding (38%) was reliably stronger than within-category decoding (33%): F1,7 = 5.69, P = 0.048, in this region. Note that no reliable within-category decoding was found for any individual object category (i.e., wine glasses, mobile phones, or apples) in any region of the PCG (left, right, or pooled; see Supplementary Fig. 5). In addition, as expected within-category decoding (59%) was significantly weaker than between category (98%) in V1 (F1,7 = 78.280, P < 0.001). Hence, our results demonstrate that information is present within the PCG that can discriminate between but not within different visual object categories. For a full depiction of within-category decoding within the PCG, see Supplementary Figure 5.
Finally, we ran a whole-brain searchlight decoding analysis (see Kriegeskorte et al. 2006) to investigate which other regions of cortex might discriminate between visual object categories. As expected, this analysis revealed high decoding performance throughout much of occipito-temporal cortex (Fig. 5A), peaking at 98% correct in early visual cortex. We note that this high decoding performance observed in early visual cortex serves as validation of our methodological approach. The searchlight analysis corroborated our earlier region-of-interest (ROI) analysis by finding reliable decoding in a cluster centered in the right postcentral sulcus bordering area 2 of the PCG and the superior extent of right S2 (Fig. 5A).
We found significant decoding, moreover, in several additional regions of posterior parietal cortex that could potentially mediate the transfer of information from visual cortex to early somatosensory brain regions (Fig. 5A). We observed significant decoding in the superior parietal lobule, especially on the left (comprising area 7 and area 5), areas known to be important for integration of proprioceptive and visual information (e.g., Hsiao 2008). We also found reliable decoding in several additional regions important in visuo-motor control: from posterior to more anterior sections of the IPS and bilateral superior parietal occipital sulcus—SPOC (e.g., Gallivan et al. 2011A, 2011B; Rossit et al. 2013). We also found smaller clusters around left dorsolateral prefrontal cortex and left premotor cortex—all areas important in visuo-motor control (e.g., Gallivan et al. 2011B). In summary, the whole-brain analysis supports our ROI-based analyses, and highlights regions of parietal cortex that may potentially transmit information regarding different visual objects to early somatosensory cortex (S1 and S2) and regions of frontal cortex that might also be involved in representing motor plans.
In a second experiment, we replicated the design of Experiment 1 but added a set of unfamiliar visual objects (Fig. 1B and Supplementary Fig. 1B) to explicitly test whether prior visuo-haptic experience with the presented visual objects is necessary to observe reliable decoding in somatosensory cortex. In addition, this experiment allowed us to perform an independent replication of the results of Experiment 1 (with familiar visual objects) while testing the generalization of our effects to a task that excluded any overt motor response during scanning (i.e., no button press response required).
As in Experiment 1, we defined individual anatomical masks of the postcentral gyri in each of our 10 participants. Decoding of familiar visual objects was above chance in each region (Fig. 4A: right PCG: 37% t(9) = 2.69, P = 0.012; left PCG: 37% t(9) = 1.89, P = 0.046; pooled: 39% t(9) = 3.43, P = 0.004), thus replicating our findings in Experiment 1. Decoding of unfamiliar objects (Fig. 4A), on the other hand, was not above chance in any region (all P's > 0.092). There were no differences between decoding performance across hemispheres (left vs. right) for either familiar or unfamiliar objects (both P's > 0.58). The confusion matrices underlying this performance can be seen in Supplementary Figure 2. Importantly, univariate signal changes did not lead to reliable differences between object categories in the PCG (Supplementary Fig. 3). In addition, in Experiment 2, we improved our finger mapping procedure to map the somatosensory representation of the digits of each hand independently, allowing us to examine whether such effects were present in the PCG when limited to “contralateral finger-sensitive voxels” (top 100 voxels responding to stimulation of the contralateral hand). Mean decoding of familiar visual objects was significant in the right PCG 38% (t(9) = 2.16, P = 0.029) and pooled PCG 36% (t(9) = 2.19, P = 0.028) but not in left PCG (34%, P = 0.28), again replicating Experiment 1. Once more, however, no significant decoding was present for unfamiliar objects in any region (all P's > 0.11). There was a nonsignificant trend for higher decoding in the right PCG (t(9) = 1.99, P = 0.078) for familiar but not unfamiliar (P = 0.34) objects. Thus, even when limiting the analysis to “contralateral finger-pad-sensitive voxels” decoding of familiar visual objects is possible in right PCG and pooled PCG.
Moreover, in an additional analysis, we selected tactile sensitive voxels in each region that responded to either contra- or ipsilateral stimulation (top 50 to each): we did this to ensure that both earlier (e.g., 3b, 1) and later regions (e.g., area 2) within the PCG would have a good chance to be selected (see Keysers et al. 2010). This analysis (Fig. 4B) revealed reliable decoding in each region again only for familiar objects: right PCG 40% (t(9) = 4.52, P = 0.0007), left PCG 37% (t(9) = 2.4, P = 0.02) and pooled 36.8% (t(9) = 2.67, P = 0.013). Again, there were no significant effects of hemisphere on decoding performance for either familiar or unfamiliar objects (both P's > 0.15). This analysis hints toward the role that higher regions (e.g., area 2) within the PCG may play in contributing to the present effects, as area 2 is the only region within the primary somatosensory cortex to receive input from ipsilateral stimulation of the hand (see Keysers et al. 2010).
Thus, the results of Experiment 2 replicate those of Experiment 1 in showing that decoding of familiar visual objects is present in bilateral PCG. The more refined somatosensory mapping of hand digits allow us to demonstrate that such decoding is still present when voxels are limited to either those active in response to contralateral finger stimulation OR when limited to those active via both contra- and ipsilateral finger stimulation. Furthermore, on no occasion is the decoding of the unfamiliar object categories above chance in the PCG.
Secondary Somatosensory Cortex (S2)
Decoding of familiar visual objects was above chance in the right S2 (Fig. 4C: mean 38% t(9) = 2.64, P = 0.013; all other P's > 0.12), replicating Experiment 1. Decoding of unfamiliar objects was not above chance in any region (Fig. 4C: all P's > 0.12). There was no difference in performance across hemispheres (both P's > 0.1). When limiting the analysis to “contralateral finger-sensitive voxels” in S2 (top 100 voxels) decoding of familiar objects was only possible in the right S2 (mean 37% t(9) = 2.82, P = 0.01). Decoding of unfamiliar objects was not possible in left or right S2, although there was a weak effect when pooled across hemispheres (mean 36%, t(9) = 1.92, P = 0.0435). Again there were no differences across hemisphere (both P's > 0.29). Hence, there is clear evidence of decoding of familiar visual objects in right S2, whether using the whole anatomical region or whether pre-selecting voxels as a function of sensitivity to tactile stimulation of the contralateral hand. There is also a weak suggestion of a reliable effect of discriminating unfamiliar objects when pooling across hemispheres, for voxels preselected as a function of sensitivity to finger-pad stimulation.
As expected in V1 decoding was robustly present in all participants for both familiar (Fig. 4D: right V1: 97% t(9) = 49.1, P < 0.0001; left V1: 96.8% t(9) = 55.2, P < 0.0001; pooled: 98% t(9) = 76.6, P < 0.0001) and unfamiliar object categories (Fig. 4D: right V1: 95.6% t(9) = 30.1, P < 0.0001; left V1: 94% t(9) = 22.7, P < 0.0001; pooled: 96.5% t(9) = 35.6, P < 0.0001). To investigate whether decoding differed for familiar and unfamiliar objects in V1, we conducted a 2-way ANOVA with the factors (region and familiarity): there were no significant main effects or interactions in this analysis although there was a trend (P = 0.065) for a main effect of region. Thus decoding of both familiar and unfamiliar objects was similar in V1, as expected.
We used probabilistic anatomical masks to define the location of several additional control regions in our participants. Decoding of familiar visual objects (Fig. 4E) was significant in both left (37% t(9) = 2.68, P = 0.013) and right (36%, t(9) = 1.97, P = 0.04) premotor cortex but not when pooled (36%, t(9) = 1.41, P = 0.097). However, no reliable decoding was present in motor cortex itself (Fig. 4F: all P's > 0.14). Decoding of unfamiliar objects was not significant in any motor or premotor region tested (Fig. 4E,F: all P's > 0.41).
To investigate whether decoding performance differed as a function of familiarity and region, we also ran the same 2-way ANOVA in each other ROI: critically only 2 regions showed a main effect of familiarity on decoding performance: in the PCG but only when limited to finger sensitive voxels from stimulation of both hands, F1,9 = 7.29, P = 0.024, mean familiar 38%, unfamiliar 33%; and in premotor cortex, F1,9 = 5.75, P = 0.04, mean familiar 37%, unfamiliar 33%. No other main effects or interactions were significant in these or any other ROIs. Hence, decoding was reliably higher for familiar compared with unfamiliar object categories in these two brain regions, suggesting a specific influence of prior knowledge.
Finally we also ran a 3-way ANOVA, with the following factors: ROI (PCG, S2, V1, premotor, motor), familiarity (familiar, unfamiliar), and side (right, left, or pooled). This analysis revealed only a main effect of ROI, F4,36 = 586.07, P < 0.0001. Follow-up t-tests revealed that decoding performance in V1 was reliably higher than that present in each other region (all t's > 29.9, all P's < 0.0001). No other effects were significant (all P's > 0.05) although there was a nonsignificant trend for decoding to be higher in PCG than in motor cortex (P = 0.0787).
Within-Category Decoding in the PCG and V1
Within-category decoding was not significant in any region of the PCG (all P's > 0.05) for familiar objects, for any particular object category independently, or when averaged across categories (see Supplementary Fig. 5). This finding therefore replicates Experiment 1. However, for unfamiliar objects, there was surprisingly a significant within-category decoding within the right PCG (38%, t(9) = 2.44, P = 0.019) and when pooled across hemispheres (37%, t(9) = 1.97, P = 0.04) but not in left PCG (34%, t(9) = 0.29, P = 0.39), for the analysis that averaged across object categories. The results of this analysis when split by object categories clearly reveals that this effect is entirely driven by the “spikies” object category (see Supplementary Fig. 5) and that hence, for 2 of the 3 unfamiliar object categories, no within-category decoding was observed (“cubies” and “smoothies”; see Supplementary Fig. 5). We discuss below the special features present in the “spikies” object category that render it, in restrospect, a less than ideal control in the present experiment (see Discussion). A 3-way ANOVA (decoding type: between vs. within; region: left, right, pooled; familiarity: familiar, unfamiliar) revealed that there was no difference across type of decoding (between vs. within; P = 0.52) or any interactions with this factor. There was a main effect of region (F2,18 = 4.02, P = 0.036) with decoding overall being higher in right than left PCG.
In V1, as expected, within-category decoding was again highly significant for familiar objects (right: 64%, t(9) = 7.22, P < 0.0001; left: 58%, t(9) = 6.78, P < 0.0001; pooled: 64%, t(9) = 7.04, P < 0.0001). The same pattern was also evident for unfamiliar objects (right: 68%, t(9) = 8.35, P < .0001; left: 65%, t(9) = 8.28, P < .0001; pooled: 70%, t(9) = 10.26, P < 0.0001). Decoding was higher for between (96%) than within (65%) category (F1,9 = 141.56, P < 0.001) classifications in V1. In addition, there was an interaction between type of decoding and familiarity (F1,9 = 10.78, P = 0.009) with between-category decoding being slightly better for familiar than unfamiliar objects but within-category decoding being slightly better for unfamiliar objects. There was also an interaction between decoding type and region (F2,18 = 4.7, P = 0.023) due to a greater decrease in decoding in left V1 relative to right V1 (or pooled) for within- not between-category decoding.
We show in Figure 5B,C the results of a whole-brain searchlight analysis independently for familiar and for unfamiliar visual object categories in Experiment 2. The broad pattern of decoding is similar in both cases: a vast region of visual cortex extending into temporal and parietal cortex has very high decoding accuracies. There is evidence of reliable decoding in the SPOC, and along the IPS and the superior parietal lobule, as in Experiment 1. This pattern is, however, broadly similar across both familiar and unfamiliar object categories. The findings for familiar objects fit well with those in Experiment 1 in terms of visual and parietal cortex although there is no evidence of significant decoding around the postcentral sulcus/postcentral gyrus in Experiment 2, for familiar objects. In addition, fewer areas are found in frontal regions in Experiment 2 for familiar objects (only left dorsolateral prefrontal and right premotor cortex). In order to increase power, we combined our data from Experiments 1 and 2 in the familiar object condition (Fig. 5D). This analysis revealed high decoding throughout visual cortex, extending again into posterior parietal (SPOC, IPS, SPL) cortex, all areas important in visuo-motor control (e.g., Gallivan et al. 2011a, 2011b; Rossit et al. 2013). In addition, reliable decoding was found around the postcentral sulcus bilaterally, extending into the postcentral gyri. Moreover, we also found reliable decoding in left premotor, motor, and dorsolateral prefrontal cortex and right premotor cortex, again all areas important in visuo-motor control (e.g., Gallivan et al. 2011B; Rossit et al. 2013) and furthermore in left posterior cingulate. Thus, our combined analysis implies that the representation of familiar object categories spans at least the visual, tactile, and motor components associated with interacting with real world objects and further suggests a network of brain areas that could play a role in transmitting visual information to early somatosensory brain regions.
Overlap Between Searchlight and ROI Analyses
For the searchlight analysis that pooled across Experiments (i.e., familiar object decoding), almost 1/5 of voxels (18%) within a group mask of the right PCG (defined as voxels shared across at least 40% of subjects; see also Meyer et al. 2011) were found to be significant (P < 0.01, uncorrected) via the searchlight analysis (13% in the left hemisphere; see Supplementary Fig. 6). This differs considerably from the number that overlap when each individual experiment is analyzed separately: only between 3% and 7% overlap for decoding of familiar objects in Experiments 1 and 2. Hence, it is clear that by pooling results across Experiments we obtain a great increase of power for the searchlight analysis. A similar story is present for S2 (overlap ranges between 1% and 9% for individual experiments, whereas it reaches 13% for the pooled across experiments analysis in right S2 (4% for left S2 in the pooled analysis).
We have shown in 2 independent experiments that early somatosensory cortex (PCG/S1) carries information that discriminates familiar visual object categories. This is true despite the fact that participants had no concurrent haptic interaction with the objects presented in the experiment nor did our stimuli, which were static photographs, depict any haptic interactions with the object. In Experiment 2, we demonstrated that this effect was not found for visual object categories in general, but only for familiar visual object categories. Thus, cross-modal connections from vision to early somatosensory cortex transmit content-specific information about familiar object categories based on visual appearance alone. We further show that right S2, a higher order somatosensory brain region, and premotor cortex also contain information about familiar object categories. Our whole-brain analyses point toward several areas in posterior parietal cortex (e.g., IPS, SPL) that might underlie the transmission of information from vision to early somatosensory cortex.
Cross-Modal Context Effects in Early Sensory Areas
The present study is in agreement with a set of studies that show the important role that cross-modal connections can play in influencing activity even in the earliest regions of supposedly unimodal sensory cortex, both in a content-specific (Meyer et al. 2010, 2011; Vetter et al. 2011) and modulatory fashion (e.g., Calvert et al. 1997; Mcintosh et al. 1998; Baler et al. 2006; Zangenehpour and Zatorre 2010). Our results are consistent with Meyer et al. (2011), but importantly show that cross-modal triggering of differential activity patterns in somatosensory cortex occurs even when participants simply view the different objects rather than view someone haptically exploring the objects. Indeed, it is conceivable that the different patterns of activation Meyer et al. (2011) observed in somatosensory cortex were driven by the differences in the depicted hand movements rather than by differences in the visual appearance of the objects themselves. In short, as those authors discuss, it is possible that what they observed is the activation of somatosensory components of mirror-neuron networks (e.g., Keysers et al. 2010). Nevertheless, both our study and that of Meyer et al. found reliable decoding of visual stimuli in primary somatosensory cortex and in S2. Moreover, in the present work, we have shown that the effect is only present for familiar but not unfamiliar object categories (all categories were familiar in Meyer et al. 2010).
In addition, the whole-brain searchlight analysis allows us uniquely to suggest several regions in parietal cortex that might underlie the visual context sensitivity observed in early somatosensory cortex. Specifically, we found reliable decoding of familiar visual objects in SPL (bilaterally), in the IPS (both posterior and anterior) and SPOC in our analysis that pooled across experiments. Importantly, these regions are all known to be crucial for the online control of actions (Milner and Goodale 2006), and in fact, parts of the SPL are thought to be a multimodal integration site for visual and proprioceptive information (e.g., Hsiao 2008). We also found reliable coding in the postcentral sulcus, extending into the PCG itself which corroborates our ROI analyses. For unfamiliar objects, in contrast, the network of brain areas that contain information discriminating the object categories is restricted to visual and posterior parietal regions (including parts of the SPL, IPS, and SPOC) not including any frontal brain regions (i.e., motor or premotor). The whole-brain results suggest the intriguing idea that visual images of familiar graspable objects may activate associated action/haptic coding routines in parietal cortex (see, e.g., Sakata et al. 1997), that subsequently feedback all the way to early somatosensory cortex. In addition, we also found reliable decoding in frontal cortices as well (premotor and dorsolateral prefrontal cortex, both areas that have been implicated in visuo-motor processing, e.g., Gallivan et al. 2011B). Although speculative, these ideas could be further tested by training a classifier on data from participants performing real actions with real objects and testing it on data acquired from visual presentation of the same objects alone. This would give an indication of whether or not the same representational space is being activated in each case. For unfamiliar objects, on the other hand, visual images trigger differential activity in posterior parietal, visual, and temporal brain areas while not leading to feedback that discriminates object categories in the early somatosensory cortices.
Moreover, our results fit well with an existing theory about the neural representation of perceptual experience (Damasio 1989; Meyer and Damasio 2009) that implies that early sensory areas (such as S1, V1, etc.) simultaneously represent the perceptual content of experience and that these sensory representations can be reactivated during recall or recognition. The present study provides converging evidence for the idea that activity in supposedly modality-specific early sensory areas can be shaped in a content-specific manner by relevant contextual information from other sensory modalities (e.g., Meyer and Damasio 2009; Meyer et al. 2010, 2011; Vetter et al. 2011).
Finger-Pad Representations Within the PCG
We demonstrated that reliable decoding of familiar visual objects was possible in the PCG even when limiting our analysis to those voxels that are highly responsive to tactile stimulation of the finger-pads. In fact, these decoding analyses worked best when we selected voxels both from contra- and ipsilateral stimulation (Experiment 2), implying that higher order areas within the PCG (area 2) may underlie the present decoding effects. In addition, in Experiment 1, these analyses produced more robust classification in the right PCG which was subject to ipsilateral finger-pad stimulation in the localizer (i.e., the left-hand finger-pads) again suggesting the involvement of area 2 within the PCG. This evidence is further supported by our searchlight decoding analyses that reveal a cluster in the right postcentral sulcus that extends into posterior PCG, that is, area 2. In any case, these analyses imply that there is fine-grained information within the finger-specific representations of the somatosensory cortices that permits decoding of visual images of familiar object categories. It will be interesting in future research to attempt to localize such effects specifically to tactile or proprioceptive representations within the PCG and specifically to finger/hand representations versus those of another effector (e.g., leg).
The Role of Prior Visuo-Haptic Experience
In Experiment 2 we demonstrated reliable decoding of familiar visual objects in bilateral PCG, right S2, bilateral premotor cortex, and of course in V1. Unfamiliar objects, on the other hand, were only reliably decoded in V1 (and on one subsampling analysis in right S2). In addition, within the PCG (when limited to finger-pad-sensitive voxels from both hemispheres) and premotor cortex, we found significantly stronger decoding for familiar than for unfamiliar visual objects. The whole-brain decoding analysis revealed reliable decoding in similar occipito-temporal and posterior parietal areas (IPS, SPL, and SPOC) for both familiar and unfamiliar object categories. However, we did not find any evidence of decoding for unfamiliar objects in more anterior parietal regions in any analysis carried out, that is, the postcentral sulcus or the PCG whereas for familiar objects, both the ROI analyses and the searchlight results (when pooled across experiments and in Experiment 1 by itself) provide evidence of reliable decoding within the PCG. We initially used familiar visual objects as we expected that cross-modal activation of object-specific activity patterns within early somatosensory cortex would be most likely to be detected with such objects since strong links already exist between different sensory modalities from joint multisensory experience for such object categories (see Meyer and Damasio 2009). Unfamiliar visual objects, however, just like familiar objects also afford certain grasping actions and tactile sensations directly from the visual appearance alone: hence it might still be useful and advantageous to activate early somatosensory representations even in this case, where such novel objects would be mapped onto our previous visuo-haptic experience and hence could still potentially trigger differential activity even in early somatosensory cortex. It may just be much harder to detect such effects in this case. This is especially true if we consider predictive coding as the basic computational strategy of the brain (see, e.g., Clark 2013; Muckli et al. 2013). Although we did not find reliable decoding of novel object categories in the present experiment, we did not consistently find greater decoding for familiar than unfamiliar object categories across our analysis: the 2 compelling exceptions to this were in premotor cortex and in the PCG (when selecting only voxels responsive to finger-pad stimulation of both hands). One prediction of this hypothesis would be the existence of a gradient of decoding accuracy as a function of the amount of visuo-haptic experience a participant has with different sets of object categories. In addition, a further important test of the role of visual haptic experience in causing the present effects would be to explicitly train participants with a set of novel objects (visuo-haptically) and then examine at what stage of learning such objects could be decoded in the PCG, from visual presentation alone.
Additional analyses demonstrated that within-category decoding effects were absent for familiar objects in the PCG in both experiments. This is what we would expect if the between-category decoding results are due to motor/grasping affordances since these are constant within a category but differ across categories. If motor/grasping affordances are indeed responsible for the present effects then this would predict that decoding accuracy in the PCG should be higher across a set of familiar objects that have more distinct motor affordances than for a set that have more similar affordances. For unfamiliar objects, on the other hand, for 2 of 3 of the object categories, no within-category decoding was detected in the PCG, as expected. However, somewhat puzzlingly, reliable within-category decoding was found for the “spikies” object category within the PCG.
Specifically, in the context of presentation of the familiar object categories that are associated with hand-grasping actions (wine glasses, mobile phones, and apples), presentation of images from the “spikies” category might be especially attention grabbing (indeed these objects do lead to qualitatively high levels of BOLD signal amplitude, see Supplementary Fig. 3). This could reflect the potential for injury that might result from performing a hand grasp to such objects. No other object category presented in the current experiments would appear to possess such a quality. Existing research has shown, furthermore, that simply viewing others in painful encounters (e.g., grasping a broken glass), leads to reliable activity within the somatosensory cortices, even within the PCG or S1 (see Keysers et al. 2010; Morrison et al. 2013). We speculate that, in the present case, this particular object category might have triggered increased sensitivity in the PCG due to the potential for pain that in turn leads to reliable within-category discrimination. Although this result might seem odd given the absence of any between-category decoding for unfamiliar objects it is possible to achieve such a result given the mathematics of machine learning algorithms.
Decoding in Premotor Regions
In Experiment 2, we found reliable decoding of familiar objects in premotor cortex. This is in contrast to Experiment 1 where no reliable decoding of familiar objects was observed in this region. Recall that Experiment 2 was explicitly designed to remove any requirements for a motor response during the fMRI scanning whereas Experiment 1 required such a response (button press). Thus, it seems reasonable that any feedback from vision to pre-motor cortex was overwritten by the need for planning a motor action. Indeed, Gallivan et al. (2013) have shown that both left and right premotor areas (both ventral and dorsal) code information during the planning phase of an action about both left and right hand actions.
The Role of Somatosensory Imagery
Visual imagery can activate the earliest sensory areas (i.e., V1) if sufficient detail is required in the mental image (see Kosslyn et al. 2000). This however requires a task optimized for generating a detailed visual image (e.g., visualizing line drawings of objects at different sizes: Kosslyn et al. 2000). Moreover, Stokes et al. (2009), using MVPA, did not detect reliable decoding in early visual areas when subjects were asked to explicitly imagine the letters X or O. In the present experiments, we used orthogonal tasks requiring participants to fixate and either to detect color changes of the fixation marker (Experiment 1) or to count such changes and report the total number at the end of a run (Experiment 2). There was hence no task (volitional) requirement to form explicit mental images regarding the somatosensory (or motor) associations that the visual object images could potentially bring to mind. Although it is conceivable that our results might be explained to some extent by somatosensory imagery mechanisms, it seems unlikely that presentation of visual object images alone, with no task instruction to form detailed somatosensory images of such objects, would be expected to generate discriminable patterns of activity in the earliest regions of somatosensory cortex (i.e., S1). If imagery did occur during the present task then it must have been automatically generated by pre-existing links between vision and somatosensation for our particular object categories (see also Zangenehpour and Zatorre 2010). Thus, although we cannot rule out imagery as a potential mechanism we argue that it is unlikely that “explicit” mental imagery underlies the present effects. In future, a rapid event-related design and short stimulus presentation durations could be used in order to further explore this question.
We have shown that familiar visual object categories can be discriminated in early regions of somatosensory cortex (S1 and S2), in the absence of concurrent somatosensory stimulation and in the absence of any object-hand interactions in the visual stimuli. Thus, cross-modal connections from vision to the primary somatosensory cortex transmit content-specific information about familiar object categories. We have further highlighted several areas in the parietal lobe (IPS, SPOC, SPL) that may underlie the transmission of information from visual to early somatosensory cortex. Our results fit well with theories on the neural representation of perceptual experience that suggest that early sensory areas jointly represent the perceptual contents of experience (e.g., Damasio 1989; Meyer and Damasio 2009).
F.W.S. was supported by a salary award from the NSERC-CREATE program. The research was supported by a grant to M.A.G. by the Canadian Institutes of Health Research. Funding to pay the Open Access publication charges for this article was provided by the University of East Anglia.
We thank Adam Mclean for help with I/O card setup and operation of the somatosensory stimulation device, and Stephanie Rossit for discussions about the functions of parietal cortex and insightful comments on the manuscript. Conflict of Interest: None declared.