Representational Similarity Mapping of Distributional Semantics in Left Inferior Frontal, Middle Temporal, and Motor Cortex

Abstract

Language comprehension engages a distributed network of frontotemporal, parietal, and sensorimotor regions, but it is still unclear how meaning of words and their semantic relationships are represented and processed within these regions and to which degrees lexico-semantic representations differ between regions and semantic types. We used fMRI and representational similarity analysis to relate word-elicited multivoxel patterns to semantic similarity between action and object words. In left inferior frontal (BA 44-45-47), left posterior middle temporal and left precentral cortex, the similarity of brain response patterns reflected semantic similarity among action-related verbs, as well as across lexical classes-between action verbs and tool-related nouns and, to a degree, between action verbs and food nouns, but not between action verbs and animal nouns. Instead, posterior inferior temporal cortex exhibited a reverse response pattern, which reflected the semantic similarity among object-related nouns, but not action-related words. These results show that semantic similarity is encoded by a range of cortical areas, including multimodal association (e.g., anterior inferior frontal, posterior middle temporal) and modality-preferential (premotor) cortex and that the representational geometries in these regions are partly dependent on semantic type, with semantic similarity among action-related words crossing lexical-semantic category boundaries.

corpus co-occurrence, fMRI, language comprehension, representational similarity analysis, semantic word category

Introduction

Learning a language requires extracting the meanings of words when they are used in different contexts to refer to objects or to speak about actions. Indeed, much previous neuropsychological and neuroimaging work has focused on the neurobiological implementation of referential word meaning (Saccuman et al. 2006; Barrós-Loscertales et al. 2012; Marconi et al. 2013), that is, the link between word forms and their related objects and actions (de Saussure 1916). However, during language acquisition, only a limited part of the human vocabulary is learnt directly in real life contexts (e.g., the word “apple” in the context of an apple), while a much larger percentage of vocabulary items is learnt in language contexts where no referent objects or related actions are available (the word “apple” in the context of “sweet”, “round”, or “grasp”). Therefore, the human brain must possess a mechanism for mapping the semantic relationships between words as they emerge from texts encountered during language learning and use (Kintsch 1974; Tomasello 2005). Hence, given that an initial set of words, the “grounding kernel”, is being related directly to, or semantically “grounded” in, action and perception information (Cangelosi and Harnad 2001; Harnad 2012; Pulvermüller 2012), contextual relationships between words in texts, dialogues, and larger communication contexts provide a further fundamental dimension for semantic learning and, thus, semantic knowledge and language comprehension (Wittgenstein 1953). For such context-based induction of lexical meaning to be translated into functional behavior, a neurocognitive mechanism must ensure that the words found in texts and discourse are accommodated adaptively within the existent semantic representation of the already learnt and grounded lexicon.

Computational language theory has proposed that the semantic information about words and their relationships can be computed from the contexts in which they appear. Accordingly, the semantic similarity between words is directly reflected by the similarity of the contexts in which they are used (Landauer and Dumais 1997). Similar word meanings are thought to be cognitively represented within a common “latent semantic space”, which maps at an abstract level the distributional properties of words, that is, how likely a given word meaning is used in combination, or co-occurs, with another one (latent semantic analysis, LSA: Landauer and Dumais 1997). However, how semantic similarity is integrated at the neural computational level and how it is represented in the human brain has been scarcely investigated, although a number of recent studies focused on the neurobiological correlates of semantic priming and activity differences between words that are close versus distant in meaning (Kiefer et al. 2008; Mahon and Caramazza 2010; Ulrich et al. 2013; Carlson et al. 2014; Grisoni et al. 2016) and on word combinations in texts (Graves et al. 2010; Price et al. 2014). We here use LSA to determine semantic similarity between words and ask which brain regions reflect, in their word category-related multivoxel activation patterns, the LSA similarity structure of words of different types.

Pre-existing neuropsychological and neuroimaging work suggests several brain regions for semantic similarity processing. If true semantic processing is restricted to a specific site, a so-called “semantic hub”, this hub should be the unique locus of semantic similarity processing. Diverging models have proposed semantic hubs in different multimodal areas, including temporal pole, posterior middle temporal gyrus (MTG) and/or inferior temporal gyrus (ITG), inferior parietal cortex (BA 39; 40) and/or left inferior frontal gyrus (LIFG BA 45-47) (Broca 1861; Wernicke 1874; Lichtheim 1885; Geschwind 1965, 1967; Binder et al. 1997, 2009; Poldrack et al. 1999; Bookheimer 2002; Catani et al. 2005; Turken and Dronkers 2011; for reviews, see Patterson et al. 2007; Binder and Desai 2011; Poeppel et al. 2012; Pulvermüller 2013). If distributional semantic learning of word meaning from contexts of language use was the only form of semantic learning and if the mechanisms for it were housed in one or more of these multimodal semantic hub areas, semantic similarity processing in the brain could be predicted in these areas exclusively.

Other work has indicated that modality-specific semantic processing is not restricted to multimodal areas but extends into modality-preferential cortical systems (Pulvermüller and Fadiga 2010; Binder and Desai 2011; Kiefer and Pulvermüller 2012; Kemmerer 2015). Interestingly, the relevance of modality-preferential cortical areas, such as the motor and premotor cortex or areas of the ventral visual object processing stream, has become particularly evident for specific semantic word categories. Among the most investigated modality-preferential semantic systems in recent years is the sensorimotor system, and in particular the precentral cortex, where it has been found that the processing of action-related words is reflected in specific neural signatures, and even subtle semantic differences between action-related verbs or nouns are manifest in different activation signatures (Hauk et al. 2004; Carota et al. 2012; Kemmerer 2015; Grisoni et al. 2016). If the motor system - and possibly other modality-preferential cortical areas - reflect semantic similarity among words, a pertinent explanatory model would posit that action- and object-related aspects of lexical meaning are grounded in sensory motor systems. Contextual learning could then lead to a hand-over of these action- and perception-grounded semantic features to the rest of the vocabulary (Harnad 1990; 2012; Kiefer and Pulvermüller 2012). Multimodal semantic “hubs” are necessary components of these grounded models too, because these areas play an important role as connection hubs and convergence zones joining together semantic information from different modalities (Damasio 1989; Damasio 1992; Damasio et al. 1996; Pulvermüller 1999; Binder and Desai 2011; Pulvermüller 2013; Tomasello et al. 2016). Apart from the sensorimotor system, for which category-specific semantic processing of action-related words has been claimed, the posterior middle temporal cortex—one of the candidate semantic hubs—has also been seen as a category-preferential area specializing in higher visual and auditory perception processes and related semantic information about visual and acoustic aspects of word-related referents.

The two competing hypotheses to be tested here are therefore 1) that semantic similarity is reflected entirely in multimodal semantic hubs consistent with earlier proposals (hub-only model) and 2) that semantic similarity is manifest in both multimodal and modality-preferential areas, including the motor system (grounded model). Further predictions are that semantic similarity representation is uniform across the vocabulary (hub-only model) or, alternatively, to some degree dependent on the type of semantic information represented (action-related semantics in frontal and sensorimotor cortices, perceptually related semantics in temporal cortex and ventral visual areas; grounded model).

Previous univariate imaging work was highly valuable in revealing the involvement of key regions of the language system in semantic processing. However, their methodological focus on spatially averaged activations to sets of words can address the brain correlates of substantial contrasts—for example, whether a word is typically used to speak about an action or an object (Martin 2007; Binder et al. 2009; Pulvermüller 2013), but not to what degree words such as “grasp,” “pick,” “take”, and “fetch” are neurocognitively represented as being gradually more or less similar in meaning. A recent methodological advance now makes it possible to scrutinize the neurobiological basis of semantic similarity. Cutting-edge semantic studies used multivariate pattern analysis became available to investigate the representational patterns of object processing in the ventral visual stream along occipital and inferior temporal areas up to the temporal pole (Carlson et al. 2014). Work using both words and object pictures suggested different cortical regions as the possible “seat” of semantic similarity, including posterior middle and inferior temporal cortex, intraparietal sulcus, perirhinal cortex, posterior and anterior cingulate gyrus and medial neocortex (Kriegeskorte et al. 2008b; Devereux et al. 2013; Fairhall and Caramazza 2013; Clarke and Tyler 2014). The variability of “semantic similarity regions” reported across these pioneering studies appears difficult to explain, although a tentative explanation could come from different semantic categories, whose similarity structure is mapped by different brain regions, and from the different tasks applied (Pulvermüller 2013). As multimodal areas were found to reflect semantic similarity, these results seem to sit well with the idea of (several) semantic hubs. One putative problem of some of these earlier studies is that activation similarities were investigated in cognitive semantic tasks that elicit different types of semantic processing, which are not necessarily relevant to semantic meaning assess per se. For example, Fairhall and Caramazza used category typicality judgments on words and pictures from different semantic categories, so that evoked multivoxel MRI pattern activation may indeed reflect aspects of the processing of the meaning of the stimuli, but may likely have also been influenced by the semantic distance of a given item to the category prototype of a given category. Therefore (and for further reasons outlined in discussion below), any activation differences between meaningful stimuli observed in this and similar tasks may reflect the true semantic differences between the stimuli presented, but may equally well relate to peripheral task factors such as the degree of matching between a given item and its category prototype. In order to disentangle the brain correlates of word meaning from those induced by a semantic task, it seems advantageous to present meaningful items in a context where the focus is on understanding them, thus discouraging additional semantic processes that could contaminate symbol-evoked semantic-conceptual activation. Therefore, we here used a passive reading task where subjects were instructed to silently read and understand words, while a non-semantic task had to be performed only occasionally on otherwise irrelevant stimuli to ascertain the subjects’ alertness.

Here we investigated the link between the computational processing and the cortical representation of lexical-semantic similarity, asking how the similarity of word usages in linguistic contexts relate to the similarity of activation patterns in specific areas. A related question is whether such similarity reflection is affected by the specific lexical category or by semantic properties of the words and, if so, in what cortical areas. As mentioned, semantic hub-only models predict that semantic similarity is manifest across categories in one or several multimodal areas, whereas semantic grounding models predict contributions of both multimodal convergence zones and modality-preferential areas to semantic similarity processing—with the latter areas being particularly sensitive to semantic type. In the current study, we included equal numbers of words typically used to speak about actions and words referring to objects, with the hypothesis that semantic similarity across the latter might be mapped in ventral visual or anterior inferior temporal cortex, whereas that of the former semantic group might be indexed primarily in frontocentral regions. Semantic similarity structure was investigated in a range of previously proposed semantic hubs, including left inferior frontal cortex (BA 45-47), temporal areas an angular gyrus. In addition, we looked at areas previously reported to process category-specific semantic information, including the posterior middle temporal cortex and the premotor and motor cortex.

fMRI responses were recorded while subjects silently read words referring to arm-, face-, and leg-related actions and animal-, food-, and tool-related objects, and occasionally pressed a button when they detected a misspelled word. Motor responses were minimized to avoid response-related activation of the precentral gyrus, which otherwise might not reveal fine-grained language-related activations (Hauk et al. 2004; Schomers and Pulvermüller 2016). We adopted multi-voxel pattern analysis (MVPA) implemented within the framework of representational similarity analysis (RSA) (Kriegeskorte et al. 2008a; Nili et al. 2014) to detect the information patterns of neural similarity between hemodynamic responses associated with single word meanings within relevant brain areas of the semantic network.

Materials and Methods

Participants

About 23 healthy volunteers participated in the study. All participants were right handed, monolingual English native speakers. Their mean age was 29 years (SE = 2.8). Participant had no history of neurological or psychiatric disorders. They had normal or corrected-to-normal vision. All participants gave their informed consent to take part in the study and were remunerated for their time. Ethical approval was obtained from the Cambridge Psychology Research Ethics Committee.

Experimental Procedure

Stimuli

96 word stimuli, 16 from each individual category of leg-, arm-, face-related actions and tool-, animal-, food-related objects, were included in the study. Stimulus word groups were matched for word length (counted in number of letters), letter bigram and trigram frequency, logarithmic word frequency, number of orthographic neighbors, and standardized lexical frequency (see Behavioral Results). Relevant values were obtained from the CELEX database (Baayen et al. 1993) and the WordSmyth Web site (www.wordsmyth.net/). About 21% of the action words were lexically unambiguous verbs and the lexically ambiguous ones that could be used as members of both lexical categories were in the average 14 times more frequently used as verbs than as nouns (according to the CELEX database). About 58% of the object words were lexically unambiguous nouns and the lexically ambiguous ones that could be used as members of both lexical categories were in the average 6 times more frequently used as nouns than as verbs. The word categories were selected based on differences in their rated semantic relationship to objects, actions, bodily sensations, emotional features, as well as their concreteness, and imageability (for discussion, see Pulvermüller 1999) (Table 1).

Table 1

Psycholinguistic properties and semantic ratings are shown for each word category

	Arm verbs	Leg verbs	Face verbs	Animal nouns	Food nouns	Tool nouns	Main effect of
word type (F)
Length	4.73 (0.16)	4.6 (0.15)	4.53 (0.12)	4.73 (0.15)	4.60 (0.12)	4.53 (0.15)	0.533 (P < 0.75)
Bigram freq.	31248.6 (3138.6)	32472.14 (4035.5)	29029.4 (11551.4)	31538.82 (2985.33)	32699.44 (3939.16)	30550.2 (3422.14)	0.155 (P < 0.98)
Trigram freq.	2475.5 (283.6)	2673.85 (367.8)	2535 (301.52)	2601.56 (1598.1)	2825.27 (1199.01)	2209.2 (1200.5)	0.401 (P < 0.85)
No. of neighbors	5.2 (0.9)	5.06 (0.6)	5.437 (0.84)	5.2 (0.76)	4.8 (0.78)	5.25(0.84)	0.071 (P < 0.99)
No. of meanings	1.06 (0.06)	1.125 (0.34)	1.125 (0.08)	1.125 (0.08)	1.11 (0.06)	1.187 (0.10)	0.333 (P < 0.89)
Log. word freq.	0.66 (0.11)	0.58 (0.08)	0.57 (0.12)	0.61 (0.15)	0.60 (0.12)	0.71 (0.15)	0.289 (P < 0.91)
Imageability	4.47 (0.17)	4.43 (0.29)	3.97 (0.26)	6.32 (0.09)	5.48 (0.27)	5.35 (0.35)	11.85 (P < .0001)
Concreteness	4.14 (0.16)	3.59 (0.19)	3.62 (0.19)	6.60 (0.08)	6.21 (0.19)	5.73 (0.21)	60.67 (P < 0.0001)
Action-relatedness	4.83 (0.23)	4.88 (0.26)	5.31 (0.25)	1.60 (0.10)	2.02 (0.29)	3.22 (0.41)	34.18 (P < 0.0001)
Face-relatedness	1.56 (0.01)	1.39 (0.09)	5.75 (0.23)	1.20 (0.07)	2.06 (0.26)	1.27 (0.10)	122.02 (P < 0.0001)
Arm-relatedness	5.68 (0.13)	1.81 (0.14)	1.33 (0.09)	1.11 (0.05)	1.37 (0.12)	2.85 (0.36)	94.88 (P < 0.0001)
Body sensation	3.74 (0.29)	3.49 (0.23)	3.92 (0.31)	1.16 (0.07)	1.35 (0.11)	1.40 (0.16)	39.65 (P < 0.0001)
Valence	3.45 (0.27)	4.05 (0.21)	3.66 (0.31)	3.52 (0.09)	4.08 (0.13)	3.85 (0.14)	1.67 (0.14)
Arousal	3.25 (0.27)	3.01 (0.21)	2.60 (0.28)	1.30 (0.21)	1.44 (0.16)	1.65 (0.19)	14.45 (P < 0.0001)

Means and standard errors (in brackets) are reported for each word category, along with results of an ANOVA comparing ratings between word groups.

Strings of meaningless hash marks matched in length to the stimulus words were used as baseline stimuli during 120 trials. These meaningless strings acted as a low-level visual baseline sharing visual properties of the written words, without involving phonetic or semantic processing. Null events consisting of a fixation cross displayed at the center of the screen were presented during additional 60 trials. About 60 trials consisting of misspelled words to be detected by the participants throughout the experimental task were presented. These “typo” trials did not include words from any of the semantic categories from which the 96 target words were taken—so as to avoid a bias towards one of these categories—and were discarded from the analysis. Statistical comparisons were carried out between brain activation patterns elicited by matched word categories.

After the fMRI experiment, participants completed an unannounced word recognition test containing both novel distractor and experimental words, to further confirm that they had attended to the word reading task. They performed above chance (average hit rate: 80% [STD: 8.3%]), indicating their attention to the words and compliance with the task.

Experimental Design

We adopted a rapid, periodic single trial, event-related paradigm. Stimulus duration was 100 ms. A fixation cross was presented at the center of the screen between two consecutive stimuli. The stimulus onset asynchrony was jittered ~3.5–4 s. This design yielded overlapping, yet detectable hemodynamic responses (Kriegeskorte et al. 2008a; Nili et al. 2014). The 96 stimulus words were presented in a different pseudo-random order in each of the 6 runs. Each stimulus occurred once per run (6 repetitions of each word in total). Stimuli were visually presented by means of E-Prime software (Psychology software Tools, Inc., Sharpsburg, USA, 2001) through a back-projection screen positioned in front of the scanner and viewed on a mirror placed on the head coil.

Task

Participants were engaged in passive word reading with occasional typo-detection (17% of trials). They were given the instruction to attend to all the stimuli, to silently read the words and to understand their meaning. In addition, they were instructed to occasionally press a button with their left hand in case a misspelt word appeared at the center of the screen.

Imaging Methods

Subjects were scanned in a Siemens 3 T Tim Trio using a head coil. Echo-planar imaging (EPI) sequence parameters were TR = 2000 ms, TE = 30 ms, and flip angle = 78°. The functional images consisted of 32 slices covering the whole brain (slice thickness 3 mm, in-plane resolution 3 × 3 mm, inter-slice distance 0.75 mm).

Data Analysis

Imaging data were analyzed using SPM8 software (Wellcome Department of Imaging Neuroscience, London, UK). Images were corrected for slice timing and re-aligned to the first image using sinc interpolation. The EPI images were co-registered to the structural T1 images using a mutual coregistration procedure (Maes et al. 1997). The structural MRI was normalized to the 152-subject T1 template of the Montreal Neurological Institute (MNI). The resulting transformation parameters were applied to the co-registered EPI images. During the spatial normalization, images were resampled with a spatial resolution of 2 mm × 2 mm × 2 mm.

Univariate Analysis

For the univariate analysis, normalised images were spatially smoothed by convolution of a 10-mm full-width half-maximum Gaussian kernel and globally normalized. Single subject statistical comparisons were computed by using the General Linear Model (Friston et al. 1994). Low-frequency noise was removed by applying a high-pass filter of 128 s. Group data were analyzed using random effects analysis. To assess the regions that were responsive to general lexical processing, we contrasted brain activation to all words against the low-level visual baseline condition (hash marks). Whole brain analysis results are displayed visually at a false discovery rate (FDR)-corrected error probability threshold of P = 0.05. Stereotaxic coordinates for voxels with maximal t-values within activation clusters are reported in the MNI standard space.

Representational Similarity Analysis

We adopted a multivariate approach to data analysis, following the framework of RSA (Kriegeskorte et al. 2008a; Nili et al. 2014).

Definition of Regions of Interest

For RSA, we focused on twelve hypothesis-driven regions of interest (ROIs), which have been shown to play a key role in semantic processing (see Introduction). These included 1) left inferior frontal gyrus pars opercularis (BA 44), 2) pars triangularis (BA 45), 3) pars orbitalis (BA47), 4) left precentral gyrus, 5) left temporal pole, 6) anterior superior, 7) anterior middle and 8) anterior inferior temporal gyrus, 9) left posterior superior, 10) posterior middle and 11) posterior inferior temporal gyrus, 12) left angular gyrus (Fig. 1b). ROIs were anatomically defined using the standard Wake Forest University (WFU) Pickatlas toolbox, which generates ROI masks in standard MNI space based on the automated anatomical labeling parcellation. In order to carry out multivariate analysis within individual subject native space, all ROI masks were subsequently transformed to subject native space by inverting the spatial normalization applied during general linear modeling (GLM) analysis.

Figure 1.

(a) Representational dissimilarity matrix (RDM). Example of RDM computed by combining the fMRI responses to words in LIFG BA 45. The 6×6 RDM corresponds to the number of experimental conditions and their corresponding semantic categories. Each cell in the matrix reflects the correlation distance (1−r) between the fMRI patterns associated with each pair of conditions. The RDM is symmetrical along a diagonal expressing maximal similarity for identical condition pairs (e.g., arm words are identical to arm words, leg words identical to leg words, etc.). The off-diagonal cells in both the upper and the lower triangle of the 6×6 RDM contain the similarity values for unique combinations of each condition with the other. The 6×6 RDM comprises a total of 6*5/2 = 15 unique combinations of conditions. (b) The 12 anatomically defined ROIs: 1 = LIFG BA44; 2 = LIFG BA45; 3 = LIFG BA47; 4 = left precentral gyrus; 5 = left temporal pole; 6–9 = left anterior and posterior STG; 7–10 = left anterior and posterior MTG; 8–11 = left anterior and posterior ITG; 12 = angular gyrus.

Estimation of fMRI Patterns

For RSA, the analysis was carried out in subject native space, using not normalized, unsmoothed functional data, which were co-registered with the MPRAGE structural image of each subject. BOLD responses were estimated by univariate GLM. The response-amplitude (β) estimate maps associated with each condition were converted into t maps and used to compute representational dissimilarity matrices (RDMs) (Misaki et al. 2010). In the RDMs, the dissimilarity between the response patterns elicited by condition pairs in a given region is expressed by correlation distances (1−r, Pearson linear correlation) (Kriegeskorte et al. 2008a). The dissimilarity value for the response patterns associated with each pairs of conditions is represented in each cell of a RDM (Fig. 1a).

Comparing fMRI Patterns and Models

RDMs were computed for each participant separately for each of the abovementioned ROIs. In each subject and for each ROI, the RDMs were calculated for brain activity patterns elicited by semantic conditions including 16 words. Subsequently, LSA semantic similarity and representational similarity were related for each pair of words/ word groups and Spearman's rank correlations were calculated. To assess the relatedness between brain activity and model RDMs, statistical inference was applied on the single subject correlations using a one-sided signed-rank test across subjects, testing whether the resulting correlation coefficients were significantly greater than zero (Wilcoxon 1945). To correct for multiple testing across several brain regions, the FDR procedure was applied; the expected FDR was less than 5% (Benjamini and Hochberg 1995).

Computational Linguistic Model Specification

In order to assess whether the meanings of the experimental stimulus words were semantically similar on the basis of an objective measure, we constructed a corpus-based, computational linguistic model of semantic similarity using LSA (Landauer and Dumais 1997). LSA is particularly adapted to design our theoretical model, because it assumes that two words are semantically related, that is, have similar meanings, if they occur in similar discourse contexts. This assumption implies that words sharing similar meanings are used in similar discourse contexts more than words that are dissimilar in meaning.

In vector space models of word meaning, such as LSA, words are represented by vectors that characterize the contexts in which those words occur. Linear algebra is then used to calculate the cosine of the angle between two vectors, to quantify the similarity or distance in meaning between words. LSA was applied to the British National Corpus (BNC), which contains 4096 texts with samples of written and spoken English from a wide range of sources and a wide variety of genres for a total of 100 million words. The words of each text were lemmatized by grouping together different inflected forms of a same word and used to construct a matrix in which each textual fragment containing a given word was represented as a row, each lemma was represented as a column, and each cell expressed the frequency with which each word occurs with all the other words in the corpus. After transformation of these word frequency values in their logarithm, singular value decomposition (SVD) was applied to this matrix. SVD mapped the word vector space to a lower dimensional vector space, effectively grouping similar contexts together. Semantic similarity between words for each condition was then measured as the cosine between two word vectors: the smallest the cosine, the greatest the similarity between word stimuli pairs. These values were averaged for each word category and expressed as a dissimilarity matrix (Fig. 2).

Figure 2.

Model of latent semantic similarity. (a) Matrix containing the latent semantic distances between each pair of conditions. The LSA model coded for combinatorial relationships of semantic similarity between the words of each category of action (leg-, arm-, face-related action verbs) and objects (animal-, food-, and tool-related object nouns). The model captures similarity between all action words and, in addition, between action words and tool words. (b) Arrangement by multidimensional scaling of the correlations among the latent semantic values specific to each experimental condition. The graph displays close relationships among action words and also between action words and tool words, while food words are closer to arm words than the remaining action words, and animal words are not closely linked to any word type. (c) Arrangement by multidimensional scaling (MDS) of the correlations among the latent semantic values specific to each word. This graph shows a larger distributional cluster of action words (from dark to light blue), with which tool-related words (in red) are intermixed. Food and animal words (in green and yellow, respectively) form two distinct groups. (d) Shepard plot displaying, for any two conditions, their distances in the 2-dimensional (2D) MDS arrangement versus their dissimilarity (i.e., their distance in the original high-dimensional space). Therefore, if the “mapping” to a 2D space does not cause a large distortion, we expect the two to be highly correlated. The Shepard plot is a way to visualize the quality of the low-dimensional projection.

Binary Models Specification

In order to further investigate within- and between-category representations in the fMRI patterns in each ROI, we compared them with four model RDMs testing specific hypotheses about the representational geometries of these patterns for the semantic word categories (see Supplementary Materials Fig. 8).

A first categorical “action words category” model tested for within-category similarity among all action words, which were also assumed to be distinct from object words (see Supplementary Materials Fig. 8a). Reversely, an “object word category” model tested for within-category similarity among all object word categories, which were also assumed to differ from action words (see Supplementary Materials Fig. 8b). A mixed “action-tools word category” model coded for similarity among all action words, tool-related and action-related words (see Supplementary Materials Fig. 8c). A final “extended action model” tested for within-category similarity among action verbs and action-related food and tool words. In this final model, we included all action verbs, plus food and tool nouns that, according to semantic ratings, were judged as carrying action-related semantic information (see Supplementary Materials Fig. 8d).

The “action”, “action-tools”, and “extended action” categorical models were expected to correlate with category-preferential semantic circuits in motor cortex, involved in the processing of action word types and also of their semantic associations with manipulable objects like tools, affording hand (lute), mouth (flute), and leg (stilt) actions.

Control Model Specification

In addition to the LSA measure of semantic similarity, we evaluated the similarity of our experimental words based on a semantic dimension long established in neuropsychological and neuroimaging research, that is, imageability (Bird et al. 2000). Imageability of written words triggers activation of higher visual object processing regions in posterior inferior temporal cortex (Hauk et al. 2008). As specified in methods above, imageability values were obtained from semantic ratings. The correlation distances among these values were calculated, which were then entered in a separate “imageability” model RDM (see Supplementary materials Fig. 9). We used this model to test for potential dissociation between the brain regions sensitive to latent semantic similarity (which reflected the similarity structure among action-related word categories) and imageability (which here captured within-category similarity for both action and object words).

Results

Univariate Results

To assess the regions which were generally activated during the processing of our stimulus words, a univariate analysis was run as a sanity check of the data, by comparing all visually presented words against the low-level visual baseline. This subtractive contrast revealed prominent clusters of activation in bilateral middle cingulate cortex, left inferior and anterior temporal gyrus, pars orbitalis of left inferior frontal gyrus (BA 47), pars opercularis of the LIFG (BA 44), pars triangularis of the LIFG (45) bilaterally, prefrontal cortex and inferior and lateral precentral gyrus (Table 2 and Fig. 3). There were however no effects in anterior or posterior inferior, middle or superior temporal cortex or in inferior parietal regions, which would have been expected based on the earlier literature on lexical-semantic processing (Binder et al. 2009; Pulvermüller and Fadiga 2010).

Figure 3.

Univariate results. Lateral and ventral views of the cortical activations for general lexical-semantic processing (all experimental words contrasted against visual baseline). Images are significant at P < 0.001, cluster level corrected at FDR 0.05.

Table 2

Table of coordinates and statistics for activation peaks produced by the contrast of all words against baseline (FDR 0.05)

Regions	Cluster extent	Voxel Z	Coordinates
			x	y	z
(A) All words—Hash marks
R middle cingulate	12 702	5.1	6	20	44
L middle cingulate		5.08	−6	18	48
R inferior frontal gyrus (BA 47)		4.98	28	28	−8
L inferior temporal (BA20)	1079	4.48	−40	−8	−28
L fusiform		4.28	−40	−32	−20
L temporal pole		4.01	−34	7	−30
R cerebellum		4.11	22	−70	−44
R cerebellum	1124	3.9	40	−58	−32
R cerebellum		3.9	38	−48	−32
L inferior frontal gyrus (BA 44)	340	3.52	−52	8	18
L precentral gyrus (BA 4)		3.41	−52	0	28
L precentral gyrus (BA 4)		3.4	−32	−10	50

We note that the baseline task involved strings of hash tags, and thus differences in activity between word conditions and baseline may index a range of different processes, including letter recognition, word form processing, activation of phonological representations, semantic understanding and even processing of aspects of the syntactic information associated with the words. In order to obtain specific information about semantics, additional work is necessary. Therefore, all further analyses used RSA to uncover genuine semantic similarity effects, by examining the word-elicited neural dissimilarity patterns within independently selected frontocentral and temporal ROIs and assessing their relatedness to computational linguistic measures of word meaning similarity.

Multivariate Results

The LSA Model Reflects Degrees of Categorical Structure

We first inspected the correlational structure of the LSA model for the stimulus words (Fig. 2a and b). LSA revealed that semantic similarity was high within each semantic group of word stimuli (diagonal with blue squares for “low dissimilarity”), and captured information about the differential semantic distances between each pair of the 6 groups. The values of within-category similarities were relatively high throughout and not significantly different and comparable across semantic types (arm–arm: r = 0.64, sd = 0.09; leg–leg: r = 0.63, sd = 0.07; face–face: r = 0.60, sd = 0.12; animal–animal: r = 0.60, sd = 0.12; food–food: r = 55, sd = 0.12; tool–tool = 0.61, sd = 0.06). Note that within-category similarity was high and could potentially have even been higher, had stimulus words been selected from narrower categories (e.g., by focusing on typical mammals or birds in the animal word selection).

We assessed the significance of the between-category similarities using a Wilcoxon rank sum test to compare the group of between-category r-values with the group of within-category r-values in the computational linguistic model RDM. The LSA model exhibited a categorical structure in which action words clustered together (r = 0.94, sd = 0.09), with no significant differences between the within-category r-values of each action type and all action types (similarity within-leg category vs. similarity within action words: P = 0.58; similarity within-arm vs. similarity within action words: P = 0.89; similarity within-face vs. similarity within action words: P = 0.78). In contrast, semantic similarities across action words were significantly larger than semantic similarity within the group of object words (r = 0.67, sd = 0.1) (P < 0.001). In addition, object nouns did not homogeneously cluster together (similarity within-tool vs. similarity within object words: P < 0.001; similarity within-food vs. similarity within object words: P < 0.001; similarity within-animal vs. similarity within object words: P < 0.001).

However, there was similarity between the large class of action words and the sub-category of object words referring to tools (r = 0.7, sd = 0.09). Also, tool words showed comparable similarity degrees as the cross-category comparison between this class and the action words (within-tool similarity vs. similarity between tool and action words: P = 0.14). All word types were significantly dissimilar from animal words (similarity within-animal vs. similarity within action words: P < 0.001).

In summary, these results indicate comparable distributional semantic similarities across each of our 6 semantic sub-categories, but substantially more semantic coherence within the action word group than across object word groups and, interestingly, similar semantic relations within words used to speak about actions and those related to tools.

Correlation Between Computational Linguistic and Neural Dissimilarity Patterns

To relate the semantic similarities captured by the latent semantic distances of the computational linguistic model to the patterns in each ROIs, we performed a condition-based analysis in which the 16 items within each semantic sub-category were averaged, resulting in 6×6 RDMs.

The relatedness between these patterns and the LSA model was assessed by statistical comparison and inference (Nili et al. 2014). Indeed, significant correlations between the linguistic dissimilarity structures assessed by the LSA-based model and the brain activity RDMs were found in the LIFG pars opercularis (BA 44) (r = 0.10, P = 0.03), LIFG pars triangularis (BA 45) (r = 0.14, P = 0.01*) (with asterisks indicating FDR corrected significance; see Table 3 and Supplementary Materials Fig. 8c), LIFG pars orbitalis (BA 47) (r = 0.11, P = 0.03), left posterior MTG (r = 0.11, P = 0.05), and left precentral gyrus (r = 0.09, P = 0.05). Only BA 45 survived FDR correction for multiple comparisons (see Table 3 and Figs. 4 and 5; a summary of the results is shown in Figs. 6 and 7) when values for all semantic sub-types were included.

Figure 4.

Multivariate RSA results. Brain activity patterns correlating with the LSA model in left inferior gyrus, left posterior middle temporal gyrus, and precentral gyrus, displayed as RDMs. Note that the categorical structure of some of these brain activity RDMs resembles the one of the LSA model, in which action words are overall represented as being similar to each other, and also similar to tool word and, to a degree, food words. The representational geometry in these regions differs from the one in a typical object processing area, like the left posterior ITG, where object words, but not action words, are represented as being similar. Therefore, the brain activity RDM in the left posterior ITG does not correlate with the LSA model.

Figure 5.

Scatter plots showing correlations across all word categories between the LSA model and the fMRI patterns in left inferior frontal gyrus (BA44: r = 0.48; P = 0.07; BA45: R = 0.62; P = 0.02; BA47: r = 0.47; P = 0.08), left posterior middle temporal gyrus (r = 0.54; P = 0.04), and precentral gyrus (r = 0.52; P = 0.05). On the rightmost panel, correlation between the LSA model and the brain activity RDM in left inferior posterior temporal (r = −0.31; P = 0.26) is also shown. The gray dots in each scatter plots refer to the 15 unique combinations of each condition with the other, which are also represented by the off-diagonal cells in the upper/lower triangle of the 6×6 RDMs.

Figure 6.

Summary of the multivariate RSA results calculated across all word categories. Regions in which significant correlations were found between categorical structures of fMRI patterns and of the LSA model are shown in red.

Figure 7.

Matrix displaying the correlations between the LSA model RDM and the brain activity RDMs in each ROI. Each entry compares two RDMs by Spearman rank correlation, from minimum (dark blue) to maximum (dark red) on the diagonal. The matrix is symmetric about a diagonal of ones. Correlations of LSA model and the RDMs in left precentral (green), left posterior MTG (green), and LIFG (BA 44-45-47) (yellow) are shown in the rightmost column. Note that the RDMs in LIFG (BA 44-45-47) and especially in BA 45 are most highly correlated with the LSA similarity measure (rightmost column). Also, these areas’ pattern similarities correlate highly with each other and with precentral and (in part) with posterior MTG (red cells).

Table 3

Table of correlation values (r) and significance values (P) between the brain activity patterns in ROIs and the 5 models

Region	LSA model		Action word category		Object word category		Action-tools word category		Action-foods-tools word category
	R	P	R	P	R	P	R	P	R	P
LIFG BA 44	0.10	0.03	0.06	0.04	−0.11	0.98	0.10	0.05	0.12	0.02
LIFG BA45	0.14	0.01*	0.08	0.02	−0.11	0.97	0.14	0.01	0.13	0.03
LIFG BA47	0.11	0.03	0.05	0.07	−0.10	0.98	0.08	0.07	0.10	0.03
L_Temp_Pole	0.03	0.24	0.01	0.52	0.01	0.52	0.01	0.61	0.04	0.30
L_ant_ITG	0.06	0.11	0.03	0.17	−0.05	0.91	0.09	0.03	0.06	0.12
L_post_ITG	0.01	0.48	0.00	0.50	0.00	0.50	0.03	0.72	0.01	0.53
L_ant_MTG	0.06	0.13	0.05	0.11	−0.05	0.95	0.05	0.17	0.05	0.78
L_post_MTG	0.11	0.05	0.06	0.08	−0.05	0.85	0.11	0.03	0.10	0.03
L_ant_STG	0.05	0.12	0.05	0.11	−0.04	0.94	0.08	0.04	0.04	0.78
L_post_STG	0.08	0.08	0.05	0.07	−0.05	0.96	0.08	0.04	0.09	0.05
L precentral	0.09	0.05	0.08	0.19	−0.08	0.87	0.18	0.02*	0.09	0.05
L_angular	0.04	0.72	0.01	0.65	−0.03	0.88	0.04	0.74	0.03	0.62

Significant correlations are marked in bold. Correlations values which survive FDR correction for multiple comparisons for the sub-set of ROIs (highlighted on gray), for which we had prior hypotheses, and the LSA MODEL are indicated by asterisk (*).

Categorical Representations in Distributed Cortical Regions

Further analyses focused on specific categorical links among subsets of the word materials examined. The “action word category” model triggered significant correlations only with the brain activity in BA 44 (r = 0.06, P = 0.04) and BA 45 (r = 0.08, P = 0.02) in the LIFG (Table 3 and Supplementary Materials Fig. 8). Correlations were only marginally significant for BA 47 (r = 0.05, P = 0.07) and left posterior middle temporal gyrus (r = 0.06, P = 0.08).

The “object word category” model did not correlate with the brain activity patterns in any of the regions (Table 3), confirming that the semantic relationships between the individual sub-types of object words were highly dissimilar, as was apparent from the visual inspection of the brain activity RDM in all regions. It is possible that these weak or absent results relate, in part, to low signal to noise ratios due to the low number of words (48) in any of the categories, although it may also be that this difference reflects the semantic heterogeneity of the object words. To reconfirm that semantic properties were reflected, we defined word groups across action and object word categories which were semantically related according to LSA, also noting that an RSA effect across semantically similar nouns and verbs cannot be due to their lexical class.

Guided by LSA-based semantic similarity across lexical category boundaries, we considered the larger class of action verbs plus nouns affording a hand action (tools). Interestingly, the semantic and activation pattern similarities in the “action-tools word category” model correlated significantly with each other within BA 44 (r = 10, P = 0.05), and BA 45 (r = 14, P = 0.01) in LIFG, left anterior ITG (r = 0.09, P = 0.03), left posterior MTG (r = 0.11, P = 0.03), left anterior superior temporal gyrus (STG) (r = 0.08, P = 0.04), left posterior STG (r = 0.08, P = 0.04) and left precentral gyrus (r = 0.18, P = 0.02*). Correlation was marginally significant in LIFG BA 47 (r = 0.08, P = 0.07). Likewise, the “action-foods-tools word category” model showed correlations between semantic similarity in terms of latent semantic distances between words and similarity of their multivoxel word-evoked activation patterns within a range of regions, including LIFG BA 44 (r = 0.12, P = 0.02), LIFG BA45 (r = 0.13, P = 0.03); LIFG BA 47 (r = 0.10, P = 0.03), left posterior MTG (r = 0.10, P = 0.03), left posterior STG (r = 0.09, P = 0.05), and left precentral gyrus (r = 0.09, P = 0.05) (Table 3 and Supplementary Materials Fig. 8c).

These results suggest that the semantic similarity structure as revealed by the LSA model was most clearly reflected by the models including both nouns and verbs semantically linked to actions (Action-Tool and Action-Foods-Tools models; see also Supplementary Material Fig. 8d). For these models, multimodal areas, including inferior frontal gyrus and posterior middle temporal gyrus and, to a similar degree, modality-preferential regions in precentral motor cortex were involved in representational similarity processing.

Control Model

The control model for imageability did not correlate with the similarity patterns of brain activity in any of the ROIs (Table 4).

Table 4

Table of correlation values (r) and significance values (P) between the brain activity patterns in ROIs and the control model

Region	Imageability
	R-value	P-value
L BA 44	0.04	0.1
L BA45	0.00	0.5
L BA47	0.01	0.37
L_Temp_Pole	0.02	0.43
L_ant_ITG	0.02	0.35
L_post_ITG	0.01	0.57
L_ant_MTG	0.00	0.53
L_post_MTG	0.05	0.12
L_ant_STG	−0.02	0.43
L_post_STG	0.02	0.32
L precentral	−0.02	0.73
L_angular	−0.02	0.66

Discussion

An RSA analysis comparing the distributional semantic similarity between the experimental words and the similarity between the corresponding fMRI response patterns revealed that relationships among lexical-semantic categories can be mapped to specific cortical regions. This was most clearly apparent for the inferior frontal cortex and did also apply, for particular categorical links among selections of stimulus words, to precentral cortex, which survived FDR correction in the action word plus tools model analysis. Other frontotemporal areas, including the posterior-middle-temporal cortex, also reflected lexical-semantic similarity. These results show that both frontal and temporal areas with an established role in multimodal language processing represent degrees of similarity among lexical-semantic categories and that also a modality-preferential region such as the precentral motor cortex contribute to this type of computation. The present findings have important implications for neurosemantic theory, arguing against grounding-free models that entirely base semantic learning on symbol co-occurrence computed in an “amodal semantic hub” and calls for an alternative explanation. Indeed, theories of semantic grounding postulated that a range of basic semantic features must originally be picked up from actions and objects during learning interaction, before distributional learning of meaning through texts is possible (Harnad 2012), and this grounding mechanism leads to lasting semantic links that reach into modality-preferential cortices such as the motor system. Our results seem to sit best with this latter position suggesting that, even though much semantic learning is due to distributional learning from linguistic contexts, grounded semantic features represented and processed in modality-preferential areas are handed over during such learning.

We found that fine-grained similarity between word-elicited fMRI response patterns emerged from predefined ROIs for which previous work had suggested a role as “semantic hubs” or a role in category-specific semantic grounding of symbol meaning. Among these, left inferior frontal gyrus, left posterior middle temporal gyrus, and precentral gyrus showed significant correlations between LSA semantic similarity and activation pattern similarity. Frontotemporal areas were also indicative of semantic similarities between the larger groupings of action-related verbs and tool nouns and the extended action word group including both tool and food nouns. Other “semantic brain areas,” including the left inferior temporal gyrus, were not found to index semantic similarity as assessed by LSA in this study (see discussion below). Our data are consistent with the proposal that neuronal circuits carrying semantic information are not focused on one “semantic hub,” as several current theories suggest, but, instead, are carried by distributed circuits of neurons scattered across multimodal and modality-preferential areas. Our results also suggest that the cortical distributions of these semantic mapping areas may depend to a degree on the semantic categories of the words under investigation.

We now discuss in detail our specific results for different vocabulary selections, their relationship to previous work and the insights they bring into the representational content of words in key semantic brain regions.

LSA as a Method for Capturing Semantic Similarity

It is an important aspect of this present work that the use of objective, corpus-based semantic similarity measures allowed us to investigate, for different cortical areas, the correlation structure for response patterns of lexical-semantic categories. Although LSA as a method is very well established (Landauer and Dumais 1997), one may argue that some aspects of our distributional semantic findings need brief comment. Our verb types were evaluated as semantically more similar than the noun sub-types, a feature becoming clearly apparent in the “semantic distance plot” in the top panel of Figure 2b. This seems to suggest that animal and food nouns such as “slug” and “soup” are more dissimilar semantically than are action verbs such as “kick” and “kiss” (even though the real life entities these words are used to speak about are in fact quite different in both cases). Still, semantic similarity within each sub-group of 16 words was relatively high and comparable between semantic sub-groups, thus indicating that the matching of these items was successful and, crucially, that subjects were able to process the close semantic relationships within each sub-group of words similarly well.

The greater semantic dissimilarity across object noun groups may, in part, relate to the fact that many of the nouns can appear in different thematic roles (as AGENT, PATIENT, SOURCE, etc.) and positions within a sentence so that their contextual variability may be generally high, thus entailing greater distributional dissimilarity. In future, it will be important to complement our current finding with other measures of semantic similarity. Carslon and colleagues (Carlson et al. 2014) recently adopted different semantic similarity measures to investigate the correlation structure of higher-level visual cortex, suggesting a range of alternative approaches, which may be more or less successful, depending on cortical area and possibly also on the part of the vocabulary the study addresses. In the present study, the lack of correlation between LSA and the representational patterns in regions known to be essential for meaning processing, such as anterior temporal and inferior parietal cortex, should therefore not be over interpreted and calls for further studies comparing semantic measures that systematically map the referential world-relationship of words, over and above their contextual distributional properties. Furthermore, as a necessary implication of the fact that broader semantic space was covered in the present work compared with other studies investigating nouns, a relatively low number of object words (48) was included here so that the statistical power available for revealing fully significant effects for those words may not have been sufficient in some of the proposed semantic hubs. Finally, the well-known susceptibility artifacts arising in anterior temporal lobes may account for the absence of significant effects for representational similarities derived for this region.

Relationship to Earlier MVPA Work

Earlier work using MVPA and RSA reported semantic similarity mapping in multimodal frontal, temporal and parietal areas, but not in precentral or other modality-preferential areas. For example, Fairhall and Caramazza's (2013) subjects had to rate the semantic typicality of object-related words (nouns) as members of their semantic type (e.g., fruit-typicality of “apple” and “coconut”) and express this judgment binaurally in each trial. Why were multimodal—and not modality preferential—regions emphasized in this study? Crucially, semantic distance was related to brain activation in a task where four or more different semantic processes were required: 1) the understanding of a word (or object) and 2) its classification into a semantic category, 3) calculation of “typical” semantic features of that category or a prototype, and 4) relating these features or prototype to the target item and estimating the “semantic distance.” Although one may argue that a semantic task may enhance semantic processing, we should emphasize the fact that such tasks bring about the danger of confounds. If semantic processing of the meaningful item (process 1) is in focus, the addition of processes 2–4 will alter the cognitive and brain activations so that the target process will be contaminated or even overridden. For example, both the words “robin” and “penguin” have each their own characteristic semantic brain signatures, which can be mapped in a passive reading/understanding task. However, the former word “robin”, which is close to its category prototype (BIRD), will lead to reduced activation in a semantic typicality task (because a robin resembles the BIRD prototype), whereas the latter untypical item will yield a double activation of exceptional category item (penguin) and prototype. Therefore, the brain correlates of semantic meaning of a symbol arise most clearly when words are “just” being understood, without additional semantic processes preceding or following upon understanding. In this sense, any additional task will add semantic processes that may work against the aim of mapping those items as semantically similar. To avoid such complication, we here chose a passive semantic comprehension task in which measures were taken to ascertain that subjects paid attention to all stimuli and were not biased by the task towards processing of some semantic features but not others. Furthermore, as much earlier work was limited to object nouns, we used an equal number of nouns and verbs in this study, thus avoiding a vocabulary bias. It must also be noted that the use of a taxing motor task in Fairhall and Caramazza's study may have worked against finding semantic activity in the motor system (Pulvermüller and Fadiga 2010; Schomers and Pulvermüller 2016), and therefore the motor output requirements were kept low in this present study. Strikingly, Mitchell et al. (2008) found that pattern classification of multivoxel activation in motor regions reliably predicted both object and action words processed by experimental subjects, although other areas of putative importance for semantic processing were not systematically investigated in this work. Overall studies using methods comparable to those applied here found semantic similarity indexes in a range of disparate areas, including intraparietal sulcus (Devereux et al. 2013), perirhinal cortex (Clarke and Tyler 2014), and prefrontal, fusiform and posterior middle and inferior temporal cortex (Fairhall and Caramazza 2013). Amongst those, the posterior middle temporal cortex is the only one with some cross validity over different studies (Fairhall and Caramazza 2013; Devereux et al. 2013, our present work). However, the present data forbid to see this region as a unitary “gateway to meaning” (Hickok 2014), because semantic similarity mapping across all categories was most prominent and robust not in posterior temporal lobe but, instead, in LIFG BA 45 and in precentral gyrus, as only the latter two areas respectively produced significant semantic similarity mapping for the latent semantic model and the action-plus-tools word category model after correction for multiple comparisons.

As mentioned, the study by Carslon et al. (2014) showed effects of latent semantic similarity in temporal cortex. However, this study focused on semantic similarity within semantic (sub-) categories, such as animals, humans, and manmade objects as revealed by a large area extending from inferior temporal cortex to lateral occipital cortex. The authors did not report significant cross-category similarity effects in this region. Our present results now show both within-category similarity mapping for all the action verbs and across category similarity mapping for the action verbs and the action-related nouns in frontocentral and middle temporal regions. This discrepancy may indicate that the frontocentral and posterior middle temporal cortex may map semantic relationships both within and across semantic kinds, whereas the inferior temporal cortex may be more specialized on discrimination within fine-grained categories. The progression of semantic dementia, which starts with fine within-category errors due to lesion in anterior inferior temporal cortex (Acosta-Cabronero et al. 2011), is consistent with this. First the fine-grained discrimination between semantically specific concepts gets lost (“horse” or “animal” instead of “zebra”) and only later-on, when the disease has spread to wider frontotemporal regions, the larger category errors are known to predominate (“thing” instead of “zebra”) (Patterson et al. 2007).

LIFG, a Key Region for the Representation of Semantic Similarity

Extensive literature has suggested that the LIFG, and in particular the anterior part of Broca's area, BA45, plays a critical role in lexical-semantic processing (Gabrieli et al. 1998; Bookheimer 2002; Mechelli et al. 2007), since this region is activated in tasks emphasizing retrieval of lexical-semantic knowledge (Petersen et al. 1990), generation of words based on semantic relationships (Klein et al. 1995), semantic decisions about written words (Demb et al. 1995; Dapretto and Bookheimer 1999) and semantic control (Thompson-Schill et al. 1997). Also, its lesion leads to semantic processing deficits (Devlin et al. 2003; Kemmerer et al. 2012). Our results bring novel evidence that the LIFG specializes in representing latent semantic information about how words that are members of semantic categories of either actions or objects, like the verb “to carve” and the tool noun “knife”, can be combined with each other in actual language usage. Interestingly, the LIFG is typically linked with the selection of the relevant semantic properties from competing semantic alternatives (Thompson-Schill et al. 1997). However, nothing in the experimental task of the present study might have overtly triggered either semantic association or selection, as participants silently read single words presented in isolation. Neurocomputational models of word meaning processing (Garagnani and Pulvermüller 2016) have indicated that inferior prefrontal cortex is one of the areas carrying the highest neuron densities of distributed word-related neuronal circuits for meaningful words. Shedding now light into the representational content of words, the present fMRI RSA results reveal that this region is essential for discriminating between these circuits and therefore for generating activation patterns that reflect lexical semantic similarity and dissimilarity.

Action-Semantic Similarity Mapping in the Frontocentral Cortex

In our data, the inferior frontal activation patterns reflected the semantic relationships across leg-, arm-, and face-related action words but not across words that denoted objects. Interestingly, the neural patterns elicited by nouns referring to tools, which activate posterior temporal regions and also hand motor regions, putatively reflecting the hand manipulation afforded by the object of the words are used to speak about (Carota et al. 2012), clustered with those of action verbs insofar as their semantic similarity was mapped in this region: the cross-category correlations (action and tools words) between semantic and neural similarities were highest in BA 45 but only marginally lower in adjacent inferior frontal BA 44 and lateral precentral cortex. BA 47 fell off for the action verb plus tool noun computation.

These results suggest that frontocentral areas are sensitive to semantic information about actions and motor features. Action semantics includes basic information, such as the body-part with which an action is typically performed and details of typical motor trajectories of stereotypical movements, temporal order for arrangement of elementary movements as well as action goals to be integrated with the movements (Jeannerod et al. 1995; Jeannerod 2006; Pulvermüller 2012). Converging evidence from recent work showed physiological signatures of semantic priming, mediated by action-semantic knowledge, in the motor system and inferior frontal cortex (Ulrich et al. 2013; Grisoni et al. 2016). Furthermore, focal lesions to the left inferior frontal regions and in the motor system are long known to impair the patients’ ability to understand action-related words and concepts (Bak et al. 2006; Tranel et al. 2003; Kemmerer et al. 2012; Bak and Chandran 2012; Dreyer et al. 2015). The present results indicate that the encoding of semantic similarity in inferior frontal and precentral cortex specifically reflects the within-between categorical links across action-related words of verbs and tool/food nouns.

One may ask to what degree lexical relatedness was mapped in our study instead of semantic similarity between words. Indeed, our action words were either unambiguous verbs or tended to be preferentially used as verbs and the object words tended towards nounhood. However, only few of our findings can be explained by lexical category distinctions. The representational geometry of the patterns in inferior frontal, posterior middle temporal, and precentral cortex conforms to the particular categorical structure captured by the LSA model. The significant correlations of the LSA for the greater action-related word categories, in which nouns and verbs were mixed, strongly argues in favor of a method sensitive to semantics but not specific to lexical category. Given that the representations of the action verb category dissociated from the patterns elicited by animal nouns but was strikingly similar to the representations of tool words in frontocentral cortex and MTG, the current data suggest a differentiation of the representational similarity patterns on the basis of a semantic property which was significantly stronger in action verbs and tools, and to a lesser degree to foods, than for animal nouns: action-relatedness.

Note that our subjects were kept naive about the purpose of the experiment until the completion of the experimental session and their attention was not attracted in any way to action-related information contained in some of the stimuli. Importantly, we also kept the overt movements of participants to a minimum, requiring one only for occasional catch trial stimuli to keep our subjects attentive. Trial-by-trail motor responses may impact on and diminish any semantically related activity in the motor system (Pulvermüller and Fadiga 2010). In addition, we balanced action and object-related word in our stimulus set to allow for the generation of action-semantic brain activity in addition to object concept processing. Furthermore, as already noted, nothing in the instructions or the procedure biased the subjects’ attention toward semantic relatedness, semantic similarity or context-based aspects of the stimuli.

Similar to the neural patterns seen in the left inferior frontal cortex, the representational structure in the precentral gyrus exhibited between-category similarity between action and tool words.

As shown by earlier studies, the motor system is a component of a distributed cortico-cortical loop underlying the neurocognitive representation of the referential link between the meaning of a word and its form (Pulvermüller et al. 2009; Pulvermüller and Fadiga 2010; see Introduction).

Our finding on the representational structure of the precentral gyrus support the view that aspects of word meaning are represented in motor areas, suggesting that these regions specifically reflect gradual semantic links among action words and action-affording object words.

These results are difficult to accommodate by a semantic account postulating an “amodal semantic system” situated in “amodal” (i.e., multimodal) hubs as the exclusive brain correlate of semantics. The finding that semantic similarity processing “reaches into” the motor system sits well with accounts positing that action-semantic information is grounded in the motor system (Pulvermüller 2005; Barsalou 2008; Pulvermüller and Fadiga 2010; Kemmerer 2015) rather than resulting from arbitrary “associations” between words and motor movements (induced, e.g., by classical conditioning, see Mahon and Caramazza 2010; Hickok 2014). According to these models, the semantic processing of object and action words requires information channeling through sensory and, crucially, motor systems. From the latter, action-related semantic features can be extracted and incorporated into semantic circuits of words distributed across the brain, which would thus extend into sensorimotor areas. Although this is certainly not the only possible explanation of our present data—one may, for example, stipulate that abstract ungrounded semantic representations are represented everywhere in the brain, the motor system included (Mahon and Hickok 2016)—, this proposal provides a tentative account how semantic similarity especially for action verbs and action-affording nouns is mapped in the precentral cortex, adjacent inferior frontal lobe and superior temporal gyrus (where action-related movements are preferentially processed).

Representation of Action and Tool Knowledge in the Posterior MTG

As already discussed above, recent multivariate voxel pattern analysis studies already suggested a role of the posterior middle temporal cortex in semantic similarity processing and some models indeed see this area the most relevant semantic hub or interface (Hickok 2014), a position we adopted earlier in this paper. However, a broad research stream has also accumulated evidence that this area is in fact contributing differentially to the processing of semantic types and is particularly important for motion related semantic information crucial for action and tool concepts (Martin 2007; Hauk et al. 2008; Deen and McCarthy 2010; Rueschemeyer et al. 2010; Saygin et al. 2010; Wallentin et al. 2011). This “category-specificity” position matches with the wisdom that loci in posterior middle temporal gyrus and sulcus are multisensory integration sites supporting bimodal (i.e., auditory and visual) representations of object motion (Fernandino et al. 2015), activated during the retrieval of information about tools, naming tool sounds, words for visual motion of manipulable objects (Chao and Martin 1999; Kiefer and Pulvermüller 2012; Martin and Weisberg 2003; Beauchamp and Martin 2006; Noppeney et al. 2005; Kable et al. 2005; Tranel et al. 2003; 2005; Kemmerer et al. 2012; Campanella et al. 2010; Kalenine et al. 2010; Humphreys et al. 2013). A novel finding of the present study is that concrete words referring to actions and manipulable objects elicit similar representational patterns in the posterior MTG. Therefore, our results corroborate previous proposals of a category-specific organization for action and tool words in this region (for a review, see Martin 2007). However, we re-emphasize that our data reveal this region as only one of several processing sites of semantic similarity.

Univariate Effects

A side result of this study is mentioned in closing: applying conventional univariate analyses, we found that the processing of all words relative to visual baseline activated a set of areas, including left prefrontal and precentral cortex and predominantly left fusiform areas, which are known to be activated in and relevant for different aspects of language processing, including semantic computations (Dapretto and Bookheimer 1999; Poldrack et al. 1999; Devlin et al. 2003; Binder et al. 2009; Binder and Desai, 2011; Pulvermüller and Fadiga 2010; Pulvermüller 2013). In addition, bilateral cingulate gyrus and pars opercularis were found active. Note that areas known to be part of the semantic system, including MTG, temporal poles and angular gyrus (Binder et al. 2009; Binder and Desai 2011; Pulvermüller 2013) were not found active. Reasons for this may relate to one or more of the following features of this experiment: First, it must be highlighted that the present study, being specifically designed for RSA, included only 16 words per category for a total of 96 test words and thus differed from classical univariate studies, which typically employ considerably higher numbers of stimulus items in order to show robust spatial-averaged activations to averaged set of these items. Furthermore, all words were repeated 6 times and preliminary analysis confirmed that brain activation decreased with repetition (Grill-Spector et al. 2006). Also, we balanced the number of object- and action-related words in this studies, therefore possibly producing more variability in the engagement of semantic areas, whereas much earlier work had investigated primarily nouns. These factors may contribute to an explanation why activation in areas such as MTG, temporal poles and angular gyrus was not detected in this study in the basic univariate analysis. However, other key semantic regions were recruited and thus confirmed by this analysis (Binder et al. 2009; Binder and Desai 2011; Pulvermüller 2013), including left inferior frontal and cingulate cortex bilaterally, left inferior temporal and fusiform cortex and the left motor regions. We should also mention once again that since a low-level visual baseline was used in this study, we cannot exclude that these univariate activations reflect a variety of parallel processes, including letter recognition, phonological and syntactic analysis, so that any conclusions on semantic processing must rely on the RSA analyses for which the present study was optimized.

Conclusions

We showed that the similarity of word-elicited multivoxel activation patterns in left inferior frontal, motor, and posterior temporal cortex is sensitive to semantic similarity between words as revealed by their distributional properties in standard text corpora. These conceptual similarity effects were physiologically manifest within and across lexico-semantic categories while subjects passively read words without a specific semantic task. The encoding of lexical-semantic information in these frontotemporal areas may be based on the distributional properties of words, thus reflecting word-word association (Landauer and Dumais 1997). However, such contextual foundation alone would not explain the representation of the semantic similarity among lexical-semantic categories in precentral cortex. Grounding models offer an account of motor system along with multimodal system involvement in semantic processing, especially of action-related words, which formed a main target of this study. Accordingly, semantic-associative learning between words would follow semantic grounding of at least one part of the vocabulary, whereby a basic set of words is semantically linked to action and perception information, a process which involves the sensorimotor system (Harnad 2012; Cangelosi and Harnad 2001; Barsalou 2008; Pulvermüller 1999; 2013). In subsequent distributional semantic learning from texts, grounded semantic features are handed over by correlation mapping between semantic word representations. Therefore, this contextual grounding perspective is consistent with the fact that, over and above multimodal areas, motor regions are part of the areas where semantic similarities are mapped. Our results confirm a widespread frontotemporal set of areas for semantic processing and representation of groups of related concepts, in which specific semantic circuits may differ in distribution according to the semantic information they bind with and carry.

Supplementary Material

Supplementary material are available at Cerebral Cortex online.

Funding

Medical Research Council (MC_US_A060_0034, U10055.04.003.00001.01 to FP and MC-A060-5PR20 to NK) and European Research Council Starting Grant (ERC-2010-StG 261352 to NK).

Notes

We thank Jeroen Geertzen, Karalyn Patterson and two anonymous reviewers for their comments at different stages of this work. Conflict of Interest: None declared.