Boundary extension as mental imagery

When we remember a scene, the scene’s boundaries are wider than the boundaries of the scene we saw. This phenomenon is called boundary extension. The most important philosophical question about boundary extension is whether it is a form of perceptual adjustment or adjustment during memory encoding. The aim of this paper is to propose a third explanatory scheme, according to which the extended boundary of the original scene is represented by means of mental imagery. And given the similarities between perception and mental imagery, the memory system encodes both the part of the scene that is represented perceptually and the part of the scene that is represented by means of mental imagery.


Boundary extension
Look at Figure 1. Now look away, do something else and try to remember the image. What you will recall is more similar to Figure 2. When we remember a scene, we remember more than what we saw. Literally more: the scene's boundaries are wider than the boundaries of the scene we saw. This phenomenon is called boundary extension.
Boundary extension is one of the most robust psychological findings about memory. It holds across age groups, experimental methods (drawing from memory vs. picking a picture that matches our memory), length of exposure time (how long we are looking at the scene), length of time gap (between seeing the scene and recollecting it), depictive style (photos vs. drawings), image content, and so on (see Hubbard et al. 2010 for a very thorough summary of these findings and Bainbridge and Baker 2020 for some recent findings that complicate this picture somewhat ).
In the philosophy of memory, boundary extension is used as an example of the constructive nature of memory. This fits into a wider set of findings about memory that all seem to demonstrate that memory formation is not a matter of copying perception into our memory. Memories are, rather, constructed on the basis of the scene we see, but their content is not determined by the scene seen (De Brigard 2014, Michaelian 2016and McCarroll 2018, see also Robins 2019).
This raises an important philosophical question about the boundary extension findings (Michaelian 2011, Fernandez 2019: 196-98 and De Brigard 2014. Is boundary extension explained by perceptual adjustment or by adjustment during memory encoding? Again, when you look at Figure   modified automatically as a memory of this scene is formed' (Michaelian 2011: 326). According to (ii), in contrast, boundary extension is a form of perceptual adjustment. There is no adjustment during memory encoding: the perceptual experience of Figure 2 leads to the memory of Figure 2. But when we look at    Figure 2. The reason for this might be that when we look at Figure 1, we supplement the stimulus with our own 'perceptual schema' and this leads to the experience of Figure 2 (Intraub 2002 andRichardson 1989). My aim is to propose a third explanatory scheme, according to which the extended boundary of the original scene is represented by means of mental imagery. And given the similarities between perception and mental imagery, the memory system encodes both the part of the scene that is represented perceptually and the part of the scene that is represented by means of mental imagery. This means that boundary extension is neither perceptual adjustment nor memory adjustment.

The two-step model of boundary extension
My claim is that boundary extension is a two-step process and we have plenty of empirical evidence for how both of these steps work. I propose the following scheme instead of (i) and (ii) above: The first step is that looking at a picture activates early cortical representations of the space immediately outside the boundaries of the picture. Note that this is a different and much weaker claim than the perceptualist's explanation that we do have perceptual experience of the space beyond the picture boundaries. But this space is nonetheless represented by the visual system. Take amodal completion as an analogy. We know that if we look at a circle partly occluded by one of the corners of a larger rectangle, our visual system completes the occluded part of the circle. This can lead to a strong quasiperceptual experience of the occluded part of the circle, but what matters from our point of view is that there is a vast amount of evidence that the occluded part of the circle is represented by direction-sensitive neurons in the early visual cortices, even in the primary visual cortex. So while on the retina the occluded contour is not present, our visual system draws on this occluded contour already in the primary visual cortex (Lee and Nguyen 2001, Komatsu 2006and Nanay 2018b. I am not suggesting that we amodally complete the scene beyond the boundaries of the picture. Depending on how we define amodal completion, this may or may not be true. If we define amodal completion as the representation of occluded parts of perceived objects, then amodal completion is out of the question as nothing occludes the part of the scene just outside the picture's boundaries. What is important from our point of view is that the early visual cortices represent the missing parts of the scene that fall just beyond the picture boundary (Chadwick et al. 2013).
Just as the amodal completion of occluded objects is not perception per se (Nanay 2010 andHelton andNanay 2019), the early cortical representation of the parts of the scene just outside the picture boundaries is not perception per se either. It is the kind of representation that is produced by perceptual processing (again, perceptual processing in the early visual cortices) without direct sensory stimulation.
Psychologists and neuroscientists have a simpler way of referring to the kind of representation that is produced by perceptual processing in the early sensory cortices without direct sensory stimulation. They call it 'mental imagery' (see Pearson et al. 2015, see also Nanay 2018a). Here is a representative quotation from a recent review article in the leading cognitive science journal Trends in Cognitive Sciences: 'We use the term "mental imagery" to refer to representations . . . of sensory information without a direct external stimulus' (Pearson et al. 2015: 590).
I will borrow this way of using the term 'mental imagery' in what follows, but I do so merely for convenience (because saying 'representation that is produced by perceptual processing in the early sensory cortices without direct sensory stimulation' is somewhat cumbersome). To put the point briefly, whenever we see a picture, we have mental imagery (in the sense used here) of the scene just outside the boundaries of the picture.
Most of the time, this mental imagery is unattended. But it can also be attended, especially in some examples of visual art, where the artists very explicitly try to evoke our mental imagery of the scene outside the frame. One famous example would be Degas, who likes to place the protagonists of his paintings in a way that only parts of them are inside the frame. The rest we need to complete by means of mental imagery. In some extreme cases (e.g. 'Dancers climbing the stairs', 1886-1890, Musee D'Orsay), we only see someone's arm or the top of their head and we need to complete those parts of her body that are outside the frame by means of mental imagery. Another example is Buster Keaton, who also uses the viewer's mental imagery of the off-screen space in his films, but normally for comical effects. One example is the first shot of his short film Cops (1922), where we see the protagonist in close up behind bars and looking depressed. The second shot reveals that he is behind an iron gate taking to a girl who does not love him back (see Burch 1973: 17-31 for more examples of this kind).
To sum up, the first step of the boundary extension process is that we represent the scene just outside the boundaries of the picture by means of mental imagery. In other words, looking at Figure 1 leads to a hybrid perception/ imagery representation of Figure 2, where the parts closer to the edges (basically the difference between Figure 2 and Figure 1) are represented by mental imagery. This is the first step.
The second step takes us from this hybrid perception/imagery representation of Figure 2 to the memory of Figure 2. And just as in the case of the first step, we have plenty of independent empirical evidence about how this step works.
We know that the perceptual system is prone to treat perception and imagery similarly. In the Perky experiment, one of the most famous and earliest experiments about mental imagery, subjects are looking at a white wall and are asked to visualize objects while keeping their eyes open. Unbeknownst to them, barely visible images of the visualized objects are projected on the wall, which they take themselves to be visualizing, not perceiving (Perky 1910, see also Nanay 2012). So their perceptual system treated the perceived images as mental imagery.
Remember that I use the term mental imagery here the way psychologists and neuroscientists use it: as representation that is produced by perceptual processing in the early sensory cortices without direct sensory stimulation. Both perception and mental imagery then amount to representation that is produced by perceptual processing in the early sensory cortices -the only difference between them is that mental imagery is not triggered by direct sensory stimulation. To go back to the amodal completion example for a second, if you look at the landscape through a mosquito net, strictly speaking, you perceive little squares of the landscape and you represent parts of the landscape that are between these little squares (that are occluded by the net itself) by means of amodal completion. This nonetheless gives rise to a unified visual experience.
The memory system takes the hybrid perception/imagery representation of the scene and transforms it wholesale into memory (regardless of what part of this representation was imagery and what was perception). It transforms the hybrid perception/imagery representation of Figure 2 into a memory of Figure  2. Last reference to amodal completion: when we look at the landscape through the mosquito net, we later recall not only the little squares we actually perceive, but the landscape that is the hybrid of amodally completed parts and perceived parts.

Best of both worlds
In this section, I will argue that this two-step explanation of boundary extension combines the explanatory benefits of the perceptual and the memory account. And it does so without inheriting their problems.
As we have seen, the most influential version of the perceptual account of boundary extension is the perceptual schema account (Intraub 2002 andRichardson 1989). One of the most important objections to this account comes from the experiments that show that boundary extension is also present for scenes with no recognizable objects. So our perceptual system is not in a position to apply a perceptual schema of the scene outside the boundaries on the basis of recognized objects inside the boundaries, for the simple reason that there are no recognized or even recognizable objects inside boundary extension as mental imagery | 5 the boundaries (only random dots, for example, see McDunn et al. 2014, see also Mamus and Boduroglu 2018 for similar results involving semantically inconsistent scenes).
While these are clearly difficulties for the perceptual schema account, they pose no problem for my account as the early cortical representations of the scene behind the boundaries of the picture are not necessarily formed in response to top-down information about the specific objects inside the frame (which would amount to a perceptual schema). They could be based solely on the geometrical features of the patterns inside the frame (see Vrins et al. 2009 for a summary).
Another consideration that militates against the perceptual account comes from subjects with bilateral hippocampal damage (which is a memory disorder). These subjects show much less boundary extension than controls ( (Mullally et al., 2012), see also (Jajdelska et al., 2019)). This seems to suggest that boundary extension has to do with memory, not perception, as memory impairment has a significant influence on it. Note, however, that the hippocampus has a well-demonstrated influence on mental imagery (again, on representations that are produced by perceptual processing in the early sensory cortices without direct sensory stimulation). In other words, my explanatory scheme predicts that bilateral hippocampal damage would interfere with boundary extension.
Finally, proponents of the memory account often dismiss the perceptual view on the basis of the fMRI findings of Park et al. 2007, which could be taken to show that the early visual cortices are not involved in boundary extension. It is important to point out that the Park et al. findings do not in fact show that the early visual cortices are not involved in boundary extension, something the authors of the study later themselves explicitly acknowledged (see especially Park and Chun 2014: 63-65). Further, more recent fMRI experiments on how the visual cortices behave in boundary extension (see Chadwick et al. 2013 andChun 2014 for summaries) show that the early visual cortices are very much involved in boundary extension, just as my account predicts.
So much for anti-perceptual considerations. But the memory account of boundary extension has also been argued to be inconsistent with some empirical findings.
While the emotional content of pictures does not seem to have an effect on boundary extension in general (Candel et al. 2003), for highly anxious subjects, negative arousal (and only negative arousal) does have a consistent effect (towards less boundary extension) (Mathews and Mackintosh 2004). One way of explaining this is that the attention of these subjects is engaged with the central part of the picture (where the emotional content is) and this takes attention away from the boundaries. More generally, it has been found that if our visual attention is engaged elsewhere, this influences boundary extension (Intraub et al. 2008).
Attention is a perceptual phenomenon, so this seems to be a point for the perceptual account and a point against the memory account. Note, however, that my account can explain these findings in a straightforward manner. Mental imagery is as sensitive to the allocation of attention as sensory stimulation-driven perception (see Nanay 2015 for a summary). So if our attention is engaged elsewhere, this has consequences for the details of the mental imagery of the scene just outside the boundaries of the picture. My account can explain the effects of attention on boundary extension as much as the perceptual account can.

Objections and some empirical support
We have seen that my account is not vulnerable to the most widespread objections to the perceptual and the memory accounts. But there is an objection that could be raised directly about my explanatory scheme (see also Gottesman and Intraub 2003).
Explicitly imagining the scene outside the frame does not increase boundary extension (Munger and Multhaup 2016). Experimenters asked the subjects to imagine what the photographer would see if she zoomed out, or to imagine the smells and sounds coming from outside the frame. One might think that this is a problem for my account, but it needs to be pointed out that mental imagery (again, by which I mean representation that is produced by perceptual processing in the early sensory cortices without direct sensory stimulation) is very different from voluntary and intended acts of imagination. Imagination may or may not presuppose the exercise of mental imagery ( (Kind, 2001)), but even if it does, it is a very specific way of using mental imagery, which is very different from the automatic, involuntary formation of mental imagery outside the boundaries of the picture. 1 On the other hand, some empirical findings seem to support my account. First, the most consistent way of cancelling out boundary extension is by making the frame extremely salient (Gottesman and Intraub 2003). It is not clear why the prominence of the frame should influence the perceptual schema or the way this scene is encoded in memory, but we can explain this effect in terms of the mental imagery of the scene outside the boundaries of the picture in a much more straightforward manner as the salience of the frame works against the early cortical representations of the scene just outside the boundaries of the picture.
Finally, boundary extension can be induced haptically (Intraub 2004). This seems very difficult both for the perceptual and the memory account to explain. On the other hand, given the vast amount of research on the visual mental imagery of (non-cortically) blind people (see, e.g., Arditi et al. 1988) and also the research on haptically induced multimodal visual imagery (where sensory stimulation in the haptic sense modality triggers mental imagery in the visual sense modality, see James et al. 2002, see also Nanay 2018a for a summary), this finding is exactly what my account would predict as multimodal mental imagery (including the haptically triggered visual imagery of blind people) is triggered in a similarly automatic and involuntary manner as the mental imagery involved in representing the scene just outside the boundaries of the picture.

Conclusion
I argued that boundary extension is not explained either by perceptual adjustments or adjustments in memory encoding. It is, in contrast, explained by the perfectly natural and understandable use of mental imagery to complete the parts of the scene just outside the boundaries of the picture and the equally natural and understandable transferring of the resulting hybrid perception/ imagery representation to memory. 2

University of Antwerp Belgium
Peterhouse University of Cambridge UK bn206@cam.ac.uk or bence.nanay@uantwerpen.be or bence.nanay@ua.ac.be