Virtual stimuli: the Next-generation Stimuli
The ideal animal stimulus is under total control of the experimenter, has visual traits, and behavior patterns that can be varied in any way, and it appears and behaves consistently between test trials that can be easily repeated many times and any time. Does this sound like wishful thinking of a biologist? Using and exploiting the potential of artificial stimuli in animal behavior research is actually not a new idea. Even in the time of Tinbergen, researchers used wood, clay, or other materials to build artificial dummy animals that would allow control over the presentation of the animal stimulus (Ter Pelkwijk and Tinbergen 1937). Since then, the idea of artificial model animals has evolved alongside technology. Even if still used today in its original form (Kim and Velando 2014), clay and wood models have inspired the use of robots, engineering objects that are specifically tuned for the standardized presentation of pre-programmed movement patterns (Martins et al. 2005; Landgraf et al. 2008; Landgraf et al. 2014). Meanwhile, researchers also started to film live animals and played back these videos to replace live animal stimuli. Rapidly these videos started to be edited, reaching a point where, today, the complete scene, including the stimulus animal is created. Continuous advances in the computer graphics sector and increasing access to high-quality free software solutions also eased these technological advances.
Nowadays, computer animation (CA) and virtual reality (VR) systems, technologies that were once exclusively created for the entertainment and education of humans, were now used by animal scientists. “Virtual” which, in the area of computing, refers to “not physically existing as such but made by software to appear to do so” (Stevenson and Lindberg 2015) captures the nature of CA and VR stimuli, which are created using computer graphics software. They are used to simulate environments for the animals to “explore” (in VR), for example, to study spatial cognition (Stowers et al. 2014), or they are meant to replace real animals with virtual counterparts that are then presented as prey, predators, conspecifics, or mating partners which can more or less interact with live test animals in the context of visual communication. Artificial animals and environments all share the fact that they can be created and varied according to specific parameters of interest for specific research questions in focus. This gives researchers a high degree of control prior and during experiments, and creates standardized conditions that facilitate testing a given hypothesis. By this, the behavior of a stimulus animal, which naturally is affected by various factors during experiments (e.g. change of motivation), is under control of the experimenter which can lead to less variation in results and thus more robust results due to the lack of side effects. Similar to the already well-established procedures for investigating mechanisms of vocal communication, in which for example audio tracks of bird songs or alarm calls can be recorded, edited, created, and played back, experimental paradigms using CA can serve to study visual communication and with the expansion to VR also spatial cognition. CA and VR even allow creating biologically novel stimuli to investigate mechanism of prey detection, predator recognition, and recognition of conspecifics, mate preferences, and other aspects of visual recognition.
Bridging a Gap between Biology and Computer Science
Although CAs and VRs arouse lots of interest among researchers for its advantages, many are reluctant to use it because of the technical requirements necessary for creating virtual stimuli, and the apparently different worlds of biology and computer science. Most of these concerns arise from lack of contact or common grounds and communication. To bridge a gap between these two worlds, we organized a symposium and workshop at the 34th International Ethological Conference known as “Behaviour 2015” in Cairns, Australia. The symposium and workshop served as a forum for researchers using CA and VR to present and discuss their current work, and show what is already possible in behavioral research, to inspire future research and to enhance further methodological development. Besides biological questions that could be answered by using CA and VR, the underlying technical background presented showed that researchers can choose from a quite diverse pool of various techniques to create CA and VR. The symposium exemplified that the most sophisticated animation techniques are not always required, instead, the type of animation needed depends on the research question as well as the test animal’s visual capacities. Hence, researchers can choose between simple 2-dimensional CAs to more complex and sophisticated 3-dimensional CAs and VR, depending on time resources and research interests.
To share and exchange this knowledge and to demonstrate today’s use of CA and VR in recent research to a broader audience, we took advantage of the possibility to guest-edit a special issue on “Computer animations and VR in animal behavior research”. We composed a collection of new articles reflecting both technical and conceptual background and the application in biological and specifically behavioral science. The papers also show problems and limitations that may arise, depending on the research question or the study animal. These articles are aimed to address a broad range of scientists looking for new ideas and new techniques in future research.
State-of-the-art and New Perspectives
In this special issue, we present a collection of new original research articles and reviews covering different techniques and approaches on how to create and present virtual stimuli to real test animals. We start with a discussion on the technical background in light of current technical development and provide an update of previous considerations that were discussed in early reviews on these methods (Oliveira et al. 2000; Baldauf et al. 2008).
Chouinard-Thuly et al. (2017) provide a complete overview of the main conceptual and technical considerations for creating, presenting, and validating a newly generated computer-based stimulus and/or VR. In their review, the authors presented possible techniques for the creation of a virtual stimulus, and emphasized the importance of the perceptual capacities of test animals. Thereby, Chouinard-Thuly et al. (2017) suggest a workflow and give advice on how to avoid common pitfalls.
Although the virtual stimuli have many advantages, the use of virtual stimuli, however, has its limits and might not be suitable to answer every research question. Powell and Rosenthal (2017) provide a thorough overview of both the benefits and fallbacks of using CA stimuli. They advise that hypotheses investigated using CA must be specific, as computer generated stimuli inherently involve many levels of visual manipulations and contain technical limitations.
Focusing specifically on constraints imposed by displays built for the human eye, Tedore and Johnsen (2017) provide a very approachable and user-friendly MATLAB tool for adjusting RGB values to simulate colors on a display. Their approach makes it possible to closely mimic real-world colors on screen, based on how they are perceived by the animal’s visual system with regard to its spectral sensitivities. Their tool is particularly useful for researchers investigating color signals with virtual stimuli and environments.
After addressing the technical and conceptual aspects of CA and VR in this special issue, different methods ranging from simple to complex (2D animation, 3D animation, and VR) stimuli or the environment are introduced and their possible application in research is highlighted.
Gerlai (2017) reviews how technologically simple 2D animations allowed for a precise description of shoaling and anti-predatory behavior of live zebrafish when driven by visual cues of virtual conspecifics or predators. Gerlai and colleagues were able to test the effect of slight changes in speed, locations, and colors on the normal behavior of zebrafish. Specifically, Gerlai and collaborators used this knowledge to investigate biological mechanisms and disease models. They provide evidence that the dopaminergic system only is activated by the sight of conspecifics, but not when the animated images were different. Furthermore, they show that zebrafish embryos exposed to alcohol show altered social behaviors as adults. In this case, CAs provided the precision required to investigate these questions.
Using 2D animations created with Microsoft PowerPoint, Balzarini et al. (2017) provide in this issue compelling evidence that cichlids respond to the darkness of facial stripes of their opponent, and that these act as a signal of aggressive status. This is a powerful example of an advantage of CAs, as they allow for a targeted manipulation of a single trait, such as the black facial stripes expressed in Neolamprologus pulcher, without the alteration of the behavior or emotional state of the stimulus fish, which often changes with visual color markings. They were, therefore, able to show that by varying only the hue of the stripes without any behavioral change of the presented virtual fish stimulus, that is, aggressive state, real test fish darkened their stripes when seeing a virtual opponent, especially if the virtual opponent had pale stripes. The study of Balzarini and colleagues nicely demonstrates the great benefits of animated stimuli to allow a systematic manipulation, simultaneously controlling for visual features and behavior, which is not possible in live fish.
As an example of a successful interdisciplinary collaboration between biologists and computer scientists, Müller et al. (2017) and Gierszewski et al. (2017) demonstrate how knowledge of both fields can be combined to create custom-made software for the use of 2D or 3D animations in behavioral science. Here, software development was enhanced by freely available high-quality design and animation tools. Introducing a technologically complex method to create 3D animation and animate virtual stimuli, Müller et al. (2017) present their innovative toolchain “FishSim” for simulating animated fish models for the use in dichotomous mate-choice experiments. Here, model design and workflow of their toolchain are described in a case study with sailfin mollies Poecilia latipinna but their toolchain can also be applied to other fish models. The toolchain developed by Müller et al. (2017) was then validated by Gierszewski et al. (2017), who presented virtual 3D sailfin mollies to live test fish in binary choice experiments. They confirmed that live sailfin mollies are as attracted to the virtual fish models as they are to video playbacks of fish and even live stimulus fish. The study design used by Gierszewski et al. (2017) exemplifies the experiments that can be performed to validate the usage of newly created CA stimuli for investigating mate choice in fish, with association time as the validation parameter.
CA stimuli are particularly useful to thoroughly investigate properties of the visual perception of a single species. Woo et al. (2017) describe in this issue how they use CA to study visual speed sensitivity in the Jacky dragon Amphibolurus muricatus using an instrumental learning paradigm. They either presented sole 2D animations of random-dot kinematograms differing in speed, coherence, and direction, or they combined these with the presentation of 3D animations of different invertebrates (cricket, mite, and spider). These animations served as secondary reinforcers to test whether lizards were able to predict their appearance based on movement direction of the animated dots. They found that Jacky dragons performed better at high speed visual motion than low speed motion, except if the strength of the signal was greater in the low speed condition. They hypothesize that Jacky dragon’s motion perception is tuned to detect and discriminate ecologically relevant motion, for example, social displays of conspecifics, from the surrounding environment. They discuss their findings in the context of signal evolution and provide implications for predator and prey detection.
Despite the numerous advantages of CA, one of the biggest limitations of CA is the lack of interaction between the presented virtual stimulus and the real test animal. For the study of visual communication in animals, interaction between both, the stimulus and the test animal, may be crucial to gain reliable results on the underlying mechanisms since some animals respond differently to non-interactive stimuli than to live stimuli which can interact with the test animal, as is revealed in the following study by Ware et al. (2017). Using a classic video playback approach, Ware and colleagues show how interactivity can be achieved with the help of a closed-loop teleprompter system, and how it can then be thoroughly manipulated to investigate social interactivity in the courtship of pigeons (Columba livia). By comparing interactive (live condition) and non-interactive (playback) presentations, Ware et al. (2017) show in this issue that, overall, social interactivity is crucial for maintaining courtship in pigeons. They could show that a behavioral response during courtship, but not during rivalry interactions, was unaffected by spatial manipulation (facing direction of the virtual partner), but that social interactivity was sensitive to temporal contiguity (delayed response). Their results show that one has to keep in mind that social interactions may rely on fine-tuned behavioral patterns in time and space and, hence, results gained in studies using virtual stimuli may be affected by a lack of interactivity and should be evaluated with care.
To understand how animals interact with their environment in terms of spatial navigation it might be beneficial for some research questions to use a system that responds to the observing animal. In this issue, Dolins et al. (2017) present the use of a first-person computer game setup in a comparative study with humans, bonobos, and chimpanzees, to investigate spatial cognition. In their article, Dolins et al. (2017) describe the training process of the test subjects to navigate a virtual 3D maze using a joystick and demonstrate that a system primarily created for humans can also be adopted for the use with non-human primates, particularly great apes. Here, they give advice on best practices and review results of a previous study where using of virtual simulated environments. They found that, overall, navigation performance of bonobos and chimpanzees was not qualitatively different from that of human subjects. The review by Dolins et al. (2017) nicely demonstrates that the use of virtual simulated environments allows a direct comparison of individual and species-specific cognitive abilities in navigation performance in 3 species using identical virtual environments under standardized conditions.
The system of VR which provides perspective correct representations of an artificial environment makes this technique unique for the study of spatial cognition. The presented stimuli always show motion parallax in response to the observing animal’s movements and, hence, it is possible to simulate the animal’s presence in the virtual world. In this issue, Thurley and Ayaz (2017) discuss the state-of-the-art for using VR in rodents, and provide a comprehensive overview of currently used VR paradigms for these animals, as well as recent results from related research. Although VR is particularly used to investigate visual modalities, the authors also highlight studies in which VR was adopted to study somatosensory processing. They emphasize the potential of exploiting VR to investigate the neural foundations of behavior in future research.
Prospects for Future Research
With this special issue, we provide useful information on the state-of-the-art of using CA and VR. We present possible merits, as well as DOs and DON’Ts for researchers interested in using CA and VR in their future research in animal behavior and related disciplines.
Since the use of these promising techniques still has some limitations, further improvements have to be achieved to overcome these boundaries (e.g. the lack of interactivity). Methodological development has to continue and software solutions need to be refined to support future research. Here, researchers could also greatly benefit from an open source policy and a research community that is willing to share their knowledge and achievements to improve techniques and experimental design when using CA and VR. Shareable and freely available solutions, for example, the program anyFish (Ingley et al. 2015), help those that are beginners in the field, with creation and animation of stimuli, and might further even increase comparability between research studies by sharing virtual models.
Overall, CA and VR offer promising possibilities for studying visually mediated behavior in a very standardized way but for application to many different contexts. Whole artificial environments can be created, and single visual traits and behavior can be varied in a way which is nearly impossible with live test animals. Instead, CA and VR provide the possibility to resign from invasive manipulations directly at the animal, for example, by surgery or drug treatments. Therefore, the implementation of CA and VR in research can serve to fulfill the 3Rs principle (Richmond 2010). The advantages are clearly that real test animals can be replaced by virtual counterparts and therefore the overall number of experimental animals can be reduced. The possibility to create highly controlled and standardized testing paradigms when applying CA or VR makes it further possible to refine experimental procedures and setups, leading to more reliable results on the one hand and to an improved reproducibility (the 4thR) of experiments on the other hand.
Especially the development of VR, which is currently highly promoted by the entertainment industry, can be expected to expand widely in the future, leading to more sophisticated and possibly cheaper technical solutions for the use in the broad spectrum of fascinating questions in animal behavior and related disciplines. We hope that this special issue will not only inspire more researchers to use virtual stimuli of any form in their research, but also cultivate a sharing and collaborative community.
We thank all the participants of the “Virtual Reality” Symposium held at the Behaviour2015 conference for their contributions to the symposium and for inspiring this special issue. We gratefully thank all authors who made this special issue possible by contributing their articles. We also thank Current Zoology for realizing this special issue and particularly Zhi-Yun Jia for his support and advice during the editorial process. S.G. and K.W. were funded by a research grant of the Deutsche Forschungsgemeinschaft (DFG; Wi 1531/12-1). L.C.T. was funded by the Quebec Center for Biodiversity Research and the Fonds de Recherche du Québec—Nature et Technologies. We thank S. Ahmad for proofreading the manuscript.