RGB displays effectively simulate millions of colors in the eyes of humans by modulating the relative amount of light emitted by 3 differently colored juxtaposed lights (red, green, and blue). The relationship between the ratio of red, green, and blue light and the perceptual experience of that light has been well defined by psychophysical experiments in humans, but is unknown in animals. The perceptual experience of an animal looking at an RGB display of imagery designed for humans is likely to poorly represent an animal’s experience of the same stimulus in the real world. This is due, in part, to the fact that many animals have different numbers of photoreceptor classes than humans do and that their photoreceptor classes have peak sensitivities centered over different parts of the ultraviolet and visible spectrum. However, it is sometimes possible to generate videos that accurately mimic natural stimuli in the eyes of another animal, even if that animal’s sensitivity extends into the ultraviolet portion of the spectrum. How independently each RGB phosphor stimulates each of an animal’s photoreceptor classes determines the range of colors that can be simulated for that animal. What is required to determine optimal color rendering for another animal is a device capable of measuring absolute or relative quanta of light across the portion of the spectrum visible to the animal (i.e., a spectrometer), and data on the spectral sensitivities of the animal’s photoreceptor classes. In this article, we outline how to use such equipment and information to generate video stimuli that mimic, as closely as possible, an animal’s color perceptual experience of real-world objects.
Key words: color vision, computer animation, perception, video playback, virtual reality.
Computer animation and video manipulation are powerful tools in the study of animal behavior, allowing for precise control of potentially correlated traits and behavior (Veen et al. 2013; Levy et al. 2014; Tedore and Johnsen 2015), and for the manipulation of morphology and movement patterns that would be impossible in live animals (Hebets and Uetz 2000; Peters and Evans 2003; Tedore and Johnsen 2013; Woo and Rieucau 2015). One of the long-standing limitations of such methodologies, however, has been that RGB displays are only designed to simulate realistic real-world colors as perceived by the human visual system (D’Eath 1998; Fleishman et al. 1998; Fleishman and Endler 2000; Westland et al. 2012).
The perception of color is neurally encoded by the ratios of quantum catches across an animal’s different photoreceptor classes. Color-matching psychophysical experiments in humans have determined how the ratio of red, green, and blue primaries, such as those found in an RGB display, corresponds to human color perception (Westland et al. 2012). Similar psychophysical experiments have been conducted in goldfish (Neumeyer 1992), but not with the spectral resolution necessary to derive the types of color matching functions used to calibrate RGB displays for humans. Few studies of this nature have been attempted because of the length of time necessary to train animals to consistently choose 1 color over another, followed by the numerous different intensities of primaries that must be tested to determine which combination of primary intensities animals perceive as equivalent to the training color. However, such lengthy training experiments are not necessary in order to approximate real-world colors for another animal’s visual system. Here we outline a simple method to simulate the color of real-world objects as seen by other animals on an RGB display. This is accomplished, in brief, by modeling the quantum catch of each photoreceptor class in response to the stimulus to be simulated and comparing these values to those generated by all possible combinations of RGB values in order to find the best match.
Although many colors can be simulated in this way across diverse taxa and types of visual systems, there are limitations inherent in the spectral locations of the RGB phosphors. A wide range of taxa have 4 or more classes of photoreceptor, and all taxa so far studied have spectral sensitivities centered over different parts of the spectrum than we do, with many having UV sensitivity as well (Land and Nilsson 2012). This means that each RGB phosphor may not stimulate each of another species’ photoreceptor classes as independently as they do ours. For example, if the wavelength of peak intensity of one of the RGB phosphors were to lie midway between the peak sensitivities of 2 photoreceptor classes, as is true of the blue phosphor in relation to the violet and blue cones of many birds (Figure 1), then it would be impossible to stimulate 1 class without stimulating the other class roughly equally. In addition, ultraviolet pigments with peak absorbances below 380 nm will catch very little of a blue phosphor’s light, and, in animals having such pigments, it will generally be impossible to simulate objects with any significant reflectance in the ultraviolet portion of the spectrum.
In the present article, we outline a method for creating RGB imagery that is as perceptually realistic as possible to an animal species whose spectral sensitivities are known (or can be inferred from related species). Although the mathematics involved is simple and can be done by anyone, we have, for user convenience, included 2 files of
Previous authors alluded to a similar method employing linear algebra, but did not guide users through the calculations or provide a program capable of doing the calculations for them (Fleishman et al. 1998). Because of the higher computational power available to most users nearly 20 years later, we are able to propose a simpler method that will be more intuitive to most users. It is our hope that by outlining a method employing simpler mathematics and by providing users with a MATLAB script capable of calculating optimal RGB values automatically, users will have the knowledge, confidence, and tools at their disposal to make their video playback experiments as color realistic as possible.
Because the method requires measurements of reflectance, radiance, and irradiance, we recommend the reader familiarize themselves with previously published introductions to these terms and how to measure them (Johnsen 2012, particularly chapters 2, 3, and 9; Johnsen 2016). The calculations outlined below assume knowledge of how to collect and interpret spectral data.
The method also requires users to calculate spectral sensitivity curves for each of their study species’ photoreceptor classes. For some species, this can be as simple as inputting a photoreceptor’s wavelength of peak absorbance into an opsin template curve (e.g., Govardovskii 2000), whereas in other species, a number of other effects must be accounted for. These may include the filtering effects of ocular media transmission, oil droplets, and adjacency or tiering of rhabdoms, as well as interreceptor differences in the lengths of photoreceptive segments and their absorption coefficients. When in doubt, it is always prudent to confer with an animal vision scientist to determine which parameters are important to one’s study system.
Measurements and calculations
A perfect match between the desired natural quantum catch values and the best possible quantum catch values from an RGB display will often not be possible. However, this mismatch can be mitigated in the eyes of the study animal by presenting the animated objects against an adapting background with a similar degree of chromatic mismatch (i.e., shift in color cast) to the animated objects. Thus, the first step is to find the best possible set of quantum catch values attainable for the background. Then one can calculate the best possible set of quantum catch values for the animated objects while specifying a similar shift in color cast for these objects, which will result in a color constant scene that will be perceived as close to a naturally lit scene as possible.
Suppose an animal with multiple photoreceptor channels is viewing a natural (denoted “N”) background (e.g., foliage) that is large enough to be an adapting stimulus. The quantum catch (to within a constant that we ignore for this analysis) of the i-th channel is:
where Ri(λ) is the spectral sensitivity of the i-th channel (including the effects of ocular media transmission, etc.), L(λ) is the background radiance, λ denotes wavelength, and λmin and λmax define the wavelength range over which the i-th channel has significant sensitivity. The background radiance can either be measured directly or (if the surface is matte and non-transmissive) calculated as the product of the background reflectance and the illuminating irradiance. An appropriate background radiance spectrum could, for example, be a measured or calculated radiance spectrum of the predominant object present in the natural background, or a wider-angle spectral radiance measurement taken in the animal’s natural habitat, directed in the animal’s typical angle of view, which would result in a spectral average of various objects (e.g., leaves, branches) weighted by the angular area subtended by each object in the detector’s field of view.
Our first goal is to find an RGB triplet for the monitor such that the quantum catches for all of the photoreceptor channels of the viewer are a constant multiple b of those for the natural background. In other words, the monitor background should look like a dimmer (or possibly brighter) version of the natural background, but with the same color as viewed by the animal. More specifically, we want to find values of R, G, and B, so that, for each photoreceptor channel of the viewer:
These bis:, which we term brightness factors, are (from Equation 3):
I(xR) is the intensity of the phosphor in arbitrary units as a function of level x (i.e., 0–255; sometimes termed “gray level”). This function depends on the color space of the monitor. For sRGB (one of the most common color spaces for LCD monitors, see International Color Consortium [ICC], www.color.org/chardata/rgb/srgb.xalter), it is:
L(λ) is the spectral radiance of the phosphor, again in arbitrary (but quantal) units. It is measured from a calibrated monitor screen where the red value is set to any approximately middle magnitude (e.g., 128) and the green and blue values are set to zero. Because the shapes of the phosphor spectra are more or less independent of level, the radiance of a given phosphor in arbitrary units is the product of I and L. The units in this situation are unimportant, because they only affect the magnitude of the brightness factors, not their relationship to each other.
As mentioned above, the quantum catches from the green and blue phosphors of the monitor are found in an analogous fashion, so the goal is to find an RGB triplet such that the bis are as close to each other as possible by minimizing Equation 4. Given the current speed of computation, this can be done by an exhaustive search of all ∼16 million RGB triplets. Two final refinements remain however. Because the RGB triplet should ideally give values in the medium range of the monitor, it is useful to set 1 of the 3 RGB values to a middle magnitude (e.g., 128). However, the final, optimal RGB triplet should also not include any values of 255 or 0, i.e., it should not saturate in either the high or low direction. The accompanying MATLAB script thus prompts the user to input the desired magnitude of the green phosphor for the background. It will then hold that value constant while testing the fit of all other possible combinations of red and blue values. The script can be rerun a number of times until none or as few as possible of the RGB outputs for the background and the various color patches contain a 0 or 255. Now that the brightness factors are chosen, and are ideally close to one another; one can then find the optimal RGB triplet to mimic any colored object that sits on the natural background. As before, the quantum catches of the natural object are calculated using Equation 1. Once these are found, we need to find an RGB triplet that results in quantum catches that match the natural values, after they have been multiplied by the brightness factors. In other words, if the monitor background has half the quantum catches as the natural background, then the monitor object must also have half the quantum catches. Specifically, one must find the RGB triplet that minimizes the sum of squared differences between monitor and nature for each photoreceptor channel:
If the user opts to predict spectral radiance from the product of habitat spectral irradiance and spectral reflectance (rather than measuring spectral radiance directly in the appropriate habitat), theEndler (1993). Although it is better to measure the habitat irradiance specific to one’s study species, these spectra are good approximations for terrestrial habitats in case users do not have easy access to their study species’ natural habitat. Aquatic habitats are much more variable than terrestrial ones due to the fact that water attenuates short and long wavelengths more than medium wavelengths, which means that the spectral composition of irradiance changes dramatically with depth. Aquatic habitats also vary according to the amount of particulate and dissolved matter suspended in the water column (Jerlov 1976). We therefore do not provide any aquatic irradiance spectra, but instead urge practitioners to measure the irradiance spectra unique to their study species’ habitat, or to contact others who have measured similar habitats.
If the study animal is meant to be viewing a horizontally facing target, then sidewelling irradiance, rather than downwelling irradiance, is the more appropriate measure to use to calculate predicted radiance from reflectance data. Another important consideration is that, in order for spectral radiance to be accurately predicted by the multiplication of spectral reflectance by spectral irradiance, the surface to be simulated has to be matte and non-transmissive. If the object to be simulated is shiny and/or transmissive (e.g., most leaves and many animal surfaces), then one can only obtain realistic radiance values by measuring radiance directly from the object in the correct orientation in its natural environment (Johnsen 2016). Indeed, direct measurements of spectral radiance will nearly always be more accurate than estimates calculated from the product of spectral reflectance and spectral irradiance, because few natural surfaces behave like perfect diffuse reflectors. We therefore suggest that users measure the radiances of the background and objects embedded in the scene directly whenever possible.
It is important to take one’s spectral measurements in such a way that the perceived relative brightness and color cast of all of the objects and the background in the virtual scene are realistically portrayed. For example, it would be ill-advised to take radiance spectra of different objects to be portrayed in the same scene during different times of day or under different weather conditions. It would also be ill-advised to mix direct measurements of spectral radiance under 1 set of conditions (including location, time of day, and weather) with spectral radiance calculated from the product of spectral reflectance and spectral irradiance measured under a different set of conditions. In addition, all radiance spectra must be in the same units; whether these are absolute or relative units does not matter. If any of these guidelines are violated, then neither chromatic nor achromatic contrasts are likely to be rendered correctly in the RGB values output by the method (and accompanying MATLAB script). In general, as stated in the previous paragraph, the best color rendering will be produced from direct measurements of background and object spectral radiance taken under constant conditions in the appropriate habitat.
Evaluating and adjusting fit
In addition to ensuring that the background optimal RGB values do not contain a 0 or 255 (as described in “Measurements and Calculations” section), it is essential to carefully inspect the relative magnitudes of each photoreceptor class’ optimum brightness factor bi, as well as the match between target and achievable quantum catches, all of which are output by the MATLAB script. If the brightness factors are very different from one another, it is worth rerunning the script with different background radiance spectra, representing the range of different backgrounds that would be acceptable to the user, to see which one provides the best fit (i.e., the most similar set of bis.). The target quantum catches for the color patches that are output by the script take into account the adapting background (i.e., the different brightness factors). This means that if the brightness factors are very different from one another, the animal’s color constancy mechanism may not be able to fully compensate for the resulting color shift even if the target versus actual quantum catch values are similar. If the brightness factors are so different from one another that the best obtainable color patch RGB values include a 0 or 255, then the brightness factors will not be consistent from 1 color patch to another and a color constant match may be impossible. To try to fix this, one can first adjust the overall brightness of the scene by inputting different levels for the G phosphor of the background. This may bring the RGB values of the saturated color patch(es) to within the dynamic range of the display device. However, if adjusting the overall brightness of the scene fails to get rid of all values of 0 or 255, one can assess whether a color patch with one or both of these values is off by a lot or just a little by checking the desired versus actual quantum catch values output by the MATLAB script. If they are similar, then the color patch is only slightly saturated and will fit into the color cast of the scene. If they are off by a lot, as will often be true for animals with peak spectral sensitivities below 380 nm, it is best to rerun the script, treating the animal as if it does not have an ultraviolet photoreceptor class. This effectively shifts the color cast of the scene to be very low in ultraviolet light and will generally result in an overall better fit for the remaining photoreceptor classes.
To convince oneself that the input spectra were measured and applied appropriately, it is always a good idea to run the MATLAB script for a human observer. If the optimal color rendering for a human observer looks similar to the object in real life, then one can rest assured that one’s spectral measurements were done correctly and that the color rendering for one’s study animal should also be accurate (within the constraints of the animal’s visual system, of course).
Finally, it is important to note that although this methodology ensures the best possible fit, it cannot guarantee a “good enough” fit. If color realistic stimuli are essential for the experimental question being investigated, as for example when studying the role of color in mate choice, it is prudent to carry out validation procedures as described in Choinard-Thuly et al. (this issue) to determine whether subjects make similar choices when presented with live stimuli.
Proofs of concept
As a proof of concept, we tested how the colors of a GretagMacbeth color chart rendered by our method versus the published RGB coordinates for the chart compared in appearance to a human observer. We used the wavelengths of peak sensitivity published by Bowmaker and Dartnall (1980), the opsin template equation developed by Govardovskii et al. (2000), and ocular media transmittance data published by Packer and Williams (2003). Color checker reflectance spectra and a D65 irradiance spectrum were taken from Ohta (1997) and the International Commission on Illumination website. A 50% reflective spectrally neutral gray was input as the background. The color cast of our calculated RGB values was a bit off from that of the standard values published by GretagMacbeth (Pascale 2006), but within-chart chromatic contrasts were similar (Figures 2 and 3).
We then tested how good of a fit we could obtain between desired and actual quantum catch values in birds having a long wavelength ultraviolet photoreceptor (“VS birds,” λmax ≈ 409) versus those having a short wavelength ultraviolet photoreceptor (“UVS birds,” λmax ≈ 370). To calculate avian spectral sensitivities, we used mean spectral sensitivity parameters for VS and UVS birds using data tabulated from a number of species in Hart and Vorobyev (2005) and optical media transmittance data provided by Almut Kelber. The spectra we attempted to simulate came from previously published plumage reflectance spectra and included a variety of different colors derived from diverse pigment types (Figure 4; Prum and Torres 2003; Shawkey and Hill 2006; Doucet et al. 2007; Mendes-Pinto et al. 2012; Ruiz-Rodriguez et al. 2013; Bleiweiss 2015). Appropriate irradiance spectra for the habitats in which each plumage patch is normally found were supplied by John Endler and were similar to those published in Endler (1993). For the purposes of the exercise, and because the color patches were not to be presented against an adapting background, we used the mean of all 4 brightness factors to calculate target quantum catch values. This ensured our target quantum catch values were as close as possible to what they would be in nature, rather than adjusted to match the color cast of the best obtainable background quantum catch values. In Figure 5, we show the differences in optimal color rendering for humans, VS birds, and UVS birds. We also include accompanying bar charts that show the differences in desired versus actual quantum catch values in VS and UVS birds. As would be expected from the small amount of overlap between the spectral sensitivity of UVS birds’ ultraviolet photoreceptors and the blue phosphor’s radiance spectrum, we find that the best attainable RGB quantum catch of UVS birds’ ultraviolet photoreceptor is a very poor match to the desired natural quantum catch for all but the most UV-absorbent color patches, e.g., Eudocimus ruber.
To demonstrate what a complete image of a bird against a natural background would look like with colors rendered for a human versus a VS bird versus a UVS bird, we collected radiance spectra of a taxidermic mount of Merops apiaster from the Lund University Biological Museum’s zoological collection. We measured its turquoise breast, yellow throat, reddish-brown crown, and green epaulettes under direct sunlight against natural vegetation. Background spectral radiance was measured immediately after we took the M. apiaster measurements. Measurements were taken with an Ocean Optics USB2000+ spectrometer fitted with a 3 degree Gershun tube and calibrated by an Ocean Optics DH-3plus calibrated light source. The reason we opted to use direct measurements of radiance, rather than the reflectance measurements published in Ruiz-Rodriguez et al. (2013), was because we found that estimates of spectral radiance from the product of spectral reflectance and irradiance did not yield as realistic color rendering for a human observer. As described in the “Measurement Considerations” section, this likely means that M. apiaster’s feathers do not act as perfect diffuse reflectors. We ran our measured radiance spectra through our MATLAB script for human, UVS, and VS avian viewers, and then used the Color Replacement tool in Adobe Photoshop to adjust the colors of an RGB photograph taken from the internet to be appropriate for these 3 different viewers. With this tool, one can select a replacement set of RGB coordinates and then paint over the color patch whose color is to be replaced. The tool preserves textural detail by varying hue, saturation, and lightness to a similar degree as the replaced color, but within the newly specified color neighborhood. It does not, however, change the overall brightness of the replaced color to match the brightness of the new color. This was accomplished by subsequently selecting each color patch individually and adjusting its brightness until its mean RGB values matched that of the specified color.
For both of the avian proof of concept exercises above, the brightness factors and desired versus actual quantum catch values calculated by the MATLAB script were a good fit for VS birds, but not for UVS birds. For example, for UVS birds viewing M.apiaster, there was a 22-fold difference between the brightness factor for the blue photoreceptor class (SWS2) and the brightness factors for the remaining photoreceptor classes (UVS, MWS, and LWS). Due to the resulting highly skewed blue color cast of the scene, the turquoise breast had to be more saturated than the dynamic range of the RGB display could allow, resulting in a blue phosphor value of 255 and a red phosphor value of 0, and a 13-fold difference between desired versus actual green photoreceptor (MWS) quantum catch values. Thus, as per our recommendation above, for the proof of concept exercises pictured in Figures 5 and 6, we reran the MATLAB script without the ultraviolet receptor, which essentially mimicked a scene with low ultraviolet irradiance and resulted in a better fit for the non-ultraviolet photoreceptor classes.
By outlining how to calculate the best possible RGB representations of real objects through animal eyes, it is our hope that behavioral researchers will now have the tools to make the colors of their animations and manipulated videos as realistic as possible to the eyes of their study species. Currently, the accompanying MATLAB script can calculate optimal RGB values for di-, tri-, and tetrachromats, but it is our intent to add functionality for animals with even more spectral classes in the future. For the most up to date version of the MATLAB script, users can visit www.cynthia.tedore.com/animation.
The authors thank Stefanie Gierszewski, Laura Chouinard-Thuly, and Klaudia Witte for organizing this Special Issue of Current Zoology on computer animation. Their interest in the topic was the impetus that led us to develop the methodology presented in this article. Thanks to Maria Mostadius, the Lund University curator of zoological collections who located and handled the taxidermic mount of M. apiaster, and to José Luís Barros for allowing us to use and modify his photograph of M. apiaster. The authors thank also John Endler for contributing his 4 major classes of daylight terrestrial habitat spectral irradiances to the user input file. Finally, the authors thank Nick Brandley, Eleanor Caves, Almut Kelber, Olle Lind, Dan Speiser, Katie Thomas, and 2 anonymous reviewers for insightful comments on the manuscript.
This work was funded by the Knut and Alice Wallenberg Foundation.