Conventional and hyperspectral time-series imaging of maize lines widely used in field trials

Abstract Background Maize (Zea mays ssp. mays) is 1 of 3 crops, along with rice and wheat, responsible for more than one-half of all calories consumed around the world. Increasing the yield and stress tolerance of these crops is essential to meet the growing need for food. The cost and speed of plant phenotyping are currently the largest constraints on plant breeding efforts. Datasets linking new types of high-throughput phenotyping data collected from plants to the performance of the same genotypes under agronomic conditions across a wide range of environments are essential for developing new statistical approaches and computer vision–based tools. Findings A set of maize inbreds—primarily recently off patent lines—were phenotyped using a high-throughput platform at University of Nebraska-Lincoln. These lines have been previously subjected to high-density genotyping and scored for a core set of 13 phenotypes in field trials across 13 North American states in 2 years by the Genomes 2 Fields Consortium. A total of 485 GB of image data including RGB, hyperspectral, fluorescence, and thermal infrared photos has been released. Conclusions Correlations between image-based measurements and manual measurements demonstrated the feasibility of quantifying variation in plant architecture using image data. However, naive approaches to measuring traits such as biomass can introduce nonrandom measurement errors confounded with genotype variation. Analysis of hyperspectral image data demonstrated unique signatures from stem tissue. Integrating heritable phenotypes from high-throughput phenotyping data with field data from different environments can reveal previously unknown factors that influence yield plasticity.


Abstract:
Background: Maize (Zea mays ssp. mays) is one of three crops, along with rice and wheat, responsible for more than 1/2 of all calories consumed around the world. Increasing the yield and stress tolerance of these crops is essential to meet the growing need for food. The cost and speed of plant phenotyping is currently the largest constraint on plant breeding efforts. Datasets linking new types of high throughput phenotyping data collected from plants to the performance of the same genotypes under agronomic conditions across a wide range of environments are essential for developing new statistical approaches and computer vision based tools.
Findings: A set of maize inbreds -primarily recently off patent lines --were phenotyped using a high throughput platform at University of Nebraska-Lincoln. These lines have been previously subjected to high density genotyping, and scored for a core set of 13 phenotypes in field trials across 13 North American states in two years by the Genomes to Fields consortium. A total of 485 GB of image data including RGB, hyperspectral, fluorescence and thermal infrared photos has been released.
Conclusions: Correlations between image-based measurements and manual measurements demonstrated the feasibility of quantifying variation in plant architecture using image data. However, naive approaches to measuring traits such as biomass can introduce nonrandom measurement errors confounded with genotype variation.
Analysis of hyperspectral image data demonstrated unique signatures from stem tissue. Integrating heritable phenotypes from high-throughput phenotyping data with field data from different environments can reveal previously unknown factors influencing yield plasticity.
is only an attempt to get our raw data out in a form that other researchers can also use it without waiting the additional 1-3 years it would take (if the timelines of prior plant phenotypic papers are a useful guide) for us to tell our own biological story with this dataset.
Additional Information:

Question Response
Are you submitting this manuscript to a special series or article collection?
No Experimental design and statistics Full details of the experimental design and statistical methods used should be given in the Methods section, as detailed in our Minimum Standards Reporting Checklist. Information essential to interpreting the data presented should be made available in the figure legends.
Have you included all the information requested in your manuscript?
Yes Resources A description of all resources used, including antibodies, cell lines, animals and software tools, with enough information to allow them to be uniquely identified, should be included in the Methods section. Authors are strongly encouraged to cite Research Resource Identifiers (RRIDs) for antibodies, model organisms and tools, where possible.
Have you included the information requested as detailed in our Minimum Standards Reporting Checklist?

Yes
Availability of data and materials All datasets and code on which the conclusions of the paper rely must be either included in your submission or deposited in publicly available repositories (where available and ethically appropriate), referencing such data using a unique identifier in the references and in the "Availability of Data and Materials" section of your manuscript.

Data Description Background
The green revolution created a signi cant increase in the yields of several major crops in the 1960s and 1970s, dramatically reducing the prevalence of hunger and famine around the world, even as population growth continued. One of the major com-ponents of the green revolution was new varieties of major grain crops produced through conventional phenotypic selection with higher yield potentially. Since the green revolution, the need for food has continued to increase, and a great deal of e ort in the public and private sectors is devoted to developing crop varieties with higher yield potential. However, as the low hanging fruit for increased yield vanish, each new increase in yield requires more time and resources. Recent studies have demonstrated that yield increases may have slowed or stopped for some major grain crops in large regions of the world [1]. New approaches to plant breeding must be developed if crop production continues to grow to meet the needs of an increasing population around the world. The major bottleneck in modern plant breeding is phenotyping. Phenotyping can be used in two ways. Firstly, by phenotyping a large set of lines, a plant breeder can identify those lines with the highest yield potential and/or greatest stress tolerance in a given environment. Secondly, su ciently detailed phenotyping measurements from enough di erent plants can be combined with genotypic data to identify regions of the genome of a particular plant species which carry bene cial or deleterious alleles. The breeder can then develop new crop varieties which incorporate as many bene cial alleles and exclude as many deleterious alleles as possible. Phenotyping tends to be expensive and low throughput, yet as breeders seek to identify larger numbers of alleles each with individually smaller e ects, the amount of phenotyping required to achieve a given increase in yield potential is growing. High throughput computer vision based approaches to plant phenotyping have the potential to ameliorate this bottleneck. These tools can be used to precisely quantify even subtle traits in plants and will tend to decrease in unit cost with scale, while conventional phenotyping, which remains a human labor intensive processes, does not.
Several recent pilot studies have applied a range of imageprocessing techniques to extract phenotypic measurements from crop plants. RGB (R: Red channel; G: Green channel; B: Blue channel) camera technology, widely used in the consumer sector, has also been the most widely used tool in these initial e orts at computer vision based plant phenotyping [2,3,4,5]. Other types of cameras including uoresence [6,7] and NIR (near-infrared) [6,8,9] have also been employed in high throughput plant phenotyping e orts, primarily in studies of the response of plant to di erent abiotic stresses.
However, the utility of current studies is limited in two ways. Firstly, current analysis tools can extract only a small number of di erent phenotypic measurements from images of crop plants. Approximately 150 tools for analyzing plant image data are listed in a eld speci c database, however the majority of these are either developed speci cally for Arabidopsis thaliana which is a model plant, or are designed speci cally to analyze images of roots [10]. Secondly, a great deal of image data is generated in controlled environments, however, there are comparatively few attempts to link phenotypic measurements in the greenhouse to performance in the eld. However, one recent report in maize suggested that more than 50% of the total variation in yield under eld conditions could be predicted using traits measured under controlled environments [5].
Advances in computational tools for extracting phenotypic measurements of plants from image data and statistical models for predicting yield under di erent eld conditions from such measurements requires suitable training datasets. Here, we generate and validate such a dataset consisting of high throughput phenotyping data from 32 distinct maize (Zea mays) accessions drawn primarily from recently o -patent lines developed by major plant breeding companies. These accessions were selected speci cally because paired data from the same lines exists for a wide range of plant phenotypes collected in 54 distinct eld trials at locations spanning 13 North American states or provinces over two years [11]. This extremely broad set of eld sites captures much of the environmental variation among areas in which maize are cultivated with total rainfall during the growing season ranging from 133.604 mm to 960.628 mm (excluding sites with supplemental irrigation) and peak temperatures during the growing season ranging from 23.5 • C to 34.9 • C. In addition, the same lines have been genotyped for approximately 200,000 SNP markers using GBS [11]. Towards these existing data, we added RGB, thermal infra-red, uorescent and hyperspectral images collected once per day per plant, as well as detailed water-use information (single day, single plant resolution). At the end of the experiment, 12 di erent types of ground-truth phenotypes were measured for individual plants including destructive measurements. A second experiment focused on interactions between genotype and environmental stress, collecting the same types of data described above from two maize genotypes under well watered and water stressed conditions [12]. We are releasing this curated dataset of high throughput plant phenotyping images from accessions where data on both genotypic variation and agronomic performance under eld conditions is already available. All data was generated using a Lemnatec designed high throughput greenhouse-based phenotyping system constructed at the University of Nebraska-Lincoln. This system is distinguished from existing public sector phenotyping systems in North America by both the ability to grow plants to a height of 2.5 meters and the incorporation of a hyperspectral camera [9]. Given the unique properties described above, this comprehensive data set should lower the barriers to the development of new computer vision approaches or statistical methodologies by independent researchers who do not have the funding or infrastructure to generate the wide range of di erent types of data needed.

Greenhouse Management
All imaged plants were grown in the greenhouse facility of the University of Nebraska-Lincoln's Greenhouse Innovation Center (Latitude: 40.83, Longitude: -96.69) between October 2nd, 2015 to November 10th, 2015. Kernels were sown in 1.5 gallon pots with Fafard germination mix supplemented with 1 cup (236 mL) of Osmocote plus 15-9-12 and one tablespoon (15 mL) of Micromax Micronutrients per 2.8 cubic feet (80 L) of soil. The target photoperiod was 14:10 with supplementary light provided by LED growth lamps from 07:00 to 21:00 each day. The target temperature of the growth facility was between 24 -26 • C. Pots were weighed once per day and watered back to a target weight of 5,400 grams from 10-09-2015 to 11-07-2015 and a target weight of 5,500 grams from 11-08-2015 to the termination of the experiment.

Experimental Design
A total of 156 plants, representing the 32 genotypes listed in Table 1 were grown and imaged, as well as 4 pots with soil but no plant which serve as controls for the amount of water lost from soil as a result of non-transpiration mechanisms (e.g. evaporation). The 156 plants plus control pots were arranged in a ten row by sixteen column grid, with 0.235 meter spacing between plants in the same row and 1.5 meters spacing between rows (Table 2). Sequential pairs of two rows were consisted of a complete replicate with either 31 genotypes and one empty control pot, or 32 genotypes. Within each pair of rows, genotypes were blocked in groups of eight (one half row), with order randomized within blocks between replicates in order to maximize statistical power to analyze within-greenhouse variation.

Plant Imaging
The plants were imaged daily using four di erent cameras in separate imaging chambers. The four types of cameras were thermal infrared, uorescence, conventional RGB, and hyperspectral [12]. Images were collected in the order that the camera types are listed in the previous sentence. On each day, plants were imaged sequentially by row, starting with row 1 column 1 and concluding with row 10, column 16 (Table 2). Plants were imaged from the side at two angles o set 90 degrees from each other as well as a top down view. On the rst day of imaging or when plants reached the two leaf stage of development, the pot was rotated so that the major axis of leaf phylotaxy was parallel to the camera in the PA0 orientation and perpendicular to the camera in the PA90 orientation. This orientation is consistent for all cameras and was not adjusted again for the remainder of the experiment. The uorescence camera captured images with a resolution of 1038 × 1390 pixels and measures emission intensity at wavelengths between 500-750 nm based on excitation with light at 400-500 nm. Plants were imaged using the same three perspectives employed for the thermal infrared camera. The RGB camera captured images with a resolution of 2454 × 2056 pixels. Initially the zoom of the RGB camera in side views was set such that each pixel corresponds to 0.746 mm at the distance of the pot from the camera. Between 2015-11-05 and 2015-11-10, the zoom level of the RGB camera was reduced to keep the entire plant in the frame of the image. As a result of a system error, this same decreased zoom level was also applied to all RGB images taken on 2015-10-20. At this reduced zoom level, each pixel corresponds to 1.507 mm at the distance of the pot from the camera, an approximate 2x change. Plants were also imaged using the same three perspectives employed for the thermal infrared camera. The hyperspectral camera captured images with a resolution of 320 horizontal pixels. As a result of the scanning technology employed, vertical resolution ranged from 494 to 499 pixels. Hyperspectral imaging was conducted using illumination from halogen bulbs (Manufacturer Sylvania, model # ES50 HM UK 240V 35W 25°GU10). A total of 243 separate intensity values were captured for each pixel spanning a range of light wavelengths between 546nm-1700nm. Data from each wavelength was stored as a separate grayscale image.

Ground Truth Measurement
Ground truth measurements were collected at the termination of data collection on November 11-12, 2015. Manually collected phenotypes included plant height, total number of visible leaves, number of total fully extended leaves, stem diameter at the base of the plant, stem diameter at the collar of the top fully extended leaf, length and width of top fully extended leaf, and presence/absence visible anthocyanin production in the stem. After these measurements, total above-ground fresh weight biomass was measured for four out of ve replicates, resulting in the destruction of the plants. Ground truth data for the drought stressed subset of this dataset was collected following the procedure previously described in [12].

RGB Image Processing
Pixels covering portions of the plant were segmented out of RGB images using a green index ((2×G)/(R+B)). Pixels with an index value greater than 1.15 [12] were considered to be plant pixels. This method produced some false positive plant pixels within the re ective metal columns at the edge of the image. To reduce the impact of false positives, these areas were excluded from the analysis. Therefore, when plant leaves cross the reective metal frame, some true plant pixels were excluded. If no plant pixels were identi ed in the image -often the case in the rst several days when the plant had either not germinated or had not risen above the edge of the pot -the value was recorded as "NA" in the output le.

Heritability Analysis
A linear regression model was used to analyze the genotype e ect (excluding genotype ZL22 which lacked replication) and greenhouse position e ect on plant traits. The responses were modeled independently for each day as where the subscript h = 1, . . . , 6 denotes the three responses extracted from the images: plant height, width and size for the two views 0 and 90 degree. The subscripts i, j and t denote the jth plant in the ith row and day t, respectively, and ν(i, j) stands for the genotype at this pot. The parameters α and γ denote row e ect and genotype e ect, respectively. The error term is h,ij,t . Let SS α,t , SS γ,t and SS ,t be the sum of squares of the regression model (1) for the row e ect, genotype e ect and the error at time t, respectively. Let SS t = SS α,t + SS γ,t + SS ,t be the total sum of squares at time t. The heritability HR t (2) of a given trait within this population was de ned as the ratio of the genotype sum of squares over the sum of genotype and error sum of squares. For the estimate of the heritability of measurement error, the row e ect term was replaced by a replicate e ect (each replicate consisted of two sequential rows) with exclusion of ZL22 as only one plant of this genotype was grown.
As the heritability index may change over the growth of the plant, an nonparametric smoothing method was provided for analyzing the time varying heritability of plants. The de nition in (3) excludes the variation brought by the greenhouse row effect, which can be considered as the percentage of the variation where similarly as (1), ν(i, j) is the genotype of the jth plant in the ith row. Let SS γ,t and SS t be the genotype sum of squares and total sum of squares under (4). The classical heritability is de ned as

Hyperspectral Image Processing
Two methods and thresholds were used to extract plant regions of interest from hyperspectral images. First, the commonly used NDVI (normalized di erence vegetation index) formula was applied to all pixels using the formula (R 750nm -R 705nm )/(R 750nm +R 705nm ), and pixels with a value greater than 0.25 were classi ed as originating from the plant [13]. Second, based on the di erence in re ectance between stem and leaves at wavelengths of 1056nm and 1151nm, the stem was segmented from other part of plants by selecting pixels where (R 1056nm /R 1151nm ) produced a value greater than 1.2. Leaf pixels were de ned as pixels identi ed as plant pixels based on NDVI but not classi ed as stem pixels. In addition to the biological variation between individual plants, overall intensity variation existed both between di erent plants imaged on the same day and the same plant on di erent days as a result of changes in the performance of the lighting used in the hyperspectral imaging chamber. To calibrate each individual image and make the results comparable, a python script (hosted on Github; see code availability section) was used to normalize the intensity values of each plant pixel using data from the non-plant pixels in the same image.
In order to visualize variation across 243 separate wavelength measurements across multiple plant images, we used a PCA (Principal Component Analysis) based approach.

Fluorescence Image Processing
A consistent area of interest was de ned for each zoom level to exclude the pot and non-uniform areas of the imaging chamber backdrop. Within that area, pixels with an intensity value greater than 70 in the red channel were considered to be plant pixels. The aggregate uorescence intensity was de ned as the sum of the red channel intensity values for all pixels classied as plant pixels within the region of interest, and the mean uorescence intensity as the aggregate uorescence intensity value divided by the number of plant pixels within the region of interest.

Plant Biomass Prediction
Two methods were used to predict plant biomass. The rst was a single variable model based on the number of zoom level adjusted plant pixels identi ed in the two RGB side view images on a given day. The second was a multivariate model based upon the sum of plant pixels identi ed in the two RGB side views, sum of plant pixels identi ed in the two RGB side views plus the RGB top view, aggregate uorescence intensity in the two side views, aggregate uorescence intensity in the two side views plus the top view, number of plant stem pixels identi ed in the hyperspectral image and number of plant leaf pixels identi ed in the hyperspectral image. Traits were selected to overlap with those employed by [14]

Validation against ground truth measurements
A total of approximately 500 GB of image data was initially generated by the system during the course of this experiment consisting of RGB images (51.1%), uorescence images (4.3%), and hyperspectral images (44.6%).
A subset of the RGB images within this dataset were previously analyzed in [18], and were made available for download from http://plantvision.unl.edu/dataset under the terms of the Toronto Agreement. To validate the dataset and ensure plants had been properly tracked through both the automated imaging system and ground truth measurements, a simple script was written to segment images into plant and not-plant pixels (Figure 1). Source codes for all validation analysis are posted online (https://github.com/shanwai1234/Maize_Phenotype_Map).
Based on the segmentation of the image into plant and nonplant pixels, plant height was scored as the y axis dimension of the minimum bounding box. Plant area was scored as the total  . Calculated values were compared to manual measurements of plant height and plant fresh biomass which were quanti ed using destructive methods on the last day of the experiment. In both cases manual measurements and image derived estimates were highly correlated, although the correlation between manual and estimated height was greater than the correlation between manually measured and estimated biomass (Figure 2A,B). Using the PlantCV software package [23], equivalent correlations between estimated and ground truth biomass were obtained (r=0.91). Estimates of biomass using both software packages were more correlated with each other (r=0.96) than either was with ground truth measurements. This suggests that a signi cant fraction of the remaining error is the result of the expected imperfect correlation between plant size and plant mass, rather than inaccuracies in easimating plant size using individual software packages. Recent reports have suggested that estimates of biomass incorporating multiple traits extracted from image data can increase accuracy [14]. We tested the accuracy of biomass prediction of four multivariate estimation techniques on this dataset (see Methods). The correlation coe cient (r value) of the estimated biomass measures with ground truth data was 0.949, 0.958, 0.925 and 0.951 for multivariate linear model, MARS, Random Forest and SVM respectively.
The residual value -di erence between the destructively measured biomass value and the predicted biomass value based on image data and the linear regression line equation -was calculated for each individual plant ( Figure 2C). Using data from the multiple replicates of each individual accession, the proportion of error which is controlled by genetic factors rather than random error can be ascertained. We determined that 58% of the total error in biomass estimate was controlled by genetic variation between di erent maize lines. As such, this error is systematic rather than random and thus more likely to produce misleading downstream results when used in quantitative genetic analysis. As mentioned above, biomass and plant size are imperfectly correlated, as di erent plants can exhibit di erent densities, for example as a result of di erent leaf to stem ratios. Recent reports have suggested that estimates of biomass incorporating multiple traits extracted from image data can increase accuracy [14]. We tested the accuracy of biomass prediction of four multivariate estimation techniques on this dataset (see Methods). The correlation of the estimated biomass measures with ground truth data was 0.949, 0.958, 0.925 and 0.951 for multivariate linear model, MARS, Random Forest and SVM respectively. However, even when employing the most accurate of these four methods (MARS), 63% of the error in biomass es-timation could be explained by genetic factors. This source of error, with the biomass of some lines systematically underestimated and the biomass of other lines systematically overestimated presents a signi cant challenge to downstream quantitative genetic analysis. Given the prevalence of plant pixel counts as a proxy for biomass [20,22,9,21,12,19].

Patterns of change over time
One of the desirable aspects of image based plant phenotyping is that, unlike destructively measured phenotypes, the same plant can be imaged repeatedly. Instead of providing a snapshot in time this allows researchers to quantify rates of change in phenotypic values over time, providing an additional set of derived trait values. Given the issues with biomass quantication presented above, measurements of plant height were selected to validate patterns of change in phenotypic values over time. As expected, height increases over time, and the patterns of increase tended to cluster together by genotype ( Figure 3A). Increases in height followed by declines, as observed for ZL26, were determined to be caused by a change in the angle of the main stalk. While the accuracy of height estimates was assessed by comparison to physical ground truth measurements only on the last day, the height of three randomly selected plants (Plant 007-26, Plant 002-7 and Plant 041-29) were manually measured from image data and compared to software based height estimates, and no signi cant di erences were observed between the manual and automated measurements ( Figure 3B; Supplementary Table 1). To perform a similar test of the accuracy of biomass estimation at di erent stages in the maize life cycle, a set of existing ground truth measurements for two genotypes under two stress treatments [12] were combined with additional later grow stage data (Supplemental Table 2). The correlation between total plant pixels observed in the two side views and plant biomass was actually substantially higher in this dataset (r=0.97) than the primary dataset, likely as a result of the smaller amount of genetic vari -1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64 ability among these plants (Supplementary Figure 1).

Heritability of phenotypes
The proportion of total phenotypic variation for a trait controlled by genetic variation is referred to as the heritability of that trait and is a good indicator of how easy or di cult it will be to either identify the genes which control variation in a given trait, or to breed new crop varieties in which a given trait is signi cantly altered. Broad-sense heritability can be estimated without the need to rst link speci c genes to variation in speci c traits [24]. Variation in a trait which is not controlled by genotype can result from environmental e ects, interactions between genotype and environment, random variance, and measurement error. Controlling for estimated row e ects on di erent phenotypic measurements signi cantly increased overall broad sense heritability ( Figure 4A,B). This result suggests that even within controlled environments such as greenhouses, signi cant micro-environmental variation exists and that proper statistically based experimental design remains critical importance in even controlled environment phenotyping e orts.
If the absolute size of measurement error was constant in this experiment, as the measured values for a given trait became larger, the total proportion of variation explained by the error term should decrease and, as a result, heritability should increase as observed ( Figure 4A). This trend was indeed observed across six di erent phenotypic measurements (three traits calculated from each of two viewing angles (Figure 4B). Plant height also exhibited signi cantly greater heritability than plant area or plant width and greater heritability when calculated solely from the 90 degree side angle photo than when calculated solely from to 0 degree angle photo.
In previous studies, uorescence intensity has been treated as an indicator for plant abiotic stress status [25,26,7,27] or chlorophyll content level [28, 29]. Using the uorescence images collected as part of this experiment, the mean uorescence intensity value for each plant image was calculated (see Methods). We found that this trait exhibited moderate heritability, with the proportion of variation controlled by genetic factors increasing over time and reaching approximately 60% by the last day of the experiment ( Figure 4B).

Hyperspectral image validation
Hyperspectral imaging of crop plants has been employed previously in eld settings using airborne cameras [30, 31, 32]. As a result of the architecture of grain crops such as maize, aerial imagery will largely capture leaf tissue during vegetative growth, and either tassels (maize) or seed heads (sorghum, millet, rice, oats, etc) during reproductive growth. The dataset described here includes hyperspectral imagery taken from the side of individual plants, enabling quanti cation of the reectance properties of plant stems in addition to leaf tissue.
Many uses of hyperspectral data reduce the data from a whole plant or whole plot of genetically identical plants to a single aggregate measurement. While these approaches can increase the precision of intensity measurements for individual wavelengths, these approaches also sacri ce spatial resolution and can in some cases produce apparent changes in re ectivity between plants that result from variation in the ratios of the sizes of di erent organs with di erent re ective properties. To assess the extent of variation in the re ectance properties of individual plants, a principal component analysis of variation in intensity values for individual pixels was conducted. After non-plant pixels were removed from the hyperspectral data cube ( Figure 5A) (See Methods), false color images were generated encoding the intensity values of the rst three principal components of variation as the intensity of the red, green, and blue channels respectively ( Figure 5B, C and D). The second principal component (green channel) marked boundary pixels where intensity values likely represent a mixture of re ectance data from the plant and from the background. The rst principal component (red channel) appeared to indicate distinctions between pixels within the stem of the plant and pixels within the leaves.
Based on this observation, an index was de ned which accurately separated plant pixels into leaf and stem (see Methods).  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63 64 65 Stem pixels were segmented from the rest of the plant using an index value derived from the di erence in intensity values observed in the 1056nm and 1151nm hyperspectral bands. This methodology was previously described [12]. The re ectance pattern of individual plant stems is quite dissimilar from the data observed from leaves and exhibits signi cantly di erent re ective properties in some areas of the near infrared ( Figure  6). Characteristics of the stem are important breeding targets for both agronomic traits (lodging resistance, yield for biomass crops) and value added traits (biofuel conversion potential for bioenergy crops, yield for sugarcane and sweet sorghum). Hyperspectral imaging of the stem has the potential to provide nondestructive measurements of these traits. The calculated pattern of leaf re ectance for the data presented here are comparable with those observed in eld-based hyperspectral studies [33, 34, 35], providing both external validation and suggesting that the data presented here may be of use in developing new indices for use under eld conditions.
In conclusion, while the results presented above highlight some of the simplest traits which can be extracted from plant image data, these represent a small fraction of the total set of phenotypes for which image analysis algorithms currently exist, and those in turn represent a small fraction of the total set of phenotypes which can potentially be scored from image data. Software packages already exist to measure a range of plant architectural traits such as leaf length, angle, and curvature from RGB images [6,36]. Tools are also being developed to extract phenotypic information on abiotic stress response patterns from uorescence imaging [6,7]. The analysis of plant traits from hyperspectral image data, while common place in the remote sensing realm where an entire eld may represent a single data point, is just beginning for single plant imaging. Recent work as highlighted the potential of hyperspectral imaging to quantify changes in plant composition and nutrient content throughout development [12,37]. While these techniques have great potential to accelerate e orts to link genotype to phenotype through ameliorating the current bottleneck of plant phenotypic data collection, it will be important to balance the development of new image analysis tools with the awareness of the potential for systematic error result- ing from genetic variation between di erent lines of the same crop species.

Availability of supporting data and materials
The image data sets from four types of cameras, pot weight records per day and ground truth measurements with corresponding documentation for 32 maize inbreds and same types of image data for two maize inbreds under two stress treatments were deposited in the CyVerse data commons under a CC0 license with [38]. All image data were stored in the following data structure: Genotype -> Plant -> Camera type -> Day. For the hyperspectral camera each photo is stored as 243 sub images, each image representing intensity values for a given wavelength, so these require one additional level of nesting in the data structure Day -> wavelength. The grayscale images from the IR camera and the hyperspectral imaging system are stored as three-channel images with all three channels in a given pixel set to identical values. The uorescence images contain almost all information in the red channel with the blue and green channel having intensities equal to or very close to zero, but data all three channels exist. Genotype data of 32 inbreds were generated as part of a separate project and SNP calls for individual inbred lines were made available either through 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64  65 [39] or the ZeaGBSv2.7 GBS SNP dataset stored in Panzea. Measurements for thirteen core phenotypes at each eld trial as well as local weather data can be retrieved from publicly released Genomes 2 Fields datasets released on CyVerse [39,40]. Data from the 2014 G2F eld trials is posted [39] and data from the 2015 G2F eld trials is posted [40]. Genetically identical seeds from the majority of the accessions used in creating both this dataset and the Genomes 2 Fields eld trial data can be ordered from public domain sources (e.g. USDA GRIN) and are listed in Table 1

Consent for publication
Not applicable.

Competing Interests
The authors declare that they have no competing interests.  The green revolution created a significant increase in the yields of several major crops in the 42 1960s and 1970s, dramatically reducing the prevalence of hunger and famine around the world, 43 even as population growth continued. One of the major components of the green revolution was 44 new varieties of major grain crops produced through conventional phenotypic selection with 45 higher yield potentially. Since the green revolution, the need for food has continued to increase, 46 and a great deal of effort in the public and private sectors is devoted to developing crop varieties 47 with higher yield potential. However, as the low hanging fruit for increased yield vanish, each 48 new increase in yield requires more time and resources. Recent studies have demonstrated that 49 yield increases may have slowed or stopped for some major grain crops in large regions of the 50 world [1] . New approaches to plant breeding must be developed if crop production continues to 51 grow to meet the needs of an increasing population around the world. 52 53 The major bottleneck in modern plant breeding is phenotyping. Phenotyping can be used in two 54 ways. Firstly, by phenotyping a large set of lines, a plant breeder can identify those lines with the 55 highest yield potential and/or greatest stress tolerance in a given environment. Secondly, 56 sufficiently detailed phenotyping measurements from enough different plants can be combined 57 with genotypic data to identify regions of the genome of a particular plant species which carry 58 beneficial or deleterious alleles. The breeder can then develop new crop varieties which 59 incorporate as many beneficial alleles and exclude as many deleterious alleles as possible. 60 Phenotyping tends to be expensive and low throughput, yet as breeders seek to identify larger 61 numbers of alleles each with individually smaller effects, the amount of phenotyping required to 62 achieve a given increase in yield potential is growing. High throughput computer vision based 63 approaches to plant phenotyping have the potential to ameliorate this bottleneck. These tools can 64 be used to precisely quantify even subtle traits in plants and will tend to decrease in unit cost 65 with scale, while conventional phenotyping, which remains a human labor intensive processes, 66 does not.

68
Several recent pilot studies have applied a range of image-processing techniques to extract 69 phenotypic measurements from crop plants. RGB (R: Red channel; G: Green channel; B: Blue 70 channel) camera technology, widely used in the consumer sector, has also been the most widely 71 used tool in these initial efforts at computer vision based plant phenotyping [2,3,4,5] . Other types of 72 cameras including fluoresence [6,7] and NIR (near-infrared) [6,8,9] have also been employed in high 73 throughput plant phenotyping efforts, primarily in studies of the response of plant to different 74 abiotic stresses.

76
However, the utility of current studies is limited in two ways. Firstly, current analysis tools can 77 extract only a small number of different phenotypic measurements from images of crop plants.

78
Approximately 150 tools for analyzing plant image data are listed in a field specific database, 79 however the majority of these are either developed specifically for Arabidopsis thaliana which is 80 a model plant, or are designed specifically to analyze images of roots [10] . Secondly, a great deal 81 of image data is generated in controlled environments, however, there are comparatively few 82 attempts to link phenotypic measurements in the greenhouse to performance in the field. 83 However, one recent report in maize suggested that more than 50% of the total variation in yield 84 under field conditions could be predicted using traits measured under controlled environments 85 [10] .

87
Advances in computational tools for extracting phenotypic measurements of plants from image 88 data and statistical models for predicting yield under different field conditions from such 89 measurements requires suitable training datasets. Here, we generate and validate such a dataset 90 consisting of high throughput phenotyping data from 32 distinct maize (Zea mays) accessions 91 drawn primarily from recently off-patent lines developed by major plant breeding companies. 92 These accessions were selected specifically because paired data from the same lines exists for a 93 wide range of plant phenotypes collected in 54 distinct field trials at locations spanning 13 North 94 American states or provinces over two years [11] . This extremely broad set of field sites captures 95 much of the environmental variation among areas in which maize are cultivated with total 96 rainfall during the growing season ranging from 133.604 mm to 960.628 mm (excluding sites 97 with supplemental irrigation) and peak temperatures during the growing season ranging from 98 23.5 ºC to 34.9 ºC. In addition, the same lines have been genotyped for approximately 200,000 99 SNP markers using GBS [11] . Towards these existing data, we added RGB, thermal infra-red, 100 fluorescent and hyperspectral images collected once per day per plant, as well as detailed water-101 use information (single day, single plant resolution). At the end of the experiment, 12 different 102 types of ground-truth phenotypes were measured for individual plants including destructive 103 measurements. A second experiment focused on interactions between genotype and 104 environmental stress, collecting the same types of data described above from two maize 105 genotypes under well watered and water stressed conditions [12] . We are releasing this curated 106 dataset of high throughput plant phenotyping images from accessions where data on both 107 genotypic variation and agronomic performance under field conditions is already available. All 108 data was generated using a Lemnatec designed high throughput greenhouse-based phenotyping 109 system constructed at the University of Nebraska-Lincoln. This system is distinguished from 110 existing public sector phenotyping systems in North America by both the ability to grow plants 111 to a height of 2.5 meters and the incorporation of a hyperspectral camera [9] . Given the unique 112 properties  Table 1 were grown and imaged, as 134 well as 4 pots with soil but no plant which serve as controls for the amount of water lost from 135 soil as a result of non-transpiration mechanisms (e.g. evaporation). The 156 plants plus control 136 pots were arranged in a ten row by sixteen column grid, with 0.235 meter spacing between plants 137 in the same row and 1.5 meters spacing between rows (Table 2). Sequential pairs of two rows 138 were consisted of a complete replicate with either 31 genotypes and one empty control pot, or 32 139 genotypes. Within each pair of rows, genotypes were blocked in groups of eight (one half row), 140 with order randomized within blocks between replicates in order to maximize statistical power to 141 analyze within-greenhouse variation. The plants were imaged daily using four different cameras in separate imaging chambers. The 159 four types of cameras were thermal infrared, fluorescence, conventional RGB, and hyperspectral 160 [12] . Images were collected in the order that the camera types are listed in the previous sentence.

161
On each day, plants were imaged sequentially by row, starting with row 1 column 1 and 162 concluding with row 10, column 16 (Table 2).

164
Plants were imaged from the side at two angles offset 90 degrees from each other as well as a top 165 down view. On the first day of imaging or when plants reached the two leaf stage of 166 development, the pot was rotated so that the major axis of leaf phylotaxy was parallel to the 167 camera in the PA0 orientation and perpendicular to the camera in the PA90 orientation. This 168 orientation is consistent for all cameras and was not adjusted again for the remainder of the 169 experiment. The fluorescence camera captured images with a resolution of 10381390 pixels and 170 measures emission intensity at wavelengths between 500-750 nm based on excitation with light 171 at 400-500 nm. Plants were imaged using the same three perspectives employed for the thermal 172 infrared camera. The RGB camera captured images with a resolution of 24542056 pixels. 173 Initially the zoom of the RGB camera in side views was set such that each pixel corresponds to 174 0.746 mm at the distance of the pot from the camera. Between 2015-11-05 and 2015-11-10, the 175 zoom level of the RGB camera was reduced to keep the entire plant in the frame of the image. As 176 a result of a system error, this same decreased zoom level was also applied to all RGB images 177 taken on 2015-10-20. At this reduced zoom level, each pixel corresponds to 1.507 mm at the 178 distance of the pot from the camera, an approximate 2x change. Plants were also imaged using 179 the same three perspectives employed for the thermal infrared camera. The hyperspectral camera 180 captured images with a resolution of 320 horizontal pixels. As a result of the scanning 181 technology employed, vertical resolution ranged from 494 to 499 pixels. Hyperspectral imaging 182 was conducted using illumination from halogen bulbs (Manufacturer Sylvania, model # ES50 183 HM UK 240V 35W 25ºGU10). A total of 243 separate intensity values were captured for each 184 pixel spanning a range of light wavelengths between 546nm-1700nm. Data from each 185 wavelength was stored as a separate grayscale image.

187
Ground Truth Measurement 188 189 Ground truth measurements were collected at the termination of data collection on November and presence/absence visible anthocyanin production in the stem. After these measurements, 194 total above-ground fresh weight biomass was measured for four out of five replicates, resulting 195 in the destruction of the plants. Ground truth data for the drought stressed subset of this dataset 196 was collected following the procedure previously described in [12] . 197 198 RGB Image Processing 199 200 Pixels covering portions of the plant were segmented out of RGB images using a green index 201 ((2G)/(R+B)). Pixels with an index value greater than 1.15 [12] were considered to be plant 202 pixels. This method produced some false positive plant pixels within the reflective metal 203 columns at the edge of the image. To reduce the impact of false positives, these areas were 204 excluded from the analysis. Therefore, when plant leaves cross the reflective metal frame, some 205 true plant pixels were excluded. If no plant pixels were identified in the image -often the case in 206 the first several days when the plant had either not germinated or had not risen above the edge of 207 the pot -the value was recorded as "NA" in the output file.

209
Heritability Analysis 210 211 A linear regression model was used to analyze the genotype effect (excluding genotype ZL22 212 which lacked replication) and greenhouse position effect on plant traits. The responses were 213 modeled independently for each day as 214 ℎ, , = µ ℎ, + ℎ, , + ℎ, ( , ), + ℎ, , (1) 215 where the subscript h=1,…,6 denotes the three responses extracted from the images: plant height, 216 width and size for the two views 0 and 90 degree. The subscripts i, j and t denote the jth plant in 217 the ith row and day t, respectively, and v(i,j) stands for the genotype at this pot. The parameters α 218 and γ denote row effect and genotype effect, respectively. The error term is h,ij,t. Let SS,t, SS,t 219 and SS,t be the sum of squares of the regression model (1) for the row effect, genotype effect 220 and the error at time t, respectively. Let SSt = SS,t + SS,t + SS,t be the total sum of squares at 221 time t. The heritability HRt (2) of a given trait within this population was defined as the ratio of 222 the genotype sum of squares over the sum of genotype and error sum of squares. For the estimate 223 of the heritability of measurement error, the row effect term was replaced by a replicate effect 224 (each replicate consisted of two sequential rows) with exclusion of ZL22 as only one plant of this 225 genotype was grown. 226 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64 HRt = , , + ,

(2) 227
As the heritability index may change over the growth of the plant, an nonparametric smoothing 228 method was provided for analyzing the time varying heritability of plants. The definition in (3) 229 excludes the variation brought by the greenhouse row effect, which can be considered as the 230 percentage of the variation in plant response that can be explained by the genotype effect after 231 adjusting the environmental effect. To compare with this definition of heritability (2) the 232 response in the model without considering the row effect was constructed as 233 ℎ, , = µ ℎ, + ℎ, ( , ), + ℎ, , where similarly as (1), v(i,j) is the genotype of the jth plant in the ith row. Let ̃ ,t and ̃t be the 235 genotype sum of squares and total sum of squares under (4). The classical heritability is defined 236 as ̃ ,t 237 Hyperspectral Image Processing 240 241 Two methods and thresholds were used to extract plant regions of interest from hyperspectral 242 images. First, the commonly used NDVI (normalized difference vegetation index) formula was 243 applied to all pixels using the formula (R 750nm -R 705nm )/(R 750nm +R 705nm ), and pixels with a value 244 greater than 0.25 were classified as originating from the plant [13] . Second, based on the 245 difference in reflectance between stem and leaves at wavelengths of 1056nm and 1151nm, the 246 stem was segmented from other part of plants by selecting pixels where (R 1056nm /R 1151nm ) 247 produced a value greater than 1.2. Leaf pixels were defined as pixels identified as plant pixels 248 based on NDVI but not classified as stem pixels. In addition to the biological variation between were selected to overlap with those employed by [14] where possible. This multivariate dataset 284 was used to predict plant biomass using linear modeling as well as MARS, Random Forest and 285 SVM [14] . MARS analysis was performed using the R package earth [15] Random Forest with the 286 R package randomForest [16] and SVM with the R package e1071 [17] . 287 288

290
Validation against ground truth measurements 291 292 A total of approximately 500 GB of image data was initially generated by the system during the 293 course of this experiment consisting of RGB images (51.1%), fluorescence images (4.3%), and 294 hyperspectral images (44.6%). A subset of the RGB images within this dataset were previously 295 analyzed in [18] , and were made available for download from http://plantvision.unl.edu/dataset 296 under the terms of the Toronto Agreement. To validate the dataset and ensure plants had been 297 properly tracked through both the automated imaging system and ground truth measurements, a 298 simple script was written to segment images into plant and not-plant pixels ( Figure 1

310
Based on the segmentation of the image into plant and non-plant pixels, plant height was scored 311 as the y axis dimension of the minimum bounding box. Plant area was scored as the total number 312 of plant pixels observed in both side view images after correcting for the area of each pixel at 313 each zoom employed (See Methods). Similar approaches to estimate plant biomass have been 314 widely employed across a range of grain crop species including [19] , wheat [20] , barley 315 [20,21] ,maize [12] , sorghum [22] and seteria [9] . Calculated values were compared to manual 316 measurements of plant height and plant fresh biomass which were quantified using destructive 317 methods on the last day of the experiment. In both cases manual measurements and image 318 derived estimates were highly correlated, although the correlation between manual and estimated 319 height was greater than the correlation between manually measured and estimated biomass 320 ( Figure 22A,B The residual value -difference between the destructively measured biomass value and the 333 predicted biomass value based on image data and the linear regression line equationwas 334 calculated for each individual plant ( Figure 2C). Using data from the multiple replicates of each 335 individual accession, the proportion of error which is controlled by genetic factors rather than 336 random error can be ascertained. We determined that 58% of the total error in biomass estimate 337 was controlled by genetic variation between different maize lines. As such, this error is 338 systematic rather than random and thus more likely to produce misleading downstream results 339 when used in quantitative genetic analysis. As mentioned above, biomass and plant size are 340 imperfectly correlated, as different plants can exhibit different densities, for example as a result 341 of different leaf to stem ratios. Recent reports have suggested that estimates of biomass 342 incorporating multiple traits extracted from image data can increase accuracy [14] . We tested the 343 accuracy of biomass prediction of four multivariate estimation techniques on this dataset (see lines systematically overestimated presents a significant challenge to downstream quantitative 350 genetic analysis. Given the prevalence of plant pixel counts as a proxy for biomass [20,22,9,21,12,19] .

357
Patterns of change over time 358 359 One of the desirable aspects of image based plant phenotyping is that, unlike destructively 360 measured phenotypes, the same plant can be imaged repeatedly. Instead of providing a snapshot 361 in time this allows researchers to quantify rates of change in phenotypic values over time, 362 providing an additional set of derived trait values. Given the issues with biomass quantification 363 presented above, measurements of plant height were selected to validate patterns of change in 364 phenotypic values over time. As expected, height increases over time, and the patterns of 365 increase tended to cluster together by genotype ( Figure 3A). Increases in height followed by 366 declines, as observed for ZL26, were determined to be caused by a change in the angle of the 367 main stalk. While the accuracy of height estimates was assessed by comparison to physical 368 ground truth measurements only on the last day, the height of three randomly selected plants 369 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64 (Plant 007-26, Plant 002-7 and Plant 041-29) were manually measured from image data and 370 compared to software based height estimates, and no significant differences were observed 371 between the manual and automated measurements ( Figure 3B; Supplementary cycle, a set of existing ground truth measurements for two genotypes under two stress treatments 374 [12] were combined with additional later grow stage data (Supplemental Table 2). The correlation 375 between total plant pixels observed in the two side views and plant biomass was actually 376 substantially higher in this dataset (r=0.97) than the primary dataset, likely as a result of the 377 smaller amount of genetic variability among these plants (Supplementary Figure 1). 378

383
Heritability of phenotypes 384 385 The proportion of total phenotypic variation for a trait controlled by genetic variation is referred 386 to as the heritability of that trait and is a good indicator of how easy or difficult it will be to 387 either identify the genes which control variation in a given trait, or to breed new crop varieties in 388 which a given trait is significantly altered. Broad-sense heritability can be estimated without the 389 need to first link specific genes to variation in specific traits [24] . Variation in a trait which is not 390 controlled by genotype can result from environmental effects, interactions between genotype and 391 environment, random variance, and measurement error. Controlling for estimated row effects on 392 1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64 different phenotypic measurements significantly increased overall broad sense heritability 393 ( Figure 4A,B). This result suggests that even within controlled environments such as 394 greenhouses, significant micro-environmental variation exists and that proper statistically based 395 experimental design remains critical importance in even controlled environment phenotyping 396 efforts.

398
If the absolute size of measurement error was constant in this experiment, as the measured values 399 for a given trait became larger, the total proportion of variation explained by the error term 400 should decrease and, as a result, heritability should increase as observed ( Figure 4A). This trend 401 was indeed observed across six different phenotypic measurements (three traits calculated from 402 each of two viewing angles ( Figure 4B). Plant height also exhibited significantly greater 403 heritability than plant area or plant width and greater heritability when calculated solely from the 404 90 degree side angle photo than when calculated solely from to 0 degree angle photo.

406
In previous studies, fluorescence intensity has been treated as an indicator for plant abiotic stress 407 status [25,26,7,27] or chlorophyll content level [28,29] . Using the fluorescence images collected as part 408 of this experiment, the mean fluorescence intensity value for each plant image was calculated 409 (see Methods). We found that this trait exhibited moderate heritability, with the proportion of 410 variation controlled by genetic factors increasing over time and reaching approximately 60% by 411 the last day of the experiment ( Figure 4B).

425
Hyperspectral image validation 426 427 Hyperspectral imaging of crop plants has been employed previously in field settings using 428 airborne cameras [30,31,32] . As a result of the architecture of grain crops such as maize, aerial 429 imagery will largely capture leaf tissue during vegetative growth, and either tassels (maize) or 430 seed heads (sorghum, millet, rice, oats, etc) during reproductive growth. The dataset described 431 here includes hyperspectral imagery taken from the side of individual plants, enabling 432 quantification of the reflectance properties of plant stems in addition to leaf tissue. 433 Many uses of hyperspectral data reduce the data from a whole plant or whole plot of 434 genetically identical plants to a single aggregate measurement. While these approaches can 435 increase the precision of intensity measurements for individual wavelengths, these approaches 436 also sacrifice spatial resolution and can in some cases produce apparent changes in reflectivity

456
Based on this observation, an index was defined which accurately separated plant pixels into leaf 457 and stem (see Methods). Stem pixels were segmented from the rest of the plant using an index 458 value derived from the difference in intensity values observed in the 1056nm and 1151nm 459 hyperspectral bands. This methodology was previously described [12] . The reflectance pattern of 460 individual plant stems is quite dissimilar from the data observed from leaves and exhibits 461   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64 significantly different reflective properties in some areas of the near infrared ( Figure 6). 462 Characteristics of the stem are important breeding targets for both agronomic traits (lodging 463 resistance, yield for biomass crops) and value added traits (biofuel conversion potential for 464 bioenergy crops, yield for sugarcane and sweet sorghum). Hyperspectral imaging of the stem has 465 the potential to provide nondestructive measurements of these traits. The calculated pattern of 466 leaf reflectance for the data presented here are comparable with those observed in field-based 467 hyperspectral studies [33,34,35] , providing both external validation and suggesting that the data 468 presented here may be of use in developing new indices for use under field conditions. 469 470 471

475
In conclusion, while the results presented above highlight some of the simplest traits which can 476 be extracted from plant image data, these represent a small fraction of the total set of phenotypes 477 for which image analysis algorithms currently exist, and those in turn represent a small fraction 478 of the total set of phenotypes which can potentially be scored from image data. Software 479 packages already exist to measure a range of plant architectural traits such as leaf length, angle, 480 and curvature from RGB images [6,36] . Tools are also being developed to extract phenotypic 481 information on abiotic stress response patterns from fluorescence imaging [6,7] . The analysis of 482 plant traits from hyperspectral image data, while common place in the remote sensing realm 483 where an entire field may represent a single data point, is just beginning for single plant imaging.

503
The image data sets from four types of cameras, pot weight records per day and ground truth 504 measurements with corresponding documentation for 32 maize inbreds and same types of image 505 data for two maize inbreds under two stress treatments were deposited in the CyVerse data 506 commons under a CC0 license with [38] . All image data were stored in the following data 507 structure: Genotype -> Plant -> Camera type -> Day. For the hyperspectral camera each photo is 508 stored as 243 sub images, each image representing intensity values for a given wavelength, so 509 these require one additional level of nesting in the data structure Day -> wavelength. The 510 grayscale images from the IR camera and the hyperspectral imaging system are stored as three-511 channel images with all three channels in a given pixel set to identical values. The fluorescence 512 images contain almost all information in the red channel with the blue and green channel having 513 intensities equal to or very close to zero, but data all three channels exist. Genotype data of 32 514 inbreds were generated as part of a separate project and SNP calls for individual inbred lines 515 were made available either through [39] or the ZeaGBSv2.7 GBS SNP dataset stored in Panzea.