Computer vision-based phenotyping for improvement of plant productivity: a machine learning perspective

Abstract Employing computer vision to extract useful information from images and videos is becoming a key technique for identifying phenotypic changes in plants. Here, we review the emerging aspects of computer vision for automated plant phenotyping. Recent advances in image analysis empowered by machine learning-based techniques, including convolutional neural network-based modeling, have expanded their application to assist high-throughput plant phenotyping. Combinatorial use of multiple sensors to acquire various spectra has allowed us to noninvasively obtain a series of datasets, including those related to the development and physiological responses of plants throughout their life. Automated phenotyping platforms accelerate the elucidation of gene functions associated with traits in model plants under controlled conditions. Remote sensing techniques with image collection platforms, such as unmanned vehicles and tractors, are also emerging for large-scale field phenotyping for crop breeding and precision agriculture. Computer vision-based phenotyping will play significant roles in both the nowcasting and forecasting of plant traits through modeling of genotype/phenotype relationships.

Full details of the experimental design and statistical methods used should be given in the Methods section, as detailed in our Minimum Standards Reporting Checklist. Information essential to interpreting the data presented should be made available in the figure legends.
Have you included all the information requested in your manuscript?
If not, please give reasons for any omissions below. as follow-up to "Experimental design and statistics Full details of the experimental design and statistical methods used should be given in the Methods section, as detailed in our Minimum Standards Reporting Checklist. Information essential to interpreting the data presented should be made available in the figure legends.
Have you included all the information requested in your manuscript? " There are no experiments and statistical analyses used in this review.

Resources
A description of all resources used, including antibodies, cell lines, animals and software tools, with enough information to allow them to be uniquely identified, should be included in the Methods section. Authors are strongly encouraged to cite Research Resource Identifiers (RRIDs) for antibodies, model organisms and tools, where possible.
Have you included the information requested as detailed in our Minimum Standards Reporting Checklist? Yes Availability of data and materials All datasets and code on which the conclusions of the paper rely must be either included in your submission or deposited in publicly available repositories (where available and ethically appropriate), referencing such data using No a unique identifier in the references and in the "Availability of Data and Materials" section of your manuscript.
Have you have met the above requirement as detailed in our Minimum Standards Reporting Checklist?
If not, please give reasons for any omissions below. as follow-up to "Availability of data and materials All datasets and code on which the conclusions of the paper rely must be either included in your submission or deposited in publicly available repositories (where available and ethically appropriate), referencing such data using a unique identifier in the references and in the "Availability of Data and Materials" section of your manuscript.
Have you have met the above requirement as detailed in our Minimum Standards Reporting Checklist? " There are no datasets and code on which the conclusions of this review.

54
Computer vision that extracts useful information from images and videos is rapidly becoming an 55 essential technique in plant phenomics [1]. Phenomics approaches to plant science aim to identify the 56 relationships between genetic diversities and phenotypic traits in plant species using noninvasive and 57 high-throughput measurements of quantitative parameters that reflect traits and physiological states 58 throughout a plant's life [2]. Recent advances in DNA sequencing technologies have enabled us to 59 rapidly acquire a map of genomic variation at a population scale [3,4]. Combining high-throughput 60 analytical platforms for DNA sequencing and plant phenotyping has provided opportunities for 61 exploring genetic factors for complex quantitative traits in plants, such as growth, environmental stress 62 tolerance, disease resistance [5] and yield, by mapping genotypes to phenotypes using statistical 63 genetics methods such as quantitative trait locus (QTL) analysis, genome-wide association study 64 (GWAS) [6]. Moreover, a model of the relationship between the genotype-phenotype map of 65 individuals in a breeding population can be used to compute genome-estimated breeding values 66 (GEBVs) to select the best parents for new crosses in genomic selection (GS) in crop breeding [7,8]. 67 Thus, high-throughput phenotyping aided by computer vision with various sensors and algorithms for 68 imagery analysis will play a crucial role for crop improvement concerning to the scenarios in 69 population demography and climate change [9]. 70 In this context, unmanned aerial vehicles (UAVs), also known as drones, can play an 71 important role for data acquisition thanks to their interesting features. UAVs are pilotless aircrafts that 72 can be launched and steered remotely in an autonomous or a semiautonomous manner, 73 notwithstanding the fact that they can be maneuvered precisely at various speed, orientation and 74 altitude levels, which suits them to various applications. Moreover, they can be deployed much faster, 75 removing thereby the need for a laborious setting in advance. They are less costly to operate (i.e., in 76 terms of manufacturing, maintenance and power consumption alike), and allow acquiring highly 77 qualitative data (e.g., extremely high-resolution imagery) since they can be flown at low attitudes. One 78 of the areas that has been benefiting from UAVs is crop phenotyping thanks to their field-friendly 79 property. In this respect, the review that was carried out in [10] surveys comprehensively the late 80 advances on UAV-oriented crop phenotyping and related sensors. In brief, recalling the lack of 81 convenient methods (i.e., normally manual) in plant phenotyping since they are rather time consuming, 82 laborious, and often costly, the paper provides an up-to-date survey of recent contributions where 83 UAVs have been promoted as a promising alternative to mitigate the earlier gaps. In this respect, 84 several phenotyping UAV platforms are envisioned such blimps, which are characterized by a 85 plausible hovering ability and a load-effective property. However, blimps remain relatively slow in 86 motion and sensitive to windy conditions. Unmanned helicopters, on the other hand, offer the 87 advantage of carrying large and heavy sensors, which enables multitasking operations. However, 88 besides being noisy, they are somewhat costly to run and maintain. Fixed-wing UAV is another 89 platform that is characterized by high velocity and long autonomy, which are essential to carry out 90 field surveying especially when the field surface is quite large. Nevertheless, they remain short of free 91 hover option. Multirotor UAVs are less costly, very flexible and easily customizable. However, one of 92 their main drawbacks refer to the rather short autonomy and limited payload. Despite such 93 shortcomings, multirotor UAVs remain the most commonly used platform. The plant phenotyping 94 endeavor dictates not only what type of UAV platform to be adopted but also what kind of sensors suit 95 the requirements. A wide range of sensors can be encountered in this respect, the basic ones are digital 96 cameras that are typically adopted for quick color and/or texture-based phenotyping operations. 97 Multispectral and hyperspectral sensors can capture richer spectral information about the plants of 98 interest, allowing thus a more in-depth phenotyping. Beyond these latter, thermal infrared sensors offer 99 another complementary and useful information especially when it comes to determining the response 100 of canopy to stress. LIDAR is another form of sensors, which however has not received much attention 101 in crop phenotyping. Synthetic aperture radar (SAR) is an imaging sensor that can acquire high 102 resolution images, which is most useful for crop identification, monitoring and yield estimation. While 103 each of the aforementioned sensors may be adopted depending on the phenotyping demands, a 104 combination of two or more sensors thereof is often a valid option. 105 Machine learning (ML), an area of computer science in which algorithm design is improved 106 automatically using experience, which aids typical steps of image analysis: preprocessing, 107 segmentation, feature extraction, and classification [11]. ML accelerates and automates image analysis, 108 which improves throughput when handling labor-intensive sensor data. Algorithms based on deep 109 learning, an emerging subfield of ML, often show more accurate performance compared to traditional 110 approaches to computer vision-based tasks, including plant identification such as PlantCLEF [12]. 111 Moreover, ML-based algorithms often provide deeper insights into discriminative features associated 112 with outputs extracted through their training process, which may enable us to dissect complex traits 113 and determine visual signatures related to traits in plants. These outcomes of ML offer us opportunities 114 for revitalizing methodologies in plant phenomics to improve throughput, accuracy, and resolution 115 ( Figure 1). 116 In this review, we provide an overview of recent advances in computer vision-based 117 approaches to plant phenotyping. Specifically, we highlight recent challenges in computer vision-118 assisted plant phenotyping for organ segmentation and species and physiological state classification, 119 as well as their applications to large-scale phenotyping in genetic studies. We also showcase recently 120 developed tools and resources for image analyses in plant phenotyping. Then, we discuss perspectives 121 and opportunities for computer vision in plant phenomics. 122 123   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64 Main text 124

Segmentation of plant organs 125
Segmentation represents a first, important and useful tool for information extraction from image data. 126 The analysis of plant organs is particularly interested in this tool (Table 1). For instance, in [13], a 127 method is presented for tracking, on a daily basis, the development of ear and silk in maize. Initially, 128 based on images acquired by means of a commanded mobile camera that is placed at about 30 cm 129 from the ear of interest, the method proceeds by selecting the plant side view that is most adequate for 130 the detection of ear positions. Subsequently, the captured images are segmented, and the stems therein 131 are labelled, and the ear position can be determined by observing changes in width along the stems. 132 Va lidated on 60 maize hybrids, the proposed pipeline scored an accuracy of 86%. Motivated by the 133 fact that counting maize tassels (which is deemed an important step towards keeping up with the 134 growth of maize plants) is still carried out manually, this paper puts forth a computer vision pipeline 135 to address the counting problem. First, a novel Maize Tassels Counting (MTC) dataset was created 136 and manually annotated from 361 field images across China within 2010-2015. Second, a deep 137 convolutional neural network is applied on the MTC dataset, and turned out to achieve plausible results 138 with respect to relevant state-of-the-art, with an absolute error of 6.6 and a mean squared error of 9.6 139 [14]. Yield estimation is one of the critical aspects in agriculture. In this context, traditional manual 140 crop counting remains rather limited especially when it comes to very large orchards, besides the fact 141 that it is costly and time-consuming. This paper suggests a deep convolutional neural network (based 142 on the Inception-ResNet) for yield estimation, which is trained on synthetic data and tested on real 143 data, and revealed a 91% of counting accuracy [15]. The paper in [16] details a method for accurate 144 extraction as well as measurement of spike and grain morphometric parameters from images acquired 145 by X-ray micro-computed tomography (μCT). The proposed method was applied to analyze the spikes 146 from an ensemble of wheat plants exposed to high temperatures under two different water regimes. 147 Interestingly, it was found out that temperature exhibits a negative impact on spike height and grain 148 number. It was also noticed that grain volume growth goes against grain number under mild stress. In 149 [17], a spike detection method in wheat plants is presented, which consists mainly of two stages. First, 150 plant segmentation is addressed via an improved color index scheme. Next, spike detection is carried 151 out by means of a neural network. Moreover, area and height thresholds were adopted for noise 152 removal, which has incurred an improvement in spike detection score, which amounted to over 80%. 153 The work in [18] puts forth a re-segmentation framework for wheat leaves. Precisely, it departs from 154 an already segmented image and turns out an improved segmentation. The underlying idea of the 155 proposed technique is that it relies on the shape of plant leaves and local orientations in order to 156 assimilate details that have been missed in the a priori segmentation. The proposed method can 157 accurately determine sharp features (e.g., leaf tips, twists and axils). 158 159

Taxonomic classification assisted by computer vision 160
Computer vision with ML-based algorithms has significantly accelerated and improved the accuracy 161 and expanded their applications to myriad areas, including computer vision [19][20][21]. Specifically, the 163 effectiveness of convolutional neural networks (CNNs) has attracted notice from researchers; the 164 significant advantages of CNNs can be attributed not only to their highly accurate recognition abilities, 165 but also to their automation of the processes of learning discriminative features. Thus, CNNs no longer 166 follow explicit feature extraction steps, unlike handcrafted feature-based algorithms. Depending on 167 tasks, however, handcrafted feature-based approaches can achieve more precise outcomes and are 168 often more efficient in their computational costs than CNNs. Because Wäldchen and Mäder (2017)  169 have thoroughly summarized the literature related to computer vision-based species identification 170 approaches published before 2016 [22], we mainly focus on studies published since then (Table 2). 171 In a handcrafted feature-based approach, Wilf et al. [23] attempted to classify leaf images into 172 labels of major groups (such as families and orders) in the taxonomic category. They used Scale-173 invariant feature transform (SIFT) and a sparse coding approach to extract the discriminative features 174 of leaf shapes and venation patterns, followed by a multiclass support vector machine (SVM) classifier 175 for grouping. A sparse representation was also used by Zhang et al. [24] as part of their process for 176 classifying plant species from RGB color leaf images, in which they used basis vectors prechosen by 177 Euclidian distances between a test sample and its nearest training samples. This selection step aimed 178 to reduce the computational cost of the sparse coding, and the authors demonstrated its superiority in 179   1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64 identification on leaf image datasets. As a case study, a Turkish research group investigated the 180 capability of computer vision algorithms in classifying wheat grains into bread wheat and durum wheat 181 based on grain images captured by high-resolution cameras [25,26]. They used two types of neural 182 networks: a multilayer perceptron (MLP) with a single hidden layer and an adaptive neuro-fuzzy 183 inference system (ANFIS). They selected seven discriminative grain features, incorporating aspects 184 of shape, color, and texture, and achieved greater than 99% accuracy on the grain classification task. 185 Another group examined two taxonomic classification tasks: the Malva alliance taxa and genus Cistus 186 taxa [27,28]. They acquired digital images of seeds using a flatbed scanner and extracted 187 morphometric, colorimetric, and textural seed features, before performing taxonomic classification 188 with stepwise linear discriminant analysis (LDA). Species identification from herbarium specimens 189 with computer vision approaches was first presented in 2016, in which Unger et al. classified German 190 trees into tens of classes with images of herbarium specimens photographed at a high resolution [29]. 191 Their analytical processes were composed of preprocessing, normalization, and feature extraction with 192 Fourier descriptors, leaf shape parameters, and vein texture, followed by SVM classification. In this 193 study, they demonstrated the potential of computer visions for taxonomic identification even when 194 using discolored leaf images of herbarium specimens. Using rather different data for species  1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53  54  55  56  57  58  59  60  61  62  63  64 dominant and most native species occur less frequently. Despite this challenge, they demonstrated that 198 the ML-based analytical approaches using SVMs and random forests (RFs) can achieve reasonable 199 tree species identification based on airborne-sensor images. 200 In the last few years, many CNN-based approaches have been developed for the taxonomic 201 classification of plants [31,32]. Dissection of trained artificial neural networks has shown that CNNs 202 can hierarchically and simultaneously learn low-, mid-, and high-level features during training [31, 203 32]. Using a dataset of accurately annotated images of wheat lines, the authors in [33] applied a CNN-204 based model to perform feature location regression to identify spikes and spikelets, as well as image-205 level classification of wheat awns, suggesting the feasibility of employing CNN-based models in 206 multiple tasks by coordinating their network architecture. In this study, the authors also suggested that 207 the images of wheat in the training dataset, which were acquired using a consumer-grade 12 MP 208 camera, can be even favorable for training the CNN-based model. A comparative assessment between 209 CNN-based and handcrafted feature-based approaches was performed in a rice kernel classification 210 task [34] In this assessment, the authors compared a deep CNN with k-nearest neighbor (kNN) 211 algorithms and SVMs, along with handcrafted features such as a pyramid histogram of oriented 212 gradients (HOG) and GIST, and showed that the CNN surpassed the kNN and SVM algorithms in 213 classification accuracy. 214 Although CNNs usually require large amounts of data and extensive computational load and time, 215 in image acquisition (perspective, illumination, and background) and preprocessing steps 234 (nonprocessed, cropped, and segmented), as well as manual efforts in these steps, and reported that 235 images taken from top-side of leaves were most effective in processing of nondestructive leaf images. 236 Interestingly in this study, the authors recorded leaf images using a smartphone, the iPhone 6, in 237 diverse situations, including natural background conditions, followed by feature extraction with the 238 pretrained ResNet-50 CNN and classification with a SVM. 239 240

Table3
Click here to download Table Table3.docx