Forest canopy mortality during the 2018-2020 summer drought years in Central Europe: The application of a deep learning approach on aerial images across Luxembourg

Efficient monitoring of tree canopy mortality requires data that cover large areas and capture changes over time while being precise enough to detect changes at the canopy level.In the development of automated approaches,aerial images represent an under-exploited scale between high-resolution drone images and satellite data. Our aim herein was to use a deep learning model to automatically detect canopy mortality from high-resolution aerial images after severe drought events in the summers 2018–2020 in Luxembourg. We analysed canopy mortality for the years 2017–2020 using the EfficientUNet ++ ,a state-of-the-art convolutional neural network.Training data were acquired for the years 2017 and 2019 only, in order to test the robustness of the model for years with no reference data. We found a severe increase in canopy mortality from 0.64 km 2 in 2017 to 7.49 km 2 in 2020, with conifers being affected at a much higher rate than broadleaf trees. The model was able to classify canopy mortality with an F1-score of 66%–71% and we found that for years without training data, we were able to transfer the model trained on other years to predict canopy mortality, if illumination conditions did not deviate severely. We conclude that aerial images hold much potential for automated regular monitoring of canopy mortality over large areas at canopy level when analysed with deep learning approaches. We consider the suggested approach a cost-efficient and -effective alternative to drone and field-based sampling.


Introduction
Forests cover about 31% of the global land area (FAO and UNEP 2020) and compose a contemporary net sink of >20% of anthropogenic CO 2 emissions (Pan et al. 2011, Le Quéré et al. 2018, Pugh et al. 2019, Friedlingstein et al. 2020).They also provide other important ecological and economic services.An increase in extreme weather events, like prolonged drought periods, poses a severe challenge for current forest ecosystems.Pronounced tree mortality incidents have been observed across many biomes around the world during the last decades (Stocker et al. 2014, Hoegh-Guldberg et al. 2018, Hari et al. 2020, Hammond et al. 2022).Despite an intensification of research on the drivers of forest canopy mortality, we still have an incomplete picture on this phenomenon and its underlying processes (Lindner et al. 2010, Hartmann et al. 2018).From 2018 to 2020, Central and Northern Europe have experienced three consecutive summer droughts coupled with high temperatures (Buras et al. 2020, Rakovec et al. 2022).These events resulted in large-scale forest damages and pronounced tree dieback across Europe (Schuldt et al. 2020, Senf & Seidl, 2021b].While some studies have explored the effects of disturbances on a broad scale (Senf & Seidl, 2021a], accessible information on canopy mortality from these events at the level of single trees and tree groups is still missing in many parts of Europe.To derive a more consistent picture on trends in canopy mortality, we hence need to further improve existing forest monitoring and mapping tools to provide information on canopy mortality that is consistent in space and time (Allen et al. 2010, Senf et al. 2015).
Traditional forest monitoring approaches via ground-level forest inventories face many challenges, such as large temporal gaps between individual measurement campaigns and high expenses for equipment and staff, while comparably small areas are actually assessed (Neeff & Piazza 2019).Meanwhile, the technological and methodological developments in remote sensing have given scientists tools to regularly acquire high-resolution data of the Earth's surface (Lechner et al. 2020).Satellite-based remote sensing products provide information on forest cover dynamics, typically at 10-30 m resolution.These data capture for instance large patches of forest die-off very well (Hansen et al. 2013, Senf & Seidl, 2021b].At the same time this resolution is too coarse to detect smaller changes such as interspersed canopy die-back in response to climatic stress (Hartmann et al. 2018).Sensors mounted on planes or unmanned aerial vehicles (UAVs) are more suitable for this task since individual tree crowns become increasingly visible at higher resolution (< 1m) and as a consequence, interspersed canopy mortality detection improves (Lu & He 2017).High-resolution data have been frequently used for instance for detailed analysis of vegetation structure and composition (Csillik et al. 2018, Kattenborn et al. 2020, Schiefer et al. 2020), but the area covered in scientific studies is typically relatively small.However, in many Central European countries aerial surveys are regularly performed to gather aerial images (i.e.orthoimagery) over large regions or entire countries, usually during the vegetation period.These data often remain behind pay-walls, but over the last years they have been made increasingly available by official sources.This trend opens opportunities for research as well as forest management.Aerial imagery could therefore be an important, yet under-explored, resource to detect and automatically monitor canopy mortality.
Recent advances in machine learning, and particularly in convolutional neural networks (CNNs), offer new pathways to automatically map canopy mortality.CNNs are deep learning algorithms that were developed for pattern analysis.They are able to extract location-invariant low-level features from images for more efficient classification (Krizhevsky et al. 2017, Hoeser & Kuenzer 2020).Over the last years CNNs and other deep learning algorithms have become increasingly popular in remote sensing and ecological research (Zhu et al. 2017, Voulodimos et al. 2018, Brodrick et al. 2019, Ma et al. 2019, Reichstein et al. 2019, Hoeser & Kuenzer 2020, Yuan et al. 2020, Kattenborn et al. 2021).The suitability of CNNs to extract various forest/tree attributes at different spatial resolutions has been demonstrated in numerous studies [e.g.Chang et al. 2019, Brandt et al. 2020, Ferreira et al. 2020, Kattenborn et al. 2020, Schiefer et al. 2020), but tree/canopy mortality has only been explored in a few studies using CNNs so far [e.g.Hamilton et al. 2021, Hickman et al. 2022).Moreover, while aerial images from governmental organizations are widely used by foresters (Fensham & Fairfax 2002), few studies have used available aerial images in combination with CNNs (Fricker et al. 2019, Sylvain et al. 2019, Chiang et al. 2020, Tao et al. 2020).Most of these studies have been limited spatially and temporally.This is because they typically employ data that were acquired specifically for that specific study and have therefore little potential to be used for long-term monitoring.Additionally, the studies using CNNs have typically been restricted to relatively small areas (Schiefer et al. 2020).Studies examining canopy mortality over larger spatial extents, e.g.countries, and exploiting available and systematically acquired aerial images are to our knowledge missing so far.In addition, in order to explore the potential for such large-scale, high-resolution aerial images to be used in forest monitoring networks we also need to test the transferability of trained algorithms across years.
In this study we fill this research gap by exploring the automatic detection of forest canopy mortality from available aerial images for the country of Luxembourg.We focus on the effects of the 2018-2020 summer droughts, which have resulted in higher than usual canopy mortality in Central Europe (Schuldt et al. 2020).Our specific objectives were to (1) investigate the application of freely available aerial images to identify canopy mortality using a state-of-the-art CNN model, (2) assess the transferability of the trained model by applying it to data from years it was not trained on, and (3) apply the model to assess canopy mortality dynamics in conifer and broadleaf trees during three extreme drought years (2017)(2018)(2019)(2020).

Study site and characteristics
The study area covers all forests of Luxembourg as defined by the LIS-L land cover data set (Korzeniowska 2020) (Fig. 1).The country of Luxembourg is located in Central Europe, between 49 • and 51 • latitude and 5 • and 7 • longitude.It has an area of 2586 km 2 of which 37% is covered by forests.Around 71% of the forests are broadleaf trees and 23% conifers; 5% are mixedstands (Korzeniowska 2020).Luxembourg has a diverse topography and while the south of the country is characterized by relatively low elevation of about 250 m a.s.l, the North is more hilly, with elevation reaching 560 m a.s.l.In 2018, central Europe was affected by a severe summer drought (Schuldt et al. 2020), which had pronounced impacts on forests including wide-spread tree mortality (Buras et al. 2020, Obladen et al. 2021, Senf & Seidl, 2021b].In the following 2 years the summers in Central Europe continued to be hotter and drier than average, which continued to impact forest conditions (Rakovec et al. 2022).According to data from the meteorological station at the Airport Findel, Luxembourg (MeteoLux), the summer of 2018 was exceptionally hot and dry.July 2018 showed an extraordinary vapour pressure deficit anomaly, alongside low precipitation resulting in the most negative monthly climatic water balance among 2016-2020 (Fig. S1).But also in 2019 and to some degree in 2020 the summers were on average drier and warmer compared with the 30-year average .

Orthoimagery and reference data
This study uses freely available true-colour (RGB) and nearinfrared (NIR) aerial orthoimagery from the government of Luxembourg (https://www.data.public.lu/).As deep learning models are able to arithmetically combine spectral bands during training, we did not provide such information a priori.The aerial images were acquired annually during the vegetation period (Table S1), mostly during July and August, but in 2017 survey f lights took place in June and in 2020 some areas were surveyed in September.For our study we selected aerial images covering four consecutive years (2017-2020) with a ground resolution of 25 cm in 2017, 20 cm in 2018 and 2019, and 10 cm in 2020 for the whole country.We z-score normalized the spectral information of the four channels to values between 0 and 1 prior to use in the model, by subtracting the pixel values of each channel by the mean and dividing it by the standard deviation of the channel in the data set (LeCun et al. 2012).To have a consistent data set with the same ground resolution we re-sampled the aerial images from 2017 and 2020 to 20 cm.
We generated the reference data for the model as follows (full protocol: see Methods S1).We defined canopy mortality as standing dead tree canopy area.Tree canopies were considered dead if we observed total browning/bleaching of the crown in conifers and the complete loss of leaves in broadleaf trees.The aerial images were visually assessed and canopy areas of standing dead trees manually labelled by drawing a polygon shape (feature) using the open Software QGIS, Version 3.12 (Development Team 2020).We did not delineate each dead tree separately but the dead forest canopy which could also consist of a cohort of dead trees.During this process we took great care to accurately label the exact shape of the dead canopy area including larger branches.Dead forest canopies were differentiated into conifer and broadleaf trees.This was done for several forested areas throughout Luxembourg (Fig. 1) with the most extensive reference data set originating from the Attert catchment where canopy mortality was mapped previously in an area of 314 km 2  (2017)(2018)(2019)(2020).The black hatched sites were additionally mapped in 2017 to increase the amount of reference data in that year.The distribution of forests is based on land use products provided by the Land Information System for Luxembourg (LIS-L) (https://data.public.lu/en/datasets/lis-l-land-cover-2018/).(Haffter 2020).Additional areas to generate reference data were chosen to cover different forest conditions across Luxembourg.The areas of these additional data sites range between ∼2 and 3 km 2 .Everything outside dead broadleaf and dead conifer trees was classified as background.Reference data were initially mapped for 2019 only, with a count of 13 378 features.Due to the lower image quality and relatively low canopy mortality in 2017, additional reference data were acquired for that year to improve model predictions.To test the transferability of the model to other years, we labelled additional testing data for 2018 and 2020 in three of the six areas (Fig. 1).In total we mapped 209 and 1991 features in 2018 and 2020, respectively (Table 1).The distribution of classes in the data set exhibited a strong imbalance between dead tree classes and the background class (the background outweighing the dead classes) as most forest stands and other land uses are part of the background.There was also an imbalance between the occurrence of dead conifers and dead broadleaf trees, as conifers made up c. 90% of the reference data.

Neural network architecture
CNNs are a family of deep learning networks that utilize convolutional layers, which are used to exploit information of spatially adjacent pixels and are therefore very efficient in detecting structural patterns or objects in images (Rawat & Wang 2017).
The depth of these networks enables the algorithm to learn how to identify complex textural features (Zhu et al. 2017).Here, we used the U-Net architecture EfficientUNet++ (Silva et al. 2021).U-Net architectures (Ronneberger et al. 2015) are a type of semantic segmentation models that are used for pixel-wise classification.They were originally developed for use in the biomedical sector, but have been successfully adopted in environmental science.U-Nets consist of an encoding and decoding branch of layers.In the encoder or contraction path, the image size is reduced with convolution and max pooling operations while also increasing the number of channels.In the decoder or expansion path the image size is again gradually increased.The layers of the encoding and decoding paths are connected through Skip Connections.Hereby, the activations of the encoder are forwarded to the decoder, providing the spatial identity of the data (Brodrick et al. 2019, Hoeser & Kuenzer 2020, Kattenborn et al. 2021).
We implemented our EfficientUNet++ model (Silva et al. 2021) in Python (Version 3.9.12),using PyTorch (Version 1.11.0) and the PyTorch Lightning framework (Version 1.5.10).As encoder we chose the EfficientNet-b5 (Tan & Le 2020), which is pretrained on the ImageNet database.We used the albumentations-package for image augmentation to rotate and f lip the images, and change brightness by a factor range of 0.2 and contrast by 0.15.The learning rate was set to 0.0003.Additionally, we implemented cosine learning rate annealing to avoid local minima during training.Due Table 1.Number of reference data features for each year.The data were divided into three datasets for training, validation, and testing of the model with a ratio of (A) 70%, (B) 20%, and (C) 10%.The numbers in brackets represent the amount of broadleaf and conifer features.For 2018 and 2019 we only mapped reference data for testing, as the U-Net model was trained on reference data from 2017 and 2019 only.

Year
Reference data (n features) to the imbalance between the two dead-tree classes (conifer and broadleaf) and the dominance of the background class we used a combination of loss terms.The total loss was defined as the sum of the generalized dice loss (Sudre et al. 2017), focal loss (Lin et al. 2017), and boundary loss (Kervadec et al. 2021).As the boundary loss grows with distance from the dead forest polygon, it helps the model delineate dead tree canopies more precisely (Kervadec et al. 2021).

CNN initialization and training
We used the WebDataset format, storing our tiled reference data as.tarfiles (Methods S2).As a result we split the images and reference data into patches of 256 px × 256 px and allocated these into batches of n=16.Each batch of the data sets was composed in equal parts from images with occurrence of dead canopy pixels and randomly chosen other patches in order to not overfit the model on the relative scarce occurrence of dead tree pixels.To balance the data in each data set, we made sure that samples from both years were allocated proportionally to the data sets.The reference data were randomly split into training (70%), validation (20%), and testing (10%) data sets (Fig. 2).We tested four input configurations for the U-Net model, which varied in spectral bands and number of predicted classes.We explored whether the inclusion of the NIR band improved model predictions, compared with using the RGB bands only.We also distinguished between a base classification scenario (dead canopy and background, referred to as binary) and a multi-class approach with three classes (dead conifers, dead broadleaf, and background, referred to as multi) to check whether splitting the dead canopy class into dead conifers and dead broadleaf would decrease the model performance.This resulted in the four model configurations binary/RGB, binary/RGBN, multi/RGB, and multi/RGBN.For each configuration the U-Net model was trained for a maximum of 500 epochs and the training was terminated if the dice score did not improve over an 100 epoch interval.
For the final canopy mortality predictions we selected the multi/RGB model because of its good performance (F1-score) and its increased thematic detail.We trained a total of three multi/RGB models and derived the inference using all three in a majority vote.For this process a class was assigned to each pixel according to the result of a majority of model outputs.Inference was run on 256 px × 256 px patches.We finally vectorized the resulting raster outputs using the gdal package (GDAL/OGR contributors 2020) to facilitate further analysis.All output features inside forests (Korzeniowska 2020) were selected and canopy mortality area was derived using QGIS V. 3.16 (Development Team 2020).We only included canopy mortality areas >0.5 m 2 to remove false positives, unlikely to represent actual canopies.

CNN performance
The overall model performance was assessed through the F1score, which is the harmonic mean of precision and recall and is robust for asymmetrical data sets (Kattenborn et al. 2021).We calculated the F1-score for each year and model input combination and created confusion matrices for the individual classes.F1-scores were calculated excluding the background class, which represents everything other than dead canopies, as this class was very dominant and would positively bias the evaluation.
In order to provide a map of canopy mortality for the whole country we split the results into grid cells with a size of 1024 px × 1024 px each (∼4 ha) and obtained the percentage of forest cover per cell.We calculated the percentage of canopy mortality based on the number of dead canopy pixels compared with forest area.We also obtained canopy mortality rates per year and created an accumulated canopy mortality map.

Model performance
We found the F1-score to range between 60% and 70% independently of model configuration or year.When comparing different combinations of spectral channels for the U-Net model, we found the inclusion of the NIR data did not improve the performance of the model (Table 2).For instance, the F1-score increased slightly from 69.7% to 71.0% for the binary classification, while it decreased from 67.6% to 66.2% in the multi-class classification.The differentiation between conifer and broadleaf trees did not result in a clearer picture on the inclusion of NIR data either, as it only improved the F1-score in the broadleaf class, but decreased it in conifers (Table 2).The more complex multi-class model differentiating into conifers and broadleaf showed that the Downloaded from https://academic.oup.com/forestry/advance-article/doi/10.1093/forestry/cpad049/7307321 by KIT Library user on 05 February 2024  model was able to separate between these two mortality classes (e.g.see Fig. 5) (F1 of 67.6%).Including the background class would result in F1-scores >0.99 in all tested model combinations.
Given that the inclusion of the NIR did not improve the performance overall, so we used the simpler RGB models for all other analyses.As training our models to differentiate between broadleaf and conifer mortality was relatively successful, we explored canopy mortality in Luxembourg based on the multiclass model in more detail.The model was able to generalize learned patterns to years with no reference data (2018 and 2020).F1-scores in those years were slightly lower than for the years with reference data (2017 and 2019), but remained above 60% (Table 3).Moreover, the model was able to detect incorrectly labelled reference data, for instance detecting canopy mortality even when it was missed in the reference data set (Fig. 3).The dead canopy pixels detected by the model did generally align better to the true dead tree pixel occurrence than the manually drawn reference data, as labellers usually cannot produce pixel-perfect dead tree canopy boundaries.
We identified areas where the model was less successful in mapping canopy mortality and noticed that the model was relatively more prone to miss areas of canopy mortality in the years 2018 and 2020.We also found that canopy mortality was more likely to be omitted if the canopy area was affected by low illumination conditions or cast shadows (Fig. 4).Broadleaf trees were less likely to be mapped correctly by the model.We found that 53.3% of mapped polygons matched the reference data and F1-scores were lower for broadleaf trees, compared with conifers (Table 2).

Canopy mortality dynamics and patterns
Analysing the multi-class model output showed a difference in canopy mortality patterns between conifer and broadleaf trees.We found that conifer mortality was typically clumped, while broadleaf mortality occurred in a more scattered pattern with mostly individual dead trees (Fig. 5).We tested this observation using a nearest neighbour analysis, which supported our findings (Fig. S4).Overall, we found a noticeable increase in canopy mortality rates in Luxembourg following the 2018 summer drought (Fig. 6).The area of dead forest canopy rose from 0.64 km 2 in 2017 to 7.49 km 2 in 2020.While the increase of dead canopy area between 2017 and 2018 was moderate, we found a substantial increase by 2.71 km 2 in 2019 and 3.85 km 2 in 2020.Most of the mortality (c.80%) could be attributed to conifers, albeit their much lower cover (c.24.5%) compared with broadleaf trees in the forests of Luxembourg.We overlaid and merged yearly canopy mortality in QGIS to account for trees that were removed between years and therefore obtain a more accurate estimation of the increase of dead canopy area.This approach suggested an increase of 11.06 km 2 instead of 7.49 km 2 from 2017 to 2020.
We created visualizations of canopy mortality patterns for the whole of Luxembourg for the time period 2017-2020 at a spatial resolution of c. 4 ha resulting in a total of 12 918 grid cells (Fig. 7).We found that canopy mortality incidences occurred in <25% of Downloaded from https://academic.oup.com/forestry/advance-article/doi/10.1093/forestry/cpad049/7307321 by KIT Library user on 05 February 2024  the grid cells in 2017 and 2018 and the percentage of dead forest canopy compared with forest cover remained low.In 2019, more than 65% of the forested grid cells experienced canopy mortality with 0.5% of forest area on average.In 2020, incidences of canopy mortality were occurring in most grid cells and mortality averaged at 1%, but in some areas canopy mortality reached values of 10% and higher.Overall, canopy mortality appeared to be approximately equally distributed within the forests of Luxembourg with hotspots in the centre-northeast and south (Fig. 7).
We also derived maps of the differences in annual canopy mortality for 2018, 2019, and 2020 by subtracting percentage of canopy mortality from the previous year.We found a clear trend of increasing canopy mortality in 2019 following the summer drought in 2018.This increase was further accelerating in 2020 and more evenly distributed throughout the country, except for an area in the northwest of Luxembourg which seemed to exhibit a decrease in canopy mortality.This decrease was, however, due to shadowing of the aerial images, which will be discussed in the following section (Fig. 8).

Discussion
In this study, we demonstrated the suitability of aerial images in combination with a semantic segmentation using a CNN, to map canopy mortality across an area of more than 2500 km 2 .We showed that (1) we could use the model to map canopy mortality from aerial images with an F1-score of 68%, (2) for years without reference data, we could successfully transfer the trained model to other years to predict canopy mortality, even though severe changes in illumination conditions may require the acquisition of (a limited amount of) additional reference data, and (3) how trends in canopy mortality from 2017 to 2020 could be derived for the study area using our map products.The trends indicated a severe rise in canopy mortality from 0.64 to 7.49 km 2 .

Dead tree classification
The overall F1-score of the model was 68%, which is in line with similar studies classifying characteristics of forests (Tao et  ), but the differences in data characteristics (resolution, spectrum, acquisition), classification models, and accuracy assessment complicate direct comparisons between the studies.Sylvain et al. (2019) who similarly classified alive and dead broadleaf and conifer trees reported global accuracy between 86% and 94%, but included alive trees, which were omitted for our reported scores.When including alive trees (background class), our models would achieve accuracies of 99%.Testing four model configurations, with varying spectral bands (RGB, RGBN) and classification targets (binary, multi), we found that distinguishing between dead conifer and broadleaf canopies led to slightly lower F1-scores of the model.This can be mostly attributed to the model's lower F1-score for dead broadleaf canopy area (0.47), compared with dead conifer area (0.76).The lower F1-score was likely caused by the relative scarcity of dead broadleaf canopies in the training data (class imbalance) and the more diverse appearance of dead broadleaf trees: while dead conifer trees were observed to be relatively uniform in size, they also commonly feature a radial/spoke-like branch structure, appearing brown or bleached in the visible spectrum.Leaf less broadleaf canopies do not exhibit a well-defined spectral signal or texture in our data set.At the same time branch thickness and canopy density may vary between broadleaf tree species.Additionally, dead conifers often appeared in groups of well distinguishable individuals, whereas dead broadleaf trees were generally sparse and loosely scattered within our study area (Fig. 5).It is unlikely that the difference in mortality patterns between conifers and broadleaf trees was caused by general spatial patterns of trees, as most forest stands in Luxembourg are homogeneous with 71% of all forests overall being described as pure broadleaf stands in the LIS-L land cover data set.We observed that the CNN would classify fewer pixels on the edges of the tree crowns as dead compared with the reference data.Consequently, the larger size and more scattered nature and therefore the greater border area of broadleaf trees can also be the cause of lower F1-scores.
We report the F1-scores in this study with caution, as they could be affected by spatial auto correlation between training and testing data sets.In our study this might be the case as both originated from the same sampling regions.Therefore the training and testing data might ref lect similar site and image conditions, leading to a higher likelihood of correct classification of dead canopies closer to the training data.This problem has been discussed in the literature and might potentially be overcome by k-fold cross-validation, or spatial blocking (Ploton et al. 2020, Meyer & Pebesma 2021, Kattenborn et al. 2022).We did not implement these approaches in our study design.Instead we focused on the proportional distribution of the available dead canopy data into training, validation, and testing data sets, due to the prevalence of the non-dead canopy class.Another known issue is temporal pseudoreplication, but even though the testing data for 2018 and 2020 was mapped in a subset of the original reference data, we do not expect this to be problematic, because the aerial images acquired across the 4 years do not align perfectly due to differences in acquisition angles and shading.
Downloaded from https://academic.oup.com/forestry/advance-article/doi/10.1093/forestry/cpad049/7307321 by KIT Library user on 05 February 2024 Inclusion of NIR data did not consistently improve the classification performance.This is in contrast to Sylvain et al. (2019) who found slight improvements to model predictions when adding near-infrared data.Other studies reported improvements when jointly using hyperspectral and structural data [i.e.Fricker et al. 2019, Mäyrä et al. 2021, Hell et al. 2022).In our case, we restrained from integrating additional information from other sensor types, mostly because in an operational monitoring setting, it is less Downloaded from https://academic.oup.com/forestry/advance-article/doi/10.1093/forestry/cpad049/7307321 by KIT Library user on 05 February 2024 realistic that multi-sensor data are regularly available.At the same time, the RGB data appeared to be sufficient in our study and allowed us to capture the vast majority of dead trees with a good accuracy.The separation into broadleaf and coniferous trees may benefit from additional integration of airborne laser scanning data, since these may help to further improve the signal related to the different branching structures of broadleaf and coniferous trees.Airborne laser scanning data are also available for Luxembourg and their potential to further increase the performance of the CNN classification could be examined in future studies.

Image Quality and Spatial Resolution
We explored if our trained model could be applied to data from subsequent years.Our results indicated that we can trust predictions from aerial images for years not included during training.One problem with respect to image quality was canopy and cast shadows of trees, which varied from year to year and across the map (likely due to difference in solar inclination as a result of varying acquisition times over the years).This is in line with Kattenborn et al. (2019) and Lopatin et al. (2019) who found shadows to negatively affect machine learning classification accuracy in UAV images.The difference in cast shadows caused systematic shifts in the appearance of individual trees over years, as well as an underestimation of the crown size, due to the edges of the trees being obscured.Differences between cast shadows in different years also caused the canopy mortality features to not align perfectly, which impeded the direct overlay of the different years.We also found that illumination conditions in general affected model performance, when no additional data were acquired.In 2020, we observed a large patch of area shadowed by clouds in the northwest of the country that corresponded with the area of decreased canopy mortality (Fig. 8C).The shade caused the model to omit a majority of dead canopy in that area, and hence resulted in an underestimation and leading to an apparent decrease in dead canopy for that area in 2020.The problem could potentially be solved by acquiring additional reference data which adequately capture the appearance of dead canopy areas in shaded image acquisitions.
A limiting factor of remote sensing studies looking at individual trees and tree species is the resolution of images.At a ground resolution of 20 cm our observations were mostly limited to coarse forest classes, that is, conifers and broadleaf trees, as structures like branches are visible but species-specific features were hard to detect.However, higher resolution images (<10 cm) from drones have been proven to be suitable to classify dead trees and even tree species (Safonova et al. 2019, Schiefer et al. 2020).Over the last years the spatial grain of Luxembourg's aerial images has become finer (from 25 to 10 cm).Hence it can be assumed that even higher spatial resolution annual images will become increasingly available and are likely to result in even better model predictions to enable a continuous mapping of canopy mortality and potentially tree species at individual tree scales.
Recently, Schiefer et al. (2023) presented an approach in which results of a high-resolution classification were extrapolated to satellite images.The application of this approach on our results opens the opportunity to map dead canopies over a far larger extent than Luxembourg and overcomes the temporal limitations of our data with satellite data that offer more frequent observations.

Reference Data
Our reference data were derived by manually delineating features from aerial images.While dead conifers were usually clearly identifiable for the human eye, it was more difficult to Downloaded from https://academic.oup.com/forestry/advance-article/doi/10.1093/forestry/cpad049/7307321 by KIT Library user on 05 February 2024 identify dead broadleaf trees, as they are heterogeneous in appearance and the small branch structures often become blurry at the spatial resolution of the available imagery.In our study, all delineations were cross-checked by at least one additional person to improve the accuracy of the reference data.We further accounted for imperfect and limited training data by choosing a U-Net architecture, which is known to produce comparably more robust results with a limited amount of reference data than other architectures (Silva et al. 2021).
The quality of reference data is a key challenge in machine and deep learning research [i.e.Geiger et al. 2021).In our study, a small experiment conducted during the reference data acquisition showed that features mapped by two people diverged in 48% of the mapped area, even though largely the same trees were delineated (Fig. S3).The large discrepancies between the two data sets was mostly the result of feature delineation.This discrepancy is highly relevant for interpreting the reported accuracy measured which base on these reference data.The effect of reference data delineation on model performance could be further examined in future studies.
At the same time, we assume that this effect is unlikely to have a notable inf luence on the model's ability to identify the target class, in our case dead trees.U-Nets have been shown to be somewhat robust when it comes to inaccurate reference data and in many cases they were quite successful in correctly mapping areas with partly wrong or missing reference data (Hamdi et al. 2019, Kattenborn et al. 2020).We found the same trend in our study.The U-Net model was able to map dead trees with missing reference data, even in the training stage (Fig. 3).Additionally, the model classified fewer pixels on the edge of dead canopies as dead than was visually determined during labelling.All these differences result in lower F1-scores of the model reported here.Based on our visual assessments, the identification and delineation results of the U-Net model looked slightly better than we would expect from the reported F1-scores.The ability of the model to predict canopy mortality in years with no reference data opens up the possibility to automatically map additional reference data for subsequent years.These additional data could be quality checked by the model user and used as additional reference data to constantly improve the model, similarly to active or semi-supervised learning methods (Settles 2009, Hady et al. 2013).The continuous annual classification of dead trees over a whole country might help forest management to identify areas for logging, for example after bark beetle outbreaks.It may also help to identify particular vulnerable areas for forest conversion.Further, the created maps may serve as inputs to ecological studies examining the drivers of tree mortality on a fine spatial scale.

The drought summers 2018-2020
Our study was conducted in the context of a severe drought in Central Europe which occurred in the years 2018 to 2020 (Schuldt et al. 2020).Our results show a steep increase in standing dead canopy area following the summer droughts.The true number of dead trees is likely to be even higher than what we have observed as many areas were harvested in between acquisition dates of the imagery.We overlaid an merged canopy mortality areas over the years in QGIS to assess cumulative canopy mortality, but this is likely an overestimation, as the aerial images did not fully overlap.This misalignment was also the reason why we were not able to calculate dead canopy areas that had been harvested between years.Nevertheless, our results clearly showed that tree mortality increased from 2017 to 2020.In future studies, the additional detection of removed trees could be implemented for example using a CNN suitable for change-detection.
Having a closer look at our mortality data set, we observed that coniferous stands were affected disproportionately more than broadleaf stands.While in 2017 70% of the mapped canopy mortality area were conifers, in 2019 the percentage of dead conifers was 83.5%.These percentages stand in strong contrast to the overall share of broadleaf and coniferous trees which amount to 71% and 23%, respectively (Korzeniowska 2020).In 2020 the proportion of dead broadleaf trees increased slightly compared with conifers, making up 20.5% of all canopy mortality.While apparent canopy mortality of broadleaf trees was sparse and usually occurred in the form of as single dead individuals, conifers tended to die in clusters of several trees, which often encompassed an entire group of conifers standing next-to-each other.This pattern is also typical for bark beetle (Ips typographus L.) infestations (Fassnacht et al. 2014), which is the most common biotic damage agent for Norway spruce in Europe and causes large dieback events (Schelhaas et al. 2003).The drought conditions in 2018 and 2019 likely weakened the trees, causing reduced carbon uptake, leaf loss, and reduced sap production, due to water deficit (Adams et al. 2017).As a result of the weakened defence, trees likely became more prone to bark beetle infestation (Kolb et al. 2019, Netherer et al. 2019, Obladen et al. 2021).Disturbance regimes, such as drought, disease, fire, and windthrow are typically not stand-alone events, because they increase the likelihood of other disturbance types to occur.As such they must be seen as interlinked (Seidl et al. 2017).Consequently, while the dead conifers detected here have most likely been finally killed by widespread bark beetle outbreaks, it is not unreasonable to conclude that the drought conditions in the previous years have played a key role.Widespread Norway spruce mortality which has been reported for large parts of Central Europe as a consequence of the 2018 and 2019 droughts (Senf & Seidl, 2021b] supports our observations. Interestingly, mortality of broadleaf trees, while increasing after 2018, rose even more after 2019.This suggests a lag effect, which has also been reported in some other studies [i.e.Bigler et al. 2007).This lag effect may relate to delayed dieback due to weakening of the trees or may also simply relate to the cumulative effect of the three subsequent drought years.

Conclusions
Overall, based on our results, we found that the quality of the aerial imagery tested in this study is suitable to map and monitor canopy mortality of broadleaf and coniferous stands.This study therefore highlights the potential of already available aerial images for national forest monitoring.The results can likely be improved in the future by adding additional reference data to the model and by exploiting the benefits of additional image quality improvements.
The data we used in this study are collected annually by the government of Luxembourg and are openly available.Many states and regional governments collect similar data sets, but the access is often restricted to government agencies or behind a paywall.We advocate for the release of this kind of data for research purposes, as it has the potential to become a valuable tool for forest monitoring.
to thank the government of the Grand Duchy of Luxembourg for contributing to public data availability and research opportunities by providing free aerial orthoimagery online (https://data.public.lu/).

Funding
This study was funded by the Center for Disaster Management and Risk Reduction Technology (CEDIM) of the Karlsruhe Institute of Technology (KIT) and was inspired by and contributes to the International Tree Mortality Network (https://tree-mortality.net/), an initiative of the IUFRO Task Force on Monitoring Global Tree Mortality Trends and Patterns.Nadine K. Ruehr acknowledges funding through the Helmholtz Initiative and Networking fund (W2/W3-156).

Figure 1 .
Figure 1.Map of the study area Luxembourg and the location of forests within the country.Shown is (A) the central location of Luxembourg in Europe, (B) the distribution of forest types within Luxembourg and sites where reference data were acquired by visual examination of aerial imagery.The black framed sites were mapped for 2017 and 2019 and sites with black dots for all years (2017-2020).The black hatched sites were additionally mapped in 2017 to increase the amount of reference data in that year.The distribution of forests is based on land use products provided by the Land Information System for Luxembourg (LIS-L) (https://data.public.lu/en/datasets/lis-l-land-cover-2018/).

Figure 2 .
Figure 2. Workf low diagram illustrating the process of reference data acquisition and usage to build and test the CNN.First, reference data were acquired through visual assessment of the aerial imagery and split into training, validation, and testing data sets.The model was first trained using the training data and tuned with the validation data.The testing data set was used to independently assess model performance and to calculate the F1-scores.

Figure 3 .
Figure 3. Example of a patch of forest where canopy mortality was not apparent in the reference data but classified correctly by the U-Net.Shown are (A) the aerial image of a forest with few dead tree canopies and some bare forest f loor, and (B) the correct detection of the dead tree canopies by the U-Net.The model was able to clearly distinguish between canopy mortality and the bare forest ground.

Figure 4 .
Figure 4. Examples of the effects of light conditions during aerial observations on the model performance.Shown are two examples in 2020 when (A) shading resulted in omission errors and dead canopies were not identified by the U-Net model, and (B) dead canopies were clearly identified by the model and no interference with shading occurred.

Figure 5 .
Figure 5. Examples of canopy mortality in 2019 detected by the U-Net model.Shown is the ability of the model to detect (A) clustered conifer canopy mortality and (B) patchy broadleaf canopy mortality.(B) also exemplifies the diversity of shape in dead broadleaf trees which challenges the U-Net model, resulting in a small misinterpretation.That is the small central feature which marks deadwood lying on the ground.

Figure 6 .
Figure 6.Development of the dead forest canopy area in Luxembourg derived from the U-Net model.Given are the annual standing broadleaf, conifer and total canopy mortality area from 2017 to 2020.Note that the total forest area of Luxembourg is 960 km 2 .

Figure 7 .
Figure 7. Map of annual canopy mortality in Luxembourg (2017-2020) derived from the U-Net model.The maps show percentage of canopy mortality for the years 2017 (A), 2018 (B), 2019 (C), and 2020 (D).Canopy mortality was calculated in percent of forest cover per grid cell (c. 4 ha; i.e. 1024 × 1024 px).Note that in 2020 some patches reached canopy mortality >50%.

Figure 8 .
Figure 8. Map of annual differences in canopy mortality in Luxembourg.The maps provide differences in the percentage of canopy mortality between 2017 and 2018 (A), 2018 and 2019 (B), and 2019 and 2020 (C).The annual differences were calculated based on the results of Fig. 7 and provide the annual difference of percentage of dead canopy area at a grid cell size of c. 4 ha (i.e.1024 × 1024 px).Underestimation of canopy mortality in the Northwest of Luxembourg is caused by poor illumination conditions.The blue colours denote a decrease in canopy mortality between years, while the red colours symbolize an increase in canopy mortality.Forested grid cells with no change in canopy mortality and areas without forest cover are given in white.Note that the apparent decrease in canopy mortality in the Northwest of Luxembourg was caused by shading of the orthoimages from clouds.

Table 2 .
F1-scores for all input combinations (binary/RGB, multi/RGB, binary/RGBN, multi/RGBN) over all years; we chose the multi/RGB setup for our final model.F1-scores for the conifer and broadleaf classes are given for the multi/RGB model setup.

Table 3 .
F1-scores of all years 2017-2020 for the multi/RGB set-up.We used n features in testing for each given year.F1-scores are reported with and without the background class.