Advances and opportunities in image analysis of bacterial cells and communities

ABSTRACT

The cellular morphology and sub-cellular spatial structure critically influence the function of microbial cells. Similarly, the spatial arrangement of genotypes and phenotypes in microbial communities has important consequences for cooperation, competition, and community functions. Fluorescence microscopy techniques are widely used to measure spatial structure inside living cells and communities, which often results in large numbers of images that are difficult or impossible to analyze manually. The rapidly evolving progress in computational image analysis has recently enabled the quantification of a large number of properties of single cells and communities, based on traditional analysis techniques and convolutional neural networks. Here, we provide a brief introduction to core concepts of automated image processing, recent software tools and how to validate image analysis results. We also discuss recent advances in image analysis of microbial cells and communities, and how these advances open up opportunities for quantitative studies of spatiotemporal processes in microbiology, based on image cytometry and adaptive microscope control.

biofilm, microbial community, single cell, segmentation, phenotyping, machine learning, data science

INTRODUCTION

Optical microscopy has long been an important technique for characterizing and understanding the microbial world. A key advantage of microscopy over other techniques for characterizing microbes is that it can acquire data for living cells with high spatial resolution. With the discovery of fluorescent proteins and improvements of fluorescent reporters, it has become possible to specifically label particular components of cells and follow cellular functions using microscopy (Specht, Braselmann and Palmer 2017). The benefits of highly specific labeling and high spatial resolution can be leveraged with a growing list of fluorescence-based live-cell microscopy techniques, which have been reviewed recently (Power and Huisken 2017; Schermelleh et al. 2019). These microscopy techniques are optimized for particular samples and imaging requirements to enable imaging from the length scale of single molecules to the length scale of microbial communities, in two and three spatial dimensions.

As an alternative to optical microscopy, spatial sequencing techniques and imaging mass spectrometry are currently emerging for fixed samples (Heacock-Kang et al. 2017; Kompauer, Heiles and Spengler 2017; Rodriques et al. 2019; Geier et al. 2020; Pareek et al. 2020), which are promising even more information than fluorescence-based microscopy. However, these techniques are technically complex and not yet widely available. Research areas in microbiology that depend on dynamics and spatial information, such as microbial cell biology or microbial community research, therefore often rely on fluorescence-based microscopy for living or fixed samples.

After the acquisition of fluorescence images, the extraction of quantitative properties from such images is a crucial, but unfortunately difficult, step in the analysis of experiments. In the past, image analysis in biology has often relied on manual quantification. However, manual analysis suffers from limited precision and becomes simply impossible when hundreds, thousands or even more images need to be analyzed. The rapidly improving accuracy and capabilities of computational image analysis are currently revolutionizing the quantification of biological processes from images (Caicedo et al. 2017; Smith et al. 2018).

This review first provides an overview for how bacterial cells and communities can be detected in images using traditional image analysis and convolutional neural networks, and how quantitative properties can be extracted after the object detection. This discussion is supported by an overview of recent image analysis tools (Box 1), general tips for using these software tools (Box 2), and guidance for quality control of the image analysis results (Box 3). A glossary of specific terms used in image processing is given in Box 4. When the terms explained in the glossary are used for the first time in the text, they are highlighted by italics. Finally, we provide an overview of new opportunities that arise from the recent improvements achieved in image analysis, with a specific focus on image analysis-driven phenotyping of microbial systems, augmented microscopy, and automated data acquisition aided by image analysis.

Box 1: Overview of recent software tools for microbial image analysis

Tremendous effort has been put into the development of many user-friendly tools for image analysis in prokaryotic cell biology and microbial community research. Below, we provide an overview of recent tools that are available for segmentation and/or data analysis following segmentation. While some of these tools are specialized for particular image types, others can be applied to a wider range of image types and research questions. It should be noted that the functionality summarized in the column ‘scope of tool’ is not a comprehensive list for the respective tool.

Many of the tools described below rely on traditional segmentation methods and offer a variety of settings to be modified by the user for optimal segmentation results. Tuning these parameters makes the methods applicable to a wide range of images, with different object sizes, signal-to-noise ratios or intensity ranges, which may vary from application to application. In many cases, default parameters are provided that yield reasonable results for most image types. However, it is beneficial for users to familiarize themselves with the options that are available and compare outputs generated by different settings for several test images. Tips for how to get started with image analysis software tools are provided in Box 2, and a guide to validating image analysis results is provided in Box 3.

Generalist image analysis tools, with an emphasis on tools for eukaryotes, have been reviewed recently (Eliceiri et al. 2012; Smith et al. 2018) including additional tools to those that are listed below. General image analysis routines, including deconvolution and powerful image visualization functionalities are also offered by commercial image analysis tools, such as cellSens (Olympus), Huygens (Scientific Volume Imaging), Imaris (Oxford Instruments), LAS X (Leica), MetaMorph (Molecular Devices), NIS Elements (Nikon), Volocity (Quorum Technologies) and Zen (Zeiss). As many commercial image analysis software tools and microscope manufacturers use their own image formats, the development of the Bio-Formats library (openmicroscopy.org/bio-formats) for standardizing image formats that are readable by most of the academia-developed software tools listed below is critically important (Linkert et al. 2010).

Name (A-Z)	Scope of tool	Reference
Single cell analysis
BacStalk	Single-cell segmentation, tracking, measurements of morphology, fluorescence properties, intracellular structure and data visualization. Focus on cells with stalks, flagella, pili.	(Hartmann et al. 2020b) https://drescherlab.org/data/bacstalk
BactMAP	Data visualization and post-processing of single-cell segmentation results. Provided as R package and therefore compatible with any other R functionality.	(van Raaphorst, Kjos and Veening 2020) https://veeninglab.com/bactmap
CellProfiler, CellProfilerAnalyst	Single-cell segmentation, classification, counting, tracking, measurements of morphology, fluorescence properties, intracellular structure, and data visualization. Widely used for eukaryotic cells.	(Jones et al. 2008; Dao et al. 2016; McQuin et al. 2018) https://cellprofiler.org https://cellprofileranalyst.org/
CellShape	Single-cell segmentation, measurements of morphology, fluorescence properties, intracellular structure and data visualization.	(Goñi-Moreno, Kim and de Lorenzo 2017) http://goo.gl/Zh0d9x
DeLTA	Single-cell segmentation, demonstrated on mother machine data. Includes neural-network based tracking and lineage reconstruction, measurements of morphology and fluorescence properties.	(Lugagne, Lin and Dunlop 2020) https://gitlab.com/dunloplab/delta
MicrobeJ	Single-cell segmentation, tracking, measurements of morphology, fluorescence properties, intracellular structure and data visualization.	(Ducret, Quardokus and Brun 2016) https://www.microbej.com
MM3	Mother machine image analysis, segmentation via thresholding or neural network based. Includes tracking, measurements of morphology, fluorescence properties, intracellular structure.	(Sauls et al. 2019) https://github.com/junlabucsd/mm3
MMHelper	Mother machine image analysis, including segmentation, tracking, measurements of morphology, fluorescence properties.	(Smith, Metz and Pagliara 2019) https://github.com/jmetz/mmhelper
MoMA	Mother machine image analysis, including segmentation, tracking, measurements of morphology, fluorescence properties and data visualization.	(Kaiser et al. 2018) https://github.com/fjug/MoMA
Oufti	Single-cell segmentation, tracking, measurements of morphology, fluorescence properties, intracellular structure and data visualization.	(Paintdakhi et al. 2016) https://oufti.org
SuperSegger	Single-cell segmentation, tracking, measurements of fluorescence properties, intracellular structure and data visualization.	(Stylianidou et al. 2016) http://mtshasta.phys.washington.edu/website/SuperSegger.php
Microbial community analysis
BiofilmQ	Cube-segmentation or import of single-cell segmentation. Includes spatial and temporal measurements of fluorescence, cytometry inside biofilms, community architecture, global measurements of microbial communities and data visualization.	(Hartmann et al. 2021) https://drescherlab.org/data/biofilmQ
Comstat, Comstat2	Global biofilm segmentation, global measurements of microbial communities.	(Heydorn et al. 2000; Vorregaard 2008) http://www.comstat.dk
Daime	Global biofilm or object segmentation. Includes fluorescence and abundance measurements, classification of cell types, spatial arrangement analysis and data visualization.	(Daims, Lücker and Wagner 2006) https://dome.csb.univie.ac.at/daime
General segmentation tools
DeepCell 2.0	Online platform for image analysis using neural networks, including pre-trained models and collection of jupyter notebooks to facilitate training and usage of own models.	(Bannon et al. 2018) http://www.deepcell.org
Fiji, ImageJ	General tool combining different image analysis workflows and many plugins for specialty applications, including segmentation, quantification and data visualization.	(Schneider, Rasband and Eliceiri 2012; Gómez-de-Mariscal et al. 2019) https://fiji.sc/https://imagej.net
ilastik	Machine-learning based segmentation, counting, tracking, classification, measurements of morphology, fluorescence properties.	(Berg et al. 2019) https://www.ilastik.org
ImJoy	Online and offline platform to apply and share deep learning methods. Various plugins, including image annotation, segmentation, classification.	(Ouyang et al. 2019) https://imjoy.io
Icy	Platform with many image analysis protocols, plugins, including segmentation, fluorescence quantification, and data visualization. Includes possibility for graphical programming.	(De Chaumont et al. 2012) http://icy.bioimageanalysis.org/
ZeroCostDL4Mic	Online platform for image analysis using neural networks via Google Colab including image segmentation, restoration, reconstruction and augmentation. Based on jupyter notebooks.	(von Chamier et al. 2020) https://github.com/HenriquesLab/ZeroCostDL4Mic

Box 2: Getting started with image analysis tools

The first step towards successfully performing quantitative image analysis is to choose the image analysis tool that is the best fit for the purpose at hand. Since this decision affects all other processing steps, it is beneficial to take the time to thoroughly research which tools are available and most suitable to perform the desired analysis. For this, the following questions may be used as a first guide.

• Is there any tool specifically developed for the desired purpose, or a similar purpose? There is a plethora of specialized tools available, some of which might fit ideally to the question. There are more image analysis tools developed for eukaryotes than for prokaryotes, but some tools that were originally developed for eukaryotes may also work for prokaryotes and vice versa.

• Is the tool still actively maintained? Is there a way to contact developers in case that questions or problems arise?

• How detailed is the documentation, and are there tutorials? Extensive documentation and tutorials are important for learning how to use a tool without constant access to the developers.

• Is there a community around the tool? Some widely used tools, especially for research on eukaryotes, have grown an active community around them, with users helping each other, e.g.via a forum, such as image.sc.

• Is the tool based on a graphical user interface or does it require coding? If tools require coding, it is important to have a look at example scripts that are (hopefully) provided, to obtain a feeling for the level of coding that is required.

• Does the tool cover all steps in the desired analysis? Some tools provide only segmentation or only data analysis and visualization, while others provide a whole workflow. Choosing a tool that combines analysis steps may save time in the future. When using several different tools for different steps, data format compatibility is critical.

Once a suitable tool has been identified for an image analysis task, a series of test images should initially be used by new users to get started and familiarize oneself with the user interface, options, analysis sensitivity, and analysis results. Some tools also offer test data for download, which can be useful for a first trial, but do not replace the user's own test images. Test images should

• be raw unmodified images, without compression,

• include examples of images which are later going to be analyzed by the tool,

• cover the full variety of expected data,

• cover the full variety of different imaging conditions.

Following the guidelines given in the tool's documentation or tutorials, the test data should be used to explore different options and analysis pipelines available with the tool. At this stage,

• explore one relevant analysis option or parameter at a time, making use of its full dynamic range to gain intuition on its effect,

• use visual feedback of the effect of each option or parameter on the analysis result, if possible after each processing step,

• use the full range of test data to ensure robustness of the processing pipeline,

• if available, use annotated image data to validate results (see Box 3).

Finally, do not hesitate to reach out for help when encountering a problem. Especially when getting started, some aspects of the program may be tricky to figure out, and advice from other users or the developers can save you a lot of work and trouble. If questions are answered in a forum-like style, exploring previous threads will also provide valuable information.

Box 3: Validation of automated image analysis results

When performing automated image analysis, it is important to be aware of the possibility of introducing artefacts and quantitative biases in several steps of the analysis. In the case of image analysis based on traditional segmentation methods, these errors can stem from inappropriate parameter settings, for example regarding filter kernel sizes, image edge effects that are unaccounted for, or an unsuitable choice of the thresholding method for the given image content or image quality. Regarding neural networks, a common risk is the over-fitting of parameters, such that training data is perfectly captured, but performance on unseen image content is poor. Uncaught, these segmentation errors propagate through the downstream analysis, skewing any quantitative result and interpretation. It is therefore important to carefully check analysis results, especially when starting to work with an unfamiliar tool or different type of data than before.

There are different approaches to validate image analysis results. Validation can be performed qualitatively, by picking examples of raw images and visually comparing them side-by-side with the final segmentation result or different processing steps. This approach will reveal if there are serious issues with the analysis. However, a systematic quantitative validation approach is often better suited, as it can also determine the segmentation accuracy when the results appear qualitatively correct. For a quantitative analysis of the segmentation accuracy, some example images need to be manually annotated to obtain a ground truth segmentation, to which the automated image analysis results can be compared. This quantitative approach helps to find the best segmentation method, or the most suitable tool for the analysis of a specific data set, and it provides unbiased guidance towards optimal parameter settings for the analysis pipeline.

An annotated test data set should include the full variety of images that the pipeline is applied to, as explained in Box 2. Tools can then be evaluated with the help of the intersection over union (IoU) measure (Fig. 4A). For two objects, their IoU is calculated by dividing their overlapping area by their unified area, yielding 1 for perfect agreement and 0 for no overlap at all. An object identified by the segmentation pipeline is considered matching if an annotated object exists, such that their IoU exceeds a certain threshold. It is then called a true positive (TP). If no match is found, the object is considered to be a false positive (FP). Similarly, each annotated object, which is not represented by a segmented object by the above measure, is a false negative (FN).

To quantify segmentation quality, several measures depending on the count of TP, FP and FN can be used, including

• precision: |$\frac{{TP}}{{TP + FP}}$|⁠,

• Jaccard index: |$\frac{{TP}}{{TP + FP + FN}}$|⁠,

• recall: |$\frac{{TP}}{{TP + FN}}$|⁠,

• Dice index: |$\frac{{2TP}}{{2TP + FP + FN}}$|⁠.

As the count of TP, FP and FN depends on the threshold set for the IoU, the performance of a segmentation method is often plotted as a function of this threshold (Fig. 4B). Other quality measures do not rely on an IoU threshold, but instead use the overall overlap between objects in the image. Denoting # as the count of pixels belonging to an object, examples are (Rubens et al. 2020):

• Overlap-based Dice index: Calculated as |$\frac{{2\# (A \cap B)}}{{\# A + \# B}}$| with A and B being the segmentation image and the annotated ground truth.

• Average Hausdorff distance: For each pixel belonging to an object in the segmentation, determine the distance to the closest pixel belonging to an annotated object. Average over these distances.

• Fraction overlap: For each segmented object A, find the ground truth object with whom it has the largest overlap B. Calculate |$\frac{{\# (A \cap B)}}{{\max (\# A,\# B)}}$| for each of those pairs and determine the mean.

• Overlap-based Jaccard index: Calculated as |$\frac{{\# (A \cap B)}}{{\# (A \cup B)}}$| with A and B being the segmentation image and the annotated ground truth, respectively.

Box 4: Glossary

Semantic segmentation: Objects are identified in images and distinguished by object type (e.g. bacterial cell vs. host cell vs. image background). Each pixel in the resulting segmented image carries the information to which type of object it belongs with the pixel value zero being assigned to background pixels (see Fig. 1). Objects of the same type (e.g. different bacterial cells) are not necessarily separated.

Instance segmentation: Objects are identified in images and individually distinguished from each other (e.g. each bacterial cell is distinguished from all other cells). The result is a label image (see Fig. 1), in which each pixel is assigned a number. This number represents the unique identifier of the object that the pixel belongs to. Background pixels are assigned the number zero.

Traditional segmentation methods: Colloquial phrase used to describe segmentation approaches that do not rely on machine learning. Traditional segmentation often includes filtering and thresholding of the raw image, followed by morphological operations.

Convolutional neural network (CNN): The basis of a neural network is a series of mathematical operations, termed layers, that transform an input (image) into an output, also called a prediction. In a CNN, most of the mathematical operations that are applied are convolutions, akin to filtering operations in traditional segmentation methods.

CNN architecture: The architecture of a neural network defines the specific arrangement and operations of layers that are applied during the process of prediction. It contains a number of free parameters that are optimized to a specific (image analysis) problem during a process called training, and it also includes several parameters that are fixed before the training (termed hyperparameters).

CNN training: During CNN training, the free parameters contained in the CNN architecture are optimized based on a comparison between the network's prediction and ‘ground truth’ data, which can be an annotated data set. This process requires significant computational resources, but ideally only needs to be performed once for a particular image analysis workflow. A trained network can perform a prediction from an input image in a short time, typically seconds or less for 2D images.

Training data: Training data for a neural network consists of image/image or image/value pairs. The first element of a pair is a raw image file, which is the same type of data that the CNN will later be applied to. The respective partner in the pair is the desired prediction of the network. This can be, e.g. a label image representing instance segmentation or a classification value (like bacterial cell/fungal cell/background). Generating training data often involves manual annotation of images, which can be tedious.

Image augmentation: By applying different transformations to the training data images, such as mirroring, rotating, or skewing, the total amount of data available for training can be dramatically increased. As a result, the CNN can make accurate predictions for a wider range of input data.

Transfer learning: After training a CNN, an optimized parameter set has been determined for the specific CNN and the specific training data. When wanting to apply the same CNN architecture to a different, but similar type of input image, it can be beneficial to start the training from the already optimized parameters, potentially reducing the amount of new training data necessary to achieve good results.

Image cytometry: Extracting a set of properties from a cell, based on an image. These features can, e.g. describe the size, shape, fluorescence properties, or internal structure of the cell. In addition, image cytometry in communities enables the quantification of each cell's position with respect to other cells or the community as a whole.

Cube cytometry: When single-cell segmentation is not achievable due to insufficient image resolution or image contrast, it is typically still possible to detect the biovolume via semantic segmentation. The detected biovolume can then be sliced into cubes of a specific size, for example the typical cell size. For each cube, properties can be extracted analogous to single-cell image cytometry (Hartmann et al. 2021). This approach enables the quantification of a variety of cellular properties and the spatial context of the community, even if single-cell resolution is not available.

Dimensionality reduction: Quantitative phenotyping often relies on measuring many properties of a sample (e.g. cellular properties in image cytometry, or the expression levels in a transcriptome). When comparing phenotypes of multiple samples, each property can be seen as an axis of a high-dimensional space, which is difficult to visualize. Each sample can then be visualized by a single point in this high-dimensional property space. Dimensionality reduction refers to a set of techniques that represent structures of the high-dimensional space by counterparts in a space with lower dimensionality, to facilitate visualization or interpretation. Depending on the method chosen for dimensionality reduction, different aspects of the point configuration in the high-dimensional space may be preserved in the low-dimensional space, for example the structure of local neighbourhoods. Popular dimensionality reduction techniques are principal component analysis (PCA) and non-linear techniques such as t-stochastic neighbourhood embedding (t-SNE) (van der Maaten and Hinton 2008 ), UMAP (McInnes et al. 2018), SPRING (Weinreb, Wolock and Klein 2017), Isomap (Tenenbaum, Silva and Langford 2000) and Diffusion map (Coifman et al. 2005).

Principal component analysis: A popular linear dimensionality reduction algorithm, resulting in a number of principal components, which represent the dimensions/axes of the lower-dimensional space. Principal components are linear combinations of the original space dimensions and therefore they are readily interpretable.

High-dimensional cluster analysis: In a high-dimensional space, clusters of points may be identified using mathematical methods, such as k-means or hierarchical clustering. In a biological context, where high-dimensional data typically represents a quantification of cellular properties or community properties, distinct clusters correspond to different phenotypes of cells or communities.

Adaptive microscopy: A microscopy method that relies on a closed-loop feedback between image acquisition, live image analysis, and motorised microscope control (see Fig. 3). The live image analysis can be used to adapt every component of the experimental acquisition, including imaging interval and imaging region. This approach can ensure automatic optimization of the experiment and tracking of rare events. One example of a useful application of adaptive microscopy is the imaging of a light-sensitive specimen, where the time resolution is increased only during specific events such as cell division (detected by image analysis).

Figure 1.

Typical segmentation workflow in traditional image analysis. A combination of filters, thresholding and morphological operations is applied to the original image (showing Bacillus subtilis cells on agar) to first achieve a semantic segmentation, followed by an instance segmentation. Here, two blurred images are subtracted to yield the image named ‘subtraction’. This modified image is thresholded to obtain the mask image, representing a semantic segmentation. A multitude of alternative operations could also lead to an accurate semantic segmentation. In an instance segmentation, individual objects are distinguished, which is represented by a label image. After segmentation, morphological operations such as a morphological dilation can help to improve the segmentation accuracy. The accuracy of segmentation results can be quantified as described in Box 3.

Detection of microbial cells and communities in images

In order to quantitatively analyze cells or communities in images, these objects have to be detected computationally. The first goal in automated image analysis is therefore to identify the regions in images where the objects are located that should be analyzed, and separate them from the background and from objects that should not be analyzed. This task is typically referred to as semantic segmentation (Box 4). Examples of semantic segmentation are the detection of all bacterial cells in an image, or the distinction of bacterial cells from fungal cells and from background in an image, or the detection of microcolonies in an image. Although this task can be obvious to a human, it is a difficult problem in automated image analysis, because computational rules have to be found for assigning each image pixel to either the foreground (or different categories of foreground, such as bacteria, fungi) or the background. Semantic segmentation can be particularly challenging when the image background is uneven or when the fluorescence signal-to-noise levels are low. For some analyses, semantic segmentation is already sufficient, e.g. if the average fluorescence of all bacterial cells in the field of view needs to be quantified.

With current fluorescence-based microscopy techniques, it has become common to acquire images with single-cell resolution, or even sub-cellular resolution (Turkowyd, Virant and Endesfelder 2016; Heintzmann and Huser 2017; Schermelleh et al. 2019). This image quality makes it often possible, at least for a human adult, to distinguish each cell in the image. The process of computationally identifying each individual cell separately in the image is termed instance segmentation. When the cells are far apart in the image, instance segmentation can be relatively easy. However, for images where the cells are close together, achieving perfect instance segmentation is a challenge (Vicar et al. 2019), which is particularly difficult for dense bacterial communities or during cell division. The availability of an accurate instance segmentation is a requirement for downstream analyses of individual cells.

There are many different methods for semantic and instance segmentation. Broadly, they can be classified as traditional image analysis methods and machine learning methods. Although most image analysis tools for microbiology applications currently rely on traditional image analysis, we expect that in the next few years, traditional object detection methods will be replaced by convolutional neural networks for many common image analysis workflows. Nevertheless, it is important to understand traditional segmentation methods in order to recognize the capabilities and limitations of the current image analysis tools. In the next section we therefore discuss traditional segmentation methods, followed by an overview and outlook for segmentation using convolutional neural networks.

Traditional methods for microbial object detection

To understand the approach on which computational object detection in images is built, it is important to appreciate some basic properties of images. Highly sensitive scientific cameras (or point-scanning confocal microscopes) typically acquire two-dimensional (2D) digital grayscale images—one such 2D image is acquired for each channel and/or imaging plane. A 2D image is simply a 2D array (or ‘matrix’) of numbers representing the intensity values of each pixel. The detection of objects in images is then generally based on mathematical operations on this matrix (Gonzalez and Woods 2018). A basic mathematical operation in image analysis is the convolution of the image matrix with a kernel, referred to as filtering. When filtering, each pixel of the image is modified based on its neighborhood, resulting in a new image. For example, noise in images can be reduced by averaging pixels over their local neighborhood, resulting in a slightly blurred image. Other choices and combinations of kernels may lead to filters that sharpen the image, emphasize edges, or amplify the signal of structures that have a particular size (e.g. the cell size) to facilitate object detection (Gonzalez and Woods 2018).

In traditional image analysis, images are typically processed by consecutively applying several different filters to achieve a background removal, an enhancement of the signal-to-noise level, and emphasis of structures of interest (illustrated in Fig. 1). Although kernel sizes can be automatically set based on the size of the cells (or colonies) that should be analyzed, the choice and order of filters is ultimately guided by the experience of the person who develops the code. If the quality or structure of the input images are changed, the parameters and filters often need to be adapted. To achieve a semantic segmentation that separates the objects of interest from the background, a threshold is then typically applied to images that have previously undergone filtering, resulting in a binary image termed ‘mask’, which only contains values of 1 and 0 for the foreground and background respectively (Fig. 1). A series of morphological operations can then be applied to the binary image so that the mask image represents an accurate semantic segmentation. These morphological operations can for example be the removal of small objects, filling of objects with holes, and widening or narrowing objects, resulting in the final semantic segmentation (Srisha and Khan 2013).

When individual cells can be visually discerned in images, it is possible to qualitatively pass judgment on the accuracy of the semantic segmentation by comparing the segmentation result with the input image. Such qualitative judgements reach their limit when individual cells cannot be visually distinguished due to crowding or lack of resolution. For example, when measuring the biovolume of a microbial community such as a biofilm without single-cell resolution, the pixel location of the biofilm edge is not uniquely defined, as the biofilm edge is typically characterized by a fluorescence signal gradient that spans several cell diameters. Different segmentation algorithms will therefore identify the biofilm edge to be at slightly different locations. As images of microbial communities without single-cell resolution are commonly analyzed in biofilm research, the need for accurate semantic segmentation has led to a remarkable number of different algorithmic solutions for biofilm biovolume detection (Heydorn et al. 2000; Yang et al. 2001; Beyenal et al. 2004; Yerly et al. 2007; Renslow, Lewandowski and Beyenal 2011; Klinger-Strobel et al. 2016). As illustrated by the development of so many solutions to a problem, traditional semantic segmentation provides users with a lot of freedom in terms of the filters, thresholding and morphological operations, which each come with a set of parameters that can be tuned. A systematic comparison of the accuracy of segmentation results based on quantitative metrics is therefore needed for any image analysis workflow, as described in Box 3.

Following semantic segmentation with traditional computational techniques as outlined above, further processing needs to be applied to achieve an accurate instance segmentation result (Fig. 1). Traditional instance segmentation typically relies on different types of edge detection algorithms (Roberts 1965; Canny 1986; Sobel 2014), the watershed algorithm (Beucher and Meyer 1993), skeleton algorithm (Lee, Kashyap and Chu 1994), or any other operation that works for a particular type of image. Traditional instance segmentation forms the basis of most current image analysis tools for microbiology (Box 1), including currently popular tools for bacterial cell biology in 2D images (Ducret, Quardokus and Brun 2016; Paintdakhi et al. 2016), and for 3D images of bacterial biofilm communities (Drescher et al. 2016; Yan et al. 2016; Wang et al. 2017; Hartmann et al. 2019).

Traditional image processing approaches can provide accurate results for semantic and instance segmentation, but they have substantial limitations. Usually the selection and order of filters that are applied, the filter parameters, as well as the thresholding and morphological operations have to be adapted in cases when the cells are close together or even touching each other. Similarly, input images that differ in terms of signal-to-noise levels, cell sizes, image background, or distribution of cell sizes, require the analysis algorithms to be adapted, limiting their range of applicability without manual parameter adjustments.

Convolutional neural networks for microbial object detection

By bypassing the limitations of traditional image analysis approaches, deep learning methods, such as convolutional neural networks (CNN) have dramatically improved segmentation accuracy in recent years not only for microscopy images, but for any type of image content (Caicedo et al. 2019). Given sufficient training data and computational resources, they can achieve highly accurate semantic or instance segmentation on a wide variety of cell types (Ronneberger, Fischer and Brox 2015; Schmidt et al. 2018; Hollandi et al. 2019; He et al. 2020; Stringer et al. 2020). With the appropriate network architectures, the applications of CNNs for biological image analysis are very diverse, ranging from classification tasks (e.g. identifying cell states), to image restoration and segmentation (Weigert et al. 2018; Moen et al. 2019a).

The performance of CNNs is closely linked to the network architecture design, but also strongly dependent on the quality and amount of training data that is available. For object detection tasks, training data consists of a pair of a raw image and its corresponding segmentation ground truth, e.g. images in which the locations of the cells are annotated (Fig. 2A). When using experimentally acquired images as raw input, this ground truth usually needs to be obtained via manual annotation, a labour-intensive task that can quickly become a bottleneck in the image analysis pipeline. Useful tools for 2D and 3D manual image annotation are available, including napari and AnnotatorJ/ImageJ (Napari Contributors 2019; Hollandi and Horváth 2020). It is important to recognize that manually annotated image segmentations are not free from error or uncertainty. Inaccuracies of the training data can be learned by the CNN and cause systematic errors in the segmentation results of the CNN.

Figure 2.

Typical segmentation workflow using convolutional neural networks. (A), A set of training data consisting of pairs of raw images and annotated images is used to train a convolutional neural network (CNN) indicated by a schematic icon of the network layers. The architecture of a CNN typically consists of several up- and downscaling layers as well as convolutions between layers. (B), After training, the resulting CNN can be used to obtain segmentation predictions for unseen raw images, which should be of the same type as the training data. Approaches for quantifying the accuracy of segmentation results are described in Box 3.

To overcome the challenge of obtaining a large amount of highly accurate manual annotations, several approaches have proven successful in the recent years. Image augmentation (Shorten and Khoshgoftaar 2019), for example, offers an option to increase the amount of training data with basic image transformations. Another common strategy to reduce the amount of new training data required for obtaining accurate segmentation results is termed transfer learning (Weiss, Khoshgoftaar and Wang 2016). Here, CNN models that have previously been trained with similar, but different, large data sets are re-trained with system-specific training data (Moen et al. 2019a).

Another possibility for generating large training data sets without the need for extensive manual annotation is to not provide experimentally obtained images, but simulated training data. With simulated training data, a perfect ground truth segmentation is automatically available, as simulated data is free from manual annotation uncertainties. This approach was successfully used, e.g. for tracking microspheres (Helgadottir, Argun and Volpe 2019) and bacterial cells (Zhang et al. 2020). It is worth noting however, that developing software to produce accurate and realistic training data resembling microscopy images is no trivial task and may initially be at least as time consuming as a manual annotation process.

Training a CNN is often computationally intensive, yet using a trained CNN for generating a segmentation of new images is typically quick and less computationally demanding (Fig. 2B). Fast and highly accurate instance segmentation provided by neural networks can be of tremendous use for microbiology. To fully utilize the potential of neural networks however, it is now necessary that user-friendly tools will be introduced allowing for easy manipulation and training of neural networks on regular personal computers, without the need for substantial programming expertise, access to high-performance computing, or complex software environments. Recent steps in this direction are already promising (Bannon et al. 2018; McQuin et al. 2018; Berg et al. 2019; Gómez-de-Mariscal et al. 2019; Ouyang et al. 2019; Moen et al. 2019b; von Chamier et al. 2020). Integration of CNN-based segmentation techniques into existing image analysis software tools for microbiology (Box 1) will be a major step towards making this technology widely available.

Image analysis beyond object detection

The improvements in image segmentation accomplished by neural networks will result in fast and accurate instance segmentation of most types of objects in microscopy images within the next few years. To obtain meaningful information from images, cells do not only have to be detected, but also queried for their quantitative properties, such as fluorescence intensity, intracellular structure, or morphology. Moreover, resolving the spatial arrangements of cells with respect to each other, nutrient sources or eukaryotic hosts, provides fruitful opportunities for understanding the cellular interactions within microbial communities, and for understanding the functions of communities.

Based on the progress in segmentation accuracy, microbial image analysis can now focus more strongly on new opportunities in cytometry, data analysis, and adaptive microscopy approaches that are enabled by highly accurate instantaneous object detection. Motivated by this shift in focus, we discuss recent progress in image-based microbial data analysis beyond object detection in the following sections.

Image cytometry of individual microbial cells

The increasing interest in bacterial cell biology, phenotypic heterogeneity and stochastic gene expression has generated the need for image-based analysis of single cells and sub-cellular structures (Kentner and Sourjik 2010; van Teeffelen, Shaevitz and Gitai 2012). A range of image analysis software tools have been made available that have increased in segmentation accuracy and functionality over the years (Guberman et al. 2008; Jones et al. 2008; Sliusarenko et al. 2011; Young et al. 2012; Mekterović, Mekterović and Maglica 2014; Vischer et al. 2015; Ducret, Quardokus and Brun 2016; Paintdakhi et al. 2016; Stylianidou et al. 2016; Balomenos et al. 2017; Goñi-Moreno, Kim and de Lorenzo 2017; Kaiser et al. 2018; Sauls et al. 2019; Smith, Metz and Pagliara 2019; Lugagne, Lin and Dunlop 2020; van Raaphorst, Kjos and Veening 2020; Hartmann et al. 2020b).

Recent software tools for microbial cell biology are highlighted in Box 1, and we provide suggestions for how to get started with using image analysis tools in Box 2. These tools are mostly designed for 2D images containing just a single layer of cells, acquired with high resolution microscope objectives. Such images can be obtained with cells growing between a glass cover slip and an agar pad (Young et al. 2012), in microfluidic chemostat chambers that are only one cell diameter high, such as the ‘mother machine’ (Wang et al. 2010; Kaiser et al. 2018) or similarly shallow but larger bioreactors (Dal Co, van Vliet and Ackermann 2019; Leygeber et al. 2019). Software tools designed for bacterial cell biology typically employ traditional image analysis approaches for instance segmentation, followed by the calculation of an intracellular coordinate system along the cell centerline, which ultimately provides various options for cytometric quantifications (based on fluorescence, absorbance, cell morphology, cell size, sub-cellular protein localization). These tools can include specific functionality for the analysis of stalks, flagella, and pili emerging from single cells (Hartmann et al. 2020b), or for particularities of the mother machine growth geometry (Kaiser et al. 2018; Sauls et al. 2019; Smith, Metz and Pagliara 2019; Lugagne, Lin and Dunlop 2020). The most recent tools also include cell tracking and lineage tracking functionality (Box 1). Generally, the existing software packages are a powerful toolset for microbiologists that are widely used for image cytometry in 2D, and are robust enough for screening large mutant libraries (Campos et al. 2018; Werner et al. 2009).

Image cytometry in communities

Image analysis of microbial communities, most notably biofilms, has typically not relied on the separate detection of individual cells. Instead, the focus has been on analyzing global community properties, such as volume, morphology, surface characterizations (Heydorn et al. 2000; Beyenal et al. 2004; Mueller et al. 2006; Vorregaard 2008; Klinger-Strobel et al. 2016; Kritikos et al. 2017), which can enable rapid screening of mutant libraries or environmental conditions (Kritikos et al. 2017; Canette, Deschamps and Briandet 2019), based on confocal microscopy (Schlafer and Meyer 2017; Jonkman et al. 2020). Tools developed for microbial ecology provide additional features including cell morphology analysis and fluorescence correlation functions (Liu et al. 2001; Daims, Lücker and Wagner 2006). Provided that images of communities with single-cell resolution are available, it is possible to compute an accurate instance segmentation and track cells during bacterial community growth in 3D using traditional image segmentation techniques (Drescher et al. 2016; Yan et al. 2016; Wang et al. 2017; Hartmann et al. 2019; Paula, Hwang and Koo 2020). CNNs will further improve the instance segmentation accuracy in biofilms in the future. Single-cell imaging of bacterial biofilm dynamics was recently used to understand the response of biofilms to antibiotics (Stewart et al. 2013; Díaz-Pascual et al. 2019), and phages (Vidakovic et al. 2017), as well as characterize biofilm dispersal (Singh et al. 2017) and mechanical interactions between cells in connection with individual based modeling (Beroz et al. 2018; Hartmann et al. 2019; Pearce et al. 2019).

For systems where single-cell resolution cannot be optically achieved, or where single-cell resolution is not necessary, a large number of community properties can still be measured spatially using cube cytometry (Hartmann et al. 2021). In this concept, after semantic segmentation, the image volume of the community is segmented into cubes (which can have the same size as single cells) to obtain a spatial context for measurements that are then performed on each cube analogous to the single-cell case. A potential drawback of the cube segmentation is that for communities with heterogeneity in cell size, the number of cells in a cube can vary. However, cube-based cytometry can enable spatially-resolved quantifications of microbial communities even when optical resolution does not permit single-cell segmentation, for example in microcolonies on agar, inside host intestines, or for other samples that are difficult to image with high-resolution microscope objectives.

The progress in single-cell segmentation and cube segmentation enables cytometry for cellular phenotyping in the context of communities. Precise phenotyping of cells in communities is important in order to characterize the species composition, phenotypic heterogeneity, and development of microbial communities from images. Phenotyping of cells within communities could be used, e.g. to reliably distinguish cells from different species and to identify cell cycle states, phenotypic subpopulations, or cellular differentiation states, which are common in spatially structured bacterial communities (Stewart and Franklin 2008).

Generally, it is desirable to be able to quantify as many different properties as possible for each cell in the community to increase the phenotyping precision of cells from images (Medyukhina et al. 2015; Hatzenpichler et al. 2020; Hartmann et al. 2021). The longer this list of distinct cytometric properties that can be accurately measured, the higher the precision with which phenotypic differences can be resolved for each cell. Therefore, it is advantageous to expand the list of different cellular properties that can be measured from images to the absolute maximum. The excellent current software tools for 2D image analysis for microbial cell biology are able to quantify several important properties per cell, such as the average fluorescence, properties of fluorescent spots, and also the location, area, and morphology of each cell (Ducret, Quardokus and Brun 2016; Paintdakhi et al. 2016; Hartmann et al. 2020b). Similar cellular fluorescence characterizations are also possible in 3D bacterial biofilms (Daims, Lücker and Wagner 2006; Hartmann et al. 2021). Yet the number of different fluorescence intensity-based measurements is ultimately limited, because only a few different fluorescent reporters can be used simultaneously, due to their overlapping fluorescence excitation and emission spectra. Even when pushed to the limit (Valm et al. 2017), cellular fluorescence measurements therefore result in only a limited number of distinct quantitative cytometric parameters per cell.

The spatial resolution that microscopy offers could be used to quantify numerous additional cytometric properties, which can be a little bit more abstract than the properties that are typically used for cytometry. Fluorescence distributions and correlations can be measured cell-internally or in the spatial context of a cell's neighborhood. Similarly, cellular orientations, density and texture can be determined for the neighbourhood of a cell and put into context with its spatial position (e.g. deep inside a microbial community vs. close to its exterior, or the relative positioning to eukaryotic host cells). Including such spatial measurements in the list of properties determined for each cell, as implemented for example in BiofilmQ (Hartmann et al. 2021), allows for a more precise characterization of single-cell and community phenotypes, to improve the resolution at which different behaviors and states can be distinguished.

Towards high-dimensional data analysis with spatial information

Once a sufficient number of cytometric properties has been extracted from the images for each cell, cell-neighbourhood, or entire community, each of these properties can be considered as a different dimension in a high-dimensional space. Each cell, neighbourhood, or entire community can then be represented by a point in this high-dimensional space. The proximity of points in this space represents how phenotypically similar particular cells, neighborhoods, or communities are to each other.

The high-dimensional data sets resulting from image cytometry therefore have a structure that is analogous to data sets generated by flow cytometry or single-cell RNA sequencing (Aghaeepour et al. 2013; Weinreb, Wolock and Klein 2017), even though the measured cellular properties are different. Therefore, similar data analysis strategies can be employed. For example, dimensionality reduction using principal component analysis (Box 4), or approaches that preserve local neighborhood structures during dimensionality reduction, such as t-SNE (van der Maaten and Hinton 2008), UMAP (Becht et al. 2019), or SPRING (Weinreb, Wolock and Klein 2017), provide a simplified visualization of the data and give an overview of the results that can sometimes directly be used for interpretation, e.g. for phenotyping cells or communities. Since distinct clusters in the high-dimensional property space typically correspond to different phenotypic states or behaviors, the application of a high-dimensional cluster-analysis enables a quantitative identification of these phenotypes, which could otherwise stay hidden. Such quantitative approaches for phenotyping avoid any biases that are inherent in manual phenotyping. A visualization of this classification in a low-dimensional representation may then further reveal transitions between cell states or community phenotypes (Haghverdi, Buettner and Theis 2015).

An important qualitative advantage of image cytometry compared with other single cell cytometry techniques is that image-based approaches can quantify the spatial context of each cell in situ within the community. The spatial location of a cell in the community, or the spatial location relative to a host cell, and perhaps the timepoint in a time series are therefore dimensions in the high-dimensional space resulting from image cytometry. These spatial and temporal dimensions, however, are qualitatively different from cellular phenotypic parameters such as fluorescence intensity or cell shape. More analysis strategies for single-cell phenotypic data in a spatiotemporal context are needed, not just for image cytometry of bacterial communities, but also for spatial RNA sequencing (Burgess 2019; ; Moncada et al.2020).

Towards image analysis-driven microscopy

The major improvements in segmentation, image cytometry, and high-dimensional data analysis described above also open up new possibilities for how microscopy and image acquisition can be performed.

By applying CNNs trained to restore high-quality microscopy images from low signal-to-noise data (Weigert et al. 2018), images can be acquired using much lower laser power, to reduce phototoxicity of the excitation illumination. In some cases, the use of fluorescence microscopy can even be completely avoided by training a neural network to predict fluorescence images from brightfield images, which dramatically reduces phototoxicity to the sample (Christiansen et al. 2018). These microscopy approaches are driven by image processing and do not rely on new optical techniques, yet they allow for faster and longer imaging of a living specimen, enabling previously unobtainable insight into developmental processes.

The possibility to detect and classify objects in real time during time lapse image acquisition routines also opens up a powerful new approach to a ‘smart’ adaptive microscopy: Specific objects, events, or developmental processes can be live-tracked by constantly monitoring and modifying the microscope imaging conditions based on live-acquired data (Fig. 3 ). This approach dramatically decreases the need for manual input and observation during an experiment and enables more specific imaging, reducing phototoxicity, and reducing the amount of unusable data, to increase the speed at which meaningful data can be acquired. Particularly the tracking of rare events could benefit tremendously from an adaptive microscopy approach. By measuring cytometric properties as described above, cells whose phenotypes differ from their ancestors or neighbors may be tracked automatically with this adaptive microscopy approach. The faster speed at which the computer can detect events compared to a human allows users to track processes that were previously impossible to follow, using such an integration of image analysis and microscope control. This concept of live-adapted microscopy was used in pioneering work on bacterial motility (Berg 1971), and more modern versions based on live image analysis were used for the imaging of a developing zebrafish (Royer et al. 2016), the expansion of a bacterial swarm (Jeckel et al. 2019), the growth of a bacterial biofilm (Hartmann et al. 2019), and can be expected to find more applications in the future.

Figure 3.

Integration of single-cell image analysis with adaptive microscopy enables highly specific imaging of communities. After capturing the raw image (panel A), cells are distinguished from background to provide a semantic segmentation as shown in panel B. These images are then further processed to identify individual cells, providing an instance segmentation (C). For each cell, a list of properties can be quantified from the imaging data (D). These properties can be interpreted as the dimensions of a high-dimensional space where each cell is represented by one point. (E), A dimensionality reduction, such as principal component analysis, may uncover distinct clusters in this image cytometry parameter space, which correspond to phenotypically different sub-populations. (F), Based on the high-dimensional cytometry analysis, the next experimental steps can be determined, for example zooming in on only those parts of the community with particular phenotypes, to increase the specificity and resolution of the experimental procedure. This concept of live image analysis and adaptation of the microscopy acquisition parameters may be repeated in a feedback loop to optimize and automate experiments.

Figure 4.

Schematic figure of the intersection over union (IoU) and the segmentation accuracy for different image analysis methods, measured as a function of the IoU threshold. (A), Visualization of the IoU, which is defined as the fraction between the size of the intersecting area between two regions and the size of their union. (B), In this schematic plot, different colors represent different segmentation methods, with different segmentation performance. High Jaccard index values indicate a high segmentation accuracy. Note that the best method for a particular choice of IoU threshold does not need to be the best method for all threshold values.

Concluding remarks

Image-based analysis of single-cell properties has been increasingly used in bacterial cell biology, following the release of several powerful image analysis tools. Similarly, software tools for image cytometry within microbial communities have become available, enabling the spatial and temporal characterization of cells inside their community niches.

The precision of cellular phenotyping from images can be improved by increasing the number of distinct properties that are quantified for each cell, and it will be important to theoretically analyze the maximum number of meaningful quantifiable properties, to explore the limits of cellular phenotyping precision from images. Image cytometry can result in a large amount of data for each cell, even without a microbial community context, so that an analysis framework that provides a general path from data to interpretation, with quantification of statistical uncertainties for hypotheses testing would be an important future development for users. Ultimately, wide-spread adoption of quantitative image analysis techniques critically relies on user-friendly software tools that require no (or minimal) low-level coding. Further tools, particularly those for machine learning based instance segmentation, spatial high-dimensional data analysis, and adaptive microscopy need to be developed, in order to harness the full potential of these emerging techniques in microbiology.

ACKNOWLEDGEMENTS

We are grateful to Eric Jelli, Takuya Ohmura and Niklas Netter for helpful discussions and important feedback on the manuscript.

FUNDING

This work was supported by the European Research Council [StG‐716 734 to K.D.]; Max Planck Society [to K.D.]; Studienstiftung des deutschen Volkes [to H.J.]; Joachim Herz Stiftung [to H.J.].

Conflicts of interest

None declared.