The Cell Tracking Challenge is an ongoing benchmarking initiative that has become a reference in cell segmentation and tracking algorithm development. Here we present a significant number of improvements introduced in the challenge since our last 2017 report. These include the creation of a new segmentation-only benchmark, the enrichment of the dataset repository with new datasets that increase its diversity and complexity, and the creation of a silver-standard reference corpus based on the most competitive results, of particular interest for data-hungry deep learning-based strategies. Furthermore, we present the up-to-date cell segmentation and tracking leaderboards, an in-depth analysis of the relationship between the performance of the state-of-the-art methods and the properties of the datasets and annotations, and two novel, insightful studies about the generalizability and the reusability of top-performing methods. These studies provide critical practical conclusions for both developers and users of traditional and machine learning-based cell segmentation and tracking algorithms.

The segmentation of cells present in microscope images is an essential step in many tasks, including determining protein concentration and analysis of gene expression per cell. In single-cell genomics studies, cell segmentations are vital to assess the genetic makeup of individual cells and their relative spatial location. Several methods and tools have been developed to offer robust segmentation, with deep learning models currently being the most promising solutions. As an alternative to developing another cell segmentation targeted model, we propose a learning ensemble strategy that aggregates many independent candidate segmentations of the same image to produce a single consensus segmentation. We are particularly interested in learning how to ensemble crowdsource image segmentations created by experts and non-experts in laboratories and data houses. We compare our trained ensemble model with other fusion methods adopted by the biomedical community and assess the robustness of the results on three aspects: fusion with outliers, missing data, and synthetic deformations. Our approach outperforms these methods in efficiency and quality, especially when there is a high disagreement among candidate segmentations of the same image.

The correct targeting and insertion of tail-anchored (TA) integral membrane proteins is critical for cellular homeostasis. TA proteins are defined by a hydrophobic transmembrane domain (TMD) at their C-terminus and are targeted to either the ER or mitochondria. Derived from experimental measurements of a few TA proteins, there has been little examination of the TMD features that determine localization. As a result, the localization of many TA proteins are misclassified by the simple heuristic of overall hydrophobicity. Because ER-directed TMDs favor arrangement of hydrophobic residues to one side, we sought to explore the role of geometric hydrophobic properties. By curating TA proteins with experimentally determined localizations and assessing hypotheses for recognition, we bioinformatically and experimentally verify that a hydrophobic face is the most accurate singular metric for separating ER and mitochondria-destined yeast TA proteins. A metric focusing on an 11 residue segment of the TMD performs well when classifying human TA proteins. The most inclusive predictor uses both hydrophobicity and C-terminal charge in tandem. This work provides context for previous observations and opens the door for more detailed mechanistic experiments to determine the molecular factors driving this recognition.

Acetylcholine (ACh) promotes various cell migrations in vitro, but there are few investigations into this nonsynaptic role of ACh signaling in vivo. Here we investigate the function of a muscarinic receptor on an epithelial cell migration in Caenorhabditis elegans We show that the migratory gonad leader cell, the linker cell (LC), uses an M1/M3/M5-like muscarinic ACh receptor GAR-3 to receive extrasynaptic ACh signaling from cholinergic neurons for its migration. Either the loss of the GAR-3 receptor in the LC or the inhibition of ACh release from cholinergic neurons resulted in migratory path defects. The overactivation of the GAR-3 muscarinic receptor caused the LC to reverse its orientation through its downstream effectors G?q/egl-30, PLC?/egl-8, and TRIO/unc-73 This reversal response only occurred in the fourth larval stage, which corresponds to the developmental time when the GAR-3::yellow fluorescent protein receptor in the membrane relocalizes from a uniform to an asymmetric distribution. These findings suggest a role for the GAR-3 muscarinic receptor in determining the direction of LC migration.

Individual cell migration requires front-to-back polarity manifested by lamellipodial extension. At present, it remains debated whether and how membrane motility mediates this cell morphological change. To gain insights into these processes, we perform live imaging and molecular perturbation of migrating chick neural crest cells in vivo. Our results reveal an endocytic loop formed by circular membrane flow and anterograde movement of lipid vesicles, resulting in cell polarization and locomotion. Rather than clathrin-mediated endocytosis, macropinosomes encapsulate F-actin in the cell body, forming vesicles that translocate via microtubules to deliver actin to the anterior. In addition to previously proposed local conversion of actin monomers to polymers, we demonstrate a surprising role for shuttling of F-actin across cells for lamellipodial expansion. Thus, the membrane and cytoskeleton act in concert in distinct subcellular compartments to drive forward cell migration.

In situ hybridization based on the mechanism of hybridization chain reaction (HCR) enables multiplexed quantitative mRNA imaging in diverse sample types. Third-generation in situ HCR (v3.0) provides automatic background suppression throughout the protocol, dramatically enhancing performance and ease of use. In situ HCR v3.0 supports two quantitative imaging modes: (1) qHCR imaging for analog mRNA relative quantitation with subcellular resolution and (2) dHCR imaging for digital mRNA absolute quantitation with single-molecule resolution. Here, we provide protocols for qHCR and dHCR imaging in mammalian cells on a slide.

We propose a new loss formulation to further advance the multiclass segmentation of cluttered cells under weakly supervised conditions. When adding a Youden's J statistic regularization term to the cross entropy loss we improve the separation of touching and immediate cells, obtaining sharp segmentation boundaries with high adequacy. This regularization intrinsically supports class imbalance thus eliminating the necessity of explicitly using weights to balance training. Simulations demonstrate this capability and show how the regularization leads to correct results by helping advancing the optimization when cross entropy stagnates. We build upon our previous work on multiclass segmentation by adding yet another training class representing gaps between adjacent cells. This addition helps the classifier identify narrow gaps as background and no longer as touching regions. We present results of our methods for 2D and 3D images, from bright field images to confocal stacks containing different types of cells, and we show that they accurately segment individual cells after training with a limited number of images, some of which are poorly annotated.

We present a weakly supervised deep learning method to perform instance segmentation of cells present in microscopy images. Annotation of biomedical images in the lab can be scarce, incomplete, and inaccurate. This is of concern when supervised learning is used for image analysis as the discriminative power of a learning model might be compromised in these situations. To overcome the curse of poor labeling, our method focuses on three aspects to improve learning: (i) we propose a loss function operating in three classes to facilitate separating adjacent cells and to drive the optimizer to properly classify underrepresented regions; (ii) a contour-aware weight map model is introduced to strengthen contour detection while improving the network generalization capacity; and (iii) we augment data by carefully modulating local intensities on edges shared by adjoining regions and to account for possibly weak signals on these edges. Generated probability maps are segmented using different methods, with the watershed based one generally offering the best solutions, specially in those regions where the prevalence of a single class is not clear. The combination of these contributions allows segmenting individual cells on challenging images. We demonstrate our methods in sparse and crowded cell images, showing improvements in the learning process for a fixed network architecture.

We present an algorithm to compute the geometric median of shapes which is based on the extension of median to high dimensions. The median finding problem is formulated as an optimization over distances and it is solved directly using the watershed method as an optimizer. We show that the geometric median shape faithfully represents the true central tendency of the data, contaminated or not. It is superior to the mean shape which can be negatively affected by the presence of outliers. Our approach can be applied to manifold and non manifold shapes, with single or multiple connected components. The application of distance transform and watershed algorithm, two well established constructs of image processing, lead to an algorithm that can be quickly implemented to generate fast solutions with linear storage requirement. We demonstrate our methods in synthetic and natural shapes and compare median and mean results under increasing outlier contamination.

The quantitative analysis of cellular membranes helps understanding developmental processes at the cellular level. Particularly 3D microscopic image data offers valuable insights into cell dynamics, but error-free automatic segmentation remains challenging due to the huge amount of data generated and strong variations in image intensities. In this paper, we propose a new 3D segmentation approach that combines the discriminative power of convolutional neural networks (CNNs) for preprocessing and investigates the performance of three watershed-based postprocessing strategies (WS), which are well suited to segment object shapes, even when supplied with vague seed and boundary constraints. To leverage the full potential of the watershed algorithm, the multi-instance segmentation problem is initially interpreted as three-class semantic segmentation problem, which in turn is well-suited for the application of CNNs. Using manually annotated 3D confocal microscopy images of Arabidopsis thaliana, we show the superior performance of the proposed method compared to the state of the art.

We present a weakly supervised deep learning strategy to perform instance segmen- tation of cells present in microscopy images. Annotation of biomedical images is scarce and often confined to the lab doing the experiments. Such images are usually acquired using specific and targeted protocols that are not necessarily standardized and thus not common to other labs working on similar experiments. In practice, this generally leads to reduced annotations by a single individual on a small set of images. Additionally, annotations might be incomplete and not as accurate as one done by a pool of candidate annotators and then reconciled based on their individual contributions. This is of concern when adopting supervised learning approaches to perform image analysis as the training of a learning model can be compromised by inaccurate, incomplete, skewed training data. The solution we present for the segmentation of individual cells in microscopy images tries to cope with these practical limitations. Specifically, we propose an augmented loss function such that it guides the optimizer to pay attention to critical regions that need to be properly identified to separate butting cells. For that to take effect on the T-cell images we have to segment, we also had to introduce the classification of pixels into three classes (see Fig.1), as the usual binary foregound-background classification is not sufficient to distinguish between cell interiors and butting regions - these get merged resulting in topologically incorrect segmentations.

We propose a new multiclass weighted loss function for instance segmentation of cluttered cells. We are primarily motivated by the need of developmental biologists to quantify and model the behavior of blood T-cells which might help us in understanding their regulation mechanisms and ultimately help researchers in their quest for developing an effective immunotherapy cancer treatment. Segmenting individual touching cells in cluttered regions is challenging as the feature distribution on shared borders and cell foreground are similar thus difficulting discriminating pixels into proper classes. We present two novel weight maps applied to the weighted cross entropy loss function which take into account both class imbalance and cell geometry. Binary ground truth training data is augmented so the learning model can handle not only foreground and background but also a third touching class. This framework allows training using U-Net. Experiments with our formulations have shown superior results when compared to other similar schemes, outperforming binary class models with significant improvement of boundary adequacy and instance detection. We validate our results on manually annotated microscope images of T-cells.

In situ hybridization based on the mechanism of the hybridization chain reaction (HCR) has addressed multi-decade challenges that impeded imaging of mRNA expression in diverse organisms, offering a unique combination of multiplexing, quantitation, sensitivity, resolution and versatility. Here, with third-generation in situ HCR, we augment these capabilities using probes and amplifiers that combine to provide automatic background suppression throughout the protocol, ensuring that reagents will not generate amplified background even if they bind non-specifically within the sample. Automatic background suppression dramatically enhances performance and robustness, combining the benefits of a higher signal-to-background ratio with the convenience of using unoptimized probe sets for new targets and organisms. In situ HCR v3.0 enables three multiplexed quantitative analysis modes: (1) qHCR imaging - analog mRNA relative quantitation with subcellular resolution in the anatomical context of whole-mount vertebrate embryos; (2) qHCR flow cytometry - analog mRNA relative quantitation for high-throughput expression profiling of mammalian and bacterial cells; and (3) dHCR imaging - digital mRNA absolute quantitation via single-molecule imaging in thick autofluorescent samples.

The quantitative analysis of 3D confocal microscopy images of the shoot apical meristem helps understanding the growth process of some plants. Cell segmentation in these images is crucial for computational plant analysis and many automated methods have been proposed. However, variations in signal intensity across the image mitigate the effectiveness of those approaches with no easy way for user correction. We propose a web-based collaborative 3D image segmentation application, SEGMENT3D, to leverage automatic segmentation results. The image is divided into 3D tiles that can be either segmented interactively from scratch or corrected from a pre-existing segmentation. Individual segmentation results per tile are then automatically merged via consensus analysis and then stitched to complete the segmentation for the entire image stack. SEGMENT3D is a comprehensive application that can be applied to other 3D imaging modalities and general objects. It also provides an easy way to create supervised data to advance segmentation using machine learning models.

Automated segmentation approaches are crucial to quantitatively analyze large-scale 3D microscopy images. Particularly in deep tissue regions, automatic methods still fail to provide error-free segmentations. To improve the segmentation quality throughout imaged samples, we present a new supervoxel-based 3D segmentation approach that outperforms current methods and reduces the manual correction effort. The algorithm consists of gentle preprocessing and a conservative super-voxel generation method followed by supervoxel agglomeration based on local signal properties and a postprocessing step to fix under-segmentation errors using a Convolutional Neural Network. We validate the functionality of the algorithm on manually labeled 3D confocal images of the plant Arabidopsis thaliana and compare the results to a state-of-the-art meristem segmentation algorithm.

Single-molecule fluorescence in situ hybridization (smFISH) is a simple and widely used method to measure mRNA transcript abundance and localization in single cells. A comparable single-molecule in situ method to measure mRNA translation would enable a more complete understanding of gene regulation. Here we describe a fluorescence assay to detect ribosome interactions with mRNA (FLARIM). The method adapts smFISH to visualize and characterize translation of single molecules of mRNA in fixed cells. To visualize ribosome-mRNA interactions, we use pairs of oligonucleotide probes that bind separately to ribosomes (via rRNA) and to the mRNA of interest, and that produce strong fluorescence signals via the hybridization chain reaction (HCR) when the probes are in close proximity. FLARIM does not require genetic manipulation, is applicable to practically any endogenous mRNA transcript, and provides both spatial and temporal information. We demonstrate that FLARIM is sensitive to changes in ribosome association with mRNA upon inhibition of global translation with puromycin. We also show that FLARIM detects changes in ribosome association with an mRNA whose translation is upregulated in response to increased concentrations of iron.

There is a great difficulty of creating robust image segmentation algorithms to automate the delineation of objects of interest present in images. While humans can seemingly do this without much effort, computers have a hard time to distinguish even simple objects. This is the case of biological images where cells might not be uniformly marked with fluorescent proteins leading to gaps and spurious data and consequently to poor segmentation results We combine Deep Convolutional Neural Network (CNN) and Structured Prediction to provide a learned joined pipeline that can segment plant stem cells from the Arabidopsis thaliana model organism.

Chromatin is a complex of DNA, RNA, and proteins that looks like beads on a string when imaged on an optical microscope. It has recently been hypothesized that local changes in chromatin conformation can lead to the spatial and temporal repositioning of cisregulatory modules, CRM for short (a segment of DNA), relative to their regulated genes which in turn may be responsible for changing patterns of gene expression. We study how the expression of a certain gene in early Drosophila embryo is controlled by different chromatin conformations and governed by the relative positioning of a 3-F¢CRM (3), a-A 5-F¢CRM (5), and a promoter proximal element (P) in a single-A DNA. To identify the position of (3,5,P) triplets, we use fluorescence insitu hybridization, FISH, where different fluorescence probes help indicate the locations of the CRMs and the promoter. A fourth probe reports structural information to help identify the image orientation relative to embryo alignment. We acquire and analyze multichannel z-stack images covering portions of the Drosophila embryo epidermis using a widefield microscope.