Mask R-CNN is a two-stage instance segmentation architecture that extends Faster R-CNN by adding a parallel mask prediction branch alongside the existing bounding box and classification heads. Published by He et al. (Facebook AI Research) in 2017, it was the first architecture to cleanly separate instance segmentation into detection (which objects, where) and mask prediction (which pixels belong to each detected object).

The pipeline works in two stages. First, a Region Proposal Network (RPN) generates candidate object regions from backbone features (typically ResNet + FPN). Second, for each proposal, three parallel heads predict: the class label, refined bounding box coordinates, and a binary pixel mask within the box. A key contribution was RoIAlign, which replaced the coarser RoIPool with bilinear interpolation to preserve exact spatial alignment between feature maps and proposals, improving mask boundary quality significantly.

Mask R-CNN became the baseline for instance segmentation research and remains widely used in production systems. Its modular design makes it easy to swap backbones, add keypoint heads (for pose estimation), or attach panoptic segmentation branches. While newer architectures like Mask2Former and SAM have pushed accuracy higher, Mask R-CNN's straightforward training and well-understood behavior make it a reliable choice for custom datasets. Datature Nexus supports Mask R-CNN for instance segmentation training.