Train an Instance Segmentation Model with Custom Dataset
Instance Segmentation: Beyond Bounding Boxes
Instance segmentation combines object detection with pixel-level masks. Each detected object gets a precise outline, not just a rectangle. This is important when objects overlap, when exact shape matters (measuring area, extracting contours), or when downstream processing needs clean masks rather than bounding boxes.
What This Tutorial Covers
Datature walks through training an instance segmentation model on a custom dataset using Nexus:
- Setting up a project with polygon annotation support
- Labeling objects with precise polygon masks
- Configuring the training workflow for instance segmentation
- Running inference and comparing predicted masks to ground truth
The tutorial takes about six minutes. The entire pipeline runs inside Nexus without switching to external tools or writing training scripts.
When to Choose Instance Segmentation
Use instance segmentation when bounding boxes are not precise enough. Examples: counting overlapping fruits in a bin, measuring crack areas on concrete surfaces, segmenting individual cells in microscopy images, or extracting product silhouettes for catalog photography. The additional annotation effort pays off when your application needs per-pixel accuracy.
For a written companion guide, read Training an Instance Segmentation Model. To speed up polygon labeling, see Using AI-Guided Segmentation to Speed Up Labelling with IntelliBrush.
