MS COCO (Common Objects in Context)

Provides 2.5 million manually-annotated object instances across 330,000 images, with each instance labeled by category (80 base classes), spatial bounding box coordinates, and pixel-level instance segmentation masks. Annotations are stored in standardized JSON format with hierarchical category taxonomy, enabling training of detection and segmentation models that understand both object identity and precise spatial boundaries. The annotation pipeline uses human annotators with quality control mechanisms to ensure consistency across the dataset.

Provides 5 diverse natural language captions per image (1.65M total captions across 330K images), each written by independent human annotators to capture different aspects of visual content. Captions are stored as free-form text in JSON annotation files and enable training of vision-language models, image-to-text systems, and evaluating caption quality through metrics like BLEU, METEOR, CIDEr, and SPICE. The multi-caption approach captures linguistic diversity and allows evaluation of caption generation systems against multiple reference descriptions.

Provides 330,000 images collected from Flickr with natural scene diversity spanning indoor/outdoor, multiple viewpoints, scales, and lighting conditions. Images are selected to contain multiple objects (average ~3.5 objects per image) and natural context, avoiding artificial or overly-controlled scenarios. The collection emphasizes 'objects in context' rather than isolated object crops, enabling models to learn detection and segmentation in realistic scenarios with occlusion, scale variation, and complex backgrounds. Image resolution and aspect ratio distribution unknown, but collection spans typical web image characteristics.

Provides keypoint annotations for the person category, marking specific anatomical joint locations (e.g., shoulders, elbows, knees, ankles) as (x, y, visibility) tuples in JSON format. Annotations cover all person instances in images, enabling training of pose estimation models that predict human skeletal structure. The visibility flag indicates whether each keypoint is visible, occluded, or outside image bounds, allowing models to handle partial visibility. Keypoint definitions follow a standardized anatomical schema (specific joint count and standard unknown from provided content).

Extends base COCO with panoptic segmentation annotations that unify instance segmentation (countable objects like people, cars) and stuff segmentation (amorphous regions like sky, grass) into a single per-pixel category prediction. Annotations include both instance IDs and semantic category labels, stored as segmentation maps with category mappings in JSON. The COCO-Stuff variant expands the taxonomy from 80 to 171 categories by adding 91 stuff classes, enabling models to predict complete scene understanding rather than just salient objects.

Provides an online evaluation infrastructure where researchers submit model predictions in standardized COCO format, and the system automatically computes metrics against withheld ground truth. The leaderboard maintains separate test sets for detection, segmentation, keypoints, panoptic, and captioning tasks, with results ranked by metric (AP, AP50, AP75 for detection; PQ for panoptic; CIDEr for captions). The withheld test set prevents overfitting to public validation data and ensures fair comparison across methods. Submission requires formatting predictions in COCO JSON format and uploading via the website interface.

Provides a single unified dataset where each image contains annotations for multiple vision tasks: object detection (bounding boxes), instance segmentation (masks), image captioning (5 captions), and human pose (keypoints). The unified JSON annotation schema maps all task annotations to the same image_id, enabling multi-task learning where models jointly optimize detection, segmentation, caption generation, and pose estimation. This integration allows researchers to train models that leverage shared visual representations across tasks, improving generalization and reducing annotation redundancy.

Extends COCO with DensePose annotations that map image pixels to 3D human body surface coordinates, enabling dense correspondence between 2D image space and 3D body model. Each person instance receives a dense map where pixels are labeled with (body_part_id, u, v) coordinates indicating which part of the 3D body model they correspond to. This enables training models for human body understanding, texture transfer, and 3D pose reconstruction. The mechanism uses a parametric body model (SMPL or similar) to define the 3D surface, and annotations map image pixels to this surface.

Provides a structured category taxonomy with 80 base thing (countable object) categories in standard COCO, expandable to 171 total categories in COCO-Stuff by adding 91 stuff (amorphous region) classes. Categories are organized hierarchically with semantic relationships (e.g., 'dog' and 'cat' are both 'animal'), enabling models to leverage category structure for improved generalization. Each category has metadata including isthing flag (distinguishing countable objects from background regions), color for visualization, and supercategory groupings. The taxonomy enables both fine-grained and coarse-grained predictions depending on model requirements.

Implements standardized evaluation metrics for each COCO task: Average Precision (AP, AP50, AP75) and Average Recall (AR) for detection/segmentation using IoU thresholds; Panoptic Quality (PQ) combining segmentation and recognition quality; Object Keypoint Similarity (OKS) for pose estimation; and caption metrics (BLEU, METEOR, CIDEr, SPICE) for image captioning. Metrics are computed against ground truth using official COCO evaluation code, ensuring reproducibility and fair comparison across methods. The metric definitions follow standard computer vision conventions (e.g., AP computed at IoU=0.5:0.95 for detection).

Manages multiple dataset variants and versions: base COCO (80 categories, detection/segmentation/captions/keypoints), COCO-Stuff (171 categories adding stuff classes), COCO Panoptic (unified instance+stuff segmentation), and DensePose extension (dense correspondence). Each variant maintains separate annotation files and evaluation leaderboards while sharing the same image collection, enabling researchers to choose appropriate variant for their task. Version history and release notes document changes across years (2015-2020+), allowing reproducibility of published results on specific dataset versions.