Prodigy

Prodigy uses active learning algorithms to rank unlabeled examples by annotation uncertainty, presenting the most informative samples first to human annotators. The system learns from each labeled example and dynamically reorders the queue, reducing labeling effort by prioritizing high-impact annotations over random sampling. This is implemented via a scoring mechanism that evaluates model confidence on incoming data and surfaces edge cases and ambiguous examples.

Prodigy provides a specialized NER annotation interface where users highlight text spans and assign entity labels (PERSON, PRODUCT, ORG, etc.) via keyboard shortcuts or UI clicks. The system supports pre-population of entity suggestions from upstream models or rule-based taggers, allowing annotators to accept/reject/correct predictions rather than labeling from scratch. Spans are stored as character offsets in the database, preserving exact positional information for downstream model training.

Prodigy stores all annotations in a local SQLite database on the user's machine. No data is transmitted to external servers or cloud services — the system is designed for complete data privacy and offline operation. The database can be backed up, version-controlled, or migrated to other machines. Prodigy includes utilities to inspect, export, and manage the database directly via Python API or CLI commands.

Prodigy is tightly integrated with spaCy, the open-source NLP library by the same creators. Users can load pre-trained spaCy models to pre-populate entity predictions, classify documents, or score examples for active learning. The system supports all spaCy model types (NER, text classification, dependency parsing, etc.) and enables fine-tuning spaCy models on annotated data. This integration eliminates the need for separate model serving infrastructure.

Prodigy enables developers to implement conditional annotation workflows where different examples are routed to different tasks based on metadata, model predictions, or custom logic. For example, high-confidence predictions can skip human review while low-confidence examples go to detailed annotation. Task routing is implemented via custom recipes that inspect example metadata and return different task configurations. This enables efficient multi-stage annotation pipelines.

Prodigy provides a statistics interface (accessible via `prodigy stats` command) that displays real-time annotation progress, including total examples annotated, annotation speed (examples/hour), dataset size, number of sessions, and per-annotator metrics. The dashboard updates as annotations are saved and can be filtered by dataset or date range. Statistics are computed from the SQLite database and include metadata like annotation duration and inter-annotator agreement.

Prodigy enables document-level text classification where annotators assign one or more category labels to entire text examples. The system supports both flat multi-label classification (example can have labels A, B, C simultaneously) and hierarchical category trees. Classification decisions are recorded with metadata (timestamp, annotator ID) and can be reviewed/corrected in subsequent passes. The interface uses button-based selection for fast labeling.

Prodigy supports computer vision annotation tasks including bounding box drawing, polygon/freehand segmentation, and point annotation on images. Annotators draw shapes directly on images using mouse/touch, and coordinates are stored as normalized or pixel-space values. The system supports batch image loading from directories or URLs and can pre-populate predictions from object detection or segmentation models for correction workflows.

Prodigy includes an evaluation mode where annotators compare two model predictions side-by-side and select the better one, or rate predictions on a scale. This is used to benchmark different models, compare annotation strategies, or evaluate model improvements. Results are aggregated to compute inter-annotator agreement, model accuracy, and ranking scores. The system records which prediction was preferred and can export evaluation metrics for statistical analysis.

Prodigy enables developers to create custom annotation workflows by writing Python functions decorated with @prodigy.recipe(). Recipes define task logic, data loading, and UI configuration. Arguments are bound via Arg() objects with type hints and validation, enabling CLI argument parsing without boilerplate. Recipes can compose multiple annotation tasks, integrate external models, or implement domain-specific workflows. The recipe system is the primary extension point for Prodigy.

Prodigy supports integrating Large Language Models into the annotation workflow via custom recipes. LLMs can be used to generate initial label suggestions, pre-populate entity predictions, or classify documents before human review. The system is model-agnostic — recipes can call any LLM API (OpenAI, Anthropic, local models) or use spaCy's built-in statistical models. Suggestions are presented to annotators for acceptance/rejection, enabling efficient correction workflows.

Prodigy provides export functionality to extract annotated data from the SQLite database in multiple formats (JSONL, JSON, CSV, spaCy training format). Exports can be filtered by dataset name, annotation status, date range, or custom metadata. The system preserves annotation history (all versions of a label) and can compute inter-annotator agreement metrics during export. Exported data is ready for model training or downstream analysis.

Prodigy supports annotating structured relationships between entities or spans, such as dependency parsing (subject-verb-object) or relation extraction (Person-WORKS_FOR-Organization). Annotators select two spans and assign a relation label, creating a directed graph of relationships. Relations are stored with head/child indices and labels, enabling training of relation extraction or dependency parsing models. The interface supports both free-form relation creation and constrained relation types.

Prodigy supports assigning the same examples to multiple annotators to measure inter-annotator agreement and identify ambiguous or controversial examples. The system tracks which annotator labeled each example, computes agreement metrics (exact match, partial overlap), and flags examples with low agreement for review. Conflicts can be resolved via a dedicated review interface where a senior annotator selects the correct label.