evaluate

Implements a factory-based module loading system that dynamically discovers and imports evaluation metrics from three sources: Hugging Face Hub (as Spaces), local filesystem, or community repositories. Uses a standardized EvaluationModule base class hierarchy with lazy loading to defer instantiation until compute time, enabling version control and caching of metric definitions across distributed environments.

Provides distributed computation infrastructure for metrics through a caching layer that stores intermediate results and supports batch processing across multiple workers. Integrates with distributed frameworks (e.g., Hugging Face Datasets) to parallelize metric computation, with automatic result aggregation and deduplication to avoid redundant calculations across runs.

Provides a command-line interface (evaluate-cli) and programmatic API for creating custom evaluation modules and publishing them to the Hugging Face Hub as Spaces. Scaffolds module structure with boilerplate code, documentation templates, and test files, then handles Hub authentication and deployment with automatic versioning and widget generation.

Provides inspect() and list_evaluation_modules() functions that query module metadata (description, inputs, outputs, citations) without loading the full module. Enables programmatic discovery of available metrics, comparisons, and measurements with filtering by type, task, or keyword, supporting both Hub and local module discovery.

Provides seamless integration with Hugging Face Transformers (model evaluation) and Datasets (distributed data loading) through shared APIs and automatic format conversion. Metrics accept Datasets objects directly, enabling zero-copy evaluation on partitioned datasets, and integrate with Transformers' Trainer class for automatic evaluation during training.

Provides EvaluationSuite class for bundling multiple metrics, comparisons, and measurements into a single reusable configuration that can be saved, versioned, and shared. Suites are defined declaratively (YAML or Python) and can be instantiated with different datasets or models, enabling reproducible evaluation across projects and teams.

Provides high-level Evaluator classes that automatically select and combine appropriate metrics for specific ML tasks (text classification, question answering, summarization, etc.) without requiring users to manually specify metrics. Each task evaluator inherits from a base Evaluator class and implements task-specific logic for metric selection, input validation, and result aggregation based on model type and dataset characteristics.

Allows bundling multiple metrics into a single CombinedEvaluations instance that computes all metrics in one pass, reducing redundant data loading and enabling efficient ensemble evaluation. The combine() function accepts multiple EvaluationModule instances and orchestrates their execution with shared input caching, returning aggregated results with optional per-metric metadata.

Provides Comparison modules (e.g., McNemar test, exact match comparison) that perform statistical significance testing between predictions from two or more models on the same dataset. Implements hypothesis testing with configurable significance levels and returns p-values, test statistics, and confidence intervals to determine if performance differences are statistically significant.

Provides Measurement modules that analyze properties of datasets or predictions without requiring ground truth labels (e.g., toxicity detection, perplexity, word length distribution). Measurements inherit from EvaluationModule and implement compute() methods that take only predictions as input, enabling analysis of dataset characteristics and model outputs independent of task-specific evaluation.

Implements a suite of classification metrics (accuracy, precision, recall, F1, confusion matrix) with built-in support for binary, multi-class, and multi-label classification scenarios. Each metric is a Metric subclass that handles different label formats (integers, strings, one-hot encodings) and averaging strategies (macro, micro, weighted) automatically based on input shape and configuration.

Provides text generation metrics (BLEU, ROUGE, METEOR, BERTScore, BLEURT) that measure quality of generated text against references or independently. Implements both reference-based metrics (comparing to gold-standard text) and reference-free metrics (evaluating intrinsic properties like fluency) with configurable tokenization, smoothing, and aggregation strategies.

Provides sequence labeling metrics (precision, recall, F1, seqeval) that evaluate token-level predictions for tasks like named entity recognition (NER) and part-of-speech tagging. Implements BIO/BIOES tag scheme handling with automatic tag parsing and entity-level evaluation, distinguishing between token-level and entity-level metrics.

Provides question answering metrics (exact match, F1, BLEU) that evaluate predicted answers against reference answers using token-level overlap and span matching. Implements SQuAD-style evaluation with automatic answer normalization (lowercasing, punctuation removal) and support for multiple reference answers per question.