Custom Task Definition Via Python Classes With Metric Registration

EleutherAI's evaluation framework — 200+ benchmarks, powers Open LLM Leaderboard.

Unique: Provides a Task base class that users can extend to implement custom evaluation logic, with automatic registration in the global task registry. Custom tasks can override request generation, metric computation, and result aggregation. Metrics are registered separately and can be reused across tasks, enabling modular metric development.

vs others: Enables arbitrary Python logic for task definition and metrics, whereas YAML-based tasks are limited to built-in capabilities; integrates custom tasks into the evaluation pipeline with automatic batching and caching support