Model Calibration Measurement With Multiple Metrics And Binning Strategies

via “evaluation metrics computation with task-specific scoring”

Microsoft's unified LLM evaluation and prompt robustness benchmark.

Unique: Automatically selects and computes task-specific metrics (accuracy for classification, BLEU/ROUGE for generation, exact match for reasoning) based on dataset type, reducing metric implementation boilerplate compared to manual metric selection

vs others: Faster than implementing metrics manually because metric selection is automatic and normalized across tasks, but less flexible than custom metric implementations