Pairwise Preference Collection Via Crowdsourced Battles

via “pairwise-preference-collection-via-crowdsourced-battles”

Crowdsourced Elo ratings from human model comparisons.

Unique: Uses continuous crowdsourced pairwise comparisons from real users rather than static expert-annotated datasets, capturing evolving preference distributions across diverse conversational tasks and languages without requiring predefined evaluation rubrics or domain expertise from annotators

vs others: Captures real-world user preferences at scale more cheaply than expert annotation while remaining more representative of actual use cases than synthetic benchmarks, though at the cost of sampling bias and preference drift

via “crowdsourced pairwise model comparison via battle mode”