Response Quality Variance Quantification Across Model Families

via “seven-model response collection and comparison”

183K multi-turn preference comparisons for alignment.

Unique: Systematically collects responses from seven different models to identical prompts rather than using single-model outputs or human-written references, enabling direct comparative analysis and preference learning from model-to-model differences.

vs others: Richer than single-model preference data because it captures relative model strengths, and more scalable than human-written reference responses while maintaining diversity through multiple model perspectives

64K preference dataset for RLHF training.

Unique: Includes responses from models with intentionally different capability levels (GPT-4 vs Llama-7B), enabling quantification of quality variance and identification of prompts where models diverge. This variance is preserved in the dataset rather than normalized away, supporting analysis of preference learning robustness to quality variation.

vs others: More informative than preference datasets with responses from similar-capability models because it captures quality variance across the capability spectrum, enabling analysis of whether preference learning methods are robust to variation in response quality or sensitive to specific model pairs.