Instruction Diversity Sampling And Stratification

via “instruction diversity sampling and deduplication”

Stanford's 52K GPT-3.5-generated instruction dataset that started it all.

Unique: Achieves diversity through implicit sampling during batch generation rather than explicit task categorization. Simplified pipeline removes classification/non-classification distinction, reducing pipeline complexity while maintaining empirical diversity through iterative sampling.

vs others: Simpler than original Self-Instruct's task-based categorization while achieving comparable diversity through batch decoding. More scalable than manual curation because diversity emerges from the generation process rather than requiring post-hoc filtering.

via “diverse-task-coverage-instruction-distribution”

300K instructions extracted directly from aligned LLM outputs.

Unique: Achieves task diversity through emergent sampling from the source model's learned instruction distribution rather than explicit stratified sampling or human task enumeration. The 300K scale naturally captures long-tail tasks without requiring domain-specific engineering.

vs others: Produces more natural task distributions than manually-curated instruction sets because it reflects what aligned models actually learn to recognize as valid tasks, rather than what humans explicitly enumerate.

Dataset by fineinstructions. 9,97,153 downloads.

Unique: Large-scale instruction dataset (546K+ examples) with inherent diversity across instruction types enables stratified sampling without losing representation; Parquet format supports efficient filtering and sampling without full dataset load

vs others: Larger instruction diversity than smaller datasets (e.g., Alpaca 52K) enables more robust stratified sampling; Parquet format enables efficient subset extraction compared to JSON/CSV alternatives