Dataset Filtering And Sampling For Model Training And Evaluation

via “synthetic and filtered training data quality optimization”

Microsoft's 3.8B model with 128K context for edge deployment.

Unique: Achieves 69% MMLU and competitive reasoning performance in 3.8B parameters through explicit focus on training data quality (synthetic + filtered) rather than scale, demonstrating that data curation can partially offset parameter count disadvantages

vs others: Prioritizes data quality over dataset size (vs. Llama 3.2 trained on broader web data), reducing bias and toxicity at the cost of potentially narrower knowledge coverage; enables stronger performance on benchmark tasks despite smaller size