News Domain Specific Text Variant With Distribution Matching

via “news-domain-specific text variant with distribution matching”

Google's cleaned Common Crawl corpus used to train T5.

Unique: Applies domain-specific filtering heuristics to C4 to create a news-distribution-matched subset, enabling news-focused pre-training without separate news corpus collection; maintains consistency with C4 cleaning pipeline while adding domain-specific selection

vs others: Simpler and more reproducible than collecting news from multiple sources; smaller and more focused than full C4, but may lack editorial quality and fact-checking standards of professional news datasets