Language Specific Corpus Filtering And Subset Selection

via “quality-filtering-with-language-specific-heuristics”

6.3T token multilingual dataset across 167 languages.

Unique: Applies language-family-aware filtering rules (separate thresholds for Latin, CJK, Indic, Arabic scripts) rather than universal heuristics, recognizing that character frequency distributions and valid repetition patterns differ dramatically across writing systems — most datasets use single global quality threshold regardless of language

vs others: More linguistically-informed than mC4's basic filtering and more transparent than OSCAR's undocumented quality pipeline, reducing the risk of removing legitimate low-resource language content while still eliminating spam and corruption

via “domain-specific dataset curation and subset extraction”

1.2M image-text pairs with GPT-4V captions.

Unique: Enables systematic curation of domain-specific subsets from 1.2M images using GPT-4V captions as semantic filters, allowing extraction of specialized datasets without manual domain annotation or external labeling services

vs others: More flexible than fixed domain-specific datasets (e.g., medical imaging datasets) which are typically small and expensive to create; leverages rich caption semantics for more accurate domain filtering than keyword-based approaches

via “domain-specific parallel corpus selection and filtering”

Massive parallel corpus for machine translation.

Unique: Curates domain-specific corpora including medical (EMEA 282.5M pairs), patents (EuroPat 252.2M), legal/institutional (Europarl 217.4M, JRC-Acquis 215.9M, DGT 1.2B), and specialized sources (Bible translations 88.3M, Ubuntu documentation) alongside general-domain subtitle and web-crawled data, enabling users to select data by source type and implied domain rather than explicit domain labels.

vs others: Provides access to specialized domain corpora (medical, legal, patents) in a single interface, whereas generic parallel corpus repositories focus on general-domain data; however, lacks explicit domain tagging, quality metrics per domain, and domain-specific preprocessing that specialized MT data providers offer.

via “language-specific-corpus-filtering-and-subset-selection”

Multilingual web corpus covering 101 languages.

Unique: Provides language-partitioned Parquet files enabling efficient columnar filtering without full corpus download. Supports both batch download and streaming APIs, allowing researchers to work with language subsets at different scales (100MB to 300GB) without infrastructure overhead.

vs others: More flexible language selection than OSCAR (which requires manual filtering) and more scalable than downloading Wikipedia dumps per language, with built-in streaming for memory-constrained environments

via “language-specific subset filtering and selective loading”

BigScience's curated multilingual dataset for BLOOM.

Unique: ROOTS organizes data with language as the primary partitioning key, enabling zero-copy subset selection at the Datasets API level — users can load only relevant languages without materializing the full corpus, a design choice that reduces memory overhead compared to post-hoc filtering on monolithic datasets.

vs others: Compared to monolithic pretraining datasets like C4, ROOTS's language-partitioned structure allows selective loading without downloading irrelevant data, reducing iteration time and storage costs for multilingual or language-specific training.

via “large-scale english text corpus filtering and deduplication”

Google's cleaned Common Crawl corpus used to train T5.

Unique: Uses deterministic heuristic-based filtering (length thresholds, keyword matching, language detection) applied at scale to 750GB of Common Crawl, enabling reproducible dataset creation without learned classifiers; includes sentence-level deduplication to remove redundant training examples

vs others: More transparent and reproducible than learned filtering approaches; larger and more thoroughly deduplicated than raw Common Crawl, but less sophisticated than newer datasets like Fineweb that use neural classifiers for quality scoring

via “language detection and english-only filtering”

Dataset by HuggingFaceFW. 6,43,166 downloads.

Unique: Applies language identification at Common Crawl scale to produce a clean monolingual English corpus, whereas raw Common Crawl contains ~50% non-English content requiring manual filtering

vs others: Provides pre-filtered English-only data out-of-the-box, eliminating need for custom language detection pipelines compared to raw Common Crawl

via “text classification dataset sampling and filtering”

Dataset by m-a-p. 4,59,057 downloads.

Unique: Leverages HuggingFace's native filtering and sampling APIs (via .filter() and .select()) to enable in-memory or streaming-based subset extraction without full corpus download; supports seed-based reproducibility for deterministic splits across experiments

vs others: More flexible than static benchmark datasets (ImageNet, MNIST) because filtering is dynamic and user-defined; faster iteration than manual annotation while maintaining reproducibility through versioned dataset snapshots

via “document-domain dataset sampling and filtering”

Dataset by mlfoundations. 8,57,357 downloads.

Unique: Provides streaming access with metadata-based filtering on trillion-token dataset without requiring full download, using Hugging Face Datasets infrastructure for efficient subset construction. Enables on-demand domain-specific corpus creation from larger collection.

vs others: More flexible than fixed-size domain datasets (e.g., ArXiv papers, legal documents) by allowing dynamic filtering from larger corpus; more efficient than downloading full dataset for subset access.

via “language-specific document filtering and sampling”

Dataset by Helsinki-NLP. 3,48,667 downloads.

Unique: Leverages HuggingFace's columnar parquet storage and streaming API to enable language-level filtering without full dataset materialization — most competing datasets require downloading entire corpus or provide only coarse-grained splits (e.g., by language family rather than individual language codes)

vs others: Faster iteration than downloading full 384K-document corpus; more granular language selection than datasets offering only pre-split language-family buckets