C4 (Colossal Clean Crawled Corpus)

Processes 750GB of raw Common Crawl data through a multi-stage heuristic filtering pipeline that removes short pages (< 100 words), deduplicates at the sentence level using fuzzy matching, filters offensive/adult content via keyword blacklists and classifier heuristics, and restricts to English-language documents via language detection. The filtering approach uses rule-based heuristics rather than learned classifiers, making it reproducible and auditable but potentially less adaptive to domain-specific quality signals.

Provides a multilingual variant of C4 covering 108 languages extracted from Common Crawl using language detection heuristics. Each language subset is independently filtered and deduplicated using the same heuristic pipeline as the English version, enabling researchers to train or evaluate multilingual models without manually collecting and cleaning language-specific corpora. Language detection is performed at document level, so mixed-language documents are assigned to a single language based on dominant language detection.

Provides a 'realnewslike' variant of C4 that filters the corpus to match the distribution of news articles from Common Crawl's news sources. This variant uses domain-specific heuristics (URL patterns, content structure, publication metadata) to identify news-domain documents and creates a subset with similar statistical properties to real news corpora. The filtering preserves the original heuristic-based approach while constraining the corpus to a specific domain distribution.

Provides C4 as a Hugging Face Dataset with native support for both streaming (on-the-fly loading without full download) and batch downloading via the Hugging Face Datasets library. The dataset is split into train/validation splits, supports efficient sampling and shuffling, and integrates with Hugging Face's caching and versioning system. Streaming uses HTTP range requests to fetch only required data, while batch access downloads and caches locally for repeated access.

Manages C4 dataset versions and train/validation splits through Hugging Face's versioning system, enabling reproducible access to specific dataset versions and splits. Each version is immutable and tied to a specific Git commit, ensuring that researchers can reproduce results by specifying the exact dataset version. Splits are pre-defined (train, validation) and deterministically generated, so the same split is always returned for the same seed.

Filters documents containing offensive, adult, or inappropriate content using a combination of keyword blacklists, pattern matching, and heuristic rules. The filtering is applied during the initial corpus curation and removes documents that match offensive content patterns, reducing but not eliminating inappropriate content. The approach is transparent and rule-based, making it auditable but potentially less effective than learned classifiers at catching nuanced offensive content.

Removes duplicate and near-duplicate sentences across the entire corpus using fuzzy string matching heuristics. The deduplication is applied at the sentence level (not document level), so documents with duplicate sentences are modified to remove the duplicates. This approach reduces data leakage and redundancy in the training corpus, improving model generalization by ensuring that the model sees diverse sentence patterns rather than repeated content.

Removes documents shorter than a minimum length threshold (typically 100 words) to filter out low-quality, stub, or boilerplate content. This filtering is applied during corpus curation and reduces the proportion of short, low-information-density documents in the training corpus. The approach is simple and transparent but may remove legitimate short-form content like abstracts, summaries, or social media posts.