Petabyte Scale Monthly Web Crawl Ingestion And Archival

via “commoncrawl-scale data aggregation from 84 dumps”

30 trillion token web dataset with 40+ quality signals per document.

Unique: Processes 84 CommonCrawl dumps (claimed as most complete coverage vs. C4, Refinedweb, Dolma, SlimPajama) into a single, consistently-annotated dataset. Eliminates user burden of managing multiple dumps and implementing aggregation logic.

vs others: Larger scale (30 trillion tokens, 84 dumps) than competitors (C4: 156B tokens, Refinedweb: limited dumps, Dolma: limited dumps); unified dataset eliminates user aggregation burden but inherits web biases from CommonCrawl.

via “petabyte-scale monthly web crawl ingestion and archival”

Largest open web crawl archive, foundation of all LLM training data.

Unique: Operates the largest open web crawl archive with 300+ billion pages spanning 15+ years, maintained as a non-profit public good with monthly refresh cycles and dual indexing (CDXJ + columnar) for both URL-based and structured queries. No commercial competitor maintains equivalent historical depth and scale.

vs others: Larger, older, and more freely accessible than commercial web archives (Wayback Machine, Archive.org) with explicit support for ML training pipelines and no rate-limiting for research use.

via “bulk-data-ingestion-and-indexing”