Books And Long Form Text Corpus Inclusion

via “books and long-form text corpus inclusion”

EleutherAI's 825 GiB diverse training dataset from 22 sources.

Unique: Explicitly includes book-focused subsets (Books3, Gutenberg) as core components rather than incidental web scrape byproducts, recognizing that long-form narrative text develops different linguistic capabilities than short web snippets. This architectural choice influences model performance on coherence, narrative structure, and long-context understanding.

vs others: More comprehensive book coverage than web-only datasets (e.g., C4); comparable to book-specific datasets (e.g., BookCorpus) but integrated into a multi-domain corpus for broader generalization rather than domain-specific pretraining

via “large-scale text corpus for language model pretraining”

Dataset by mlfoundations. 8,57,357 downloads.

Unique: Derives 1 trillion tokens specifically from PDF documents rather than generic web crawls, capturing formal, structured writing with higher information density than typical web text. Preserves document-level context and structure signals that web-only corpora lose.

vs others: Complements web-text corpora (C4, The Pile) by providing document-sourced content with different statistical properties, useful for models requiring strong document understanding capabilities.