MINT-1T-PDF-CC-2024-18
DatasetFreeDataset by mlfoundations. 10,34,415 downloads.
Capabilities6 decomposed
large-scale multimodal document-image dataset curation and indexing
Medium confidenceProvides a 1 trillion token-scale dataset of PDF documents paired with extracted images and text, curated from Common Crawl with deduplication and quality filtering applied at scale. The dataset uses HuggingFace's distributed dataset infrastructure to enable efficient streaming and sampling of 1M+ document-image pairs without requiring full local storage, with metadata indexing for retrieval by document type, language, and content characteristics.
Combines PDF-level document structure preservation with extracted image-text pairs at 1T token scale, using Common Crawl's distributed crawl infrastructure and HuggingFace's streaming dataset format to avoid centralized storage bottlenecks — most competitors (e.g., LAION) focus on web images or require full downloads
Larger and more document-focused than LAION-5B or Conceptual Captions, with native PDF structure metadata enabling document-aware training; more accessible than proprietary datasets like Google's internal document corpora due to CC-BY-4.0 licensing and HuggingFace Hub distribution
streaming dataset access with lazy loading and memory-efficient batching
Medium confidenceImplements HuggingFace Datasets' streaming protocol to load document-image pairs on-demand without downloading the full 1T token dataset, using memory-mapped Arrow format and distributed sharding across multiple processes. Batching is handled through configurable DataLoader wrappers that respect image tensor dimensions and text sequence lengths, enabling training on machines with limited VRAM through dynamic batch size adjustment.
Uses HuggingFace's Arrow-based streaming format with automatic shard distribution and epoch-level determinism, enabling true lazy loading without requiring dataset mirroring — most competitors (Petastorm, TFRecord) require pre-sharding or local caching
More memory-efficient than downloading full datasets and faster to iterate than manual data pipelines; integrates natively with PyTorch/TensorFlow without custom serialization code
document-image pair extraction and alignment from pdf sources
Medium confidenceExtracts text and images from PDF documents using OCR and layout analysis, then aligns extracted text with corresponding page images through spatial coordinate matching and text-region association. The extraction pipeline handles multi-page PDFs, preserves document structure metadata (headers, footers, sections), and deduplicates near-identical documents using perceptual hashing and text similarity metrics to ensure dataset quality.
Combines PDF text extraction with rendered page images and spatial alignment metadata at scale, using perceptual hashing for deduplication — most document datasets (DocVQA, RVL-CDIP) are manually curated or use simpler extraction without alignment preservation
Preserves document structure and layout information unlike text-only datasets; larger and more diverse than manually-curated document benchmarks; automated extraction enables continuous updates from Common Crawl
common crawl-sourced dataset with quality filtering and language detection
Medium confidenceIngests documents from Common Crawl's WARC archives, applies language detection (likely using fastText or similar) to filter for English content, and runs quality heuristics (text-to-image ratio, document length, spam detection) to remove low-quality or malicious PDFs. The filtering pipeline is applied during dataset construction, reducing the raw crawl from billions of documents to 1M+ high-quality document-image pairs with reproducible filtering criteria.
Applies reproducible quality filtering to Common Crawl at scale, with transparent filtering criteria and public provenance — most proprietary datasets (Google, OpenAI) do not disclose filtering methods; most academic datasets are manually curated at smaller scale
Larger and more diverse than manually-curated datasets; more transparent and reproducible than proprietary web-scale datasets; enables research on real-world document distributions
multimodal dataset sampling and stratification for balanced model training
Medium confidenceProvides mechanisms to sample subsets of the 1T token dataset with control over document type distribution, image-text ratio, and content characteristics. Sampling can be stratified by document category (academic papers, web pages, forms, etc.) or by content properties (text length, image density, language) to ensure training data reflects desired distributions rather than raw web frequencies, which are heavily skewed toward common document types.
Enables stratified sampling across document types and content properties at scale, allowing researchers to control training data distribution — most large datasets provide raw access without built-in stratification mechanisms
More flexible than fixed dataset splits; enables targeted evaluation on specific document categories; supports research on dataset bias and distribution effects
metadata-rich document records with source attribution and quality scores
Medium confidenceEach dataset record includes rich metadata beyond image and text: source URL, crawl date, document type classification, quality score, OCR confidence, text-image alignment score, and deduplication information. Metadata is structured as JSON and queryable, enabling filtering and analysis without loading full images/text, and providing traceability for reproducibility and copyright attribution.
Provides queryable metadata with quality scores and source attribution for every record, enabling transparent dataset analysis and reproducibility — most large datasets provide minimal metadata or require custom extraction
More transparent than proprietary datasets; enables reproducible research and copyright compliance; supports dataset bias analysis and quality-aware training
Capabilities are decomposed by AI analysis. Each maps to specific user intents and improves with match feedback.
Related Artifactssharing capabilities
Artifacts that share capabilities with MINT-1T-PDF-CC-2024-18, ranked by overlap. Discovered automatically through the match graph.
MINT-1T-PDF-CC-2023-06
Dataset by mlfoundations. 5,39,406 downloads.
MINT-1T-PDF-CC-2023-23
Dataset by mlfoundations. 6,33,111 downloads.
MINT-1T-PDF-CC-2023-14
Dataset by mlfoundations. 5,72,108 downloads.
MINT-1T-PDF-CC-2023-40
Dataset by mlfoundations. 8,57,357 downloads.
Chat With PDF by Copilot.us
An AI app that enables dialogue with PDF documents, supporting interactions with multiple files simultaneously through language models.
MINT-1T-PDF-CC-2023-50
Dataset by mlfoundations. 7,96,577 downloads.
Best For
- ✓ML researchers training large vision-language models (LLaVA, GPT-4V competitors)
- ✓Teams building document processing pipelines requiring diverse training data
- ✓Organizations developing OCR and document understanding systems
- ✓Researchers with GPU clusters but limited NVMe storage
- ✓Teams using cloud training (AWS SageMaker, GCP Vertex AI) with per-instance bandwidth constraints
- ✓Iterative model development requiring rapid experimentation cycles
- ✓Teams building document understanding systems (invoice processing, form extraction, contract analysis)
- ✓Researchers developing layout-aware vision-language models
Known Limitations
- ⚠1T tokens requires significant computational resources for full training — most practitioners sample subsets
- ⚠PDF extraction quality varies by document structure; scanned/image-heavy PDFs may have degraded text extraction
- ⚠Dataset is English-dominant; limited multilingual coverage despite CC-BY-4.0 license allowing derivative works
- ⚠No built-in document type stratification — requires custom filtering to balance document categories
- ⚠Streaming introduces ~50-200ms latency per batch due to network I/O and decompression — not suitable for real-time inference
- ⚠Deterministic shuffling requires maintaining epoch-level state; distributed training needs careful synchronization to avoid duplicate batches
Requirements
Input / Output
UnfragileRank
UnfragileRank is computed from adoption signals, documentation quality, ecosystem connectivity, match graph feedback, and freshness. No artifact can pay for a higher rank.
About
MINT-1T-PDF-CC-2024-18 — a dataset on HuggingFace with 10,34,415 downloads
Categories
Alternatives to MINT-1T-PDF-CC-2024-18
Are you the builder of MINT-1T-PDF-CC-2024-18?
Claim this artifact to get a verified badge, access match analytics, see which intents users search for, and manage your listing.
Get the weekly brief
New tools, rising stars, and what's actually worth your time. No spam.
Data Sources
Looking for something else?
Search →