MINT-1T-PDF-CC-2023-23
DatasetFreeDataset by mlfoundations. 6,33,111 downloads.
Capabilities7 decomposed
multimodal image-text pair extraction from pdf documents at scale
Medium confidenceExtracts aligned image-text pairs from 1T+ tokens of PDF documents using a structured pipeline that preserves document layout and semantic relationships. The dataset uses WebDataset format for efficient streaming access to 633K+ samples, enabling distributed training without requiring full dataset materialization in memory. Implements MLCroissant metadata standards for reproducible dataset discovery and versioning.
Combines 1T+ tokens of PDF-native multimodal data with WebDataset streaming architecture and MLCroissant metadata standards, enabling efficient distributed training without full dataset materialization — unlike image-text datasets that require pre-downloaded image files or separate text corpora
Larger scale and document-native structure than LAION or similar web-scraped image-text datasets, with preserved layout context that benefits document-specific tasks; more efficient streaming than datasets requiring separate image downloads
streaming access to large-scale multimodal samples via webdataset format
Medium confidenceImplements WebDataset tar-based streaming protocol that allows sequential access to image-text pairs without downloading the entire 633K-sample dataset. Uses tar archive sharding and lazy loading to enable training on machines with limited disk space, with built-in support for distributed data loading across multiple GPUs/TPUs via HuggingFace datasets library integration.
Uses tar-based streaming with HuggingFace datasets integration and automatic caching, enabling efficient distributed training without pre-extraction — unlike traditional image-text datasets that require separate image file downloads and manual sharding logic
More memory-efficient than datasets requiring full image materialization; faster startup than downloading 500GB+ before training; simpler distributed setup than custom tar streaming implementations
reproducible dataset versioning and metadata discovery via mlcroissant standard
Medium confidenceEncodes dataset structure, provenance, and licensing metadata in MLCroissant format, enabling automated discovery, citation, and reproducible dataset loading across different tools and frameworks. Metadata includes source URLs, extraction timestamps, license information (CC-BY-4.0), and data schema definitions that allow downstream tools to validate data integrity and understand dataset composition without manual inspection.
Implements MLCroissant standard for machine-readable dataset metadata with automated schema validation and provenance tracking, enabling reproducible dataset loading and citation without manual documentation — unlike datasets with only README files or unstructured metadata
Standardized metadata format enables automated discovery and validation; better reproducibility than datasets relying on informal documentation; supports automated data pipeline validation that custom metadata formats cannot provide
pdf-native image-text alignment extraction with layout preservation
Medium confidenceExtracts image-text pairs from PDF documents while preserving spatial layout information, semantic relationships, and document structure (e.g., captions near figures, text flowing around images). Uses PDF parsing to identify image boundaries and associated text blocks, maintaining coordinate information that enables downstream tasks like layout understanding and spatial reasoning without requiring separate OCR or layout analysis steps.
Preserves PDF-native layout coordinates and document structure during extraction, enabling spatial reasoning tasks without separate layout analysis — unlike generic image-text datasets that discard layout information or require post-hoc layout detection
Maintains document structure and spatial relationships that improve downstream model performance on layout-aware tasks; reduces preprocessing overhead compared to datasets requiring separate layout analysis steps
common crawl 2023 pdf document filtering and quality curation
Medium confidenceFilters and curates 1T+ tokens of PDF documents from Common Crawl 2023 snapshot using quality heuristics (document completeness, text-image ratio, language detection, format validity) to create a high-quality subset of 633K samples. Implements multi-stage filtering pipeline that removes corrupted PDFs, non-English content, and documents with poor image-text alignment, producing a dataset suitable for training vision-language models without extensive downstream cleaning.
Applies multi-stage quality filtering to Common Crawl 2023 PDFs using document completeness, text-image ratio, and language detection heuristics, reducing 1T+ tokens to 633K high-quality samples — unlike raw Common Crawl data requiring extensive downstream cleaning
Pre-filtered dataset eliminates need for manual quality assessment; curated subset is more suitable for training than raw Common Crawl; reduces data cleaning overhead compared to unfiltered web-scale datasets
english-language document filtering and multilingual dataset composition
Medium confidenceFilters dataset to English-language documents using language detection heuristics applied during curation, ensuring consistent language composition for training English-focused vision-language models. Implements language identification at document and sample level, removing non-English PDFs and mixed-language content to maintain dataset homogeneity and training stability.
Applies language detection filtering to ensure English-only composition, removing multilingual and non-English documents from Common Crawl — unlike multilingual datasets that require language-specific handling during training
Simpler training pipeline for English models without multilingual complexity; consistent language composition improves training stability; reduces need for language-specific preprocessing
cc-by-4.0 licensed dataset with attribution and commercial use compliance
Medium confidenceDataset is released under Creative Commons Attribution 4.0 (CC-BY-4.0) license, enabling commercial use with attribution requirements. License metadata is embedded in MLCroissant format and HuggingFace Hub, providing clear terms for usage, redistribution, and derivative works. Requires attribution to original sources and compliance with underlying Common Crawl and source document licenses.
Provides clear CC-BY-4.0 licensing with embedded metadata in MLCroissant format, enabling transparent commercial use with documented attribution requirements — unlike proprietary datasets with unclear licensing or datasets with restrictive licenses
Clear commercial use terms reduce legal uncertainty; CC-BY-4.0 is more permissive than restrictive licenses; embedded metadata simplifies compliance tracking
Capabilities are decomposed by AI analysis. Each maps to specific user intents and improves with match feedback.
Related Artifactssharing capabilities
Artifacts that share capabilities with MINT-1T-PDF-CC-2023-23, ranked by overlap. Discovered automatically through the match graph.
MINT-1T-PDF-CC-2023-14
Dataset by mlfoundations. 5,72,108 downloads.
MINT-1T-PDF-CC-2023-50
Dataset by mlfoundations. 7,96,577 downloads.
MINT-1T-PDF-CC-2023-40
Dataset by mlfoundations. 8,57,357 downloads.
MINT-1T-PDF-CC-2024-18
Dataset by mlfoundations. 10,34,415 downloads.
banned-historical-archives
Dataset by banned-historical-archives. 17,46,771 downloads.
MINT-1T-PDF-CC-2023-06
Dataset by mlfoundations. 5,39,406 downloads.
Best For
- ✓ML researchers training vision-language foundation models
- ✓Teams building document AI and OCR systems
- ✓Organizations developing multimodal RAG systems for enterprise document processing
- ✓Researchers studying document layout understanding and spatial reasoning
- ✓Teams with limited GPU memory or storage running large-scale training jobs
- ✓Researchers using cloud infrastructure (AWS, GCP, Azure) with per-GB egress costs
- ✓Distributed training setups requiring efficient multi-worker data loading
- ✓Rapid prototyping scenarios where full dataset download is prohibitive
Known Limitations
- ⚠633K samples represent filtered subset of Common Crawl — not exhaustive coverage of all document types or languages
- ⚠PDF extraction quality depends on source document structure; scanned/image-based PDFs may have degraded text alignment
- ⚠WebDataset streaming format requires compatible data loading libraries; not directly compatible with standard PyTorch DataLoader without adapters
- ⚠CC-BY-4.0 license requires attribution; commercial use requires compliance with original source licenses
- ⚠No built-in deduplication or quality filtering — downstream users must implement their own data cleaning pipelines
- ⚠Sequential tar streaming introduces ~5-15% throughput overhead vs pre-extracted image files due to decompression
Requirements
Input / Output
UnfragileRank
UnfragileRank is computed from adoption signals, documentation quality, ecosystem connectivity, match graph feedback, and freshness. No artifact can pay for a higher rank.
About
MINT-1T-PDF-CC-2023-23 — a dataset on HuggingFace with 6,33,111 downloads
Categories
Alternatives to MINT-1T-PDF-CC-2023-23
Are you the builder of MINT-1T-PDF-CC-2023-23?
Claim this artifact to get a verified badge, access match analytics, see which intents users search for, and manage your listing.
Get the weekly brief
New tools, rising stars, and what's actually worth your time. No spam.
Data Sources
Looking for something else?
Search →