doc-build
DatasetFreeDataset by hf-doc-build. 2,82,022 downloads.
Capabilities5 decomposed
documentation-source-code-pair extraction and indexing
Medium confidenceExtracts aligned pairs of documentation text and source code from HuggingFace repositories and related projects, organizing them into a structured dataset with 282,022 examples. The dataset uses a collection pipeline that crawls public repositories, parses documentation files (Markdown, RST, HTML), correlates them with corresponding source code files through AST analysis and file path heuristics, and stores the pairs in a standardized format (typically Parquet or JSON Lines) with metadata including source repository, file paths, and documentation type. This enables downstream models to learn the relationship between natural language documentation and code implementation.
Specifically curated from HuggingFace ecosystem repositories (Transformers, Datasets, Diffusers, etc.) rather than generic GitHub crawl, ensuring high-quality, well-maintained code-documentation pairs with consistent documentation standards and active community maintenance
More focused and higher-quality than generic GitHub code-documentation datasets because it filters for actively-maintained HuggingFace projects with professional documentation standards, whereas alternatives like CodeSearchNet include abandoned repositories and inconsistent documentation practices
multi-language code-documentation corpus filtering and sampling
Medium confidenceProvides mechanisms to filter and sample the documentation-code pairs by programming language, documentation format (docstring, API docs, README), and repository characteristics. The dataset supports stratified sampling to create balanced subsets across languages and documentation types, and includes metadata fields that enable downstream filtering without re-downloading the full dataset. Filtering is performed at the HuggingFace dataset level using the library's built-in map() and filter() operations, which are optimized for lazy evaluation and streaming to avoid loading the entire dataset into memory.
Integrates with HuggingFace dataset streaming and lazy evaluation, allowing efficient filtering of 282k examples without materializing the full dataset; supports both eager and streaming modes for memory-constrained environments
More memory-efficient than downloading and filtering locally because it leverages HuggingFace's distributed dataset infrastructure and streaming APIs, whereas alternatives require downloading the full dataset before filtering
documentation-code pair validation and quality assessment
Medium confidenceEnables assessment of alignment quality between documentation and code pairs through structural validation and heuristic scoring. The dataset includes metadata that can be used to compute alignment metrics: code-to-documentation length ratios, presence of code examples in documentation, consistency of function/class names between documentation and implementation, and documentation coverage (percentage of public APIs documented). These metrics are computed via post-processing scripts that parse code ASTs and documentation text, comparing extracted identifiers and structure to measure alignment strength.
Provides structural validation specific to code-documentation pairs by comparing AST-extracted identifiers and documentation text, rather than generic text quality metrics; enables alignment-aware filtering that other datasets lack
More sophisticated than simple length-based filtering because it performs structural comparison between code and documentation using AST analysis, whereas generic code datasets only validate code syntax or documentation readability
dataset versioning and reproducible data splits
Medium confidenceSupports reproducible train/validation/test splits through deterministic seeding and version-pinned dataset snapshots on HuggingFace Hub. The dataset is versioned with Git-based revision tracking, allowing researchers to specify exact dataset versions in their experiments (e.g., 'revision=main' or 'revision=v1.0'). Splits are created using seeded random sampling, ensuring that the same split configuration produces identical results across different machines and time periods. This enables reproducibility in research and allows teams to compare models trained on identical data subsets.
Leverages HuggingFace Hub's Git-based versioning system to provide full dataset version history and reproducible splits, enabling researchers to pin exact dataset versions in code rather than relying on external version management
More reproducible than manually-downloaded datasets because version pinning is built into the HuggingFace infrastructure and automatically tracked, whereas alternatives require manual version management or external tools like DVC
batch dataset export and format conversion
Medium confidenceEnables efficient export of the documentation-code dataset to multiple formats (Parquet, JSON Lines, CSV, Arrow) for integration with different ML frameworks and data pipelines. Exports are performed using HuggingFace's built-in save_to_disk() and to_csv()/to_json() methods, which support streaming and batching to avoid memory overflow on large datasets. The export process preserves all metadata fields and supports optional compression (gzip, snappy) to reduce storage footprint. Exported datasets can be directly loaded into PyTorch DataLoaders, TensorFlow tf.data pipelines, or processed with pandas/Polars for analysis.
Integrates with HuggingFace's streaming and batching infrastructure to support efficient export of large datasets without materializing full dataset in memory; supports multiple formats natively without external conversion tools
More efficient than manual export scripts because it leverages HuggingFace's optimized I/O and batching, whereas alternatives require custom code to handle streaming and memory management
Capabilities are decomposed by AI analysis. Each maps to specific user intents and improves with match feedback.
Related Artifactssharing capabilities
Artifacts that share capabilities with doc-build, ranked by overlap. Discovered automatically through the match graph.
xCodeEval
Dataset by NTU-NLP-sg. 6,96,087 downloads.
Stenography
Automatic code...
Swimm
AI code documentation — auto-generates from code, auto-syncs on changes, IDE integration.
CodeSearchNet
6M functions across 6 languages paired with documentation.
doc-build-dev
Dataset by hf-doc-build. 2,71,754 downloads.
StarCoderData
250GB curated code dataset for StarCoder training.
Best For
- ✓ML researchers training neural models for code documentation tasks
- ✓Teams building IDE plugins that auto-generate docstrings from code
- ✓Organizations developing code-to-documentation search engines
- ✓Academic groups studying the relationship between code and natural language
- ✓ML engineers fine-tuning models on specific programming languages
- ✓Researchers studying cross-language documentation patterns
- ✓Teams with limited compute budgets needing representative subsets
- ✓Data scientists performing exploratory analysis on code-documentation relationships
Known Limitations
- ⚠Dataset is static snapshot — does not automatically update as source repositories evolve; requires periodic re-crawling to capture new documentation patterns
- ⚠Documentation-code alignment is heuristic-based (path matching, AST correlation) and may have false positives/negatives, especially for complex multi-file documentation
- ⚠Heavily skewed toward Python and JavaScript projects due to HuggingFace ecosystem composition; limited coverage of Java, C++, Rust documentation patterns
- ⚠No built-in deduplication — may contain near-duplicate pairs from forked repositories or similar projects
- ⚠Metadata is minimal — lacks information about documentation quality, code complexity metrics, or temporal relationships
- ⚠Filtering operations require loading metadata for all 282k examples into memory; full-dataset filtering may use 1-2GB RAM on machines with limited resources
Requirements
Input / Output
UnfragileRank
UnfragileRank is computed from adoption signals, documentation quality, ecosystem connectivity, match graph feedback, and freshness. No artifact can pay for a higher rank.
About
doc-build — a dataset on HuggingFace with 2,82,022 downloads
Categories
Alternatives to doc-build
Are you the builder of doc-build?
Claim this artifact to get a verified badge, access match analytics, see which intents users search for, and manage your listing.
Get the weekly brief
New tools, rising stars, and what's actually worth your time. No spam.
Data Sources
Looking for something else?
Search →