StarCoder Data

Aggregates 783 GB of source code across 86 programming languages from publicly available repositories, filtering exclusively for permissively licensed code (MIT, Apache 2.0, BSD, etc.) to ensure legal trainability. Uses license detection via SPDX identifiers and repository metadata scanning to validate licensing status at collection time, preventing inclusion of GPL or proprietary code that would create legal friction for downstream model training.

Applies two-stage deduplication: exact string matching to remove byte-for-byte duplicates, followed by near-deduplication using MinHash/Jaccard similarity (typically threshold ~0.85) to identify and remove near-identical code blocks that differ only in whitespace, comments, or minor variable renames. This reduces redundancy while preserving legitimate code diversity, preventing the model from overweighting common boilerplate or copy-pasted snippets.

Scans the entire 783 GB corpus for PII patterns including email addresses, IP addresses (IPv4/IPv6), API keys, private keys, and other sensitive credentials using regex-based pattern matching and entropy-based detection. Redacts or removes identified PII before dataset release, protecting developer privacy and preventing accidental exposure of secrets in the training data that could be memorized and leaked by the model.

Extracts and preserves code cells and markdown text from Jupyter notebooks as interleaved sequences, maintaining the pedagogical structure where explanatory text precedes or follows code blocks. This allows models trained on the dataset to learn the relationship between natural language documentation and code implementation, improving code generation quality when models can reference explanatory context.

Provides a mechanism for developers to request exclusion of their repositories from the dataset, respecting developer autonomy and addressing concerns about code being used for AI training without consent. Maintains an opt-out registry that is checked during dataset construction and updates, allowing developers to remove their code retroactively or prevent future inclusion.

Organizes and represents code across 86 programming languages, applying language-specific parsing and tokenization strategies to preserve syntactic structure. Enables downstream models to learn language-specific patterns (e.g., Python indentation, Rust ownership, JavaScript async/await) rather than treating all code as generic text, improving language-specific code generation quality.

Incorporates GitHub issues and Git commit messages alongside source code, providing natural language context about code changes, bug fixes, and feature requests. This allows models to learn the relationship between code changes and their motivations, improving code generation quality by training on examples where code is paired with explanatory intent.

Implements distributed processing pipeline for 783 GB of code using frameworks like Spark or Ray, enabling efficient deduplication, PII redaction, and language detection across multiple machines. Provides streaming/chunked access patterns (Hugging Face Datasets format) to allow downstream users to load and process the dataset without requiring full 783 GB in memory, using lazy evaluation and batch processing.

Maintains versioned snapshots of the dataset with full provenance tracking, including data processing pipeline parameters, deduplication thresholds, PII redaction patterns, and opt-out exclusions applied to each version. Enables reproducible model training by documenting exact dataset composition, enabling researchers to cite specific dataset versions and understand how dataset changes affect model behavior. Supports rollback to previous versions and comparison of dataset statistics across versions.