markdown-to-llm-context extraction
Parses markdown and MDX files from a documentation source directory and extracts semantic content blocks (headings, paragraphs, code blocks, lists) into a structured format optimized for LLM consumption. Uses AST-based parsing to preserve document hierarchy and metadata, then flattens content into a single llms.txt file with clear delimiters and context markers that help LLMs understand document structure without needing to parse raw markdown syntax.
Unique: Specifically targets the llms.txt convention (emerging standard for LLM-friendly documentation) rather than generic markdown-to-text conversion, with awareness of documentation site generators (Next.js, Astro, Docusaurus) and their directory structures
vs alternatives: Purpose-built for LLM context generation unlike generic markdown converters; understands documentation site conventions and preserves semantic hierarchy better than simple text extraction
multi-framework documentation source detection
Automatically detects and adapts to different documentation framework conventions (Next.js, Astro, Docusaurus, VitePress, Gatsby) by identifying framework-specific directory patterns, configuration files, and content organization schemes. Uses heuristic-based framework detection (checking for framework config files like next.config.js, astro.config.mjs, docusaurus.config.js) to determine the correct source directory and content structure without requiring explicit configuration.
Unique: Implements framework-agnostic detection logic that recognizes multiple documentation generators' conventions and automatically resolves content paths, eliminating the need for manual configuration across different tech stacks
vs alternatives: Eliminates configuration overhead compared to generic markdown processors that require explicit path specification; handles framework-specific quirks automatically
recursive directory traversal with file filtering
Walks through nested directory structures starting from a detected or configured source directory, recursively discovers all markdown and MDX files, and applies filtering rules to include/exclude content based on file patterns, directory names, and metadata. Uses file system APIs with configurable glob patterns or ignore rules to skip common non-content directories (node_modules, .git, build output) and focus only on documentation source files.
Unique: Combines recursive traversal with framework-aware filtering that understands documentation site conventions (e.g., skipping build directories, node_modules) without explicit configuration
vs alternatives: More intelligent than generic file globbing because it understands documentation project structure; faster than shell-based find commands for large trees
markdown-to-plaintext semantic conversion
Transforms markdown syntax into plain text while preserving semantic meaning and document structure through strategic formatting choices. Converts markdown headers to uppercase labels with separators, converts lists to indented plain text, strips inline formatting (bold, italic) while keeping content, removes markdown-specific syntax (backticks, brackets), and preserves code blocks as indented text blocks. This approach ensures LLMs can understand content hierarchy without needing to parse markdown syntax.
Unique: Prioritizes semantic clarity for LLM consumption over markdown fidelity; uses structural formatting (uppercase headers, indentation, delimiters) instead of markdown syntax to signal document hierarchy
vs alternatives: Better for LLM context than raw markdown (which adds parsing overhead) or naive text extraction (which loses structure); optimized for the specific use case of LLM-friendly documentation
mdx component and jsx handling
Processes MDX files containing embedded JSX components and React code by extracting text content from component props, rendering component descriptions, and handling interactive elements as plain text descriptions. Parses JSX syntax to identify component boundaries, extracts meaningful text from component children and props, and generates fallback text descriptions for components that don't have direct text equivalents (e.g., 'Interactive Code Example' for a CodeSandbox embed).
Unique: Handles MDX-specific content (React components, JSX) which generic markdown tools cannot process; extracts semantic meaning from component structures rather than treating them as unparseable syntax
vs alternatives: Enables MDX documentation to be included in llms.txt unlike markdown-only tools; better than stripping JSX entirely because it preserves component intent through fallback descriptions
configurable output formatting and delimiters
Generates llms.txt output with customizable formatting options including configurable section delimiters, header formatting styles, content separators, and metadata inclusion. Allows users to specify how headers are formatted (e.g., '# HEADER' vs '=== HEADER ==='), what separators divide sections, whether to include file paths or metadata, and how to structure the final output. Supports multiple output format presets (compact, verbose, structured) to optimize for different LLM consumption patterns.
Unique: Provides format customization specifically for LLM consumption patterns rather than generic text formatting; includes preset formats optimized for different LLM architectures and use cases
vs alternatives: More flexible than fixed-format tools; allows optimization for specific LLM providers unlike one-size-fits-all markdown converters
batch processing and file aggregation
Processes multiple markdown and MDX files in a single operation, aggregates their content into a unified llms.txt output, and maintains file-level organization through metadata or section markers. Reads all discovered files, parses each independently, concatenates converted content with clear file boundaries, and optionally includes file path information or table of contents to help LLMs navigate the aggregated content. Handles file ordering (alphabetical, by modification time, or custom) to ensure consistent output.
Unique: Designed specifically for documentation aggregation with awareness of file boundaries and logical organization; maintains context about source files unlike naive concatenation
vs alternatives: More efficient than processing files individually; preserves file-level context better than simple text concatenation
npm package distribution and cli integration
Distributes get-llms-txt as an npm package with a command-line interface that can be invoked directly or integrated into build scripts and CI/CD pipelines. Provides both programmatic API (for Node.js projects) and CLI commands (for shell scripts and automation), supports configuration via command-line arguments or config files, and integrates with npm scripts in package.json for automated llms.txt generation during builds or deployments.
Unique: Provides both CLI and programmatic API for maximum flexibility; integrates seamlessly with npm-based workflows and CI/CD systems through standard Node.js conventions
vs alternatives: More accessible than standalone tools because it leverages existing npm infrastructure; easier to integrate into existing Node.js projects than external utilities
+2 more capabilities