PocketFlow-Tutorial-Codebase-Knowledge

Orchestrates a six-node sequential workflow (FetchRepo → IdentifyAbstractions → AnalyzeRelationships → OrderChapters → WriteChapters → CombineTutorial) using PocketFlow's node-chaining pattern with the >> operator. Each node implements a prep-exec-post lifecycle, passing results through a shared dictionary that acts as a central state store. Nodes are executed sequentially with automatic data threading between stages, eliminating manual context passing.

The FetchRepo node ingests code from GitHub repositories or local directories, applying include/exclude glob patterns to filter files before processing. Implements dual crawling strategies: GitHubRepositoryCrawler for remote repos (clones via git CLI) and LocalDirectoryCrawler for local paths (filesystem traversal). Outputs a files dictionary mapping file paths to source code content, with language detection based on file extensions.

The pipeline implements caching at two levels: (1) prompt-level caching in call_llm() to avoid regenerating identical LLM responses, and (2) file-level caching in FetchRepo to avoid re-cloning unchanged repositories. Cache keys are derived from repository URL/path and file content hashes. Cached results are stored in a local cache directory (.pocketflow_cache by default) and reused across pipeline runs, enabling fast iteration and cost reduction.

The pipeline outputs abstractions and relationships as structured JSON/dict objects, not just markdown text. Each abstraction includes name, description, file location, and type (class, function, module, pattern). Each relationship includes source, target, type (uses, imports, extends, calls), and strength. This structured output enables downstream processing, visualization, and integration with other tools. The JSON format is documented and stable across versions.

The IdentifyAbstractions node uses an LLM to analyze source code files and extract core abstractions (classes, functions, modules, patterns) that form the conceptual foundation of the codebase. Sends the files dictionary and detected language to the LLM with a prompt engineered to identify pedagogically relevant abstractions. Returns a structured list of abstractions with descriptions, enabling downstream nodes to build relationships and ordering.

The AnalyzeRelationships node uses an LLM to map dependencies and relationships between identified abstractions (e.g., 'ClassA uses ClassB', 'FunctionX calls FunctionY', 'ModuleA imports ModuleB'). Takes abstractions list and source files as input, prompts the LLM to analyze call graphs and dependency patterns, and outputs a relationships graph. This graph is used by downstream nodes to determine pedagogical ordering and chapter structure.

The OrderChapters node uses the relationships graph to determine optimal chapter ordering for the tutorial. Applies topological sorting to the dependency graph to ensure prerequisites are covered before dependent concepts. Uses an LLM to refine the ordering based on pedagogical principles (e.g., 'start with simple examples before complex patterns'). Outputs a chapter_order list that sequences abstractions from foundational to advanced, with grouping suggestions for related concepts.

The WriteChapters BatchNode generates tutorial content for each chapter in the ordered sequence using batch LLM calls. For each abstraction in chapter_order, constructs a detailed prompt including the abstraction description, related code snippets, dependencies, and pedagogical context. Implements caching via call_llm(prompt, use_cache=True) to avoid regenerating identical chapters. Outputs chapters dictionary mapping chapter names to markdown content with code examples, explanations, and learning objectives.

The CombineTutorial node assembles generated chapters into multiple output formats: Markdown files (one per chapter), Mermaid diagrams (architecture and dependency visualizations), and a Jekyll-compatible documentation site structure. Takes all pipeline outputs (chapters, abstractions, relationships, chapter_order) and generates a complete tutorial package. Outputs to a configurable output_dir with subdirectories for chapters, diagrams, and static site assets. Supports multi-language documentation via language-specific templates.

The call_llm(prompt, use_cache) utility function abstracts LLM provider differences, supporting OpenAI, Anthropic, and local Ollama models. Reads model configuration from environment variables or config file (MODEL_PROVIDER, MODEL_NAME, API_KEY). Routes prompts to the appropriate provider's API with consistent request/response handling. Implements caching via prompt-hash-based key lookup in a local cache store, returning cached responses without API calls for identical prompts.

The main.py CLI interface parses command-line arguments for repository URL/local path, output directory, include/exclude patterns, and LLM configuration. Initializes a shared dictionary with all configuration parameters and invokes create_tutorial_flow() to execute the pipeline. Supports Docker containerization via Dockerfile, enabling reproducible execution across environments. Configuration can be provided via CLI flags, environment variables, or config file (config.yaml).

The pipeline detects programming language from file extensions and passes language context through all nodes (IdentifyAbstractions, AnalyzeRelationships, OrderChapters, WriteChapters). LLM prompts are language-aware, using language-specific terminology and patterns (e.g., 'Python decorators', 'JavaScript closures', 'Java interfaces'). Supports Python, JavaScript, Java, Go, Rust, and other common languages. Language detection is automatic and transparent to users.