hierarchical parent-child document chunking with dual-embedding indexing
Splits PDF documents into small child chunks (512 tokens) nested within larger parent chunks (2048 tokens), then indexes both layers separately using dense embeddings (sentence-transformers) and sparse BM25 embeddings via FastEmbedSparse. At retrieval time, the system fetches child chunks for precision but returns their parent context for completeness, solving the precision-vs-context tradeoff inherent in flat RAG systems. This two-tier indexing strategy is orchestrated through a DocumentChunker and VectorDatabaseManager that maintains parent-child relationships in Qdrant.
Unique: Implements explicit parent-child chunk relationships with dual-embedding (dense + sparse BM25) indexing in a single Qdrant instance, rather than maintaining separate indices or flattening chunks. The VectorDatabaseManager and ParentStoreManager classes coordinate retrieval to return child chunks for ranking but parent context for generation, a pattern not standard in LangChain's default RecursiveCharacterTextSplitter.
vs alternatives: Outperforms naive chunking strategies by reducing context loss (vs flat chunks) and retrieval latency (vs separate vector stores) while maintaining both semantic and keyword search capabilities in one index.
agentic multi-turn query reasoning with langgraph state machine
Orchestrates a multi-node LangGraph workflow where an LLM-powered agent reasons about user queries, decides whether to retrieve documents, clarifies ambiguous questions via human-in-the-loop prompts, and iteratively refines search strategies based on retrieval results. The graph implements conditional routing (via graph.add_conditional_edges) to branch between retrieval, clarification, and response generation nodes. State is maintained across turns in a TypedDict that tracks conversation history, retrieved documents, and agent decisions, enabling the agent to learn from previous retrieval failures and adjust its approach.
Unique: Uses LangGraph's graph.add_conditional_edges() to implement branching logic where an LLM node decides routing (retrieve vs clarify vs respond) based on query analysis, rather than hard-coded rule-based routing. The state machine pattern with TypedDict enables stateful reasoning across conversation turns, allowing the agent to learn from retrieval failures and adjust strategy dynamically.
vs alternatives: Provides more flexible agent reasoning than rule-based RAG pipelines by letting the LLM decide when retrieval is needed, and more transparent than black-box agent frameworks by exposing the graph structure for debugging and customization.
document indexing pipeline with batch processing and incremental updates
Processes PDF documents through a multi-stage pipeline: PDF-to-text conversion (with smart routing), hierarchical chunking (parent-child), embedding generation (dense + sparse), and storage in Qdrant. The DocumentManager orchestrates this pipeline, supporting batch indexing of multiple documents and incremental updates (adding new documents without re-indexing existing ones). The pipeline is modular, enabling custom PDF processing strategies or embedding models to be swapped without changing the core indexing logic.
Unique: Implements document indexing as a modular pipeline (PDF conversion → chunking → embedding → storage) with support for incremental updates, rather than requiring full re-indexing on each document addition. The DocumentManager class abstracts pipeline orchestration, enabling custom strategies to be plugged in without changing core logic.
vs alternatives: More efficient than re-indexing all documents on each update and more flexible than monolithic indexing scripts; the modular design enables easy customization for different document types and embedding strategies.
vector database abstraction with qdrant backend and parent-child relationship management
Abstracts vector database operations (insert, search, delete) behind a VectorDatabaseManager class that handles both dense and sparse vector storage in Qdrant. The manager maintains parent-child chunk relationships using Qdrant's metadata filtering, enabling retrieval of child chunks while returning parent context. Supports both in-process (local) and remote Qdrant instances, enabling development on local machines and production on cloud deployments without code changes.
Unique: Implements VectorDatabaseManager as an abstraction layer that handles both dense and sparse vectors, parent-child relationships, and supports both in-process and remote Qdrant instances. The abstraction enables swapping vector database backends (in theory) without changing agent code, though current implementation is Qdrant-specific.
vs alternatives: More flexible than direct Qdrant client usage and more maintainable than scattered vector database calls throughout the codebase; the abstraction layer enables easier testing and backend swapping.
notebook-based tutorial with interactive cells for learning rag concepts
Provides a Jupyter notebook that walks through RAG concepts step-by-step: document loading, chunking, embedding, retrieval, and agent workflows. Each cell is self-contained and executable, enabling learners to understand concepts incrementally and experiment with parameters (chunk sizes, embedding models, LLM providers). The notebook includes visualizations of the indexing pipeline and agent graph, making abstract concepts concrete. This is distinct from the production modular system, serving as an educational tool rather than a deployment artifact.
Unique: Provides an interactive Jupyter notebook that teaches RAG concepts through executable cells, distinct from the production modular system. The notebook includes visualizations of the indexing pipeline and agent graph, making abstract concepts concrete and enabling experimentation with parameters.
vs alternatives: More accessible than reading documentation and more hands-on than static tutorials; enables learners to modify code and see results immediately, accelerating understanding of RAG concepts.
human-in-the-loop clarification prompting for ambiguous queries
Implements a dedicated agent node that detects ambiguous or under-specified user queries and generates clarification prompts asking the user to provide additional context (e.g., 'Which department's budget are you asking about?'). The clarification node is triggered via conditional routing when the agent's reasoning indicates insufficient query specificity. User responses are appended to the conversation state and the query is re-processed with the clarified context, enabling iterative refinement without requiring the user to restart the conversation.
Unique: Embeds clarification as a first-class agent node in the LangGraph workflow, triggered by conditional routing, rather than implementing it as a pre-processing step or external validation layer. The clarified context is merged back into the conversation state, enabling the agent to learn from the clarification in subsequent reasoning steps.
vs alternatives: More user-friendly than silent retrieval failures and more efficient than always retrieving multiple interpretations; clarification is integrated into the agent loop rather than bolted on as a separate validation step.
multi-strategy pdf-to-text conversion with smart routing
Implements three PDF processing strategies (simple text extraction via PyMuPDF4LLM, OCR+table detection for medium-complexity PDFs, and vision-language model analysis for complex layouts) with automatic routing based on PDF characteristics. The DocumentManager analyzes PDF structure (text density, table presence, image complexity) and selects the appropriate strategy, falling back to simpler methods if advanced processing fails. This avoids unnecessary computation (vision models are expensive) while ensuring complex PDFs are handled correctly.
Unique: Implements adaptive PDF processing with three-tier strategy selection (simple extraction → OCR+tables → vision models) based on PDF analysis, rather than requiring users to specify strategy upfront or always using the most expensive approach. The DocumentManager class encapsulates routing logic, enabling cost-aware processing without manual intervention.
vs alternatives: More cost-effective than always using vision models and more robust than simple text extraction; the smart routing avoids both unnecessary expense and processing failures by matching strategy to PDF complexity.
two-stage retrieval with dense-sparse hybrid search
Combines dense vector embeddings (sentence-transformers) and sparse BM25 embeddings (FastEmbedSparse) in a two-stage retrieval pipeline: first, both dense and sparse searches are executed in parallel against Qdrant, then results are merged using reciprocal rank fusion (RRF) to balance semantic relevance and keyword matching. This hybrid approach retrieves child chunks for ranking but returns parent chunks for generation, addressing both semantic gaps (where BM25 fails) and keyword-specific queries (where dense embeddings alone miss exact matches).
Unique: Implements parallel dense+sparse search with reciprocal rank fusion (RRF) merging in a single Qdrant query, rather than maintaining separate indices or sequentially executing searches. The VectorDatabaseManager class abstracts the hybrid search logic, enabling transparent switching between retrieval strategies without changing the agent code.
vs alternatives: Outperforms pure dense retrieval on keyword-heavy queries and pure BM25 on semantic queries; the hybrid approach captures both signal types in a single retrieval pass, reducing latency vs sequential search strategies.
+5 more capabilities