llama_index

LlamaIndex ingests documents from 50+ sources (files, web, cloud APIs, databases) through a pluggable NodeParser system that intelligently chunks content based on document type and semantic boundaries. The framework uses a unified Document/Node abstraction that preserves metadata and relationships, enabling downstream RAG systems to maintain context fidelity. Parsers support hierarchical chunking, sliding windows, and semantic-aware splitting via language-specific tokenizers.

LlamaIndex abstracts vector store operations through a standardized VectorStore interface, supporting 15+ backends (Milvus, Qdrant, PostgreSQL pgvector, Azure AI Search, Pinecone, Weaviate) without changing application code. The framework handles embedding generation, vector insertion, and similarity search through a unified QueryEngine that routes queries to the appropriate index type. Index creation is lazy — vectors are generated on-demand during ingestion using configurable embedding models.

LlamaIndex provides LlamaPacks — pre-built, production-ready application templates for common use cases (document Q&A, multi-document analysis, research agents, code analysis). Each pack includes optimized configurations, prompt templates, and best practices. Packs are composable — developers can combine multiple packs or customize individual components. The framework provides a registry of community-contributed packs with versioning and dependency management.

LlamaIndex supports hybrid retrieval combining vector similarity search with BM25 keyword matching, optionally followed by semantic reranking using cross-encoder models or LLM-based ranking. The framework provides configurable fusion algorithms (reciprocal rank fusion, weighted combination) to merge results from multiple retrieval strategies. Reranking can use built-in models (Cohere, BGE) or custom LLM-based rankers that consider query-document relevance and other criteria.

LlamaIndex supports metadata filtering at the document and node level, enabling structured queries that combine semantic search with metadata constraints (date ranges, document type, author, custom tags). The framework provides a query language for expressing complex filters and integrates filtering with all retrieval strategies (vector, keyword, graph). Metadata is preserved through the ingestion pipeline and can be used for post-retrieval filtering or pre-filtering to reduce search scope.

LlamaIndex supports streaming LLM responses at the token level, enabling real-time response display and early termination based on token content or count. The framework provides streaming abstractions for both LLM calls and query engines, with configurable buffering and batching. Streaming works across all LLM providers and integrates with observability for tracking streamed token usage.

LlamaIndex supports batch processing of documents and async execution for scalable ingestion and querying. The framework provides batch APIs for ingesting multiple documents in parallel, with configurable concurrency limits and error handling. Async execution is available throughout the stack (LLM calls, retrievals, agent steps), enabling efficient resource utilization. Batch operations support progress tracking and resumable processing for long-running jobs.

LlamaIndex's QueryEngine system orchestrates queries across multiple index types (vector, keyword, graph, structured) using a composable strategy pattern. The framework supports hybrid retrieval (combining vector similarity with BM25 keyword search, graph traversal, or SQL queries) through a unified query interface. Query routing is configurable — developers can implement custom routers that select the optimal index based on query semantics, or use built-in routers that combine results from multiple indices.

LlamaIndex's Workflow system provides an event-driven architecture for building multi-step LLM applications using a declarative step-based model. Workflows are defined as a graph of Steps that emit and consume Events, with built-in state management for maintaining context across steps. The framework handles event routing, step scheduling, and error recovery automatically. Workflows support both synchronous and asynchronous execution, with optional persistence for long-running operations.

LlamaIndex's Agent system supports both single-agent and multi-agent architectures with configurable memory backends and tool calling patterns. Agents can be composed hierarchically (sub-agents delegating to other agents) or coordinated through a central orchestrator. The framework provides memory abstractions (chat history, summary memory, hybrid memory) that persist across agent interactions. Tool calling is standardized through a schema-based registry supporting OpenAI, Anthropic, and Ollama function-calling APIs.

LlamaIndex's Knowledge Graph system automatically extracts entities and relationships from documents using LLM-based extraction, building a Property Graph Index that supports both semantic and structural queries. The framework provides graph store abstractions (Neo4j, Kuzu, Nebula) and enables hybrid retrieval combining graph traversal with vector search. Graph construction is configurable — developers can customize entity/relationship extraction prompts, define custom schemas, or use pre-built extractors.

LlamaIndex abstracts LLM interactions through a unified LLM interface supporting 20+ providers (OpenAI, Anthropic, AWS Bedrock, Google GenAI, Ollama, Azure OpenAI, Hugging Face, etc.) without changing application code. The framework standardizes tool calling across providers with different native formats (OpenAI functions, Anthropic tools, Ollama function calling) through a schema-based registry. LLM selection is configurable at runtime — applications can switch models or providers without code changes.

LlamaIndex supports structured data extraction from documents using LLM-based extraction with optional schema validation. The framework can extract data into Pydantic models, JSON, or SQL tables, with configurable extraction prompts and validation rules. Structured indices enable SQL-like querying over extracted data, combining semantic search with structured filters. The system supports both single-document extraction and batch extraction across document collections.

LlamaIndex provides end-to-end fine-tuning support including automatic training data generation from documents, fine-tuning orchestration across providers (OpenAI, Hugging Face), and evaluation metrics for retrieval and generation quality. The framework generates synthetic question-answer pairs from documents, supports custom evaluation metrics, and tracks fine-tuning experiments. Fine-tuning can target embedding models, LLMs, or ranking models depending on application needs.

LlamaIndex provides comprehensive observability through an instrumentation framework that captures events across the entire application lifecycle (LLM calls, retrieval operations, agent steps, workflow transitions). The framework integrates with observability platforms (Langfuse, Arize, Datadog, New Relic) and provides structured event logging with automatic context propagation. Developers can define custom events and metrics, and the framework handles event batching and async transmission.