llama-index-core

Ingests documents from diverse sources (files, web, cloud APIs) through a modular reader architecture that abstracts source-specific logic. Each reader implements a common interface that normalizes heterogeneous data formats (PDF, markdown, HTML, JSON, databases) into a unified Document object with metadata preservation. The framework uses a registry pattern to discover and instantiate readers, enabling extensibility without core framework changes.

Splits documents into chunks using multiple strategies (fixed-size, recursive, semantic) that preserve document structure and relationships. The NodeParser abstraction allows pluggable chunking logic; implementations include SimpleNodeParser (basic splitting), HierarchicalNodeParser (preserves heading hierarchy), and SemanticSplitter (uses embeddings to find natural boundaries). Chunk metadata includes parent-child relationships, document source, and custom attributes for context-aware retrieval.

Provides utilities for fine-tuning LLMs on domain-specific data generated from RAG systems. The framework can generate synthetic training data from retrieval results, format it for fine-tuning APIs (OpenAI, Anthropic), and manage fine-tuning jobs. Fine-tuned models can be used as drop-in replacements in RAG pipelines, improving performance on domain-specific tasks without retraining from scratch. The system tracks fine-tuning experiments and enables comparison of base vs fine-tuned model performance.

Enables LLMs to generate structured outputs (JSON, Pydantic models, dataclasses) with schema validation. The framework uses provider-specific structured output APIs (OpenAI JSON mode, Anthropic structured output) or LLM-based parsing with validation fallback. Output schemas are defined as Pydantic models or JSON schemas; the framework automatically formats prompts to guide LLM generation and validates outputs against schemas. Failed validations trigger retries with corrected prompts.

Integrates with the Model Context Protocol (MCP) standard for tool definition and execution, enabling standardized tool calling across applications. MCP servers expose tools through a standard interface; the framework discovers and registers MCP tools for use in agents and workflows. This enables reuse of tools across different LLM applications and providers without reimplementation. MCP integration handles authentication, request/response serialization, and error handling transparently.

Manages LLM context windows by tracking token usage and automatically summarizing or truncating context when approaching limits. The framework estimates token counts for prompts, retrieved context, and conversation history using provider-specific tokenizers. When context approaches the model's limit, it applies strategies: summarization (condense context with LLM), truncation (remove oldest messages), or hierarchical retrieval (fetch higher-level summaries). This enables long conversations and large document sets without hitting context limits.

Provides LlamaDatasets and evaluation utilities for benchmarking RAG systems. Datasets include pre-built question-answer pairs for common domains (finance, medical, legal). The framework supports custom dataset creation from documents, automatic evaluation metrics (BLEU, ROUGE, semantic similarity), and comparison of different RAG configurations. Evaluation results are tracked and can be exported for analysis. This enables systematic optimization of RAG pipelines.

Provides multiple index types (VectorStoreIndex, SummaryIndex, TreeIndex, PropertyGraphIndex, KeywordTableIndex) that organize ingested nodes for different retrieval patterns. Each index implements a common Index interface with a query_engine() method that returns a QueryEngine for executing retrieval. Indices are backed by pluggable storage (vector stores, graph databases, in-memory) and support hybrid retrieval combining multiple strategies. The framework handles index construction, persistence, and updates transparently.

Executes queries against indices through a multi-stage pipeline: retrieval (fetch candidate nodes), reranking (score/filter candidates), synthesis (generate response from top nodes). QueryEngine implementations (RetrieverQueryEngine, RouterQueryEngine, SubQuestionQueryEngine) support different retrieval patterns. Rerankers (Cohere, LLM-based, similarity-based) re-score retrieved nodes to improve relevance. The synthesis stage uses an LLM to generate grounded responses from retrieved context, with configurable prompts and response modes (compact, tree_summarize, accumulate).

Abstracts LLM interactions behind a common LLM interface supporting 20+ providers (OpenAI, Anthropic, Google, AWS Bedrock, Ollama, Azure, etc.). Each provider implements complete_message() and stream_message() methods accepting ContentBlock messages (text, image, tool calls). The framework handles provider-specific details: API authentication, request formatting, response parsing, streaming, and error handling. Tool calling is standardized across providers through a schema-based function registry that maps to native provider APIs (OpenAI functions, Anthropic tools, etc.).

Abstracts embedding generation and storage behind pluggable Embedding and VectorStore interfaces. Embedding implementations support 15+ providers (OpenAI, Cohere, HuggingFace, local models via Ollama). VectorStore implementations support 10+ backends (Pinecone, Weaviate, Milvus, Qdrant, PostgreSQL, Azure AI Search, etc.). The framework handles embedding generation during indexing, storage in vector databases, and similarity search during retrieval. Batch embedding operations optimize API calls; caching prevents redundant embeddings for identical text.

Provides a Workflow abstraction for building event-driven, stateful LLM applications using a step-based execution model. Workflows are defined as classes with step methods decorated with @step; each step is an async function that processes input and emits events triggering downstream steps. The framework manages event routing, step scheduling, and state persistence across step executions. Workflows support branching (conditional step execution), loops (iterative processing), and error handling with retry logic. State is managed through a unified context object passed between steps.

Provides Agent abstraction for building autonomous LLM agents that use tools to accomplish goals. Agents implement a reasoning loop: observe (read state/context), think (LLM generates reasoning + tool calls), act (execute tools), and repeat until goal achieved or max iterations reached. Tool calling is standardized through a schema-based function registry that maps to LLM provider APIs. The framework supports multiple agent types: ReActAgent (reasoning + acting), OpenAIAgent (native OpenAI function calling), and custom agents. Memory management tracks conversation history and tool execution results. Multi-agent orchestration enables agent-to-agent communication and delegation.

Builds knowledge graphs from documents by extracting entities and relationships using LLMs, then storing them in a graph database (Neo4j, Nebula, Kuzu). The PropertyGraphIndex uses an LLM to extract structured triples (subject, predicate, object) from document chunks, deduplicates entities across chunks, and builds a connected graph. Query execution traverses the graph to find relevant entities and relationships, then retrieves associated document chunks. The framework supports graph-based reasoning: multi-hop traversal, relationship filtering, and entity-centric retrieval.

Provides instrumentation hooks throughout the framework (LLM calls, embeddings, retrievals, agent steps) that emit structured events for monitoring and debugging. Events are captured through a pluggable event handler system supporting multiple backends (console, file, cloud services like Datadog, New Relic). The framework tracks latency, token usage, cost, and errors for each operation. Integration with observability platforms enables real-time monitoring, tracing, and alerting. Custom event handlers can be registered to implement application-specific logging or metrics.