Letta (MemGPT)

Implements a sliding-window context management system that maintains unlimited conversation history by automatically summarizing older messages and archiving them when the LLM's context window approaches capacity. Uses a tiered memory architecture where recent messages stay in the active context, mid-range messages are compressed via LLM summarization, and older messages are moved to archival storage with vector embeddings for semantic retrieval. The system tracks token counts per message and dynamically decides what to keep in-context vs. archive based on configurable thresholds and message importance scoring.

Provides a multi-block memory architecture where agents maintain distinct, editable memory sections: persona (agent identity/instructions), human (user profile/preferences), and custom context blocks. Each block is independently versioned, searchable, and can be modified by the agent itself through dedicated memory-editing tools (core_memory_append, core_memory_replace). The system uses a Git-backed storage model for memory versioning, allowing rollback and audit trails. Memory blocks are injected into the system prompt at runtime, and the agent can introspect and modify its own memory based on conversation context.

Implements a message persistence layer that stores all agent-user conversations in a database with support for full-text search, filtering, and retrieval. Messages are stored with metadata (timestamp, sender, message type, tool calls, etc.) and indexed for efficient querying. Supports searching conversations by content, date range, sender, or message type. Provides APIs for retrieving conversation history, exporting conversations, and analyzing conversation patterns. Integrates with the archival memory system to automatically extract and index important passages from conversations.

Provides batch processing capabilities for running agents on large datasets or executing agents on schedules. Supports batch job submission with input data (CSV, JSON, etc.), parallel execution across multiple agent instances, and result aggregation. Integrates with job scheduling systems (APScheduler, Celery) to enable periodic agent execution (e.g., daily reports, periodic data processing). Batch jobs can be monitored for progress, paused/resumed, and results can be exported or streamed to external systems.

Implements human-in-the-loop (HITL) workflows where agents can request human approval before executing sensitive operations, and humans can provide feedback to improve agent behavior. The system pauses agent execution at designated checkpoints, routes requests to human reviewers, and resumes execution based on approval/rejection. Supports feedback collection (ratings, corrections, suggestions) that can be used to fine-tune agent behavior or update memory. Integrates with the tool execution system to gate sensitive tool calls, and with the memory system to incorporate human feedback.

Provides voice interaction capabilities for agents with audio input/output streaming and automatic speech-to-text transcription. Agents can receive audio streams, transcribe them to text using speech recognition services, process the text, and generate audio responses using text-to-speech. Supports streaming audio for low-latency voice interactions and integrates with voice providers (OpenAI Whisper, Google Speech-to-Text, etc.). Handles audio format conversion and quality management.

Implements multi-tenant architecture where multiple organizations/users can use the same Letta instance with isolated data and access control. Each tenant has isolated agents, conversations, and data. The system implements role-based access control (RBAC) with roles like admin, agent-creator, viewer, etc., and fine-grained permissions for agent management, conversation access, and tool execution. Supports API key-based authentication and OAuth integration. Tenant isolation is enforced at the database and API levels.

Provides a unified LLM client interface that abstracts over 10+ LLM providers (OpenAI, Anthropic, Google Gemini, Ollama, local models, etc.) with automatic message format transformation. The system implements a provider-agnostic message schema internally, then transforms messages to each provider's specific format (OpenAI's chat completion format, Anthropic's native format, etc.) at request time. Handles provider-specific features like prompt caching (OpenAI), thinking tokens (o1), tool-use schemas, and reasoning models. Includes built-in retry logic, error handling, and fallback mechanisms for provider failures.

Provides a tool management and execution system where agents can call custom Python tools with configurable sandboxing and access control rules. Tools are registered with schemas (input/output types, descriptions) and executed in isolated environments with resource limits (CPU, memory, execution time). The system includes a rule engine that evaluates tool-use policies before execution — agents can be restricted from calling certain tools, tools can be rate-limited, and execution can require human approval. Supports both synchronous and asynchronous tool execution with streaming support for long-running operations.

Integrates with the Model Context Protocol (MCP) standard, allowing agents to discover and use tools exposed via MCP servers. The system implements native MCP client bindings that communicate with MCP servers over stdio or HTTP transports, automatically translating MCP tool schemas into Letta's internal tool format. Agents can dynamically load tools from MCP servers at runtime, and the system handles MCP-specific features like resource management and sampling. Supports both local MCP servers and remote MCP endpoints.

Implements a vector-backed archival storage system for long-term memory that stores conversation passages, documents, and knowledge with semantic embeddings. When agents need to retrieve relevant information, the system performs vector similarity search across archived passages and returns the most relevant results. Passages are automatically chunked from documents and conversations, embedded using configurable embedding models, and stored in a vector database (Chroma, Pinecone, etc.). The system supports hybrid search (semantic + keyword) and can rank results by relevance, recency, or custom scoring functions.

Provides a multi-agent system framework where agents can be organized into groups and coordinated through different patterns (sequential, parallel, hierarchical, broadcast). Agents within a group can communicate through shared memory, message passing, or tool calls. The system manages agent lifecycle (creation, activation, sleep/wake), handles inter-agent communication, and provides coordination primitives like barriers and message queues. Supports 'sleeptime agents' that wake up based on time or event triggers, enabling long-running multi-agent workflows.

Exposes a comprehensive REST API for agent management, messaging, and streaming with support for long-running operations through a job/run system. The API supports streaming responses (Server-Sent Events) for real-time agent output, background job execution with status tracking, and webhook callbacks for job completion. Implements a SyncServer abstraction layer that handles request routing, database persistence, and service orchestration. The job system decouples request handling from execution, allowing agents to run asynchronously with status polling or webhook notifications.

Provides an end-to-end file processing pipeline that ingests documents (PDF, text, code, images) and makes them searchable by agents. The pipeline includes OCR for image-based PDFs, intelligent chunking strategies (semantic, fixed-size, sliding-window), and automatic embedding generation. Processed documents are stored in a vector database with metadata (source, page number, chunk boundaries) and indexed for semantic search. Agents can query documents through the archival memory system or directly through file-based tools. Supports batch processing of large document collections.

Provides serialization and deserialization of agent configurations, memory state, and conversation history for backup, migration, and sharing. Agents can be exported to JSON/YAML with full state (memory blocks, tools, LLM configuration, conversation history) and imported to recreate the agent in a new environment. Supports partial exports (e.g., memory only, configuration only) and selective import (e.g., restore memory but keep new conversation history). Enables version control of agent configurations and facilitates agent cloning and templating.