holmesgpt

Executes a closed-loop reasoning cycle that alternates between LLM inference and tool execution, using structured tool-calling APIs (OpenAI, Anthropic native function calling) to invoke observability and infrastructure tools. The loop maintains conversation state across iterations, processes tool outputs through transformers, and implements context window management to handle large observability datasets. Tool execution is gated by an approval/security model that validates tool calls before execution against configured RBAC policies.

Aggregates real-time observability data from heterogeneous sources (Kubernetes API, Prometheus, Grafana, Loki, Tempo, DataDog, cloud provider APIs) through a pluggable toolset architecture. Each toolset encapsulates source-specific query logic, authentication, and data transformation. The system uses a factory-based loader (holmes/plugins/toolsets/__init__.py) to dynamically instantiate toolsets from configuration, and applies tool output transformers to normalize disparate data formats into a consistent schema for LLM processing.

Provides an interactive CLI interface (holmes/interactive.py) for conversational investigation with multi-turn dialogue support. The CLI maintains conversation history, supports tool execution with user approval workflows, displays investigation results with formatting, and integrates with the agentic loop for iterative investigation. Supports both interactive mode (human-in-the-loop) and batch mode (automated investigation) through the same codebase.

Exposes investigation capabilities through a REST API (server.py) with streaming support for long-running investigations. The API supports investigation triggering (alerts, issues, custom queries), result polling or streaming via Server-Sent Events (SSE), and webhook integration for alert/issue sources. Implements authentication, rate limiting, and request validation. Supports both synchronous (request-response) and asynchronous (streaming) investigation patterns.

Implements a tool approval and security model that gates tool execution based on RBAC policies and approval workflows. The system supports multiple approval modes: auto-approve (for safe tools), require-approval (for sensitive operations like pod deletion), and deny (for prohibited tools). Integrates with Kubernetes RBAC and custom authorization providers. Logs all tool executions for audit trails and supports dry-run mode for previewing tool effects without execution.

Implements scheduled investigation capabilities for proactive health checks and periodic analysis. The system supports cron-like scheduling (e.g., daily health checks on critical services), automatic investigation triggering based on conditions (e.g., investigate when error rate exceeds threshold), and result persistence to external systems (Jira, Slack, databases). Integrates with the agentic loop for investigation execution and supports custom investigation templates per schedule.

Provides a plugin system for developing custom toolsets that extend HolmesGPT with domain-specific tools. The system uses a base Toolset class and factory pattern (holmes/plugins/toolsets/__init__.py) to enable custom tool definitions without modifying core code. Custom toolsets can integrate with proprietary systems (internal APIs, custom databases, specialized monitoring tools) and are loaded dynamically from configuration. Includes documentation and examples for common integration patterns.

Implements Model Context Protocol (MCP) server support, enabling HolmesGPT to be deployed as an MCP server and integrated with other MCP clients (Claude Desktop, other LLM applications). The MCP integration exposes HolmesGPT tools as MCP resources, enabling external LLM applications to invoke investigations without direct API calls. Supports both standalone MCP server deployment and embedded MCP server within HolmesGPT.

Integrates with Robusta platform (the company behind HolmesGPT) for enhanced automation, alerting, and incident management capabilities. The integration enables bi-directional data flow between HolmesGPT and Robusta platform, automatic alert routing to HolmesGPT for investigation, and result synchronization back to Robusta for incident tracking. Supports Robusta-specific features like playbook execution and alert enrichment.

Provides a specialized toolset for Kubernetes cluster investigation with tools for querying cluster state (nodes, pods, deployments, services), analyzing resource health, executing diagnostic commands in containers, and retrieving logs. The toolset uses the Kubernetes Python client library to interact with the cluster API, supports both in-cluster (service account) and out-of-cluster (kubeconfig) authentication, and implements resource filtering by namespace, label selectors, and resource type. Tool outputs are transformed into structured formats suitable for LLM analysis.

Integrates Prometheus as a native observability data source through a specialized toolset that executes PromQL queries, retrieves time-series metrics, and analyzes metric trends. The toolset handles Prometheus API authentication, query execution with configurable time ranges and step intervals, and transforms metric results into structured formats. Supports both instant queries (current metric values) and range queries (time-series data), enabling the agent to analyze metric patterns and anomalies.

Implements a specialized investigation workflow triggered by alert payloads (Prometheus AlertManager format, Grafana alerts, or custom alert sources). The workflow extracts alert metadata (firing rules, labels, annotations), automatically constructs investigation context (related metrics, logs, Kubernetes resources), and executes the agentic loop to determine root causes and recommend remediation. Integrates with issue sources/destinations to create tickets in Jira, GitHub Issues, or Slack notifications.

Implements an investigation workflow triggered by issues/tickets from external sources (Jira, GitHub Issues, linear.app) or direct API calls. The workflow extracts issue metadata (title, description, labels, assignee), automatically constructs investigation context from observability data sources, and executes the agentic loop to analyze the issue and provide recommendations. Results can be written back to the issue as comments or create linked tickets. Supports scheduled investigation of issues (e.g., daily health checks on critical services).

Integrates organizational knowledge sources (Confluence, Notion, Runbooks, internal wikis) into the investigation context through a knowledge retrieval system. The system indexes runbooks and documentation, retrieves relevant knowledge based on investigation context (alert type, service name, error patterns), and injects retrieved knowledge into the LLM prompt to guide investigation and remediation recommendations. Supports both static knowledge (indexed documents) and dynamic knowledge (API-based retrieval).

Implements a provider-agnostic LLM abstraction layer (holmes/core/llm.py) that supports multiple LLM providers (OpenAI, Anthropic, Ollama, custom providers) through a factory pattern and model registry. The system abstracts provider-specific API differences (function calling schemas, token counting, streaming formats) behind a unified interface, enabling seamless provider switching via configuration. Supports both cloud-hosted models (GPT-4, Claude) and self-hosted models (Ollama, vLLM) with configurable parameters (temperature, max tokens, system prompts).

Implements intelligent context window management to handle large observability datasets that exceed LLM token limits. The system uses multiple strategies: summarization of large tool outputs, pagination of results, prioritization of recent/relevant data, and automatic context trimming based on token budgets. Supports configurable context window sizes per LLM provider and implements token counting to track context usage across conversation history, tool outputs, and system prompts.

Implements structured output processing to extract and validate investigation results from LLM responses. The system uses JSON schema validation to ensure LLM outputs conform to expected structures (root cause analysis, remediation steps, confidence scores), handles partial or malformed outputs gracefully, and provides fallback mechanisms for invalid responses. Supports custom output schemas per investigation type and integrates with issue sources/destinations for structured result writing.