Haystack

Haystack provides a decorator-based component system (@component) where any Python class can be registered as a reusable pipeline node with typed inputs/outputs. Pipelines are constructed as directed acyclic graphs (DAGs) where components connect via socket-based routing, enabling explicit control flow definition. The Pipeline class validates component compatibility at graph construction time, performs type checking across connections, and supports both synchronous and asynchronous execution paths through separate Pipeline and AsyncPipeline implementations.

Haystack abstracts document persistence through a DocumentStore interface supporting Elasticsearch, Pinecone, Weaviate, and in-memory implementations. Documents are stored with both dense embeddings (for semantic search) and sparse keyword indices, enabling hybrid retrieval strategies. The abstraction layer handles backend-specific query translation, filtering, and result ranking without exposing provider APIs to pipeline code, allowing seamless backend swapping via configuration.

Haystack provides AsyncPipeline as a parallel implementation to Pipeline, enabling non-blocking execution of components with async/await syntax. Async components can perform I/O-bound operations (API calls, database queries) without blocking the event loop, improving throughput in high-concurrency scenarios. The AsyncPipeline validates component compatibility with async execution and manages event loop lifecycle, allowing developers to write async pipelines with the same component-based architecture as synchronous pipelines.

Haystack enforces type safety at multiple levels: component input/output types are specified via Python type hints, pipeline connections are validated at graph construction time to ensure type compatibility, and runtime type conversion is performed automatically for compatible types (e.g., str to List[str]). The component system uses socket-based routing (haystack/core/component/sockets.py) where each output socket has a declared type, and connections are validated before pipeline execution. This prevents type mismatches that would cause runtime errors.

Haystack enables developers to create custom components by decorating Python classes with @component, defining typed inputs and outputs via method signatures. The framework validates component contracts at pipeline construction time, ensuring type compatibility with connected components. Custom components can be stateful (holding model instances), async, and integrated seamlessly into pipelines without special handling.

Haystack provides a unified GenerativeModel interface supporting OpenAI, Azure OpenAI, Anthropic, Cohere, Hugging Face (API and local), and AWS Bedrock. Prompts are built using a ChatMessage-based abstraction (haystack/lazy_imports.py, chat message classes) that normalizes role/content across providers, and a PromptBuilder component enables Jinja2-based templating with variable injection. The framework handles provider-specific serialization (e.g., OpenAI's function_call vs Anthropic's tool_use), token counting, and error handling without exposing provider APIs.

Haystack's Agent system (AGENTS.md, Advanced Features) implements autonomous agents that iteratively reason about tasks, invoke tools, and update state based on results. Agents use an Agent component that wraps an LLM with a tool registry, manages conversation history, and implements a loop that continues until a termination condition is met (e.g., max iterations, tool returns final answer). Tool invocation is handled through a schema-based function registry that converts tool definitions to LLM-compatible formats (OpenAI function_call, Anthropic tool_use) and executes them with error handling.

Haystack provides a modular document processing stack (Document Converters, Document Preprocessing and Retrieval) supporting multiple input formats (PDF, HTML, DOCX, Markdown, etc.) through format-specific converters. Documents are converted to a unified Document object, then processed through a pipeline of cleaning, splitting, and embedding stages. The DocumentSplitter component implements multiple strategies (sliding window, recursive character splitting, semantic splitting) with configurable chunk size and overlap, enabling fine-grained control over document segmentation for retrieval.

Haystack provides Embedder components that generate dense vector representations using OpenAI, Hugging Face, Cohere, and local models. Embedders are integrated into document processing pipelines to embed documents at ingestion time and queries at retrieval time. Ranker components (e.g., SentenceTransformerRanker, LLMRanker) re-rank retrieved documents using semantic similarity or LLM-based scoring, improving retrieval quality. The abstraction allows swapping embedding models without changing pipeline code, enabling experimentation with different embedding strategies.

Haystack includes built-in evaluation components (Evaluation Components) for assessing RAG pipeline quality. Evaluators measure retrieval metrics (recall, precision, NDCG) by comparing retrieved documents against ground truth, and generation metrics (BLEU, ROUGE, semantic similarity) by comparing generated answers against reference answers. Evaluators are implemented as pipeline components, enabling evaluation to be integrated into training and validation workflows. The framework supports custom evaluators through a standard interface.

Haystack provides built-in observability through component-level tracing (Observability and Tracing) that captures execution flow, timing, and data at each pipeline step. Traces include component inputs/outputs, execution duration, and error information, enabling debugging and performance analysis. The framework integrates with external observability platforms (e.g., Datadog, New Relic) through a pluggable tracer interface, allowing production deployments to send traces to centralized monitoring systems without code changes.

Haystack supports human-in-the-loop (HITL) patterns through components that pause pipeline execution for human feedback, collect user ratings or corrections, and use feedback to improve model performance. HITL components integrate with evaluation frameworks to measure impact of feedback on pipeline quality. The framework enables workflows where humans review and correct LLM outputs, with corrections fed back into training or fine-tuning pipelines.

Haystack pipelines can be serialized to YAML/JSON format, capturing component definitions, connections, and parameters. Serialized pipelines are reproducible — they can be loaded and executed in different environments without code changes. The serialization format is human-readable, enabling version control and code review of pipeline definitions. Deserialization reconstructs the pipeline graph and component instances, enabling deployment of pipelines as configuration files rather than Python code.