Temporal

Executes application workflows as code with automatic checkpointing to a persistence layer (PostgreSQL, MySQL, Cassandra, or in-memory), enabling workflows to survive process crashes, network failures, and server restarts without losing execution state. Uses event sourcing via a History Service that maintains an immutable event log of all workflow decisions and state transitions, allowing deterministic replay of workflow logic from any point in the execution timeline.

Implements configurable retry policies with exponential backoff, jitter, and maximum retry counts at both the activity and workflow levels. The History Service generates retry tasks when activities fail, and the Matching Service re-queues them to available workers with backoff delays. Timeouts (start-to-close, schedule-to-close, heartbeat) are enforced server-side via the History Service's task generation engine, preventing zombie tasks from consuming resources indefinitely.

Enforces rate limits and quotas at the Frontend Service level via a configurable Rate Limiting and Quotas system. Supports per-namespace limits (max workflows/sec, max activities/sec) and per-task-queue limits (max concurrent activities). Rate limiting uses token bucket algorithms with configurable refill rates, and quota enforcement is applied before tasks are dispatched to workers, preventing overload.

Provides a pluggable request interceptor chain in the Frontend Service that allows custom logic to be applied to all incoming requests. Built-in interceptors handle authentication (JWT, mTLS), request logging, and distributed tracing (OpenTelemetry). Interceptors are applied in order before the request reaches the handler, enabling cross-cutting concerns without modifying handler code.

Enables workflows in one namespace to invoke workflows or activities in another namespace or even another Temporal cluster via the Nexus Operations system. Nexus provides a service-oriented interface for cross-namespace communication, with built-in retry logic, timeout management, and result caching. Invocations are routed through the Frontend Service and can span multiple clusters if configured.

Provides batch operations for managing large numbers of workflows without overwhelming the system. Supports batch termination, batch signaling, and batch visibility queries via the Batch Operations system. Batch operations are processed asynchronously by the Worker Service, with progress tracking and error handling. Enables operators to manage thousands of workflows efficiently (e.g., terminate all workflows for a customer).

Provides a built-in Scheduler Workflow that enables recurring workflow execution (cron-like schedules) and delayed execution without requiring external schedulers. Schedules are defined with cron expressions or interval-based patterns, and the Scheduler Workflow automatically spawns workflow executions at the scheduled times. Supports timezone-aware scheduling, backfill for missed executions, and pause/resume of schedules.

Routes workflow and activity tasks to workers via a task queue abstraction managed by the Matching Service. Workers poll task queues via long-polling gRPC connections, and the Matching Service dispatches tasks to available workers based on queue depth and worker availability. Supports multiple workers per queue for horizontal scaling, with built-in load balancing that prevents queue starvation and ensures fair task distribution across workers.

Supports deploying new versions of workflow and activity code without stopping running workflows via the Worker Versioning system. Workers register their version set (e.g., 'v1.0', 'v1.1') when connecting to a task queue, and the server routes tasks to compatible worker versions based on compatibility rules. Enables gradual rollouts where old and new worker versions coexist, with the server ensuring that in-flight workflows continue on their original version until completion.

Enables complex distributed workflows by supporting child workflows (nested workflow execution) and signals (asynchronous notifications sent to running workflows). Child workflows are spawned from parent workflows and inherit the parent's context, allowing hierarchical workflow composition. Signals are delivered via the Frontend Service and persisted in the History Service, ensuring that signals sent to workflows are not lost even if the workflow is temporarily paused or the worker crashes.

Provides multi-region disaster recovery via the Replication System, which asynchronously replicates workflow execution history from a primary cluster to standby clusters. Each cluster is isolated by namespace, allowing different teams or applications to run independently with their own retention policies and replication rules. The History Service coordinates replication by sending history events to remote clusters via gRPC, with built-in conflict resolution for concurrent updates.

Provides full-text search and filtering of workflow executions via a Visibility Store (Elasticsearch, SQL database, or in-memory index). Workflows can emit custom attributes (key-value pairs) that are indexed and searchable, enabling queries like 'find all workflows for customer X with status RUNNING'. The Frontend Service exposes search APIs that query the Visibility Store, decoupling search from the main execution path to avoid impacting workflow throughput.

Automatically archives completed workflows to long-term storage (S3, GCS, Azure Blob Storage, or local filesystem) based on retention policies configured per namespace. The Archiver service runs as a background process that reads completed workflows from the History Service and writes them to the archive backend in a compressed, queryable format. Archived workflows can be retrieved for audit or debugging purposes without keeping them in the main execution database.

Provides dynamic configuration management via the Dynamic Configuration system, allowing operators to change server behavior (rate limits, timeouts, feature flags) without restarting the server. Configuration values are stored in a database and polled periodically by each service, with support for per-namespace and per-task-queue overrides. Changes propagate within seconds, enabling rapid response to operational issues (e.g., disabling a buggy activity type).

Emits detailed metrics about workflow execution, task processing, and system health via a pluggable metrics reporter system. Built-in reporters support Prometheus, StatsD, and custom implementations. Metrics include workflow throughput, latency percentiles, error rates, task queue depth, and worker availability. The Metrics and Observability system is integrated throughout the codebase via dependency injection, ensuring comprehensive instrumentation without scattered metric collection code.