Determined AI

Enables multi-GPU and multi-node PyTorch training through a custom trial harness that wraps the standard PyTorch training loop. The system intercepts the training process via the PyTorchTrial base class, automatically handles distributed data loading, gradient aggregation across nodes, and checkpoint management without requiring users to manually implement DistributedDataParallel or write boilerplate synchronization code. Integration points include custom callbacks, learning rate schedulers, and context managers that inject distributed training logic transparently.

Implements a pluggable hyperparameter optimization framework that supports grid search, random search, Bayesian optimization, and population-based training (PBT). The system decomposes the search space into a configuration schema, spawns multiple trials with different hyperparameter combinations, and uses a search algorithm backend to generate the next set of hyperparameters based on trial results. The master service orchestrates trial scheduling and metric collection, feeding results back to the search algorithm via a standardized interface.

Provides a metrics collection API that training code can use to report metrics (loss, accuracy, custom metrics) during training. Metrics are streamed to the master service in real-time via gRPC, enabling live monitoring and early stopping decisions. The system supports both scalar metrics and structured metrics (e.g., confusion matrices), and automatically aggregates metrics across distributed trials. Metrics are persisted to PostgreSQL and can be queried via the API or visualized in the web UI.

Provides a pluggable early stopping framework that monitors trial metrics and stops trials that are unlikely to improve. The system supports multiple stopping policies (e.g., no improvement for N steps, metric threshold, PBT-based stopping) that can be configured in the experiment YAML. The master service evaluates stopping conditions after each metric report and sends a stop signal to the trial if conditions are met. Early stopping decisions are logged and can be reviewed in the web UI.

Provides an interactive notebook and command execution environment that runs on the cluster with GPU access. Users can launch Jupyter notebooks or shell commands that are scheduled as tasks on the cluster, with resource allocation managed by the same scheduler as training jobs. Notebooks and commands have access to the Determined Python SDK, enabling programmatic experiment submission and result analysis. Output (notebooks, logs) is persisted and accessible via the web UI.

Provides a model registry that tracks trained model checkpoints, their performance metrics, and associated metadata (training configuration, hyperparameters, etc.). Checkpoints can be tagged with semantic versions or custom labels, and the registry maintains a history of all versions. The system supports querying the registry to find best-performing models, comparing model versions, and downloading checkpoints for deployment. Integration with the web UI enables browsing and managing models without CLI commands.

Manages GPU and CPU resources across a cluster using a two-tier scheduling system: the master service maintains a global resource pool view and uses a pluggable resource manager (agent-based or Kubernetes-native) to allocate resources to tasks. The allocation service implements fairness policies (round-robin, priority queues) and bin-packing algorithms to maximize cluster utilization. Tasks (trials, notebooks, commands) are assigned to resource pools, and the scheduler respects constraints like GPU type, memory requirements, and node affinity. Integration with Kubernetes enables dynamic scaling and native resource quotas.

Provides a state machine-based experiment lifecycle that tracks trials from creation through completion, with automatic checkpoint saving at configurable intervals. The system persists experiment metadata, trial state, and model checkpoints to PostgreSQL and cloud storage (S3, GCS, etc.). On failure, the master service can restore experiments from the last checkpoint and resume training without losing progress. The checkpoint garbage collection service automatically prunes old checkpoints based on retention policies, freeing storage while preserving the best-performing models.

Provides a YAML-based configuration language for declaring experiments, including model architecture, training hyperparameters, distributed settings, and resource requirements. The configuration is parsed by the master service and used to instantiate trials with the specified settings. The system supports configuration inheritance, environment variable substitution, and validation against a schema. Configuration changes trigger experiment re-runs without requiring code changes, enabling reproducible experimentation and easy sharing of experiment definitions.

Provides a React-based web interface for visualizing experiment progress, trial metrics, and cluster status in real-time. The UI connects to the master service via REST and gRPC APIs, streaming metric updates and task status changes. Users can interactively pause/resume/kill trials, adjust resource allocations, and view detailed logs and checkpoint metadata. The UI includes dashboards for comparing trial performance, visualizing hyperparameter importance, and tracking resource utilization across the cluster.

Exposes the master service functionality via dual REST and gRPC APIs, enabling programmatic access to experiments, trials, and cluster resources. The APIs are auto-generated from Protocol Buffer definitions, ensuring consistency between REST and gRPC interfaces. Clients can submit experiments, query trial status, retrieve metrics, and manage resources without using the CLI or web UI. The API supports streaming responses for real-time metric updates and includes authentication via API tokens.

Provides a command-line interface for submitting experiments, querying cluster status, managing tasks, and retrieving results. The CLI communicates with the master service via REST/gRPC APIs and supports both interactive and scripted workflows. Commands include experiment submission, trial inspection, checkpoint download, and resource pool management. The CLI supports configuration file loading, environment variable substitution, and output formatting (JSON, table, etc.) for integration with shell scripts and automation tools.

Provides Helm charts and Kubernetes manifests for deploying Determined on Kubernetes clusters, with native integration for resource quotas, pod scheduling, and dynamic scaling. The master service runs as a Kubernetes deployment, and worker tasks are scheduled as Kubernetes pods with resource requests/limits. The system supports multiple resource pools mapped to Kubernetes namespaces or node selectors, enabling multi-tenant deployments. Horizontal pod autoscaling can be configured to scale worker pods based on cluster load.

Provides a Keras-compatible training harness that wraps TensorFlow/Keras models and enables distributed training without requiring users to manually implement tf.distribute strategies. The system intercepts the training loop via a custom callback, handles data distribution across GPUs/nodes, and manages gradient synchronization. The harness supports both eager execution and graph mode, and integrates with Determined's checkpoint and metric collection systems.