Accelerate

Abstracts PyTorch's distributed training backends (DDP, FSDP, DeepSpeed, Megatron-LM) behind a unified Accelerator class that auto-detects hardware and selects the appropriate backend without code changes. The Accelerator wraps models, optimizers, and dataloaders with backend-specific logic while preserving the user's training loop structure, enabling the same script to run on single GPU, multi-GPU, TPU, or multi-node clusters by only changing launch configuration.

Implements FP16, BF16, and FP8 mixed-precision training by wrapping the backward pass and optimizer step with automatic casting logic that varies by backend and hardware. Uses native PyTorch autocast for DDP, DeepSpeed's native FP16 handler for DeepSpeed training, and FSDP's built-in mixed-precision APIs for FSDP, automatically selecting the optimal implementation based on detected hardware capabilities (e.g., BF16 support on newer GPUs).

Wraps PyTorch's Fully Sharded Data Parallel (FSDP) with automatic sharding strategy selection based on model size and available hardware. Handles FSDP-specific configuration (sharding strategy, backward prefetch, CPU offloading) transparently, and provides utilities for saving/loading sharded checkpoints and managing FSDP-specific state (e.g., full_state_dict for inference).

Wraps DeepSpeed's ZeRO optimizer with automatic stage selection (Stage 1: gradient partitioning, Stage 2: optimizer state partitioning, Stage 3: parameter partitioning) based on model size and available memory. Handles DeepSpeed-specific configuration (activation checkpointing, gradient accumulation, communication hooks) transparently, and provides utilities for DeepSpeed checkpoint management and inference optimization.

Provides a notebook_launcher function that detects the notebook environment (Jupyter, Colab, Kaggle) and launches distributed training within the notebook process, handling process spawning and environment setup automatically. Enables distributed training experimentation in notebooks without manual process management, with support for multiple GPUs and TPUs.

Wraps PyTorch optimizers with AcceleratedOptimizer that handles distributed gradient synchronization, gradient accumulation step counting, and backend-specific optimizer state management. Automatically defers optimizer steps until gradient accumulation threshold is reached, and handles gradient scaling for mixed-precision training without requiring manual loss scaling logic.

Wraps PyTorch DataLoaders to automatically partition data across distributed processes using DistributedSampler under the hood, with support for multiple sharding strategies (by-index, by-node, custom). Maintains DataLoader state (current batch index, epoch) across checkpoints, enabling exact resumption from a checkpoint without data duplication or skipping, even in distributed settings where process counts may change between runs.

Implements gradient accumulation by deferring gradient synchronization across processes until the accumulation step count is reached, reducing communication overhead. Uses backend-specific synchronization hooks (DDP's no_sync context manager, DeepSpeed's gradient accumulation steps, FSDP's reduce-scatter timing) to avoid redundant all-reduce operations, enabling effective batch size scaling without proportional communication cost.

Implements automatic device mapping for models larger than GPU memory by partitioning model layers across GPUs and CPU using a cost model that estimates layer memory and compute time. Supports CPU offloading (layer swapping between GPU and CPU) and NVMe offloading (via DeepSpeed) for models that exceed total system memory, with hooks to manage data movement and minimize latency impact during inference.

Detects and optimizes models with tied parameters (e.g., shared embeddings in language models) by tracking parameter aliases and ensuring gradients are correctly accumulated across all uses without duplication. Implements a hook system that intercepts backward passes to merge gradients from tied parameters before optimizer steps, reducing memory overhead and preventing gradient inconsistencies in distributed settings.

Provides a unified checkpoint API that serializes model, optimizer, and DataLoader state across distributed processes, with support for custom checkpoint hooks and project-level configuration. Handles backend-specific state serialization (e.g., DeepSpeed's checkpoint format, FSDP's sharded checkpoints) transparently, and supports resuming from checkpoints with different process counts or hardware configurations.

Integrates with experiment tracking platforms (Weights & Biases, TensorBoard, Comet, MLflow) via a unified Tracker API that handles multi-process logging coordination, ensuring only the main process logs to avoid duplicate entries. Provides context managers and decorators for tracking custom metrics, and automatically logs training hyperparameters and system information across all supported backends.

Provides high-level APIs for distributed collective operations (all-reduce, all-gather, broadcast, scatter) that abstract backend differences (DDP, FSDP, DeepSpeed) and handle tensor type conversions automatically. Includes utilities for gathering metrics across processes, broadcasting model state, and synchronizing random number generators to ensure reproducibility in distributed settings.

Provides the accelerate launch CLI that configures and launches distributed training scripts with automatic environment variable setup. Detects available hardware (GPUs, TPUs, CPU count) and prompts for configuration (backend, precision, etc.), then sets up process groups and environment variables before launching the training script, eliminating manual torchrun or torch.distributed.launch setup.