Fsdp Integration For Distributed Quantized Model Training

via “distributed training with fsdp and model parallelism across multi-gpu and tpu”

Lightning AI's LLM library — pretrain, fine-tune, deploy with clean PyTorch Lightning code.

Unique: Integrates FSDP with PyTorch Lightning's distributed training callbacks, providing automatic rank management and checkpoint coordination, vs raw PyTorch FSDP which requires manual rank initialization and synchronization

vs others: Simpler distributed training setup than raw PyTorch FSDP, with automatic gradient synchronization and checkpoint management; more flexible than DeepSpeed which requires custom training loops

via “fsdp integration with automatic sharding strategies”

Easy distributed training — abstracts PyTorch distributed, DeepSpeed, FSDP behind simple API.

Unique: Automatically selects FSDP sharding strategy (FULL_SHARD, SHARD_GRAD_OP, NO_SHARD) based on model size and hardware, and provides utilities for managing FSDP-specific state (full_state_dict, sharded checkpoints) that raw FSDP requires manual handling for

vs others: More automatic than raw FSDP (which requires manual strategy selection) and more memory-efficient than DDP for very large models; integrates checkpoint management for FSDP's sharded state format

via “quantization with fp8, fp4, int8, and modelopt support”

Fast LLM/VLM serving — RadixAttention, prefix caching, structured output, automatic parallelism.

Unique: Provides a quantization registry that maps quantization types to optimized kernel implementations, with automatic fallback to slower kernels on unsupported hardware. Supports per-layer and per-channel quantization strategies with integrated calibration.

vs others: Supports more quantization schemes (FP8, FP4, INT8, MXFP4) than vLLM's INT8-only support, with optimized kernels for each scheme and automatic hardware-aware fallbacks.

8-bit and 4-bit quantization enabling QLoRA fine-tuning.

Unique: Implements custom hooks in GlobalOptimManager to synchronize QuantState metadata across FSDP ranks, enabling distributed training of quantized models without requiring users to write custom distributed code. Handles parameter sharding and gathering transparently.

vs others: Enables distributed training of quantized models with minimal code changes vs manual FSDP integration, and maintains quantization efficiency across multiple GPUs by properly synchronizing metadata.

via “distributed training with fsdp and multi-gpu synchronization”

PyTorch-native LLM fine-tuning library.

Unique: Wraps FSDP initialization and process group setup in a recipe-level abstraction, so users never directly call torch.distributed APIs. Torchtune automatically detects the number of available GPUs, initializes FSDP with optimal sharding strategies (FULL_SHARD, SHARD_GRAD_OP), and handles rank-aware checkpoint saving/loading without user intervention.

vs others: Simpler FSDP setup than raw PyTorch because torchtune handles process group initialization, device assignment, and checkpoint consolidation automatically, whereas users must manually write distributed boilerplate code with native PyTorch.

via “distributed training with adapter synchronization”

Parameter-efficient fine-tuning — LoRA, QLoRA, adapter methods for LLMs on consumer GPUs.

Unique: Leverages PyTorch DDP's gradient synchronization to coordinate adapter training across devices while keeping base model weights frozen and non-communicating. Reduces communication bandwidth by 99%+ compared to full model distributed training because only adapter parameters (0.1-2% of model) are synchronized across devices.

vs others: Enables efficient multi-GPU training with minimal communication overhead compared to full model DDP, achieving near-linear scaling efficiency (90%+) because adapter parameters are orders of magnitude smaller than full model weights.

via “distributed training with fsdp and gradient checkpointing”

Meta's library for music and audio generation.

Unique: Integrates FSDP with gradient checkpointing to enable training of large models on limited per-GPU memory; automatically handles parameter sharding, gradient synchronization, and activation recomputation across distributed devices through PyTorch's native APIs.

vs others: More memory-efficient than data parallelism alone; enables training of models that would not fit on single GPU. Simpler to implement than custom model parallelism while maintaining reasonable scaling efficiency.

via “multi-gpu distributed fine-tuning with fsdp orchestration”

Welcome to the Llama Cookbook! This is your go to guide for Building with Llama: Getting started with Inference, Fine-Tuning, RAG. We also show you how to solve end to end problems using Llama model family and using them on various provider services

Unique: Cookbook includes FSDP launch templates with automatic GPU detection, gradient checkpointing configuration, and mixed-precision (bfloat16) setup that works across different cluster topologies — most tutorials assume homogeneous setups

vs others: Simpler than DeepSpeed or Megatron for Llama fine-tuning because it uses PyTorch native FSDP without external dependency chains, reducing debugging surface area and enabling faster iteration on hyperparameters

via “distributed training with deepspeed and fsdp support”

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Unique: Integrates both DeepSpeed (with ZeRO-1/2/3 stages) and PyTorch FSDP through a unified distributed training interface that auto-detects hardware and configures the appropriate backend. Handles checkpoint sharding/unsharding transparently.

vs others: Supports both DeepSpeed and FSDP with automatic backend selection vs. alternatives like Hugging Face Trainer which requires manual DeepSpeed config, reducing setup complexity for distributed training.

via “multi-gpu distributed video generation with fsdp”

Phantom: Subject-Consistent Video Generation via Cross-Modal Alignment

Unique: Uses PyTorch FSDP to automatically shard model parameters, optimizer states, and gradients across 8-GPU clusters, enabling 14B parameter models to run where single-GPU approaches would fail. The implementation abstracts away manual sharding logic through PyTorch's native distributed primitives.

vs others: More efficient than naive data parallelism for large models because FSDP reduces per-GPU memory by 8x through weight sharding, and simpler to implement than custom model parallelism strategies that require manual layer partitioning.

via “distributed training with ddp and fsdp for multi-gpu scaling”

SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformer

Unique: Implements both DDP and FSDP strategies with automatic selection based on model size and hardware configuration, with integrated checkpoint management that handles distributed state serialization and conversion to single-GPU format

vs others: Provides flexible distributed training with both data parallelism (DDP) and model parallelism (FSDP) options, enabling efficient scaling from 2 GPUs to 100+ GPUs without code changes

via “fsdp (fully sharded data parallel) integration with automatic sharding configuration”

Accelerate

Unique: Implements automatic FSDP sharding strategy selection based on model size and hardware, eliminating manual strategy tuning. Integrates FSDP with mixed precision and gradient checkpointing for maximum memory efficiency.

vs others: More automated than raw PyTorch FSDP because it selects sharding strategy automatically; more flexible than DeepSpeed ZeRO because it allows fine-grained control over sharding strategy and integrates with other Accelerate features.