Diffusers

Provides a unified DiffusionPipeline base class that orchestrates end-to-end inference by composing models (UNet, VAE, text encoders), schedulers, and adapters into a single callable interface. The pipeline system uses ConfigMixin for serialization and ModelMixin for device management, enabling users to swap components (e.g., different schedulers or LoRA adapters) without rewriting inference logic. Pipelines automatically handle component initialization, device placement, and memory management across CPU/GPU/multi-GPU setups.

Implements a SchedulerMixin base class with pluggable scheduler implementations (DDPM, DDIM, DPM++, Euler, Karras, LCM) that decouple the noise schedule from the diffusion model. Each scheduler manages timestep ordering, noise scaling, and step prediction via a unified interface (set_timesteps(), step()). The scheduler system supports custom noise schedules (linear, cosine, sqrt) and enables runtime switching without reloading the model, allowing users to trade off speed vs quality by selecting different schedulers for the same checkpoint.

Implements a unified model loading system via from_pretrained() that handles multiple checkpoint formats (.safetensors, .bin, .pt, .pth) and automatically downloads models from Hugging Face Hub or loads from local paths. The system supports single-file loading (loading entire pipelines from .safetensors files) and checkpoint conversion utilities that transform weights from other frameworks (Stability AI, Civitai, etc.) into Diffusers format. ModelMixin provides device management (CPU/GPU/multi-GPU) and gradient checkpointing for memory optimization.

Provides training scripts for DreamBooth (fine-tuning the full UNet on a few images of a subject to learn a unique identifier) and Textual Inversion (learning a new token embedding for a concept using a few examples). Both approaches use a small number of images (3-10) and produce lightweight checkpoints (LoRA-style weights for DreamBooth, embedding vectors for Textual Inversion) that can be loaded into any base model. The system includes regularization techniques (prior preservation loss) to prevent overfitting and supports multi-GPU training.

Implements multiple guidance mechanisms to steer generation toward specific concepts: classifier-free guidance (CFG) uses unconditional predictions to amplify conditional signals; CLIP guidance uses CLIP embeddings to align generated images with text; Perturbed Attention Guidance (PAG) modulates attention weights to enhance concept alignment. Each guidance type has different computational costs and quality tradeoffs. The system supports combining multiple guidance types and enables per-step guidance scale adjustment for fine-grained control.

Provides memory optimization techniques to reduce VRAM usage for large models: attention slicing computes attention in chunks to reduce peak memory; VAE tiling processes large images in overlapping tiles to avoid OOM errors; gradient checkpointing trades computation for memory by recomputing activations during backprop. The system enables these optimizations via simple API calls (enable_attention_slicing(), enable_vae_tiling(), enable_gradient_checkpointing()) and supports combining multiple techniques for cumulative memory savings.

Implements automatic device management and distributed inference support via ModelMixin, enabling models to be moved across CPU/GPU/multi-GPU setups without code changes. The system supports data parallelism (replicating models across GPUs) and pipeline parallelism (splitting models across GPUs) for large models. Device management handles memory transfers, synchronization, and gradient aggregation automatically, with support for mixed precision (float16, bfloat16) to reduce memory and increase speed.

Integrates PEFT (Parameter-Efficient Fine-Tuning) library to load and merge LoRA (Low-Rank Adaptation) weights into UNet and text encoder models without modifying the base model architecture. The system uses load_lora_weights() to inject LoRA layers and set_lora_scale() to dynamically adjust LoRA influence (0.0 = base model, 1.0 = full LoRA) during inference. LoRA weights are stored as separate checkpoints and merged on-the-fly, enabling users to compose multiple LoRAs or switch between them without reloading the base model.

Implements ControlNet integration as a conditional generation system that injects spatial guidance (edge maps, depth, pose, segmentation) into the diffusion process via cross-attention mechanisms. ControlNet models are loaded separately and their outputs are added to the UNet's cross-attention layers during the denoising loop, allowing precise spatial control without modifying the base model. The system supports multiple ControlNet types (Canny edges, depth estimation, OpenPose) and enables ControlNet stacking (multiple spatial conditions simultaneously) with per-ControlNet scale adjustment.

Implements IP-Adapter (Image Prompt Adapter) as a lightweight cross-modal conditioning system that encodes reference images via CLIP image encoder and injects their embeddings into the UNet's cross-attention layers, enabling style transfer and content-guided generation without text prompts. IP-Adapter weights are separate from the base model and use a projection layer to map CLIP image embeddings to the UNet's embedding space. The system supports multiple IP-Adapter variants (standard, plus, face) and enables IP-Adapter stacking with per-adapter scale control for blending multiple reference images.

Implements StableDiffusionPipeline as a text-to-image system that encodes text prompts via CLIP text encoder, passes embeddings to the UNet denoising loop, and applies classifier-free guidance (CFG) to amplify text-image alignment. The pipeline supports negative prompts (anti-guidance) to suppress unwanted concepts, guidance scale tuning (1.0 = no guidance, 7.5+ = strong alignment), and prompt weighting via syntax like '(concept:weight)'. The system handles tokenization, embedding truncation, and multi-prompt composition automatically.

Implements StableDiffusionImg2ImgPipeline and StableDiffusionInpaintPipeline for controlled image editing by encoding reference images into VAE latent space, adding noise, and denoising with text guidance. Image-to-image uses strength parameter (0.0 = no change, 1.0 = full regeneration) to control how much the output deviates from the input. Inpainting uses a mask to selectively edit regions while preserving masked-out areas. Both pipelines support LoRA, ControlNet, and IP-Adapter conditioning for fine-grained control.

Implements StableDiffusionXLPipeline as a two-stage generation system using a base model (768x768) for initial generation and an optional refiner model for detail enhancement. The base model generates latents with reduced steps (e.g., 30), then the refiner model denoises the same latents with additional steps (e.g., 20) to add fine details and improve quality. The system supports high-resolution output (1024x1024+) by using larger latent dimensions and enables skipping the refiner stage for faster inference. Both stages support text and image conditioning (LoRA, ControlNet, IP-Adapter).

Implements FluxPipeline and StableDiffusion3Pipeline to support transformer-based diffusion models (Flux, Stable Diffusion 3) that replace the UNet with Transformer blocks (DiT architecture). These models use different attention mechanisms (multi-head attention, RoPE positional encoding) and require separate schedulers optimized for transformer inference. The system automatically detects model architecture and selects the appropriate pipeline, supporting the same conditioning mechanisms (text, ControlNet, IP-Adapter) as UNet-based models but with different computational characteristics.

Implements video generation pipelines (e.g., AnimateDiffPipeline, VideoToVideoPipeline) that extend image diffusion to temporal sequences by adding temporal attention layers or using frame-by-frame generation with optical flow-based consistency. The system supports both latent-space video generation (encoding full video into VAE latents, then denoising temporally) and frame-by-frame approaches (generating frames sequentially with consistency constraints). Video pipelines support motion control via motion embeddings and enable frame interpolation for smooth transitions.