HunyuanVideo-1.5

Generates videos from natural language text prompts using a Diffusion Transformer (DiT) architecture with 8.3B parameters. The system encodes text via CLIP-style embeddings, processes them through a two-stage transformer block design (MMDoubleStreamBlock for parallel text-visual processing, MMSingleStreamBlock for unified fusion), and iteratively denoises latent video representations via diffusion steps. Outputs are decoded from 3D causal VAE latent space (16× spatial, 4× temporal compression) to pixel-space video frames at native 480p/720p resolutions.

Animates static images by encoding them via a vision encoder (CLIP ViT), concatenating with text prompt embeddings, and processing through the same DiT architecture to synthesize plausible motion and scene evolution. The 3D causal VAE ensures temporal coherence by maintaining causal dependencies across frames, preventing temporal artifacts. The system preserves image content fidelity while generating smooth, physically-plausible motion conditioned on the text instruction.

Integrates HunyuanVideo-1.5 into the Hugging Face Diffusers library, providing a standardized StableDiffusionPipeline-like interface. Users can load the model via `diffusers.AutoPipelineForText2Video.from_pretrained()`, call the pipeline with text prompts, and access standard features like scheduler selection, safety checkers, and callback hooks. This integration enables seamless composition with other Diffusers components and community tools.

Provides ComfyUI nodes that wrap HunyuanVideo-1.5 pipelines, enabling visual node-based workflow construction. Users can build complex generation pipelines by connecting nodes for text encoding, video generation, super-resolution, and post-processing. The integration includes custom nodes for prompt engineering, seed management, and parameter sweeping, allowing non-technical users to create sophisticated workflows.

Offers an optional prompt rewriting service that transforms user-provided text prompts into optimized prompts that better align with the model's training data and capabilities. The service uses heuristics or a separate language model to expand vague descriptions, add visual details, and correct common phrasing issues. Rewritten prompts typically produce higher-quality videos with better adherence to user intent.

Provides a comprehensive CLI tool (`hyvideo generate`) that accepts text prompts, image inputs, and configuration parameters, enabling batch video generation and integration into shell scripts or CI/CD pipelines. The CLI supports reading prompts from files, saving outputs to specified directories, and logging generation metadata. Configuration can be specified via command-line arguments or YAML files, enabling reproducible generation workflows.

Implements activation checkpointing (gradient checkpointing) to reduce peak memory usage during inference by recomputing activations instead of storing them. Additionally, the system uses key-value (KV) caching in attention layers to avoid recomputing attention outputs for unchanged tokens, reducing memory and computation. These techniques are applied selectively to balance memory savings vs. inference speed.

Generates videos natively at 480p (848×480) or 720p (1280×720) resolutions by configuring the transformer's latent space dimensions and VAE decoder output size. The 3D causal VAE's 16× spatial compression means 480p input maps to ~53×30 latent tokens, enabling efficient diffusion without excessive memory. Resolution selection is a configuration parameter passed to the pipeline class, allowing runtime switching without model reloading.

A separate HunyuanVideo_1_5_SR_Pipeline class upscales generated videos from 480p or 720p to 1080p using a specialized diffusion transformer trained on super-resolution tasks. The pipeline takes the low-resolution video latents from the main generation pipeline, encodes them via the SR VAE, and applies a diffusion-based refinement process conditioned on the original text prompt. This two-stage approach avoids the computational cost of native 1080p generation while maintaining quality.

Implements classifier-free guidance (CFG) to strengthen prompt adherence by computing unconditional and conditional diffusion predictions, then interpolating with a guidance scale. The system includes CFG distillation, a technique that trains a smaller model to approximate the CFG computation, reducing the number of forward passes required during inference. This allows trading off some quality for 30-50% faster generation without retraining the base model.

Trains a distilled model to predict multi-step diffusion trajectories in a single forward pass, reducing the number of sampling steps from 50-100 to 4-8 while maintaining quality. The distillation process uses knowledge distillation from the full model, training the student to match the teacher's output distribution across multiple timesteps. This is applied post-training and requires no changes to the base model architecture.

Implements sparse attention variants (e.g., local attention, strided attention) in the transformer blocks to reduce the quadratic memory complexity of full self-attention. The system allows swapping attention mechanisms via configuration without changing the core model, enabling trade-offs between memory usage and quality. Sparse attention is particularly effective for longer videos (100+ frames) where full attention becomes prohibitive.

A variational autoencoder with 3D convolutions and causal masking ensures temporal coherence by preventing future frames from influencing past frames during encoding. The VAE achieves 16× spatial compression and 4× temporal compression, mapping 480p video to ~53×30×8 latent tokens. Causality is enforced via causal padding in temporal convolutions, ensuring the latent representation respects temporal ordering and enabling efficient diffusion in latent space.

Implements Low-Rank Adaptation (LoRA) to fine-tune the transformer and text encoder with minimal additional parameters (~1-5% of base model size). LoRA decomposes weight updates as low-rank matrices, enabling efficient adaptation to custom styles, objects, or concepts without full model retraining. Fine-tuned LoRA weights can be merged or kept separate, allowing easy switching between styles or concepts at inference time.

Implements distributed training across multiple GPUs using PyTorch DistributedDataParallel (DDP) with gradient accumulation and mixed precision (AMP). The Muon optimizer is used instead of Adam, providing better convergence properties and lower memory overhead for large models. Training pipeline includes data loading, loss computation, gradient synchronization, and checkpoint management across distributed workers.