LTX-Video

Generates videos directly from natural language prompts using a Diffusion Transformer (DiT) architecture with a rectified flow scheduler. The system encodes text prompts through a language model, then iteratively denoises latent video representations in the causal video autoencoder's latent space, producing 30 FPS video at 1216×704 resolution. Uses spatiotemporal attention mechanisms to maintain temporal coherence across frames while respecting the causal structure of video generation.

Transforms static images into dynamic videos by conditioning the diffusion process on image embeddings at specified frame positions. The system encodes the input image through the causal video autoencoder, injects it as a conditioning signal at designated temporal positions (e.g., frame 0 for image-to-video), then generates surrounding frames while maintaining visual consistency with the conditioned image. Supports multiple conditioning frames at different temporal positions for keyframe-based animation control.

Implements classifier-free guidance (CFG) to improve prompt adherence and video quality by training the model to generate both conditioned and unconditional outputs. During inference, the system computes predictions for both conditioned and unconditional cases, then interpolates between them using a guidance scale parameter. Higher guidance scales increase adherence to conditioning signals (text, images) at the cost of reduced diversity and potential artifacts. The guidance scale can be dynamically adjusted per timestep, enabling stronger guidance early in generation (for structure) and weaker guidance later (for detail).

Provides a command-line inference interface (inference.py) that orchestrates the complete video generation pipeline with YAML-based configuration management. The script accepts model checkpoints, prompts, conditioning media, and generation parameters, then executes the appropriate pipeline (text-to-video, image-to-video, etc.) based on provided inputs. Configuration files specify model architecture, hyperparameters, and generation settings, enabling reproducible generation and easy model variant switching. The script handles device management, memory optimization, and output formatting automatically.

Converts video frames into patch tokens for transformer processing through VAE encoding followed by spatial patchification. The causal video autoencoder encodes video into latent space, then the latent representation is divided into non-overlapping patches (e.g., 16×16 spatial patches), flattened into tokens, and concatenated with temporal dimension. This patchification reduces sequence length by ~256x (16×16 spatial patches) while preserving spatial structure, enabling efficient transformer processing. Patches are then processed through the Transformer3D model, and the output is unpatchified and decoded back to video space.

Provides multiple model variants optimized for different hardware constraints through quantization and distillation. The ltxv-13b-0.9.7-dev-fp8 variant uses 8-bit floating point quantization to reduce model size by ~75% while maintaining quality. The ltxv-13b-0.9.7-distilled variant uses knowledge distillation to create a smaller, faster model suitable for rapid iteration. These variants are loaded through configuration files that specify quantization parameters, enabling easy switching between quality/speed trade-offs. Quantization is applied during model loading; no retraining required.

Extends existing video segments forward or backward in time by conditioning the diffusion process on video frames from the source clip. The system encodes video frames into the causal video autoencoder's latent space, specifies conditioning frame positions, then generates new frames before or after the conditioned segment. Uses the causal attention structure to ensure temporal consistency and prevent information leakage from future frames during backward extension.

Generates videos constrained by multiple conditioning frames at different temporal positions, enabling precise control over video structure and content. The system accepts multiple image or video segments as conditioning inputs, maps them to specified frame indices, then performs diffusion with all constraints active simultaneously. Uses a multi-condition attention mechanism to balance competing constraints and maintain coherence across the entire temporal span while respecting individual conditioning signals.

Transforms existing video content by conditioning generation on the source video while applying text-guided modifications. The system encodes the source video into latent space, uses it as a conditioning signal, then applies diffusion with a text prompt describing desired transformations (style changes, object modifications, scene alterations). The conditioning strength parameter controls the balance between preserving source content and applying text-guided changes, enabling style transfer, object replacement, or scene reinterpretation while maintaining temporal coherence.

Encodes and decodes videos using a causal video autoencoder (CausalVideoAutoencoder) that compresses video into a latent space while preserving temporal structure. The encoder uses 3D convolutions with causal masking to ensure frames only depend on past frames, reducing spatial resolution by 8x and temporal resolution by 4x while maintaining motion information. The decoder reconstructs video from latent representations with high fidelity. This compression enables efficient diffusion in latent space rather than pixel space, reducing memory requirements and generation time by orders of magnitude.

Implements a rectified flow scheduler that optimizes the diffusion process by mapping noise schedules to straight-line trajectories in latent space, enabling fewer denoising steps while maintaining quality. The scheduler computes optimal timestep sequences that minimize the path length through noise space, reducing the number of required inference steps from typical 50-100 down to 20-30 steps. Uses linear interpolation between noise and signal rather than exponential schedules, improving convergence speed and enabling real-time generation without quality degradation.

Implements a 3D transformer architecture (Transformer3D) that processes video as spatiotemporal tokens using causal attention mechanisms. The model applies self-attention across spatial dimensions (height, width) and temporal dimensions (frames) simultaneously, with causal masking preventing frames from attending to future frames. Uses grouped query attention and flash attention optimizations to reduce memory overhead and computation time. The architecture enables efficient processing of long video sequences while maintaining temporal coherence through causal constraints.

Implements LTXMultiScalePipeline for generating videos at higher resolutions through progressive multi-pass generation. The system first generates low-resolution video (e.g., 1216×704), then upscales and refines at progressively higher resolutions (e.g., 2432×1408, 4864×2816) using the same diffusion process with additional refinement steps. Each pass conditions on the previous resolution's output, enabling coherent upscaling while adding fine details. This approach avoids the memory and computation overhead of single-pass high-resolution generation.

Processes natural language prompts through semantic enhancement to improve video generation quality and coherence. The system tokenizes prompts, encodes them through a text encoder (typically CLIP or similar), and optionally applies prompt expansion or rewriting to clarify ambiguous descriptions. Enhanced prompts are converted to embeddings that condition the diffusion process. The text encoder's semantic understanding enables the model to interpret complex descriptions, temporal narratives, and stylistic directives, translating them into coherent video generation constraints.