MotionDirector

Adapts pre-trained text-to-video diffusion models using Low-Rank Adaptation (LoRA) applied selectively to temporal layers to extract and encode specific motion patterns from reference video clips. The system decomposes the adaptation into spatial (appearance) and temporal (motion) paths, allowing independent training of motion concepts without full model fine-tuning. This approach reduces trainable parameters by orders of magnitude while preserving the base model's text-to-video generation capabilities.

Trains a single LoRA adapter from multiple reference videos depicting the same motion concept (e.g., different subjects performing the same sport), extracting the motion pattern that generalizes across subjects and appearances. The training process uses a shared temporal LoRA module that learns motion invariant to spatial variations, enabling the learned motion to transfer to new subjects and scenes specified via text prompts.

Generates multiple videos in sequence with different text prompts, LoRA scales, or random seeds, enabling systematic exploration of the motion-text-seed space. The system manages GPU memory and inference scheduling to process batches efficiently, with configurable output organization (one video per prompt, per scale, per seed combination) and optional result aggregation for comparison.

Provides a unified interface for training and inference across different pre-trained text-to-video models (ZeroScope, ModelScopeT2V) by abstracting model-specific details (architecture, tokenizer, latent dimensions) behind a common API. The system automatically detects the base model type from configuration and loads appropriate model weights, adapters, and preprocessing pipelines, enabling seamless switching between models without code changes.

Ensures reproducible training by managing random seeds across PyTorch, NumPy, and CUDA, logging all hyperparameters and training metrics to files, and saving model checkpoints at regular intervals. The system records training loss, validation metrics, and LoRA weight statistics to enable analysis of training dynamics and recovery from interrupted training sessions.

Learns camera movement and cinematic techniques (dolly zoom, orbit shots, follow shots) from a single reference video by training LoRA on temporal layers to capture the specific camera trajectory and framing dynamics. The system preserves the spatial content of the reference while extracting pure motion information, enabling the learned camera movement to be applied to new scenes and subjects via text prompts.

Animates static images by combining a learned motion LoRA with a spatial appearance LoRA, enabling the system to apply motion patterns to new subjects while preserving their appearance. The inference pipeline injects both LoRA adapters into the diffusion model, with the spatial path controlling appearance and temporal path controlling motion dynamics, allowing seamless composition of appearance and motion from different sources.

Combines multiple spatial LoRAs (for different character appearances) with a single temporal LoRA (for motion) to generate videos of specific characters performing learned motions. The system allows mixing appearance from one training set with motion from another, enabling fine-grained control over both subject identity and action dynamics through separate text prompts and LoRA weight combinations.

Provides unified dataset interfaces (MultiVideoDataset, SingleVideoDataset, ImageDataset) that handle diverse input types (multiple videos, single video, images) with automatic preprocessing including frame extraction, resizing, normalization, and temporal sampling. The system abstracts dataset heterogeneity through a common DataLoader interface, enabling seamless switching between training modes (multi-video motion, single-video cinematic, image animation) without code changes.

Centralizes all training and inference hyperparameters (LoRA rank, learning rate, batch size, video paths, motion labels) in YAML configuration files, enabling reproducible experiments and easy parameter sweeping without code modification. The system parses YAML configs into Python objects that are passed through the training and inference pipelines, supporting environment variable substitution and config inheritance for managing complex experimental setups.

Implements gradient checkpointing, mixed-precision training (FP16), and selective LoRA freezing to reduce GPU memory footprint during training, enabling MotionDirector to run on GPUs with 24GB VRAM instead of requiring 40GB+. The system automatically applies memory optimizations based on available GPU memory, with configurable trade-offs between memory usage and training speed (e.g., gradient checkpointing adds ~20% training time overhead).

Generates videos by combining a text prompt with a trained motion LoRA, using the base text-to-video diffusion model's text encoder to condition generation on semantic descriptions while the temporal LoRA injects learned motion patterns. The inference pipeline performs iterative denoising in the latent space, with cross-attention layers modulated by both text embeddings and motion LoRA weights to produce coherent videos matching both the text description and learned motion.

Provides configurable LoRA weight scaling during inference to control the strength of learned motion effects, enabling users to blend between the base model's default motion and the learned motion by adjusting a single parameter (typically 0.0 to 1.0 scale factor). The system applies the scaled LoRA weights to attention layers during the diffusion process, allowing fine-grained control over motion intensity without retraining.