Spatiotemporal Attention With Cross Frame Relationships

via “spatiotemporal attention with cross-frame relationships”

Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch

Unique: Combines spatial and temporal attention in a unified module rather than applying them sequentially, enabling direct modeling of spatiotemporal relationships; integrates Flash Attention for kernel-fused computation reducing memory bandwidth bottlenecks

vs others: More memory-efficient than standard multi-head attention (40-50% reduction with Flash Attention) while capturing richer temporal dependencies than frame-independent spatial attention, enabling longer coherent video generation

via “temporal consistency modeling with frame-to-frame attention”

text-to-video model by undefined. 39,484 downloads.

Unique: Implements spatiotemporal attention blocks that jointly model spatial relationships (within-frame) and temporal relationships (across frames) in a single attention computation, rather than alternating between spatial and temporal attention. This unified approach enables more efficient and coherent temporal modeling compared to separate spatial/temporal attention streams.

vs others: Produces smoother, more coherent motion than frame-by-frame generation approaches (e.g., stacking image generation models), while remaining more efficient than full bidirectional temporal attention used in some research models.