Temporal Consistency Modeling With Frame To Frame Attention

via “spatiotemporal attention with cross-frame relationships”

Implementation of Make-A-Video, new SOTA text to video generator from Meta AI, in Pytorch

Unique: Combines spatial and temporal attention in a unified module rather than applying them sequentially, enabling direct modeling of spatiotemporal relationships; integrates Flash Attention for kernel-fused computation reducing memory bandwidth bottlenecks

vs others: More memory-efficient than standard multi-head attention (40-50% reduction with Flash Attention) while capturing richer temporal dependencies than frame-independent spatial attention, enabling longer coherent video generation