Efficient Block Local Attention With Spatial Locality Bias

via “efficient block-local attention with spatial locality bias”

* ⭐ 04/2022: [Hierarchical Text-Conditional Image Generation with CLIP Latents (DALL-E 2)](https://arxiv.org/abs/2204.06125)

Unique: Uses learnable 2D relative position biases within fixed-size windows to encode spatial locality, enabling efficient local attention with explicit geometric inductive bias — distinct from absolute positional encodings and from attention without position bias

vs others: More efficient than full self-attention for high-resolution images while maintaining stronger spatial locality than global attention, and provides better inductive bias for vision tasks than position-free local attention