Frozen Vision Encoder Integration With Efficient Parameter Tuning

via “frozen-encoder visual feature extraction with querying transformer bridging”

Salesforce's efficient vision-language bridge model.

Unique: Uses learnable query tokens with cross-attention to frozen image features instead of direct feature projection or fine-tuning, enabling parameter-efficient bridging between any frozen vision encoder and any LLM without modifying either component's weights

vs others: More parameter-efficient than CLIP-based adapters (LoRA, prefix-tuning) because Q-Former learns task-specific visual abstractions rather than just adapting LLM layers, and more flexible than ALBEF because it doesn't require vision encoder fine-tuning

via “clip-vision-encoder-integration”

Open multimodal model for visual reasoning.

Unique: Uses frozen CLIP ViT-L/14 encoder with a simple learned projection matrix rather than fine-tuning the vision encoder, trading visual adaptability for training efficiency and stability; this design choice enables 1-day training on 8 A100s

vs others: Simpler and faster to train than models that fine-tune vision encoders (like BLIP-2 with ViT-G), but sacrifices domain-specific visual adaptation; ideal for general-purpose applications where CLIP's visual understanding is sufficient

via “transfer learning and domain-specific fine-tuning with frozen vision encoder”

image-to-text model by undefined. 5,97,442 downloads.

Unique: Enables parameter-efficient fine-tuning by freezing the ViT encoder (which contains ~86M parameters) and only updating Q-Former (~190M) and OPT decoder (~2.7B), reducing memory footprint and training time by ~40% compared to full model fine-tuning while maintaining strong performance on downstream tasks.

vs others: More efficient than fine-tuning full vision-language models like BLIP-2-OPT-6.7B; more flexible than fixed-feature extraction because the Q-Former and decoder can adapt to domain-specific patterns.

* ⭐ 05/2022: [A Generalist Agent (Gato)](https://arxiv.org/abs/2205.06175)

Unique: Freezes the entire vision encoder while training only fusion and language layers, reducing training parameters by ~90% compared to end-to-end fine-tuning — a design choice that trades off vision encoder adaptability for training efficiency and preservation of pre-trained visual knowledge

vs others: Achieves competitive few-shot performance with 10-20× fewer trainable parameters than models that fine-tune vision encoders, enabling training on consumer GPUs and reducing training time from weeks to days