PaliGemma

Extracts and recognizes text embedded in images using a SigLIP vision encoder that processes images at 224×224, 448×448, or 896×896 pixel resolutions, feeding visual features into a Gemma language decoder that generates character-level text output. The multi-resolution pipeline allows trade-offs between accuracy (higher resolution) and latency (lower resolution), with the vision encoder producing dense spatial features that preserve text layout and structure for downstream language modeling.

Answers natural language questions about image content by encoding the image through SigLIP to produce spatial feature maps, then conditioning a Gemma language model decoder on those features to generate free-form text answers. The architecture treats VQA as a sequence-to-sequence task where the vision encoder provides context and the language model generates answers token-by-token, allowing complex reasoning over visual content without explicit object detection or scene graph extraction.

Offers three parameter-count variants (3B, 10B, 28B) based on Gemma language model sizes, enabling deployment on hardware with different memory and compute constraints. The 3B variant is optimized for edge devices and latency-sensitive applications; the 10B variant balances capability and resource requirements; the 28B variant maximizes capability for high-resource environments. All variants share the same architecture and training approach, differing only in Gemma decoder size, allowing developers to select the appropriate trade-off for their deployment target.

Identifies objects in images and predicts their spatial locations by leveraging SigLIP's dense spatial feature maps (from 224×224 to 896×896 resolution) and using the Gemma decoder to generate structured or free-form descriptions of object positions. Rather than explicit bounding box regression, the model encodes spatial information implicitly through the vision encoder's feature resolution and the language model's ability to describe locations using natural language (e.g., 'top-left corner', 'center-right') or coordinate-like tokens.

Performs pixel-level classification to segment images into semantic regions by using SigLIP's dense spatial features as input to the Gemma decoder, which generates segmentation outputs either as natural language descriptions of regions or as structured token sequences representing pixel classes. The vision encoder's multi-resolution support (up to 896×896) preserves fine-grained spatial detail needed for accurate segmentation boundaries, while the language model can incorporate semantic context and reasoning about region relationships.

Generates natural language descriptions of image content and short video sequences by encoding visual frames through SigLIP and decoding with Gemma to produce fluent, contextually appropriate captions. For images, the model generates single captions; for short videos, it likely processes multiple frames and generates descriptions that capture temporal dynamics or key events. The language decoder produces captions token-by-token, allowing variable-length outputs and incorporation of visual context into natural language.

Enables customization of PaliGemma for specific visual understanding tasks by freezing or partially updating the SigLIP vision encoder and fine-tuning the Gemma language decoder (or both components) on task-specific datasets. The pre-trained vision encoder provides strong feature representations that transfer across tasks, reducing fine-tuning data requirements and training time. Three model variants support different fine-tuning strategies: PT (pre-trained, fully fine-tunable), FT (research-specific, task-locked), and mix (multi-task, ready-to-use).

Processes images at three supported resolutions (224×224, 448×448, 896×896 pixels) without retraining, allowing developers to dynamically select resolution based on accuracy requirements and latency constraints. Higher resolutions preserve fine-grained visual details (beneficial for OCR, small object detection) at the cost of increased inference time and memory; lower resolutions reduce latency and memory footprint at the cost of detail loss. The SigLIP vision encoder and Gemma decoder are resolution-agnostic, supporting this flexibility through positional encoding or patch-based processing.

Distributes PaliGemma model weights through open-source repositories (Kaggle, Hugging Face) in a format compatible with standard inference frameworks (PyTorch, JAX), enabling developers to download, run, and fine-tune models locally without cloud dependencies or API keys. The open-source architecture (SigLIP vision encoder + Gemma language model) provides full transparency into model design, training approach, and inference pipeline, supporting custom modifications and integration into proprietary systems.

Provides a 'mix' variant of PaliGemma that is pre-fine-tuned on multiple vision-language tasks (OCR, VQA, object detection, segmentation, captioning) and ready for immediate use without additional fine-tuning. This variant represents a middle ground between the general-purpose PT (pre-trained) variant and task-specific FT variants, offering reasonable performance across common applications while maintaining some customization capability. The multi-task training approach allows the model to leverage shared representations across tasks, potentially improving generalization.

Exposes the SigLIP vision encoder as a feature extractor that can be used independently of the Gemma language decoder, enabling integration into custom vision-language pipelines or non-language tasks. The encoder produces dense spatial feature maps at multiple resolutions that can be fed to custom classification heads, detection heads, or other downstream models. This modular approach allows developers to leverage pre-trained visual representations without committing to the full PaliGemma architecture or language generation paradigm.