Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning (CM3Leon) vs PostHog

CM3Leon implements a decoder-only, token-based multimodal architecture that unifies text and image modalities into a single autoregressive sequence. The model uses a retrieval-augmented approach during pretraining where both text and image tokens are processed through the same transformer decoder, enabling bidirectional generation (text→image and image→text) without separate encoder-decoder branches. This is achieved by tokenizing images into discrete tokens and treating them identically to text tokens in the autoregressive sequence, allowing the model to learn cross-modal dependencies through standard language modeling objectives.

Unique: Uses a single decoder-only transformer with unified token representation for both modalities rather than separate vision encoders and text decoders, eliminating the need for cross-modal fusion layers and enabling true bidirectional generation through standard autoregressive training

vs alternatives: More parameter-efficient than encoder-decoder multimodal models (CLIP, BLIP) because it eliminates separate vision encoders; achieves 5x better training efficiency than comparable text-to-image methods while maintaining competitive zero-shot quality

CM3Leon's pretraining stage incorporates retrieval augmentation where relevant text-image pairs are retrieved and concatenated into the training sequences. During pretraining, the model learns to predict both text and image tokens in context of retrieved examples, enabling the model to leverage external knowledge without explicit fine-tuning. The retrieval mechanism operates at the sequence level, pulling related examples from a large corpus and interleaving them with the primary sequence, allowing the autoregressive model to learn in-context patterns and improve generalization through exposure to diverse multimodal contexts.

Unique: Integrates retrieval augmentation directly into the pretraining loop rather than as a post-hoc inference technique, allowing the model to learn retrieval-aware representations during training and achieve 5x training efficiency gains compared to non-retrieval baselines

vs alternatives: More efficient than scaling model size alone because retrieval provides external knowledge without parameter growth; outperforms standard pretraining by exposing the model to diverse in-context examples during training rather than only at inference

CM3Leon frames semantic segmentation as a token prediction task within the unified decoder, enabling the model to generate segmentation masks by predicting special segmentation tokens conditioned on image input. During multi-task SFT, the model learns to output segmentation tokens that correspond to semantic classes, converting the segmentation task into sequence prediction. This approach integrates segmentation into the multimodal model without separate segmentation heads or decoders.

Unique: Frames semantic segmentation as token prediction within the unified decoder, enabling segmentation without separate segmentation heads or architectures, though at potential cost of resolution compared to specialized models

vs alternatives: More parameter-efficient than maintaining separate segmentation models; unified architecture enables knowledge transfer from other multimodal tasks, though likely trades off segmentation quality for architectural simplicity

CM3Leon supports image infilling where partial images with missing regions are completed based on surrounding context and optional text descriptions. The model conditions on the visible image tokens and text instructions, predicting tokens for the masked regions autoregressively. This capability is learned during multi-task SFT and enables tasks like object removal, hole filling, and content-aware completion without requiring explicit mask inputs or separate inpainting models.

Unique: Performs image infilling within the unified decoder by conditioning on visible image tokens and text, enabling context-aware completion without separate inpainting models or explicit mask processing

vs alternatives: More flexible than traditional inpainting because it supports optional text guidance; more efficient than ensemble approaches because it uses a single model for multiple completion strategies

CM3Leon's multi-task SFT stage trains the model on diverse downstream tasks (text-to-image, image-to-text, infilling, editing, segmentation) using instruction-tuning approaches where each task is framed as following natural language instructions. This enables the model to learn task-specific behaviors while maintaining a unified architecture, allowing a single model to handle multiple vision and language tasks. The instruction tuning approach enables the model to generalize to new tasks and instructions not seen during training.

Unique: Applies instruction tuning to diverse vision and language tasks within a single unified decoder, enabling flexible task specification through natural language while maintaining a consolidated model architecture

vs alternatives: More flexible than task-specific models because instructions enable dynamic task specification; more parameter-efficient than maintaining separate models for each task, though with potential performance trade-offs

After retrieval-augmented pretraining, CM3Leon undergoes multi-task supervised fine-tuning (SFT) on diverse downstream tasks including text-to-image generation, image infilling, language-guided image editing, image-controlled generation, and segmentation. The SFT stage uses task-specific training data where each task is framed as a sequence prediction problem, allowing the unified decoder to learn task-specific behaviors while maintaining the shared multimodal representation. Contrastive decoding methods are applied during this stage to improve generation quality by contrasting high-quality and lower-quality outputs.

Unique: Frames diverse vision tasks (generation, editing, segmentation, infilling) as unified token prediction problems within a single decoder, using contrastive decoding to improve quality without task-specific auxiliary models or separate decoders

vs alternatives: More parameter-efficient than maintaining separate specialized models for each task; contrastive decoding improves quality without requiring additional discriminator networks or separate quality models like DALL-E 3's approach

CM3Leon implements a self-contained contrastive decoding method that improves generation quality by contrasting predictions from the model with a reference distribution during inference. Rather than requiring a separate quality model or discriminator, the method operates within the single multimodal decoder by sampling multiple candidate sequences and selecting or reranking them based on contrastive objectives. This approach is integrated into the SFT stage and applied during inference to improve both image and text generation without architectural modifications.

Unique: Implements contrastive decoding as a self-contained inference-time method within the single decoder rather than requiring separate quality models or ensemble approaches, enabling quality improvements without architectural overhead

vs alternatives: Lighter-weight than ensemble-based quality improvement (e.g., DALL-E 3's approach) because it reuses the same model for candidate generation and selection; more practical than training separate discriminators or quality models

CM3Leon achieves zero-shot image generation capability (without task-specific fine-tuning) through its retrieval-augmented pretraining and unified multimodal architecture. The model generates images directly from text prompts by predicting image tokens autoregressively, achieving MS-COCO FID score of 4.88 without any COCO-specific training. This zero-shot capability emerges from the large-scale pretraining on diverse text-image pairs and the model's ability to leverage retrieved examples during inference, enabling competitive performance on standard benchmarks without task-specific adaptation.

Unique: Achieves competitive zero-shot image generation (FID 4.88) through unified autoregressive architecture with retrieval augmentation, rather than specialized diffusion models or task-specific fine-tuning, demonstrating that token-based approaches can match diffusion-based quality

vs alternatives: More parameter-efficient than maintaining separate specialized text-to-image models; retrieval augmentation enables zero-shot performance without COCO-specific training, whereas most competing models require task-specific fine-tuning

PostHog/posthog | DeepWiki Loading... Index your code with Devin DeepWiki DeepWiki PostHog/posthog Index your code with Devin Edit Wiki Share Loading... Last indexed: 28 May 2026 ( 4a5e38 ) Overview Monorepo Structure and Build System Frontend Workspace and Product Packages Python Dependencies and Configuration CI/CD Pipeline Schema and Type System Cross-Language Schema Synchronization Query Schema Definitions Database Migrations Data Storage and Ingestion ClickHouse Architecture Kafka to ClickHouse Pipeline PostgreSQL and Database Pools Query Log Archive System Event Ingestion Pipeline (Node.js) Backend Services Django Middleware System Feature Flags Service (Rust) API Layer and Authentication Rust Microservices LLM Gateway Service Agentic Provisioning and OAuth Max AI Assistant Architecture and Agent Modes Query Execution and Streaming Frontend Integration MCP Server Tasks (AI Coding Agent) Feature Flags System Feature Flag Management API Flag Evaluation and Dependencies Frontend Interface Product Features Logs Viewer Session Recordings Insights and Analytics Surveys and Scheduled Changes Experiments (A/B Testing) Web Analytics Error Tracking LLM Analytics Frontend Architecture Kea State Management Product Module System Build System and Tooling Testing and Quality Test Infrastructure Backend and Rust Tests Frontend and E2E Tests Data Platform and Workf

Monorepo Structure and Build System | PostHog/posthog | DeepWiki Loading... Index your code with Devin DeepWiki DeepWiki PostHog/posthog Index your code with Devin Edit Wiki Share Loading... Last indexed: 28 May 2026 ( 4a5e38 ) Overview Monorepo Structure and Build System Frontend Workspace and Product Packages Python Dependencies and Configuration CI/CD Pipeline Schema and Type System Cross-Language Schema Synchronization Query Schema Definitions Database Migrations Data Storage and Ingestion ClickHouse Architecture Kafka to ClickHouse Pipeline PostgreSQL and Database Pools Query Log Archive System Event Ingestion Pipeline (Node.js) Backend Services Django Middleware System Feature Flags Service (Rust) API Layer and Authentication Rust Microservices LLM Gateway Service Agentic Provisioning and OAuth Max AI Assistant Architecture and Agent Modes Query Execution and Streaming Frontend Integration MCP Server Tasks (AI Coding Agent) Feature Flags System Feature Flag Management API Flag Evaluation and Dependencies Frontend Interface Product Features Logs Viewer Session Recordings Insights and Analytics Surveys and Scheduled Changes Experiments (A/B Testing) Web Analytics Error Tracking LLM Analytics Frontend Architecture Kea State Management Product Module System Build System and Tooling Testing and Quality Test Infrastructure Backend and Rust Tests Frontend a

Schema and Type System | PostHog/posthog | DeepWiki Loading... Index your code with Devin DeepWiki DeepWiki PostHog/posthog Index your code with Devin Edit Wiki Share Loading... Last indexed: 28 May 2026 ( 4a5e38 ) Overview Monorepo Structure and Build System Frontend Workspace and Product Packages Python Dependencies and Configuration CI/CD Pipeline Schema and Type System Cross-Language Schema Synchronization Query Schema Definitions Database Migrations Data Storage and Ingestion ClickHouse Architecture Kafka to ClickHouse Pipeline PostgreSQL and Database Pools Query Log Archive System Event Ingestion Pipeline (Node.js) Backend Services Django Middleware System Feature Flags Service (Rust) API Layer and Authentication Rust Microservices LLM Gateway Service Agentic Provisioning and OAuth Max AI Assistant Architecture and Agent Modes Query Execution and Streaming Frontend Integration MCP Server Tasks (AI Coding Agent) Feature Flags System Feature Flag Management API Flag Evaluation and Dependencies Frontend Interface Product Features Logs Viewer Session Recordings Insights and Analytics Surveys and Scheduled Changes Experiments (A/B Testing) Web Analytics Error Tracking LLM Analytics Frontend Architecture Kea State Management Product Module System Build System and Tooling Testing and Quality Test Infrastructure Backend and Rust Tests Frontend and E2E Tests

PostHog/posthog | DeepWiki Loading... Index your code with Devin DeepWiki DeepWiki PostHog/posthog Index your code with Devin Edit Wiki Share Loading... Last indexed: 28 May 2026 ( 4a5e38 ) Overview Monorepo Structure and Build System Frontend Workspace and Product Packages Python Dependencies and Configuration CI/CD Pipeline Schema and Type System Cross-Language Schema Synchronization Query Schema Definitions Database Migrations Data Storage and Ingestion ClickHouse Architecture Kafka to ClickHouse Pipeline PostgreSQL and Database Pools Query Log Archive System Event Ingestion Pipeline (Node.js) Backend Services Django Middleware System Feature Flags Service (Rust) API Layer and Authentication Rust Microservices LLM Gateway Service Agentic Provisioning and OAuth Max AI Assistant Architecture and Agent Modes Query Execution and Streaming Frontend Integration MCP Server Tasks (AI Coding Agent) Feature Flags System Feature Flag Management API Flag Evaluation and Dependencies Frontend Interface Product Features Logs Viewer Session Recordings Insights and Analytics Surveys and Scheduled Ch