Multi Task Visual Policy Learning With Task Agnostic World Models

via “multi-task dataset enabling transfer learning across detection, segmentation, captioning, and pose tasks”

330K images with object detection, segmentation, and captions.

Unique: Single dataset with annotations for 7+ vision tasks enables multi-task learning and transfer learning; shared image set allows models to learn task-agnostic visual representations and transfer knowledge across tasks

vs others: More comprehensive than single-task datasets; enables multi-task learning unlike separate datasets for each task; shared image set ensures fair comparison across tasks unlike different image distributions

via “cross-task knowledge transfer through shared representations”

Microsoft's unified model for diverse vision tasks.

Unique: Achieves knowledge transfer across 6+ vision tasks through a single unified seq2seq architecture, where shared visual encoding and decoder parameters enable cross-task learning without task-specific branches or ensemble methods

vs others: Outperforms task-specific models on low-data scenarios through knowledge transfer, though with 5-10% lower peak performance on high-data tasks compared to specialized models

via “multimodal-dataset-integration-for-vision-language-models”

108K images with dense scene graphs and 5.4M region descriptions.

Unique: Provides unified integration of 5 complementary annotation types (scene graphs, region descriptions, object instances, attributes, QA pairs) across 108K images, enabling multi-task learning from diverse supervision signals. Dataset structure supports joint optimization for detection, grounding, reasoning, and attribute prediction in a single training pipeline.

vs others: More comprehensive than single-task datasets (COCO, Flickr30K) and enables multi-task learning unlike datasets with isolated annotation types; supports training unified models that leverage complementary supervision signals

via “multi-task visual policy learning with task-agnostic world models”

* ⏫ 02/2023: [Grounding Large Language Models in Interactive Environments with Online RL (GLAM)](https://arxiv.org/abs/2302.02662)

Unique: DreamerV3's task-agnostic world model learns shared visual representations without explicit task conditioning, relying on the policy learning objective to extract task-relevant information from the shared latent space. This contrasts with task-conditioned approaches (e.g., MTRL baselines) that explicitly encode task identity, making DreamerV3 more flexible for discovering emergent task structure.

vs others: Achieves better sample efficiency and generalization than task-conditioned baselines by learning task-invariant visual dynamics, while avoiding the computational overhead of task-specific world models or explicit task embeddings.

via “multi-task adapter composition for vision-language understanding”

* ⭐ 04/2022: [Winoground: Probing Vision and Language Models for Visio-Linguistic... (Winoground)](https://arxiv.org/abs/2204.03162)

Unique: Implements task-specific adapter composition for multimodal models with explicit routing logic, enabling independent training of task adapters while maintaining shared backbone — distinct from single-task adapter approaches and multi-task learning methods that require joint training

vs others: More memory-efficient than training separate full models per task and more flexible than single-task adapters, enabling dynamic task switching without model reloading

via “robust terrain perception and adaptation through visual feature learning”

* ⭐ 02/2022: [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://proceedings.mlr.press/v164/jang22a.html)

Unique: Learns terrain understanding implicitly through end-to-end visuomotor training rather than using explicit terrain classifiers or segmentation networks. The approach allows the policy to discover task-relevant visual features without human annotation of terrain types, creating a unified perception-action system optimized for locomotion success.

vs others: More robust than hand-crafted terrain classifiers because learned features adapt to the specific locomotion task, and more efficient than separate perception and control pipelines by jointly optimizing visual features with motor control objectives.

via “multi-task vision model with shared representation”

* ⏫ 12/2023: [VideoPoet: A Large Language Model for Zero-Shot Video Generation (VideoPoet)](https://arxiv.org/abs/2312.14125)

Unique: Uses single encoder-decoder backbone with shared parameters across all vision tasks, trained on 5.4B diverse annotations to learn unified representation handling variable spatial hierarchies and semantic granularities. Contrasts with ensemble or task-specific approaches by consolidating capabilities into one model.

vs others: Reduces deployment complexity and memory footprint compared to maintaining separate detection (YOLO), segmentation (DeepLab), grounding (ALBEF), and captioning (BLIP) models, though individual task performance vs specialized baselines unknown.

via “multi-task robot policy learning from diverse demonstrations”

## Historical Papers <a name="history"></a>

Unique: Trains a single transformer model on 700+ diverse tasks without task-specific heads or explicit multi-task loss weighting, relying on language conditioning and shared token embeddings to learn task-agnostic manipulation primitives. This contrasts with prior multi-task approaches that use separate output heads or task-specific adapters.

vs others: Achieves better generalization to novel objects and scenes than task-specific policies trained on equivalent data, and scales more efficiently than ensemble or modular approaches by sharing all transformer parameters across tasks.