Octo

Loads a pretrained OctoModel trained on 800K diverse robot trajectories from Open X-Embodiment dataset and performs action prediction by processing multimodal inputs (camera observations, proprioception, language instructions or goal images) through a causal transformer backbone followed by action head decoding. The model uses tokenized representations of observations and task specifications, processes them through the OctoTransformer's attention layers, and outputs continuous action distributions via diffusion or L1 action heads.

Adapts pretrained Octo models to new robot morphologies and sensor configurations through parameter-efficient fine-tuning that reuses the transformer backbone while replacing or retraining tokenizers and action heads. The system supports selective layer freezing, custom observation/action tokenizer training, and task-specific data augmentation, enabling adaptation with 10-100x less data than training from scratch.

Enables deployment of Octo policies to physical robots through standardized control loops that execute actions, collect observations, and monitor performance in real-time. Supports multiple control modes (open-loop trajectory execution, closed-loop feedback control, receding horizon control) and provides hooks for safety monitoring, action filtering, and emergency stops.

Provides extensible callback system for monitoring training progress, logging metrics, and triggering actions during training (e.g., checkpointing, evaluation, learning rate scheduling). Callbacks integrate with standard logging frameworks (Weights & Biases, TensorBoard) and support custom metrics computation (action prediction accuracy, trajectory success rates in simulation).

Computes quantitative metrics for policy evaluation (action prediction accuracy, trajectory success rates, action smoothness, task completion time) and provides visualization tools (trajectory playback, attention weight visualization, action distribution plots). Metrics are computed on validation datasets or in simulation, enabling quantitative comparison of policies and identification of failure modes.

Converts heterogeneous robot sensor inputs (RGB/grayscale images from multiple cameras, proprioceptive state vectors, depth maps) into fixed-size token sequences using modular tokenizer components (image tokenizers via learned codebooks or pretrained vision models, proprioception tokenizers via linear projections or MLPs). Tokenizers are composed in a pipeline that handles variable numbers of cameras and sensor modalities, enabling the transformer to process observations in a unified sequence format.

Encodes task specifications (natural language instructions or goal images) into token sequences using task-specific tokenizers (language tokenizers via pretrained text models like BERT, goal image tokenizers via vision models). These task tokens are concatenated with observation tokens in the transformer input sequence, enabling the model to condition action prediction on either linguistic task descriptions or visual goal states without architectural changes.

Processes tokenized observation and task sequences through a causal transformer architecture (OctoTransformer) that applies masked self-attention to prevent attending to future tokens, enabling autoregressive action prediction. The transformer uses standard components (multi-head attention, feedforward layers, layer normalization) with causal masking to ensure actions depend only on past and current observations, not future information.

Decodes transformer hidden states into robot actions using pluggable action heads that support diffusion-based action prediction (iterative denoising of action distributions) or L1 regression (direct action prediction). Diffusion heads enable multi-modal action distributions and uncertainty quantification, while L1 heads provide deterministic, low-latency action prediction. Both heads are trained jointly with the transformer backbone.

Loads and preprocesses the Open X-Embodiment dataset (800K robot trajectories across 22+ platforms) through a standardized data pipeline that handles heterogeneous data formats (HDF5, TFRecord, RLDS), performs observation normalization, action space conversion, and trajectory filtering. The data system supports lazy loading and on-the-fly augmentation to handle the dataset's scale and diversity.

Applies configurable transformations to training data including observation normalization, action space conversion, image augmentation (resizing, cropping, color jittering), and task augmentation (language paraphrasing, goal image transformations). Transformations are composed in a pipeline that can be applied during data loading or training, enabling efficient on-the-fly augmentation without storing augmented data.

Provides standardized gym-compatible wrappers (NormalizeProprio, HistoryWrapper, RHCWrapper) that interface Octo policies with robot environments and simulators. Wrappers handle observation normalization, history buffering for temporal context, and receding horizon control (RHC) for closed-loop execution. This abstraction enables the same policy code to work across different robot platforms and simulators.

Integrates with simulation environments (MuJoCo, PyBullet, IsaacGym) through gym-compatible wrappers, enabling policy evaluation in simulation before deployment to physical robots. Supports rendering, trajectory logging, and metrics collection (success rates, trajectory lengths, action smoothness) for quantitative policy evaluation.