OPT

OPT implements a decoder-only transformer architecture trained with causal language modeling (predicting next tokens given previous context). The model uses standard transformer components including multi-head self-attention, feed-forward layers, and layer normalization, trained on 180B tokens of diverse text data. Unlike encoder-decoder models, it processes sequences unidirectionally, making it efficient for autoregressive text generation without requiring separate encoder preprocessing.

OPT provides a family of pre-trained models spanning 350M to 175B parameters, allowing developers to select variants optimized for specific latency, throughput, and accuracy requirements. Each variant uses identical architecture and training approach but with different layer counts and hidden dimensions, enabling direct performance comparisons and staged deployment strategies where smaller models handle high-volume requests and larger models handle complex queries.

OPT's open-source weights enable knowledge distillation where a smaller student model learns to mimic the larger teacher model's behavior. Developers can train smaller models (e.g., 125M parameters) to match 350M or 1.3B model outputs, reducing inference latency and memory requirements while preserving task performance. Distillation uses KL divergence loss between student and teacher logits, typically requiring 10-50% of the teacher's training data.

OPT's open-source architecture enables extraction and visualization of attention weights, allowing analysis of which tokens the model attends to when making predictions. Developers can extract attention heads from any layer, visualize attention patterns as heatmaps, and analyze how different heads specialize in different linguistic phenomena (syntax, semantics, discourse). This enables interpretability research and debugging of model behavior.

OPT supports efficient batch processing of variable-length sequences through padding and attention masking, allowing multiple prompts of different lengths to be processed simultaneously without wasting computation on padding tokens. The implementation uses standard PyTorch batching with causal attention masks that prevent tokens from attending to future positions, enabling both single-sample and batch inference with identical model behavior.

OPT can be fine-tuned on downstream tasks using standard supervised learning approaches (full fine-tuning, LoRA, prefix tuning) by loading pre-trained weights and training on task-specific datasets. The model exposes all parameters for gradient computation, enabling both full-model fine-tuning for high-resource teams and parameter-efficient methods (LoRA adds ~0.1% trainable parameters) for resource-constrained scenarios. Fine-tuning typically requires 1-10 epochs on task data with learning rates 1e-5 to 5e-5.

OPT can perform few-shot learning by including task examples in the prompt context, allowing the model to adapt to new tasks without parameter updates. The model uses in-context learning where examples are concatenated with the query, and the model's causal attention mechanism learns to recognize patterns from examples and apply them to the query. This approach works best with 1-8 examples and requires no training, making it suitable for rapid prototyping and zero-resource-cost adaptation.

OPT outputs logits for each token position, enabling calculation of per-token probabilities, confidence scores, and uncertainty estimates. The model's softmax-normalized logits reveal which tokens the model considers likely continuations, and the entropy of the probability distribution indicates model confidence. This enables applications like confidence-based filtering, uncertainty sampling for active learning, and detection of hallucinated or low-confidence generations.

OPT was trained on diverse internet text including non-English content, enabling generation in multiple languages though with English-dominant performance. The model uses a shared vocabulary across languages (50,272 BPE tokens) and can generate coherent text in Spanish, French, German, Chinese, and other languages, though quality degrades compared to English. The model shows code-switching behavior where it may mix languages in a single generation.

OPT can generate code snippets and understand programming languages due to training on diverse internet text including GitHub repositories and Stack Overflow. The model can complete code functions, generate SQL queries, write shell scripts, and explain code, though performance is lower than models specifically trained on code (Codex, CodeLLaMA). Code generation uses the same causal language modeling approach as text generation, with the model learning syntax and common patterns from training data.

OPT can generate factual text about topics covered in its training data (April 2021 cutoff), leveraging learned knowledge from pretraining. The model encodes world knowledge in its parameters through next-token prediction on diverse text, enabling generation of factually accurate text about historical events, scientific concepts, and common knowledge. However, the model has no mechanism to retrieve external knowledge or verify facts, leading to hallucinations and outdated information.

OPT supports context windows up to 2048 tokens, enabling generation that considers up to ~1500 tokens of input context (with ~500 tokens reserved for generation). The model uses standard causal attention where each token attends to all previous tokens, with quadratic complexity in sequence length. This enables multi-turn conversations, long document summarization, and context-aware generation, though latency increases quadratically with context length.