Transformers

Provides AutoModel, AutoTokenizer, AutoImageProcessor, and AutoProcessor classes that automatically detect model architecture and framework (PyTorch/TensorFlow/JAX) from a model identifier, then instantiate the correct class without explicit architecture specification. Uses a registry-based discovery pattern where model_type metadata in config.json maps to concrete model classes, enabling single-line model loading across 1000+ architectures and eliminating framework-specific boilerplate.

Provides PreTrainedTokenizer and PreTrainedTokenizerFast classes that handle text-to-token conversion with support for subword tokenization (BPE, WordPiece, SentencePiece), special tokens, and padding/truncation strategies. Fast tokenizers are backed by the Rust-based tokenizers library for 10-100x speedup over pure Python implementations, while maintaining API compatibility. Automatically handles vocabulary loading, token type IDs, attention masks, and position IDs in a single encode() call.

Provides distributed training support via Trainer class integration with accelerate library, handling multi-GPU (DDP), multi-node, TPU, and mixed precision training automatically. Supports gradient accumulation to simulate larger batch sizes on limited memory, automatic mixed precision (AMP) with float16/bfloat16, and gradient checkpointing to trade compute for memory. Automatically synchronizes gradients across devices and handles loss scaling for numerical stability in mixed precision.

Provides utilities to inspect model architecture (layer names, parameter counts, shapes) and extract intermediate layer outputs (hidden states, attention weights) for analysis or downstream tasks. Supports registering forward hooks to capture activations from specific layers without modifying model code. Enables feature extraction by freezing early layers and training only later layers, useful for transfer learning and representation learning.

Provides seamless integration with Hugging Face Hub for downloading and caching pretrained models, tokenizers, and datasets. Automatically manages model versioning via git-based revision system (branches, tags, commits), enabling reproducible model loading. Supports remote code execution to load custom modeling code from Hub repositories without local installation. Caches downloaded files locally to avoid re-downloading, with configurable cache directory and automatic cleanup.

Provides implementations of multiple attention mechanisms (standard scaled dot-product, multi-head, grouped-query, multi-query) and positional embedding strategies (absolute, relative, rotary, ALiBi) that can be selected per model. Supports efficient attention implementations (FlashAttention, memory-efficient attention) that reduce memory usage and latency. Allows swapping attention mechanisms without retraining by modifying model config.

Provides implementations of Mixture-of-Experts layers where each token is routed to a subset of expert networks based on learned routing weights, enabling sparse computation and scaling to very large models. Supports load balancing to ensure experts are used evenly, and auxiliary loss to prevent router collapse. Enables training models with 1000s of experts without proportional increase in compute per token.

Provides Whisper model for automatic speech recognition (ASR) that supports 99 languages with a single model, and audio feature extraction utilities (MFCC, mel-spectrogram, Wav2Vec2 features) for audio processing. Whisper is trained on 680k hours of multilingual audio and handles various audio qualities and accents robustly. Supports both PyTorch and TensorFlow inference, with optional quantization for faster inference.

Provides Vision Transformer (ViT) and CNN-based image classification models (ResNet, EfficientNet, DeiT) that can be fine-tuned on custom datasets or used for feature extraction. Supports image preprocessing (resizing, normalization) via ImageProcessor, and automatic model selection via AutoModel. Enables transfer learning by freezing early layers and training only later layers, reducing training time and data requirements.

Provides encoder-decoder architectures (BART, T5, mBART, mT5) for sequence-to-sequence tasks like machine translation, summarization, and question answering. Encoder processes input sequence and produces context, decoder generates output sequence token-by-token using beam search or other decoding strategies. Supports cross-attention between encoder and decoder outputs, and shared vocabulary between encoder and decoder.

Provides high-level pipeline() function that wraps model + tokenizer/processor + postprocessing into a single callable interface for 20+ NLP/vision/audio tasks (text-classification, token-classification, question-answering, image-classification, object-detection, speech-recognition, etc.). Pipelines automatically handle input validation, preprocessing (tokenization/image resizing), model inference, and output formatting without exposing model internals. Supports batching, device management, and framework selection transparently.

Provides Trainer class that abstracts the training loop for PyTorch/TensorFlow/JAX, handling gradient accumulation, mixed precision, distributed training (DDP, DeepSpeed, FSDP), learning rate scheduling, checkpoint management, and evaluation. Trainer accepts TrainingArguments config object that specifies hyperparameters, and automatically manages device placement, gradient synchronization, and loss scaling. Supports custom callbacks for logging, early stopping, and metric computation without modifying core training code.

Provides generate() method on language models that supports multiple decoding strategies (greedy, beam search, nucleus sampling, contrastive search, assisted decoding) with configurable stopping criteria, logits processors, and token selection. Implements KV cache (key-value cache) to avoid recomputing attention for previously generated tokens, reducing inference latency by 5-10x. Supports speculative decoding (draft model + verification) and continuous batching for serving multiple sequences with different lengths efficiently.

Provides quantization utilities for reducing model size and inference latency by converting weights from float32 to lower precision (int8, int4, float16, bfloat16). Supports multiple quantization methods: post-training quantization (PTQ) via bitsandbytes, quantization-aware training (QAT), and dynamic quantization. Integrates with GPTQ and AWQ quantization schemes for LLMs. Automatically handles quantization during model loading without explicit conversion code, and supports inference on quantized models with minimal accuracy loss.

Provides AutoProcessor and task-specific processors (ImageProcessor, AudioProcessor, VideoProcessor) that handle preprocessing for multi-modal models (vision-language, audio-language, video-language). Processors combine tokenization, image resizing, audio feature extraction, and normalization into a single call, returning a dict with all required model inputs (pixel_values, input_ids, attention_mask, etc.). Supports batch processing with automatic padding/truncation for heterogeneous input sizes.

Provides utilities for converting model weights between PyTorch, TensorFlow, JAX, and ONNX formats, enabling inference on different frameworks without retraining. Includes conversion scripts for specific architectures (e.g., convert_pytorch_checkpoint_to_tf2.py) that handle weight name mapping, shape transformations, and framework-specific quirks. Supports exporting models to ONNX for hardware acceleration and mobile deployment. Automatically validates converted weights by comparing outputs between source and target frameworks.

Integrates with PEFT (Parameter-Efficient Fine-Tuning) library to enable LoRA, prefix tuning, and adapter-based fine-tuning that trains only 0.1-1% of model parameters instead of full fine-tuning. Automatically wraps model layers with adapter modules during loading, reducing memory usage and training time by 10-100x. Supports merging adapters back into base model weights for inference without additional overhead.

Provides chat template system that automatically formats multi-turn conversations into the correct prompt format for instruction-tuned models (ChatGPT, Llama 2 Chat, Mistral, etc.). Each model has a jinja2 template that specifies how to format system messages, user messages, and assistant responses. Handles special tokens (e.g., BOS, EOS) and role markers automatically, eliminating manual prompt engineering. Supports streaming responses by yielding tokens as they are generated.