PEFT

Injects trainable low-rank decomposition matrices (A and B) into transformer attention and feed-forward layers, reducing trainable parameters from billions to millions while maintaining model capacity through rank-based factorization. Uses a registry-based dispatch mechanism (src/peft/mapping.py) to instantiate LoRA tuners that wrap base model layers, enabling selective parameter freezing and gradient computation only on adapter weights during backpropagation.

Enables fine-tuning of 4-bit and 8-bit quantized models by training adapters on top of frozen quantized weights, using bitsandbytes integration to handle quantized forward passes while computing gradients only through adapter parameters. The architecture freezes the quantized base model and routes gradients exclusively through LoRA layers, eliminating the need to dequantize weights during training.

Automatically detects model architecture and applies adapter-specific optimizations for popular model families (LLaMA, Mistral, GPT-2, BERT, ViT, etc.) through architecture-aware tuner selection. The integration layer (src/peft/mapping.py) maps model classes to appropriate tuner implementations, enabling seamless adapter injection without manual layer specification. Supports automatic target module detection for different model architectures, reducing configuration complexity.

Integrates with PyTorch's gradient checkpointing to reduce memory footprint during training by recomputing activations during backpropagation instead of storing them. Works seamlessly with adapter training by checkpointing the base model while maintaining gradient flow through adapter parameters. Reduces peak memory usage by 30-50% during training with minimal computational overhead (10-15% slower training).

Manages adapter lifecycle through add_adapter(), set_adapter(), delete_adapter(), and disable_adapter() methods, enabling programmatic control over which adapters are active during inference or training. The state management system maintains a registry of adapters and their activation status, enabling dynamic adapter switching without model reloading. Supports adapter enable/disable without deletion, allowing temporary deactivation and reactivation.

Enables training adapters in mixed precision (float16 or bfloat16) with automatic loss scaling to prevent gradient underflow, reducing memory usage by 50% and improving training speed by 1.5-2x. Integrates with PyTorch's automatic mixed precision (AMP) and transformers' native mixed-precision support to maintain numerical stability while reducing precision.

Enables selecting and routing to different adapters at inference time based on input characteristics or external signals, without reloading base model weights. Implements set_adapter() method that switches active adapter in-place, enabling dynamic adapter selection in production systems where different inputs may require different task-specific adapters.

Manages multiple independent adapters attached to a single base model, enabling runtime switching between task-specific adapters via set_adapter() and composition of multiple adapters through add_adapter(). The architecture maintains a registry of named adapters and routes forward passes through the active adapter(s), supporting both sequential and parallel adapter composition patterns defined in the configuration system.

Saves and loads adapter weights and configurations independently from base model weights using save_pretrained() and from_pretrained() methods, storing only the trainable adapter parameters (~19MB) rather than full model checkpoints (multi-GB). The serialization format uses JSON for configuration and safetensors for weights, enabling portable adapter distribution and version control without base model dependencies.

Automatically adjusts LoRA rank during training based on parameter importance scores, pruning low-importance parameters and allocating rank budget to high-importance dimensions. Uses a parametrized rank matrix with importance-weighted pruning to dynamically reduce the effective rank, optimizing the rank-performance tradeoff without manual hyperparameter tuning. The mechanism computes importance scores via gradient-based analysis and applies structured pruning to adapter matrices.

Adds learnable prompt tokens (prompt tuning) or prefix embeddings (prefix tuning) to the input sequence or hidden states, enabling model adaptation without modifying model weights. Prompt tuning prepends learnable soft prompts to the input embeddings, while prefix tuning injects learnable prefix vectors into each transformer layer's key-value cache. Both methods freeze all model parameters and train only the prompt/prefix embeddings, reducing trainable parameters to 0.01-0.1% of model size.

Merges trained adapter weights into base model weights via merge_adapter(), combining adapter parameters with base weights to create a single unified model without separate adapter modules. Unmerging via unmerge_adapter() restores the original base model weights and adapter separation, enabling reversible adapter composition. The merge operation computes merged_weight = base_weight + adapter_weight, eliminating the adapter module from the forward pass.

Integrates with PyTorch Distributed Data Parallel (DDP) and Hugging Face Accelerate to synchronize adapter gradients across multiple GPUs/nodes during training. The architecture freezes base model weights and distributes only adapter parameters across devices, reducing communication overhead and enabling efficient multi-GPU training. Gradient synchronization occurs only for adapter parameters, not the full model, reducing communication bandwidth by 99%+ compared to full model distributed training.

Uses a declarative configuration system (PeftConfig subclasses) to specify adapter type, hyperparameters, and target modules, enabling adapter instantiation via get_peft_model(model, config) without manual layer wrapping. The configuration system maps adapter types to tuner classes via a registry (src/peft/mapping.py), enabling extensible adapter support. Configurations are serializable to JSON, enabling reproducible adapter creation and version control.

Injects learnable scaling vectors into transformer feed-forward and attention layers to modulate intermediate activations, enabling parameter-efficient adaptation through element-wise scaling rather than low-rank decomposition. IA3 learns multiplicative masks applied to inner activations, reducing trainable parameters to 0.01% of model size while maintaining model capacity through activation modulation. The mechanism is simpler than LoRA, requiring only vector-scale parameters instead of matrix decomposition.