PEFT

Injects trainable low-rank decomposition matrices (LoRA) into transformer attention and feed-forward layers by wrapping linear modules with a registry-based dispatch system. Uses PeftModel wrapper pattern to intercept forward passes and compose base weights with adapter weights via matrix multiplication, enabling training of only 0.1-2% of parameters while maintaining architectural compatibility with HuggingFace transformers.

Extends LoRA with automatic rank discovery by computing importance scores for adapter parameters during training and pruning low-importance weights. Implements a parametric allocation algorithm that adjusts per-layer ranks dynamically based on gradient statistics, reducing manual hyperparameter tuning while maintaining task performance with fewer total parameters than fixed-rank LoRA.

Uses a declarative configuration system (PeftConfig subclasses) that specifies adapter type, hyperparameters, and target modules, enabling adapter creation without writing custom code. Implements a registry-based factory pattern (src/peft/mapping.py) that maps configuration objects to concrete tuner implementations, supporting 25+ PEFT methods through unified configuration interface.

Allows fine-grained control over which model layers receive adapters through pattern matching on module names (e.g., 'q_proj', 'v_proj' for attention, 'mlp' for feed-forward). Implements regex-based and exact-match module selection that enables adapting only specific layers (e.g., attention layers only) without modifying feed-forward layers, reducing parameters and enabling layer-specific optimization.

Integrates with PyTorch's gradient checkpointing to trade computation for memory by recomputing activations during backward pass instead of storing them. Automatically enables gradient checkpointing for adapter training, reducing peak memory usage by 30-50% while adding ~20-30% training time overhead, enabling larger batch sizes on memory-constrained hardware.

Enables training adapters in mixed precision (float16 or bfloat16) with automatic loss scaling to prevent gradient underflow, reducing memory usage by 50% and improving training speed by 1.5-2x. Integrates with PyTorch's automatic mixed precision (AMP) and transformers' native mixed-precision support to maintain numerical stability while reducing precision.

Enables selecting and routing to different adapters at inference time based on input characteristics or external signals, without reloading base model weights. Implements set_adapter() method that switches active adapter in-place, enabling dynamic adapter selection in production systems where different inputs may require different task-specific adapters.

Prepends learnable prefix tokens to input embeddings that are optimized during fine-tuning, allowing the model to learn task-specific prompts without modifying base model weights. Implements a shallow feed-forward network that projects prefix parameters to full embedding dimension, enabling efficient adaptation by training only prefix embeddings (typically 0.1-1% of model size).

Learns a small set of soft prompt tokens (typically 20-100) that are concatenated with input embeddings and optimized via gradient descent. Unlike prefix tuning, prompt tuning operates only on the input layer and uses a simpler optimization approach, making it the most parameter-efficient method (0.01-0.1% of model size) while maintaining competitive performance on classification and generation tasks.

Manages multiple independent adapters on a single base model, enabling dynamic switching between task-specific adapters at inference time or composition of multiple adapters for ensemble effects. Implements adapter registry and routing logic (add_adapter, set_adapter, delete_adapter methods in PeftModel) that maintains separate parameter sets while sharing the frozen base model, enabling efficient multi-task deployment.

Combines 4-bit or 8-bit quantization of base model weights with low-rank adapter training, enabling fine-tuning of billion-parameter models on consumer GPUs (e.g., 70B model on 24GB VRAM). Integrates with bitsandbytes quantization library to load base weights in low-bit format while keeping adapters in full precision, using gradient checkpointing and paged optimizers to manage memory.

Saves and loads adapter weights independently from base model weights using a standardized format (JSON config + safetensors binary), enabling portable adapter distribution and composition. Implements save_pretrained() and from_pretrained() methods that serialize only adapter parameters and configuration, producing ~19MB checkpoints vs multi-GB full model checkpoints, with support for loading adapters onto different base model versions.

Fuses adapter weights into base model weights (merge_adapter) to eliminate adapter inference overhead, or separates merged adapters back to original form (unmerge_adapter) to recover adapter-only parameters. Implements weight composition logic that adds scaled adapter outputs to base weights, producing a single model file that requires no adapter loading at inference time, with optional unmerging to recover original adapter parameters.

Enables multi-GPU and multi-node training of adapters using PyTorch DistributedDataParallel (DDP) and DeepSpeed integration, with automatic gradient synchronization across devices. Implements adapter-aware distributed training that synchronizes only adapter gradients (0.1-2% of parameters) instead of full model gradients, reducing communication overhead and enabling efficient scaling across multiple GPUs.

Introduces learnable scaling vectors on intermediate activations (feed-forward and attention outputs) without adding new parameters to weight matrices. Implements element-wise scaling of activations via learned vectors, achieving parameter efficiency comparable to LoRA (0.1-1% of model size) while using a different architectural approach that scales activations rather than weights.