QLoRA: Efficient Finetuning of Quantized LLMs (QLoRA)
Product* ⭐ 05/2023: [Voyager: An Open-Ended Embodied Agent with Large Language Models (Voyager)](https://arxiv.org/abs/2305.16291)
Capabilities6 decomposed
4-bit quantization with nf4 data type for llm weight compression
Medium confidenceImplements a novel 4-bit quantization scheme using NF4 (Normal Float 4), a data type optimized for normally-distributed weight matrices in neural networks. The approach uses block-wise quantization with absmax scaling to compress 70B+ parameter models into 24-48GB GPU memory, enabling fine-tuning on consumer hardware. Quantization is applied to the base model weights while LoRA adapters remain in full precision, creating a hybrid precision architecture that maintains training stability.
Introduces NF4 (Normal Float 4) data type specifically designed for normally-distributed LLM weights, combined with block-wise absmax scaling and double quantization of quantization constants, achieving 4x compression with minimal accuracy loss — prior work used uniform or symmetric quantization schemes that were less suited to weight distributions
Outperforms standard 8-bit quantization (e.g., QAT, post-training quantization) by enabling 4-bit precision without significant accuracy degradation, and surpasses naive 4-bit approaches by using NF4 data type optimized for neural network weight distributions rather than generic floating-point formats
lora adapter fine-tuning with frozen quantized base model
Medium confidenceCombines Low-Rank Adaptation (LoRA) with quantized base weights to enable parameter-efficient fine-tuning. Only LoRA adapter matrices (rank r, typically 8-64) are trained in full precision while the 4-bit quantized base model remains frozen. This approach reduces trainable parameters from billions to millions (0.1-1% of model size), dramatically lowering memory and compute requirements for gradient computation and optimizer state storage.
Combines LoRA with 4-bit quantization in a unified framework where adapters are trained in full precision while base weights remain frozen and quantized, enabling end-to-end fine-tuning without dequantization — prior LoRA work assumed full-precision base models or required dequantization during training
Achieves 10x lower memory consumption than standard LoRA on full-precision models by freezing quantized weights, and enables fine-tuning of 70B models on single GPUs where full-precision LoRA would require multi-GPU setups or gradient checkpointing
double quantization of quantization constants for nested compression
Medium confidenceApplies a second level of quantization to the quantization constants (scales and zero-points) themselves, reducing their memory footprint by an additional 2-4x. The quantization constants from the first quantization pass are themselves quantized to 8-bit precision and stored with their own scales, creating a nested quantization hierarchy. This technique is particularly effective for large models where quantization constant storage becomes a bottleneck (typically 2-5% of total model size).
Introduces nested quantization where quantization constants themselves are quantized to 8-bit precision with separate scales, reducing constant overhead by 2-4x — prior quantization work treated constants as full-precision metadata, not subject to further compression
Reduces total model size by an additional 2-4% compared to single-level quantization, enabling 70B models to fit in 24GB memory where standard 4-bit quantization alone would require 28-32GB
paged optimizers with unified memory management for gradient updates
Medium confidenceImplements a paged optimizer system that manages gradient and optimizer state (momentum, variance) using a unified memory pool with automatic paging between GPU and CPU memory. During backward passes, gradients are computed for LoRA parameters only and stored in a paged buffer; optimizer state is similarly paged, allowing the system to dynamically allocate memory based on batch size and gradient sparsity. This eliminates the need to pre-allocate large optimizer state buffers and enables dynamic batch sizing.
Introduces paged optimizer state management where gradient and optimizer buffers are dynamically allocated and paged between GPU and CPU memory based on runtime requirements, rather than pre-allocating fixed buffers — enables adaptive memory usage patterns not possible with static buffer allocation
Reduces peak GPU memory by 20-30% compared to standard optimizers with pre-allocated state buffers, and enables dynamic batch sizing that would otherwise require manual memory management or gradient accumulation
unified memory-efficient training pipeline with mixed-precision gradient computation
Medium confidenceOrchestrates an end-to-end training pipeline that combines 4-bit quantized base weights, full-precision LoRA adapters, and mixed-precision gradient computation. During forward passes, quantized weights are dequantized on-the-fly in a block-wise manner; during backward passes, gradients are computed only for LoRA parameters in full precision. The pipeline automatically manages precision conversions, gradient accumulation, and loss scaling to maintain numerical stability across the mixed-precision hierarchy.
Unifies 4-bit quantization, LoRA, double quantization, and paged optimizers into a single coherent training pipeline with automatic precision management and gradient stability mechanisms — prior work treated these techniques independently or required manual integration
Enables single-GPU fine-tuning of 70B models where alternatives (full-precision LoRA, standard quantization + LoRA) would require multi-GPU setups, gradient checkpointing, or significant accuracy loss
adapter composition and inference with merged weight strategies
Medium confidenceProvides mechanisms to compose multiple LoRA adapters trained on the same quantized base model and merge them into a single unified model for inference. Supports both sequential composition (adapter1 → adapter2) and weighted ensemble composition (w1*adapter1 + w2*adapter2). During inference, adapters can be merged into the base model weights (creating a standalone checkpoint) or applied dynamically at inference time. The system handles precision conversions and ensures numerical stability when merging full-precision adapters with quantized base weights.
Provides systematic adapter composition strategies (sequential, weighted ensemble) with automatic precision handling when merging full-precision adapters into quantized base weights, enabling flexible multi-task model construction — prior LoRA work focused on single-adapter inference
Enables multi-task inference without maintaining separate models or adapter routing logic, and supports weighted ensemble composition that would otherwise require custom inference code or model ensembling infrastructure
Capabilities are decomposed by AI analysis. Each maps to specific user intents and improves with match feedback.
Related Artifactssharing capabilities
Artifacts that share capabilities with QLoRA: Efficient Finetuning of Quantized LLMs (QLoRA), ranked by overlap. Discovered automatically through the match graph.
bitsandbytes
8-bit and 4-bit quantization enabling QLoRA fine-tuning.
LlamaFactory
Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)
peft
Parameter-Efficient Fine-Tuning (PEFT)
Transformers
Hugging Face's model library — thousands of pretrained transformers for NLP, vision, audio.
airllm
AirLLM 70B inference with single 4GB GPU
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
Best For
- ✓researchers and practitioners with limited GPU memory budgets
- ✓teams building domain-specific LLM variants without enterprise infrastructure
- ✓organizations seeking to reduce fine-tuning costs by 75%+
- ✓multi-task learning scenarios where separate adapters are needed per domain
- ✓teams with limited GPU memory seeking to fine-tune 13B-70B models
- ✓practitioners building adapter libraries for model composition and ensemble methods
- ✓practitioners deploying very large models (65B-70B parameters) on memory-constrained hardware
- ✓scenarios where quantization constant storage is a measurable bottleneck (>500MB for 70B models)
Known Limitations
- ⚠4-bit quantization introduces ~0.5-1% accuracy degradation on downstream tasks compared to full-precision fine-tuning
- ⚠Inference speed gains are modest (10-15%) because dequantization overhead partially offsets memory bandwidth savings
- ⚠Requires careful hyperparameter tuning (learning rate, warmup steps) to maintain convergence with quantized weights
- ⚠LoRA rank selection requires empirical tuning; rank too low (r=4) may underfit, rank too high (r=256) reduces memory savings
- ⚠Inference latency increases by 5-10% due to additional matrix multiplications for adapter projection and merging
- ⚠Adapter composition (merging multiple adapters) is non-trivial and may require retraining for optimal performance
Requirements
Input / Output
UnfragileRank
UnfragileRank is computed from adoption signals, documentation quality, ecosystem connectivity, match graph feedback, and freshness. No artifact can pay for a higher rank.
About
* ⭐ 05/2023: [Voyager: An Open-Ended Embodied Agent with Large Language Models (Voyager)](https://arxiv.org/abs/2305.16291)
Categories
Alternatives to QLoRA: Efficient Finetuning of Quantized LLMs (QLoRA)
Are you the builder of QLoRA: Efficient Finetuning of Quantized LLMs (QLoRA)?
Claim this artifact to get a verified badge, access match analytics, see which intents users search for, and manage your listing.
Get the weekly brief
New tools, rising stars, and what's actually worth your time. No spam.
Data Sources
Looking for something else?
Search →