Nf4 Normal Float 4 Bit Quantization With Information Theoretic Optimality

via “nf4 (normal float 4-bit) quantization with information-theoretic optimality”

8-bit and 4-bit quantization enabling QLoRA fine-tuning.

Unique: Uses information-theoretically optimal quantization levels derived from inverse normal CDF, allocating more precision to high-probability regions of weight distributions. Achieves better accuracy than uniform FP4 quantization on transformer weights without requiring per-layer calibration.

vs others: Outperforms FP4 quantization on transformer models by 1-2% accuracy while maintaining same memory footprint, and requires no calibration unlike post-training quantization methods.

via “quantization-aware training with 2/4/8-bit precision and bitsandbytes integration”

Unified Efficient Fine-Tuning of 100+ LLMs & VLMs (ACL 2024)

Unique: Integrates bitsandbytes quantization kernels with LoRA adapter system to enable 4-bit training with NF4 format, supporting nested quantization (double_quant) for additional memory savings. Automatically handles quantization/dequantization in forward/backward passes without user intervention.

vs others: Native 4-bit quantization with NF4 format vs. alternatives like GPTQ which requires post-training quantization, enabling QLoRA training on consumer GPUs without pre-quantized models.

via “4-bit quantization with nf4 data type for llm weight compression”

* ⭐ 05/2023: [Voyager: An Open-Ended Embodied Agent with Large Language Models (Voyager)](https://arxiv.org/abs/2305.16291)

Unique: Introduces NF4 (Normal Float 4) data type specifically designed for normally-distributed LLM weights, combined with block-wise absmax scaling and double quantization of quantization constants, achieving 4x compression with minimal accuracy loss — prior work used uniform or symmetric quantization schemes that were less suited to weight distributions

vs others: Outperforms standard 8-bit quantization (e.g., QAT, post-training quantization) by enabling 4-bit precision without significant accuracy degradation, and surpasses naive 4-bit approaches by using NF4 data type optimized for neural network weight distributions rather than generic floating-point formats