LlamaFactory

Provides a single configuration-driven interface to fine-tune 100+ model families (LLaMA, Qwen, GLM, Mistral, Gemma, Yi, DeepSeek, etc.) by abstracting model-specific loading logic through a centralized model registry and adapter system. The framework uses HuggingFace Transformers as the base loader, then applies model-specific patches and configurations via a modular patching system that handles architecture variations, attention mechanisms, and special token handling without requiring separate codebases per model.

Implements multiple parameter-efficient fine-tuning (PEFT) methods through a pluggable adapter architecture that wraps model layers without modifying base weights. Supports LoRA (low-rank decomposition), QLoRA (quantized LoRA for 4-bit models), and OFT (orthogonal fine-tuning) by integrating with HuggingFace PEFT library and extending it with custom implementations. The adapter system allows selective application to specific layer types (attention, MLP) and supports merging adapters back into base weights or keeping them separate for inference.

Enables exporting fine-tuned models and adapters in multiple formats (PyTorch, SafeTensors, GGUF, GPTQ) and merging adapters back into base model weights for deployment. The export system handles format conversion, quantization during export (e.g., exporting to GPTQ format), and adapter merging which combines LoRA weights with base model weights through a weighted sum operation. Supports exporting to HuggingFace Hub for easy sharing, and includes format-specific optimizations (e.g., GGUF export includes quantization and can target specific hardware like CPU or mobile).

Integrates custom optimizers (GaLore, BAdam, APOLLO) that improve training efficiency beyond standard Adam by reducing memory usage or improving convergence. GaLore (Gradient Low-Rank Projection) projects gradients into a low-rank subspace, reducing optimizer state memory by 50-70%. BAdam (Block-wise Adam) partitions parameters into blocks and maintains separate optimizer states per block, improving convergence on large models. APOLLO applies adaptive learning rates per parameter group. These optimizers are pluggable through the training system and can be selected via configuration.

Provides a flexible dataset loading system that supports 50+ dataset formats (Alpaca, ShareGPT, OpenAI, JSONL, CSV, Parquet, etc.) through a template-based approach that maps raw data to standardized training formats. Each dataset format has a corresponding template that defines how to extract instruction, input, output, and history fields from the raw data. The system handles dataset discovery (from HuggingFace Hub or local paths), automatic format detection, and data validation. Custom templates can be defined in YAML to support new formats without code changes.

Integrates training callbacks that track metrics, log to external services (TensorBoard, Weights & Biases, Wandb), and trigger custom actions during training. The callback system hooks into the training loop at key points (step, epoch, validation) and enables custom metric computation, early stopping, learning rate scheduling, and model checkpointing. Built-in callbacks include loss tracking, gradient norm monitoring, learning rate logging, and stage-specific metrics (e.g., reward model accuracy, PPO policy divergence). Custom callbacks can be defined by extending a base class.

Orchestrates sequential training stages (pre-training, supervised fine-tuning, reward modeling, PPO, DPO, KTO, ORPO, SimPO) through a stage-aware trainer system that swaps loss functions, data collators, and optimization strategies based on the selected training_stage parameter. Each stage has a dedicated trainer class (SFTTrainer, RewardTrainer, PPOTrainer, etc.) that inherits from HuggingFace Trainer and implements stage-specific logic like preference pair handling for reward models or policy gradient computation for PPO. The configuration system validates stage transitions and manages data format expectations per stage.

Centralizes all training, inference, and data parameters through a unified configuration parser (hparams/parser.py) that accepts YAML/JSON files and validates inputs against typed argument classes (ModelArguments, DataArguments, TrainingArguments, etc.). The parser converts flat configuration dictionaries into strongly-typed Python dataclasses, performs cross-field validation (e.g., ensuring adapter_name_or_path exists if adapter_type is set), and distributes validated arguments to the appropriate subsystems. This eliminates the need for command-line argument parsing and enables reproducible training via version-controlled config files.

Extends the data pipeline to handle multimodal inputs (images, videos, audio) alongside text through specialized data processors that convert visual/audio tokens into embeddings compatible with LLM training. The system uses vision transformers (e.g., CLIP, Qwen-VL) to encode images and videos into token sequences, and audio processors to convert audio into spectrograms or embeddings. Data templates define how to interleave text and multimodal tokens (e.g., <image>token_sequence</image>text), and the collator handles variable-length multimodal sequences with padding/truncation.

Integrates bitsandbytes library to enable training with reduced precision (2-bit, 4-bit, 8-bit) through quantization-aware training (QAT) and post-training quantization (PTQ). The system loads models in quantized format using bitsandbytes' quantization kernels, then applies LoRA adapters on top of frozen quantized weights. For 4-bit quantization, it uses NF4 (normalized float 4) format which preserves more information than standard INT4. The training loop computes gradients only for adapter weights while keeping base model weights frozen in quantized format, reducing memory usage by 75-90% compared to full precision training.

Enables distributed training across multiple GPUs/TPUs through integration with DeepSpeed and PyTorch FSDP (Fully Sharded Data Parallel). The system detects available hardware and automatically configures the appropriate distributed backend, handling gradient accumulation, gradient synchronization, and model sharding across devices. DeepSpeed integration includes support for ZeRO-1/2/3 optimization stages which partition optimizer states, gradients, and model parameters across devices to reduce per-GPU memory usage. FSDP provides pure PyTorch distributed training without external dependencies.

Provides a pluggable inference backend system that abstracts away differences between inference engines (HuggingFace Transformers, vLLM, SGLang, KTransformers) through a unified ChatModel interface. Each backend implements the same generation API but with different optimization strategies: HuggingFace Transformers is the baseline, vLLM adds paged attention and continuous batching for throughput, SGLang adds structured generation and multi-modal support, KTransformers adds kernel-level optimizations for specific models. The system auto-selects the best backend based on model type and available hardware, or allows manual override via configuration.

Exposes fine-tuned models through an OpenAI-compatible REST API server that implements the Chat Completions and Embeddings endpoints, enabling drop-in replacement for OpenAI's API. The server uses the inference engine abstraction to support multiple backends (vLLM, SGLang, etc.) and handles request routing, batching, and streaming responses. Clients written for OpenAI's API can use LlamaFactory's server without modification, reducing integration friction. The server supports authentication via API keys and includes request logging and metrics collection.

Provides a browser-based interface (LLaMA Board) built with Gradio/Streamlit that enables non-technical users to configure training jobs, monitor progress, run inference, and evaluate models without command-line interaction. The UI includes a training configuration builder that generates YAML configs, a real-time training monitor showing loss curves and metrics, a chat interface for testing models, and an evaluation dashboard for comparing model outputs. The backend communicates with the training system via a REST API, enabling remote training on a separate machine.