TensorRT-LLM

Implements a pluggable quantization system that converts model weights to lower-precision formats (FP8, INT4, AWQ, GPTQ) with per-layer scale management and weight loading pipelines. The quantization configuration system integrates with the Linear Layer abstraction, allowing selective quantization of different layer types while maintaining numerical stability through dynamic scaling and per-channel quantization strategies. Supports both symmetric and asymmetric quantization with automatic scale computation during model compilation.

Implements a memory-efficient KV cache system using paged allocation (similar to OS virtual memory) that decouples cache pages from request lifetimes, enabling dynamic reuse across batches. The KV cache is managed by the PyExecutor runtime with explicit transfer semantics for disaggregated serving architectures where prefill and decode phases run on separate GPU clusters. Supports context parallelism where KV cache is sharded across GPUs with efficient all-gather operations during attention computation.

Provides an automated model onboarding pipeline (AutoDeploy) that takes a pre-trained model and automatically applies transformations (quantization, sharding, kernel fusion) to optimize for target hardware. The system includes model architecture detection, automatic sharding strategy selection, and performance profiling to validate optimizations. Supports custom transformation rules via pattern matching and fusion transforms.

Supports multimodal inference by processing image inputs through vision encoders that produce visual embeddings, which are then merged with text tokens before passing to the LLM. Implements token merging strategies (e.g., average pooling, learned projection) to reduce the number of visual tokens while preserving semantic information. Supports multiple vision encoder backends (CLIP, DINOv2, custom encoders) with configurable preprocessing pipelines.

Provides a comprehensive benchmarking framework (trtllm-bench) that measures inference latency, throughput, and memory usage across different configurations (batch sizes, sequence lengths, quantization strategies). Includes regression detection that compares performance against baseline metrics and alerts on performance degradation. Supports custom benchmark scenarios and metrics collection via pluggable backends.

Compiles inference workloads into CUDA graphs that capture the entire computation and communication pattern as a single graph, eliminating kernel launch overhead and enabling static scheduling. The compilation pipeline analyzes the model and generates optimized CUDA graphs for different batch sizes and sequence lengths. Supports dynamic CUDA graphs for variable-length sequences with minimal overhead.

Provides a Triton Inference Server backend that wraps TensorRT-LLM models, enabling deployment via Triton's standardized model serving interface. Includes automatic model configuration generation from TensorRT engine metadata and support for Triton's ensemble models for complex inference pipelines. The backend handles request batching, response formatting, and metrics collection compatible with Triton's monitoring infrastructure.

Implements a request scheduling system in the PyExecutor runtime that dynamically batches requests during both prefill and decode phases, allowing new requests to join ongoing batches without waiting for previous requests to complete. The scheduler uses an event loop that interleaves prefill and decode operations, with configurable batch sizes and scheduling policies (FCFS, priority-based). Requests are tracked through a state machine with explicit transitions between prefill, decode, and completion states.

Provides a pluggable kernel fusion system that combines multiple operations (e.g., attention + softmax + output projection) into single CUDA kernels to reduce memory bandwidth and kernel launch overhead. The AutoTuner component profiles different kernel implementations (TRTLLM native kernels, FlashInfer, FlashAttention) and selects optimal kernels based on model architecture, batch size, and sequence length. Custom ops can be registered via the Tunable Runners interface, enabling integration of third-party kernels.

Implements automatic tensor parallelism by decomposing model weights across multiple GPUs using pattern-matching transformations that insert collective communication ops (all-reduce, all-gather) at appropriate points in the computation graph. The sharding transformation pipeline analyzes the model architecture and applies sharding rules (e.g., split linear layers column-wise for matrix multiplication) while preserving numerical correctness. Communication backends (NCCL, MPI) are abstracted behind a unified interface.

Implements pipeline parallelism by partitioning model layers across GPUs and executing stages in a pipelined fashion, where GPU-0 processes prefill while GPU-1 processes decode of previous requests. The ModelEngine coordinates stage execution with explicit synchronization points and bubble minimization strategies. Supports both GPipe-style (synchronous) and asynchronous pipeline execution with configurable micro-batch sizes.

Implements speculative decoding where a smaller draft model generates candidate tokens that are verified in parallel by the main model, reducing the number of main model forward passes. Supports multiple speculation strategies: EAGLE3 (learns to predict next tokens), MTP (multi-token prediction), and custom strategies via the SpeculativeDecodingConfig. The verification phase uses efficient batch processing to check multiple candidates simultaneously, with fallback to standard decoding if verification fails.

Implements efficient MoE inference with support for multiple backend strategies: expert parallelism (distribute experts across GPUs), expert tensor parallelism (combine expert and tensor parallelism), and expert pipeline parallelism. The MoE interface abstracts different communication strategies (all-to-all, all-gather + scatter) and load balancing approaches (token-to-expert assignment, expert capacity management). Supports quantization of expert weights independently from shared layers.

Provides an OpenAI-compatible HTTP server (trtllm-serve) that exposes inference via standard /v1/chat/completions and /v1/completions endpoints. Supports tool calling with automatic function routing where the model can invoke registered tools, with response post-processing that extracts tool calls and executes them. Includes reasoning parsers for models with chain-of-thought capabilities and Harmony adapter for GPT-OSS model compatibility.

Implements disaggregated serving where prefill and decode phases run on separate GPU clusters, optimizing for different compute patterns (prefill is compute-bound, decode is memory-bound). The serving infrastructure includes service discovery and routing logic that directs prefill requests to prefill cluster and decode requests to decode cluster, with KV cache transfer between clusters. Supports dynamic cluster scaling based on load.