TensorRT-LLM

Implements a pluggable quantization system that converts model weights to lower precision formats (FP8, INT4, AWQ, GPTQ) with per-layer scale management and weight loading pipelines. The quantization configuration system allows fine-grained control over which layers use which quantization methods, with automatic scale computation during model compilation. Supports mixed-precision strategies where different layers can use different quantization schemes optimized for their numerical characteristics.

Implements a memory-efficient KV cache system that pages attention key-value tensors into fixed-size blocks, enabling dynamic allocation and reuse across requests without fragmentation. The cache is managed by the PyExecutor runtime which tracks block allocation, deallocation, and reuse across the request queue. Supports disaggregated serving architectures where KV cache can be transferred between encoder and decoder workers via IPC, enabling horizontal scaling of inference workloads.

Implements an AutoDeploy system that automatically converts Hugging Face models to optimized TensorRT engines through a transformation pipeline. The pipeline applies sharding transformations, pattern-matching fusion, quantization, and kernel optimization in sequence. Supports model discovery from Hugging Face Hub and automatic configuration of optimal settings based on model architecture and target hardware.

Implements multimodal inference where images are encoded using vision encoders (CLIP, SigLIP) and their embeddings are injected into the token sequence for processing by the LLM. Supports multiple image formats (JPEG, PNG, WebP) and automatic image resizing/normalization. Vision encoder outputs are cached to avoid redundant computation when the same image is processed multiple times.

Implements a comprehensive benchmarking framework that measures inference latency, throughput, memory usage, and accuracy across different configurations. Includes regression detection that compares performance against baseline metrics and flags significant degradations. Supports both synthetic benchmarks (fixed batch sizes, sequence lengths) and realistic workload simulation (variable request patterns, arrival rates).

Implements a flexible sampling system through the SamplingParams configuration that controls token generation behavior. Supports multiple sampling strategies: temperature-based softmax scaling, top-k filtering, nucleus (top-p) sampling, and beam search. Parameters can be set per-request, enabling fine-grained control over generation diversity and quality. Integrates with the Sampler component in PyExecutor to apply sampling decisions at token generation time.

Provides a Triton Inference Server backend that wraps TensorRT-LLM models, enabling deployment via Triton's standardized model serving interface. Includes automatic model configuration generation from TensorRT engine metadata and support for Triton's ensemble models for complex inference pipelines. The backend handles request batching, response formatting, and metrics collection compatible with Triton's monitoring infrastructure.

Implements a request scheduler in the PyExecutor runtime that dynamically batches requests at the token level, allowing new requests to join ongoing batches mid-inference without waiting for current batches to complete. The scheduler uses an event loop that processes requests in priority order, allocates KV cache blocks, and schedules forward passes through the ModelEngine. Supports heterogeneous batch composition where requests with different sequence lengths, batch sizes, and sampling parameters execute in the same batch.

Implements a pattern-matching and fusion transformation pipeline that identifies subgraphs of operations (e.g., linear layer + activation + layer norm) and replaces them with fused custom CUDA kernels. The AutoTuner system profiles different kernel implementations and selects the fastest variant for each operation based on input shapes and hardware. Supports vendor-specific kernels (Triton, CUTLASS) and allows registration of custom ops through a tunable runner interface.

Implements distributed tensor parallelism where model weights are sharded across multiple GPUs and each forward pass requires all-reduce communication to synchronize partial results. The sharding transformation pipeline automatically partitions linear layers, attention operations, and MoE layers across GPUs based on a sharding strategy. Uses NCCL for efficient GPU-to-GPU communication and supports multiple communication backends (NCCL, GLOO, MPI).

Implements pipeline parallelism where model layers are partitioned across multiple GPUs and each GPU processes a different stage of the pipeline. Uses a bubble-minimization scheduling algorithm (similar to GPipe) to overlap computation and communication across stages. Supports both synchronous and asynchronous pipeline execution with configurable pipeline depth and micro-batch sizes.

Implements speculative decoding where a smaller draft model generates candidate tokens and a larger verifier model validates them in parallel, reducing the number of forward passes required. Supports multiple speculation strategies: EAGLE3 (learned draft model), MTP (multi-token prediction), and custom strategies. The verification process uses batch processing to validate multiple candidate sequences in a single forward pass, amortizing compute cost.

Implements efficient MoE inference with expert parallelism where experts are distributed across GPUs and routing decisions are made per token. Supports multiple MoE backends (TRTLLM native, custom implementations) and communication strategies (all-to-all, hierarchical). Includes expert load balancing to minimize GPU idle time and communication overhead. Supports quantization of expert weights independently from dense layers.

Implements a Triton Inference Server backend that exposes TensorRT-LLM models via OpenAI-compatible REST API endpoints. Supports function calling through a schema-based function registry where tools are defined as JSON schemas and the model generates function calls that are executed and fed back into the context. Includes response post-processing to extract structured outputs (JSON, function calls) from model generations.

Implements a disaggregated serving architecture where prefill (prompt processing) and decode (token generation) are separated into independent worker pools that communicate via gRPC. Prefill workers process incoming requests and generate initial KV cache, which is transferred to decode workers for token generation. Includes service discovery and load balancing to route requests to available workers and handle worker failures.