Request Scheduling With Prefill Decode Disaggregation

via “request scheduling with prefill-decode disaggregation”

Fast LLM/VLM serving — RadixAttention, prefix caching, structured output, automatic parallelism.

Unique: Separates prefill and decode scheduling with different batch sizes and priorities, enabling continuous batching where new requests are added to the decode queue without blocking prefill operations.

vs others: Achieves lower time-to-first-token than vLLM through prefill-decode disaggregation and continuous batching, with higher decode throughput by using larger decode batch sizes.

via “disaggregated prefill-decode serving with service discovery”

NVIDIA's LLM inference optimizer — quantization, kernel fusion, maximum GPU performance.

Unique: Implements disaggregated prefill-decode architecture with gRPC-based inter-worker communication and integrated service discovery. Separates compute-intensive prefill from memory-intensive decode, enabling independent scaling and hardware optimization for each stage.

vs others: More efficient than monolithic serving for high-throughput workloads; achieves 2-3x higher throughput than single-worker setups by overlapping prefill and decode across different GPU pools. Service discovery integration enables auto-scaling and fault tolerance.