Serving engines & model servers
Serving engines & model servers covers open-source LLM serving engines (vLLM, SGLang, TGI) plus general and Kubernetes-native model servers (KServe, Triton, BentoML, Ray Serve, Seldon Core).
In brief
Serving engines & model servers, in brief.
What it isServing engines & model servers covers open-source LLM serving engines (vLLM, SGLang, TGI) plus general and Kubernetes-native model servers (KServe, Triton, BentoML, Ray Serve, Seldon Core).
Where it sitsOne of 13 categories in the Inference and serving layer. Inference and serving is everything that turns a trained model into a running service: hosted inference providers, gateways that route between them, serving engines, and the tools that make models smaller and faster.
What we track31 products, including AIBrix, Aphrodite Engine, BentoML, DeepSpeed-MII and Gateway API Inference Extension.
The products
31 products in Serving engines & model servers.
Each name opens a search for that product in our index.
ProductMakerIts own site
From our census of 2026-11-08. A product listed here is one that belongs in this category; the search answer says whether we have checked it.
Questions
Serving engines & model servers, answered.
What is Serving engines & model servers in AI infrastructure?Serving engines & model servers covers open-source LLM serving engines (vLLM, SGLang, TGI) plus general and Kubernetes-native model servers (KServe, Triton, BentoML, Ray Serve, Seldon Core).
Which layer of AI infrastructure is Serving engines & model servers in?Serving engines & model servers is in the Inference and serving layer. Inference and serving is everything that turns a trained model into a running service: hosted inference providers, gateways that route between them, serving engines, and the tools that make models smaller and faster.
Which products are in Serving engines & model servers?We track 31 products in Serving engines & model servers, including AIBrix, Aphrodite Engine, BentoML, DeepSpeed-MII, Gateway API Inference Extension, Infinity (embedding/reranker server), KServe and KTransformers. The full list is on this page.
Inference and serving
The other categories in Inference and serving.
AI gateways & model routersBatch inferenceGenerative media inference platformsGenerative media workflow toolsInference benchmarking & load testingInference kernels & attention librariesInference providersKV-cache & prompt-caching infrastructureLoRA serving & adapter hubsModel compilers & graph optimizersOn-device / edge runtimesQuantization / optimization