LayersLayer 3 of 15
Inference and serving
Inference and serving is everything that turns a trained model into a running service: hosted inference providers, gateways that route between them, serving engines, and the tools that make models smaller and faster.
The categories
13 categories and 347 products in Inference and serving.
CategoryWhat it coversProducts
AI gateways & model routersAI gateways & model routers covers services that put one address in front of many model providers and route each request between them.34
Batch inferenceBatch inference covers asynchronous large-scale offline inference over datasets at reduced cost.23
Generative media inference platformsGenerative media inference platforms covers hosted inference clouds optimized for image, video, audio and 3D generation models, with a large media-model catalogue behind one API and per-image/per-second pricing.20
Generative media workflow toolsGenerative media workflow tools covers open-source node/canvas applications and pipelines for running image and video diffusion models locally, with custom-node ecosystems.25
Inference benchmarking & load testingInference benchmarking & load testing covers tools that measure the speed, capacity and cost of inference engines and providers.15
Inference kernels & attention librariesInference kernels & attention libraries covers low-level GPU kernels and DSLs that engines call for attention, GEMM, MoE.28
KV-cache & prompt-caching infrastructureKV-cache & prompt-caching infrastructure covers systems that store and reuse a model's earlier work across requests, so repeated prompts cost less and answer faster.24
LoRA serving & adapter hubsLoRA serving & adapter hubs covers systems that serve many fine-tuned variants on one base model, and the hubs where those variants are shared.22
Model compilers & graph optimizersModel compilers & graph optimizers covers compilers that convert and optimise a model to run fast on specific hardware, such as TensorRT, TVM, OpenVINO and XLA.24
On-device / edge runtimesOn-device / edge runtimes covers software runtimes that execute models on phones, laptops, browsers and edge chips (llama.cpp, ExecuTorch, MLC LLM, LiteRT, Ollama).36
Quantization / optimizationQuantization / optimization covers tools that compress models and make them faster to run.27
Serving engines & model serversServing engines & model servers covers open-source LLM serving engines (vLLM, SGLang, TGI) plus general and Kubernetes-native model servers (KServe, Triton, BentoML, Ray Serve, Seldon Core).31
Questions
Inference and serving, answered.
What is the Inference and serving layer of AI infrastructure?Inference and serving is everything that turns a trained model into a running service: hosted inference providers, gateways that route between them, serving engines, and the tools that make models smaller and faster.
What does the Inference and serving layer include?It has 13 categories: AI gateways & model routers, Batch inference, Generative media inference platforms, Generative media workflow tools, Inference benchmarking & load testing, Inference kernels & attention libraries, Inference providers, KV-cache & prompt-caching infrastructure, LoRA serving & adapter hubs, Model compilers & graph optimizers, On-device / edge runtimes, Quantization / optimization and Serving engines & model servers.
How many Inference and serving products does Unfragile track?347 products across 13 categories, each category with its own page.
What works here
The three products on this layer.
SearchLive
RecommendNot live yet.
RouteNot live yet.