For running AI models at the edge, the strongest candidates are ONNX Runtime, built for accelerated inference across platforms 2, and Ollama, which runs open models locally on macOS, Windows and Linux 3. NVIDIA's Jetson-focused inference tooling covers edge LLM and VLM serving on NVIDIA hardware 1, and LocalAI runs models on any hardware without a GPU 9.

1. 1

   ONNX Runtimeonnxruntime.aiverified and signed

   A cross-platform accelerated machine learning engine that speeds up inferencing across existing technology stacks, usable from Python via pip 2.

   WebsiteDocs

   for agents: machine · onnx · learning

2. 2

   Ollamaollama.comwebsite read, tools not yet checked

   Runs open AI models locally or in the cloud on macOS, Windows and Linux 3.

   WebsiteDocsPricingGet started

   +141 community tools built for it

3. 3

   NVIDIA Jetson inference toolingnvidia.comverified and signed

   Picks the serving stack and memory settings (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) for LLM and VLM workloads on NVIDIA Jetson devices 11.

   WebsiteDocsPricingGet started

   official:MCP server · NVIDIA CUDA Docs+84 community tools built for it

   for agents: tao-run-inference-service · jetson-inference-mem-tune · inference

4. 4

   LocalAIlocalai.ioverified and signed

   An open source AI engine that runs text, vision, voice and image models on any hardware, with no GPU required 9, and exposes model listing and embedding actions 19.

   WebsiteDocs

   +6 community tools built for it

   for agents: list_models · ask_localai · create_embedding

5. 5

   SGLangsglang.iowebsite read, tools not yet checked

   An open-source serving framework for large language and multimodal models, supporting a range of hardware platforms 7.

   WebsiteDocsPricingGet started

   +2 community tools built for it