onnxruntime

Loads ONNX-format models and executes inference through a pluggable execution provider architecture that automatically partitions computation graphs across available hardware accelerators (CPU, GPU, NPU). The InferenceSession abstraction handles model validation, graph optimization, and provider selection without requiring explicit hardware configuration. Supports tensor-based I/O compatible with numpy arrays across Python, C#, C++, Java, JavaScript, and Rust bindings.

Accepts pre-trained models from PyTorch, TensorFlow/Keras, TFLite, scikit-learn, and Hugging Face model hub, converting them to ONNX canonical representation for runtime execution. The conversion process validates model structure against ONNX specification and applies graph-level optimizations (operator fusion, constant folding, dead code elimination) before runtime execution. Enables single-model-artifact deployment across frameworks without retraining.

Provides InferenceSession API that loads ONNX models and executes inference with named input/output tensors managed as dictionaries. The API abstracts tensor shape and type handling, allowing users to pass numpy arrays (Python), typed arrays (JavaScript), or native arrays (C++) without explicit type conversion. Session manages model state (weights, buffers) and caches optimizations across multiple inference calls. Supports batch inference with variable batch sizes without model reloading.

Provides profiling capabilities to measure inference latency, memory usage, and per-operator execution time. The profiling system instruments the inference pipeline to collect detailed metrics (operator execution time, memory allocation, cache hits) and generates performance reports. Metrics can be exported for analysis and optimization. Profiling is optional and can be enabled/disabled at runtime without model recompilation.

Supports saving and loading model checkpoints during training, enabling resumable training and model versioning. The checkpoint system preserves model weights, optimizer state, and training metadata (epoch, loss, metrics) for recovery from training interruptions. Checkpoints are saved in ONNX format for compatibility with inference runtime. Enables training workflows that span multiple sessions or machines without losing progress.

The onnxruntime-genai module provides optimized inference for large language models (LLMs) with support for token-by-token streaming, dynamic batching, and state management across inference steps. Implements efficient attention mechanisms (KV-cache management, grouped query attention) and supports popular model families (Llama-2, Phi, Mistral, Qwen) with automatic quantization and graph optimization. Handles variable-length sequences and manages model state (past key-value tensors) across generation steps without explicit user management.

Enables training and fine-tuning of models directly on edge devices (mobile, IoT) or local machines without cloud infrastructure, supporting large model training acceleration and parameter-efficient fine-tuning methods. The training runtime applies graph-level optimizations (gradient checkpointing, mixed precision) and manages memory constraints on resource-limited devices. Supports personalization workflows where models adapt to user data without uploading sensitive information to cloud services.

Provides platform-specific runtime distributions (ONNX Runtime Mobile for iOS/Android, ONNX Runtime Web for browsers, cloud-optimized builds for Linux/Windows) that package the core inference engine with platform-appropriate dependencies and APIs. Each platform distribution includes language bindings (Swift/Objective-C for iOS, Kotlin/Java for Android, JavaScript for Web, C# for Windows) and applies platform-specific optimizations (CoreML integration on iOS, NNAPI on Android, WebGL/WebAssembly on browsers). Enables single ONNX model to run across desktop, mobile, web, and cloud with minimal code changes.

Applies compile-time graph optimizations to ONNX models before execution, including operator fusion (combining multiple operators into single fused kernel), constant folding (pre-computing constant subexpressions), dead code elimination, and layout optimization. The optimization pipeline is applied uniformly across all execution providers and hardware targets, reducing memory bandwidth, improving cache locality, and decreasing kernel launch overhead. Optimizations are transparent to user code — no explicit API calls required.

Supports inference on quantized models (INT8, INT4, FP16) with automatic precision selection based on hardware capabilities and model requirements. The runtime handles dequantization transparently during inference, applying quantized operations on hardware that supports them (e.g., INT8 on NVIDIA GPUs) and falling back to higher precision on unsupported hardware. Quantized models reduce memory footprint and improve inference latency without requiring explicit quantization code from users.

Provides language-specific API bindings (Python, C#/.NET, C++, Java, JavaScript/Node.js, Rust) that wrap the core C++ inference engine with language-idiomatic interfaces. Each binding implements the same InferenceSession abstraction (load model, run inference, retrieve outputs) with language-specific conventions (numpy arrays in Python, typed arrays in JavaScript, native arrays in C++). Enables teams to use ONNX Runtime across polyglot codebases without learning framework-specific APIs.

Validates ONNX models against specification compliance and scans for potentially malicious patterns (excessive memory allocation, unbounded loops, unsafe operations) before execution. The validation process checks model structure, operator compatibility, tensor shape consistency, and data type correctness. Security scanning identifies models that could trigger denial-of-service attacks through resource exhaustion (memory bombs, infinite loops) or unsafe operations. Validation is applied at InferenceSession creation time before model execution.

Pluggable execution provider architecture that abstracts hardware-specific inference implementations (CPU, NVIDIA GPU, AMD GPU, Intel GPU, Apple Neural Engine, ARM NNAPI, Qualcomm Hexagon NPU) behind unified interface. Each provider implements hardware-specific optimized kernels for common operators (convolution, matrix multiplication, attention) and applies provider-specific graph optimizations. The runtime automatically selects available providers and partitions computation graph across multiple providers if beneficial. Providers are loaded dynamically at runtime without recompilation.