bentoml

BentoML uses Python decorators (@bentoml.service) to declaratively define ML service endpoints with type hints and dependency injection. The framework parses decorator metadata to auto-generate OpenAPI schemas, request/response validation, and service routing without boilerplate. Services are defined as Python classes with methods decorated as endpoints, enabling IDE autocomplete and static type checking while maintaining runtime flexibility for model loading and inference logic.

BentoML packages trained models, preprocessors, and dependencies into immutable Bento artifacts with semantic versioning and content-addressed storage. Each Bento is a self-contained bundle containing the model binary, Python environment specification (via pip/conda), custom code, and metadata. The framework uses a local model store (by default ~/.bentoml) with tag-based retrieval, enabling reproducible deployments and easy model rollback without re-training.

BentoML automatically infers model input/output signatures from type hints and generates OpenAPI schemas without manual specification. The framework inspects service method signatures, IODescriptor types, and model metadata to generate complete API documentation. Generated schemas include request/response examples, validation rules, and are served via /docs (Swagger UI) and /openapi.json endpoints.

BentoML integrates with BentoCloud (managed hosting platform) for one-command deployment of Bento artifacts. The framework provides CLI commands (bentoml deploy) that package services, authenticate with BentoCloud, and deploy with automatic scaling, monitoring, and API endpoint provisioning. Deployments are tracked with version history, and rollback is supported via CLI commands.

BentoML provides a local development server (bentoml serve) that runs services locally with automatic hot-reload on code changes. The server watches service files and reloads the service without restarting, enabling rapid iteration during development. The server exposes the same API endpoints, health checks, and metrics as production deployments, enabling local testing before containerization.

BentoML captures Python dependencies (via pip or conda) in the Bento artifact and automatically includes them in generated Docker images. Dependencies are specified in requirements.txt or environment.yml and are resolved during Bento creation. The framework validates that all imports in service code are declared as dependencies, preventing runtime import errors in production.

BentoML automatically generates Dockerfiles and builds OCI-compliant container images from Bento artifacts without manual Docker configuration. The framework introspects the service definition, dependencies, and model artifacts to create optimized multi-stage Dockerfiles with minimal image size. Generated images include the BentoML runtime, service code, model binaries, and all dependencies, ready for deployment to Kubernetes, Docker Swarm, or cloud platforms.

BentoML implements server-side request batching that automatically groups incoming inference requests and processes them together to maximize GPU/CPU utilization. The framework uses configurable batch windows (time-based or size-based) to accumulate requests before invoking the model, reducing per-request overhead and improving throughput. Batching is transparent to the client — individual requests are queued, batched, and responses are returned asynchronously without client-side coordination.

BentoML enables defining services with multiple models and composing them into inference pipelines where outputs from one model feed into another. Services can declare multiple model dependencies, load them with different configurations, and orchestrate their execution through explicit method calls or implicit dependency injection. The framework handles model lifecycle (loading, caching, unloading) and enables conditional routing (e.g., route to model A or B based on input features) without external orchestration tools.

BentoML abstracts model loading through a Runner abstraction that supports any ML framework (PyTorch, TensorFlow, scikit-learn, XGBoost, ONNX, custom Python code) without framework-specific code in the service definition. Runners are initialized with model artifacts and expose a predict() interface; the framework handles model lifecycle (lazy loading, GPU memory management, multi-process execution). Custom runners can be implemented for proprietary models or non-standard inference logic by subclassing bentoml.Runner.

BentoML uses IODescriptor classes to define how requests and responses are serialized/deserialized (JSON, binary, images, numpy arrays, pandas DataFrames, etc.). Descriptors are attached to service methods via type hints and automatically handle content-type negotiation, validation, and conversion. Custom IODescriptors can be implemented for domain-specific formats (e.g., medical imaging DICOM, audio WAV) by subclassing bentoml.io.IODescriptor.

BentoML supports multi-process runners that distribute inference across multiple worker processes, enabling true parallelism on multi-core CPUs and avoiding Python GIL limitations. Runners can be configured with a process pool, and requests are automatically distributed across workers. The framework handles inter-process communication, request queuing, and response aggregation transparently, enabling horizontal scaling within a single container.

BentoML services expose standard health check endpoints (/healthz, /readyz) compatible with Kubernetes liveness and readiness probes. Health checks verify that the service is running and models are loaded; readiness probes confirm the service is ready to accept traffic. Custom health check logic can be implemented by overriding the health_check() method, enabling checks for external dependencies (database, cache, API availability).

BentoML automatically collects inference metrics (request count, latency, error rate) and exports them in Prometheus format via a /metrics endpoint. Metrics are collected per endpoint and can be scraped by Prometheus or other monitoring systems. Custom metrics can be added by instrumenting service code with bentoml.metrics APIs, enabling tracking of business metrics (e.g., model confidence, prediction distribution) alongside infrastructure metrics.