Kubeflow

Kubeflow Pipelines enables users to define, compile, and execute multi-step ML workflows as directed acyclic graphs (DAGs) using Python SDK or YAML manifests. Workflows are compiled into Argo Workflows CRDs and executed on Kubernetes, with built-in support for artifact passing between steps, conditional execution, and loop constructs. The platform provides a web UI for pipeline versioning, run history, and artifact lineage tracking.

Kubeflow Training Operators provide Kubernetes custom resources (PyTorchJob, TFJob, MPIJob) that abstract distributed training orchestration across multiple nodes and GPUs. Each operator handles framework-specific concerns: PyTorch uses torch.distributed.launch, TensorFlow manages parameter servers and workers, MPI uses OpenMPI. Operators manage pod creation, network setup, failure recovery, and graceful shutdown, exposing a declarative YAML interface that hides distributed training complexity.

Kubeflow implements a three-layer architecture pattern: User Interface Layer (web applications for Notebooks, Pipelines, Katib), Controller Layer (Kubernetes controllers managing custom resources), and Resource Layer (CRDs representing ML workloads). This separation enables independent scaling and evolution of each layer — UI changes don't affect controllers, and new controllers can be added without modifying the UI. Controllers use the Kubernetes watch API to react to resource changes, implementing the operator pattern for declarative resource management.

Kubeflow Notebooks provides managed Jupyter, RStudio, and VS Code server instances running in Kubernetes pods, with Profile Controller enforcing per-user namespace isolation and resource quotas. Users access notebooks through the Central Dashboard web UI, which handles authentication, namespace routing, and ingress management. Notebooks persist user code and data to PVCs, enabling long-running development sessions with automatic pod restart on failure.

Katib provides a Kubernetes-native hyperparameter optimization platform supporting multiple search algorithms (grid, random, Bayesian optimization, genetic algorithms, population-based training). Users define search spaces in YAML, and Katib spawns trial jobs (using Training Operators or custom containers) in parallel, collecting metrics from each trial and iteratively refining the search space. The platform integrates with TensorBoard for visualization and supports early stopping policies to terminate unpromising trials.

KServe provides a Kubernetes-native model serving platform supporting multiple inference frameworks (TensorFlow, PyTorch, Scikit-learn, XGBoost, ONNX) through standardized InferenceService CRDs. KServe handles model loading, request routing, auto-scaling based on traffic, and canary deployments via traffic splitting between model versions. The platform abstracts framework-specific serving concerns (e.g., TensorFlow Serving vs TorchServe) behind a unified REST/gRPC API, with built-in support for request batching and GPU acceleration.

Kubeflow's Profile Controller implements multi-tenancy by creating isolated Kubernetes namespaces per user/team with automatic RBAC, network policies, and resource quotas. Each profile maps to a namespace with pre-configured role bindings, allowing users to access only their own resources. The controller also manages PVC provisioning for user storage and integrates with the Central Dashboard for profile creation and management, enforcing resource limits to prevent noisy neighbor problems.

Kubeflow's Central Dashboard serves as the single entry point for all platform components, providing unified authentication (OIDC, LDAP, Kubernetes RBAC), role-based access control, and navigation to specialized web applications (Notebooks, Pipelines, Katib, KServe). The dashboard handles session management, namespace routing, and ingress configuration, abstracting away Kubernetes complexity from end users. It integrates with the Profile Controller to enforce namespace isolation and provides a unified view of user resources across components.

Kubeflow Model Registry provides a centralized repository for ML models with versioning, metadata tracking, and lineage information. Models are registered with framework type, training dataset references, hyperparameters, and evaluation metrics, enabling reproducibility and audit trails. The registry integrates with Kubeflow Pipelines to automatically capture model lineage (which pipeline produced which model), and with KServe to enable model deployment directly from the registry. Metadata is stored in a backend database (MySQL, PostgreSQL) with REST API access.

Kubeflow's Admission Webhook intercepts Kubernetes API requests for Kubeflow resources (Notebooks, Training Jobs, InferenceServices) and performs validation and mutation before persistence. The webhook enforces policies (e.g., resource limits, image whitelisting, namespace restrictions) and automatically injects sidecar containers, environment variables, or volume mounts based on cluster configuration. This enables centralized policy enforcement without modifying user manifests and provides a hook for custom business logic (e.g., cost tracking, compliance checks).

Kubeflow integrates the Spark Operator to enable declarative Spark job submission on Kubernetes via SparkApplication CRDs. Users define Spark jobs in YAML with driver/executor pod specifications, and the operator manages pod creation, driver-executor communication, and job lifecycle. The operator handles Spark-specific concerns (e.g., dynamic executor scaling, shuffle service) and integrates with Kubeflow Pipelines to enable Spark jobs as pipeline steps, enabling data processing workflows alongside ML training.