Comet API

Captures and stores hyperparameters, training metrics, and evaluation scores from ML training runs via SDK instrumentation that hooks into popular frameworks (PyTorch, TensorFlow, scikit-learn). Uses a client-side buffer that batches logged data and sends it to Comet's backend via REST/gRPC, enabling real-time metric streaming with configurable flush intervals and automatic deduplication of repeated values.

Automatically captures source code, Git metadata (commit hash, branch, diff), Python environment (installed packages, versions), system information (GPU/CPU specs, OS), and dependency graphs at experiment start time. Uses Git integration to extract version control context and pip/conda introspection to build environment manifests, storing immutable snapshots linked to each experiment for reproducibility.

Integrates with hyperparameter optimization libraries (Optuna, Ray Tune, Hyperopt) to automatically log trial configurations, metrics, and results. Provides visualization of optimization progress (parameter importance, trial history) and enables resuming optimization from previous runs by querying best parameters from Comet. Uses callback-based integration to capture optimization metadata without modifying optimization code.

Aggregates metrics and logs from distributed training runs (multi-GPU, multi-node) into a single experiment record, handling clock skew and out-of-order metric arrivals. Uses a distributed ID scheme to correlate metrics from different processes; backend aggregates metrics by timestamp and handles missing values via interpolation. Supports logging from multiple processes simultaneously without conflicts via process-safe locking.

Provides web-based dashboard for side-by-side comparison of experiments using interactive visualizations (line charts, scatter plots, parallel coordinates) that dynamically filter and aggregate metrics across runs. Backend indexes experiment metadata and metrics in a columnar store, enabling fast queries across thousands of experiments; frontend uses React with WebGL rendering for large datasets.

Centralized registry that stores trained model artifacts (weights, checkpoints, ONNX exports) with versioning, metadata tagging, and stage transitions (staging → production → archived). Uses content-addressable storage (SHA-256 hashing) to deduplicate identical model files; supports linking models to source experiments and tracking lineage through training pipeline stages.

Monitors deployed models in production by logging predictions, ground truth labels, and feature distributions; detects data drift (input distribution changes), prediction drift (output distribution changes), and performance degradation (metric decline) using statistical tests (KL divergence, Kolmogorov-Smirnov). Triggers configurable alerts via email/Slack when thresholds are exceeded, with root cause analysis linking drift to specific feature changes.

Allows logging of arbitrary custom metrics beyond standard scalars (histograms, confusion matrices, ROC curves, custom plots) via a flexible logging API that accepts JSON-serializable objects and renders them in the dashboard. Backend stores custom metrics in a document store (MongoDB-like) with schema inference; frontend renders custom visualizations using Plotly/D3.js templates.

Provides full-text and structured search across experiment metadata (parameters, metrics, tags, timestamps) using an inverted index backend that supports boolean queries, range filters, and regex matching. Enables finding experiments by parameter values, metric ranges, tags, or free-text search; results are ranked by relevance and can be sorted by any metric.

REST API and Python/JavaScript SDKs for programmatic experiment logging and querying, enabling integration with custom training scripts, notebooks, and CI/CD pipelines. API supports batch logging (multiple metrics in single request), async operations (non-blocking logging), and streaming (WebSocket-based real-time metric updates). Uses standard HTTP methods (POST for logging, GET for querying) with JSON payloads and supports pagination for large result sets.

Stores arbitrary experiment artifacts (training logs, plots, model checkpoints, datasets) as binary blobs with content-addressed storage (SHA-256) and automatic deduplication. Supports uploading files via SDK, downloading via API/web UI, and organizing artifacts into logical folders. Uses S3-compatible backend storage with configurable retention policies and automatic cleanup of old artifacts.

Enables sharing experiments, models, and dashboards with team members via role-based access control (RBAC) with granular permissions (view, edit, delete). Supports team workspaces with shared experiment history, collaborative comments on experiments, and audit logs tracking all user actions. Uses JWT tokens for API authentication and supports SSO integration (SAML, OAuth) for enterprise deployments.