Replicate

Execute any of thousands of hosted ML models through a stateless HTTP API with granular time-based billing. Requests are routed to shared or dedicated hardware pools depending on model type, with automatic queue management and scaling. The platform abstracts away container orchestration, GPU allocation, and billing calculation—developers submit input, receive output, and pay only for compute seconds consumed.

A public marketplace hosting thousands of community-contributed ML models alongside official models from creators like Meta, Google, and OpenAI. Each model displays total run counts, creator attribution, and hardware requirements. The registry is searchable and filterable by model type (image generation, LLM, video, etc.), enabling developers to discover and compare models before deployment.

Create organizations to manage team access, billing, and model deployments. Members can be assigned roles (admin, member, viewer) with granular permissions for creating models, managing billing, and accessing predictions. Organizations enable shared billing, centralized credential management, and audit trails for team activities.

Automatically build and deploy Cog-based models to Replicate when code is pushed to GitHub. A GitHub Action monitors the repository, runs Cog build, pushes the resulting image to Replicate's registry, and updates the deployed model. Developers define deployment workflows in .github/workflows/deploy.yml, enabling GitOps-style model deployments with version control and audit trails.

Train custom image generation models by fine-tuning base models (e.g., Flux, Stable Diffusion) on user-provided datasets. Replicate handles data preprocessing, training orchestration, and model packaging. Developers can also upload pre-trained LoRA (Low-Rank Adaptation) weights to customize model behavior without full fine-tuning. Fine-tuned models are deployed as private endpoints with dedicated hardware.

Replicate retains prediction inputs, outputs, and metadata for a configurable period, accessible via the API and dashboard. Developers can query prediction history, export results, and configure retention policies (e.g., delete after 30 days). This enables audit trails, debugging, and compliance with data retention regulations.

Expose Replicate models as tools within the Model Context Protocol (MCP) framework, enabling AI agents and LLMs to invoke models as part of multi-step reasoning. The MCP server translates agent tool calls into Replicate API invocations, handles streaming responses, and returns results to the agent. This enables agents to use image generation, video, or other models as composable building blocks.

Enforce per-user and per-organization rate limits to prevent abuse and manage resource consumption. Developers can configure request limits (e.g., 100 requests/minute), burst allowances, and quota thresholds. Rate limit headers in API responses indicate remaining capacity, enabling clients to implement backoff strategies. Exceeding limits returns HTTP 429 (Too Many Requests) with retry-after guidance.

Package custom ML models (PyTorch, TensorFlow, Transformers, Diffusers) into Cog containers—a standardized format that abstracts GPU setup, dependency management, and API exposure. Developers define model inputs/outputs in YAML, write Python prediction code, and push to Replicate via GitHub Actions or CLI. Cog handles container building, registry management, and auto-scaling on Replicate's infrastructure.

Return model outputs incrementally as they are generated, rather than waiting for full completion. Implemented via HTTP streaming (chunked transfer encoding) or WebSocket connections, enabling real-time feedback for text generation, video frame-by-frame output, or progressive image rendering. Clients receive partial results immediately, reducing perceived latency and enabling interactive UX patterns.

Submit long-running predictions asynchronously and receive HTTP POST callbacks when results are ready. Replicate signs webhooks with HMAC-SHA256 and includes prediction metadata (status, output, error details) in the payload. Developers can verify webhook authenticity, retry failed deliveries, and decouple prediction submission from result handling—enabling background job patterns and decoupled microservices.

Automatically select and scale hardware based on model requirements and traffic. Public models run on shared hardware pools (CPU, A100, H100) with dynamic allocation; private models can be pinned to dedicated hardware (always-on) or use fast-booting fine-tunes (pay-per-use). Replicate's orchestration layer monitors queue depth and scales instances up/down to meet demand, abstracting capacity planning from developers.

Tag and version model deployments using semantic versioning (e.g., creator/model:v1.0), enabling reproducible inference and A/B testing across versions. Each version pins specific model weights, code, and dependencies, ensuring consistent outputs over time. Developers can reference specific versions in API calls, and Replicate maintains version history for rollback or comparison.

Alternative to time-based billing for models where output size is predictable. LLMs (Claude 3.7-Sonnet, DeepSeek-R1) charge per input/output token; image models (Flux 1.1-Pro, Ideogram) charge per output image; video models charge per second of output video. This enables cost predictability for high-volume applications and aligns pricing with actual resource consumption rather than wall-clock time.

Built-in safety checks flag potentially harmful outputs (NSFW content, violence, hate speech) before returning results to users. Implemented as a post-processing step on model outputs, with configurable thresholds and filtering policies. Developers can enable/disable safety checks per prediction and receive metadata indicating which safety rules were triggered.

Store API keys, authentication tokens, and other sensitive credentials as encrypted secrets within Replicate, accessible to custom models at runtime via environment variables. Secrets are scoped to models or organizations and never logged or exposed in prediction outputs. Developers define secrets in the Replicate dashboard or via API, and Cog-based models reference them as standard environment variables.