Fireworks AI

Provides on-demand inference across 40+ text generation models (DeepSeek, Kimi, GLM, Qwen, Mixtral, DBRX, Gemma) via a unified REST API with per-token billing. Models are pre-optimized and globally distributed with zero cold starts; requests are routed to the nearest inference cluster and billed only for input and output tokens consumed, with 50% discounts on cached input tokens. Supports context windows up to 262,144 tokens and handles streaming responses for real-time output.

Enables structured tool invocation across supported models via OpenAI-compatible function calling API. Developers define tool schemas (name, description, parameters) in JSON; the model receives the schema, reasons about which tool to call, and returns structured function calls with arguments. Fireworks handles schema validation and supports parallel function calling (multiple tools invoked in a single response). Works with DeepSeek, Kimi, GLM, Qwen, and other models that support tool-use.

Processes inference requests asynchronously in batches with 50% cost reduction vs. serverless pricing. Supports text generation and speech-to-text (STT batch API has 40% discount). Ideal for non-urgent workloads (document processing, bulk transcription, batch classification). Requests are queued and processed when resources are available; results are retrieved via polling or webhook (webhook support not documented). Reduces costs significantly for high-volume, latency-tolerant applications.

Provides access to DeepSeek R1, a reasoning-focused model that performs chain-of-thought reasoning before generating answers. The model explicitly shows its reasoning process, making it suitable for complex problem-solving, math, code generation, and multi-step reasoning tasks. Pricing and context window not documented. Reasoning models are slower than standard models due to extended thinking; latency tradeoff is not quantified.

Provides a unified REST API and SDK that abstracts away differences between multiple LLM providers (OpenAI, Anthropic, open-source models). Developers write code once and can switch between providers or models without changing application logic. Supports the same function calling, structured output, and streaming interfaces across all providers. Enables A/B testing different models and providers without code refactoring.

Claims 'globally distributed virtual cloud infrastructure' with 'no cold starts' for serverless inference, implying models are pre-loaded across multiple geographic regions. Specific regions not documented. Cold-start elimination suggests persistent model loading or aggressive caching, but implementation details unknown. Latency claims ('industry-leading throughput and latency') unquantified. Distributed infrastructure presumably enables geographic load balancing and reduced latency for global users.

Constrains model output to valid JSON or custom grammar formats without post-processing. JSON mode forces the model to generate only valid JSON matching a provided schema; grammar mode uses GBNF (GBNF format) to define arbitrary output structures (e.g., YAML, custom DSLs). Both modes prevent invalid output at generation time by restricting token selection during decoding, eliminating the need for output parsing or validation.

Provides image understanding and document analysis via vision-capable models (Kimi K2.5/K2.6, GLM-5/5.1, Qwen3 VL 30B) with context windows up to 262,144 tokens. Supports multiple images per request, OCR-like document analysis, and reasoning over visual content. Images are encoded as base64 or URLs; the model processes them alongside text prompts and returns text descriptions, extracted data, or answers to visual questions.

Transcribes audio to text using Whisper V3 Large or Whisper V3 Large Turbo models. Supports diarization (speaker identification) with a 40% cost surcharge. Offers two pricing tiers: serverless (per-minute billing) and batch API (40% discount, async processing). Audio is sent as file upload or URL; output includes transcription text and optional speaker labels. Batch API processes multiple audio files asynchronously, ideal for high-volume transcription.

Generates images from text prompts using FLUX.1 (dev, schnell, Kontext Pro/Max) and SDXL models. Pricing is per-inference-step (SDXL ~30 steps, FLUX dev ~28 steps, FLUX schnell ~4 steps) or flat-rate per image (Kontext variants). Supports prompt engineering, negative prompts, and seed control for reproducibility. Requests are processed asynchronously; output is a URL to the generated image.

Generates dense vector embeddings for text using models up to 350M parameters (e.g., Qwen3 8B). Embeddings are fixed-dimensional vectors (dimension size not documented) suitable for semantic search, clustering, and similarity comparison. Supports batch embedding of multiple texts in a single request. Embeddings can be stored in vector databases (Pinecone, Weaviate, etc.) for retrieval-augmented generation (RAG) or recommendation systems.

Enables fine-tuning of open-source models (Llama, Mixtral, etc.) using supervised fine-tuning (SFT) or direct preference optimization (DPO). Supports both LoRA (parameter-efficient) and full-parameter fine-tuning. Fine-tuned models are immediately deployable on Fireworks' serverless or on-demand infrastructure at the same price as base models. Training is managed (no GPU provisioning required); pricing is per 1M training tokens, with separate costs for LoRA vs. full-parameter methods.

Allows deployment of custom models or base models on dedicated GPU infrastructure with auto-scaling. Billing is per GPU-second (exact rates not documented). Deployments support custom Docker containers, enabling arbitrary model architectures or inference code. Auto-scaling adjusts GPU count based on traffic; minimal cold starts (faster than serverless but slower than pre-warmed). Suitable for high-throughput, latency-sensitive applications.

Caches repeated input tokens (system prompts, context, documents) and charges only 50% of the base input token price for cached tokens on subsequent requests. Caching is automatic for identical token sequences; no explicit cache management required. Ideal for RAG systems, multi-turn conversations, or applications with large static context (e.g., system prompts, knowledge bases). Reduces both latency and cost for repeated queries.