Whisper

Converts audio in 99+ languages to text using a transformer-based encoder-decoder architecture trained on 680,000 hours of multilingual and multitask supervised data from the web. The model learns from weak supervision (noisy labels from automatic captions) rather than hand-annotated data, enabling robust generalization across accents, background noise, technical language, and low-resource languages without language-specific fine-tuning.

Automatically detects the spoken language in audio segments using the same transformer encoder that processes speech, outputting ISO 639-1 language codes with confidence scores. The model learns language identification as a multitask objective during training, enabling detection of code-switching and mixed-language segments without separate language classifiers.

Outputs transcription with word-level or segment-level timestamps by decoding the audio in overlapping chunks and aligning predicted tokens to their temporal positions in the spectrogram. The model generates timestamps as special tokens during decoding, enabling precise alignment without post-hoc forced alignment algorithms.

Provides open-source model weights in multiple sizes (tiny, base, small, medium, large) ranging from 39M to 1.5B parameters, with support for quantization (int8, fp16) and ONNX export for optimized inference on CPU, GPU, and edge devices. The base implementation uses PyTorch with automatic mixed precision, and community implementations provide TensorRT, CoreML, and WebAssembly variants for deployment flexibility.

Supports task tokens (transcribe, translate) and optional prompt text during decoding to guide model behavior, enabling conditional generation of translations, punctuation/capitalization correction, and style adaptation. The model learns to condition on task tokens and prompt prefixes during training, allowing zero-shot adaptation to new tasks without fine-tuning.

Achieves low word error rates on audio with background noise, accents, and technical jargon due to training on 680,000 hours of diverse web audio with weak supervision. The model learns robust acoustic representations that generalize across speaker variation, environmental noise, and non-standard pronunciations without explicit noise robustness training or data augmentation.

OpenAI-hosted API endpoint that accepts audio files via HTTP multipart upload and returns transcription results synchronously or asynchronously. The API handles audio preprocessing, model inference, and result formatting server-side, with support for batch processing and webhook callbacks for long-running jobs.