Whisper Large v3

Transcribes audio in 98 languages to text in the original language using a Transformer sequence-to-sequence architecture trained on 680,000 hours of diverse internet audio. The system uses mel spectrogram feature extraction via FFmpeg integration, processes audio through an AudioEncoder that generates embeddings, then applies an autoregressive TextDecoder with task-specific tokens to produce language-native transcriptions. Language-specific models (e.g., tiny.en, base.en) optimize for English-only workloads with reduced parameter count.

Translates non-English speech directly to English text in a single forward pass using the same Transformer architecture as transcription, but with a translation task token prepended to the decoder input. The model learns to skip intermediate transcription and generate English output directly from audio embeddings, avoiding cascading errors from intermediate transcription steps. Supports 98 source languages translating to English only.

Normalizes variable-length audio to exactly 30 seconds via `whisper.pad_or_trim()`: audio shorter than 30 seconds is padded with silence (zeros) to reach 30 seconds, audio longer than 30 seconds is trimmed to first 30 seconds. This ensures consistent input shape (80×3000 mel spectrogram) for the model, avoiding shape mismatches and enabling batch processing. Padding strategy is simple zero-padding rather than sophisticated techniques like repetition or interpolation.

Returns transcription results as structured JSON objects containing: transcribed text, language code, duration, segments (with timing and text), and optional confidence metrics. The `model.transcribe()` API returns a dictionary with keys like 'text' (full transcript), 'language' (detected language), 'segments' (list of segment objects with start/end times and text). This structured format enables downstream processing (subtitle generation, database storage, API responses) without string parsing.

Detects the spoken language in audio by processing mel spectrograms through the AudioEncoder and using a language classification head that outputs probability distributions over 98 supported languages. The model leverages 680K hours of multilingual training data to recognize language characteristics from acoustic features alone, without requiring transcription. Language detection occurs as a preliminary step in the transcription pipeline and can be called independently via the language detection task token.

Provides six model variants (tiny 39M, base 74M, small 244M, medium 769M, large 1550M, turbo 809M) with different parameter counts, VRAM requirements (1-10GB), and inference speeds (10x-1x relative to large). Each size trades accuracy for speed — tiny runs ~10x faster but with ~5-10% lower WER (word error rate), while large provides best accuracy at 10GB VRAM cost. Turbo variant (809M params) optimizes large-v3 for 8x speedup with minimal accuracy loss but lacks translation support.

Automatically segments audio longer than 30 seconds into overlapping windows, processes each window independently through the transcription pipeline, and merges results with overlap handling to produce seamless full-length transcripts. The system uses `whisper.pad_or_trim()` to normalize each segment to exactly 30 seconds (padding with silence if needed), then applies the decoder to each segment and concatenates outputs while managing word-level boundaries and timestamp continuity across segment edges.

Generates precise word-level timestamps (start and end times in milliseconds) for each word in the transcript by leveraging the decoder's attention weights and token alignment information. The system maps output tokens back to audio frames using the attention mechanism, then converts frame indices to millisecond timestamps based on the mel spectrogram hop length (20ms per frame). Timestamps are returned as part of the structured output alongside transcribed text.

Converts raw audio files in multiple formats (MP3, WAV, M4A, FLAC, OGG) to mel spectrograms via a two-stage pipeline: (1) FFmpeg decodes audio to 16kHz mono PCM, (2) `whisper.log_mel_spectrogram()` applies a mel-scale filterbank (80 mel bins) and log compression to produce 80×3000 feature matrices (30 seconds at 100Hz frame rate). The mel spectrogram is the input to the AudioEncoder, making this preprocessing critical for model accuracy.

Provides `whisper.decode()` function accepting `DecodingOptions` object to control inference behavior: beam search width, temperature for sampling, language/task specification, no-speech threshold, and initial prompt. The decoder implements autoregressive token generation with optional beam search (width 1-5) for exploring multiple hypotheses, temperature-based sampling for diversity, and early stopping when no-speech probability exceeds threshold. Developers can tune these parameters per-file for accuracy-latency tradeoffs.

Automatically detects CUDA-capable GPU and moves model weights and inference computations to GPU for 30-60x speedup vs CPU. The system uses PyTorch's device management to handle GPU memory allocation, model loading to GPU via `.to('cuda')`, and batch processing of mel spectrograms on GPU. Falls back to CPU if CUDA unavailable, with transparent API (developers don't explicitly specify device).

Uses special task tokens prepended to decoder input to control model behavior: `<|transcribe|>` for speech-to-text in original language, `<|translate|>` for speech-to-English translation, and language tokens (e.g., `<|en|>`, `<|es|>`) to specify source language. The same model weights handle all three tasks (transcription, translation, language detection) by conditioning on these tokens, avoiding separate model checkpoints and enabling task switching without model reloading.