Whisper CLI

Transcribes audio in 98 languages to text using a unified Transformer sequence-to-sequence architecture with a shared AudioEncoder that processes mel spectrograms and a language-agnostic TextDecoder that generates tokens autoregressively. The system handles variable-length audio by padding or trimming to 30-second segments and uses FFmpeg for format normalization, enabling end-to-end transcription without language-specific model switching.

Translates non-English audio directly to English text by injecting a translation task token into the decoder, bypassing intermediate transcription steps. The model learns to map audio embeddings from the shared AudioEncoder directly to English token sequences, leveraging the same Transformer decoder used for transcription but with different task conditioning.

Loads audio files in any format (MP3, WAV, FLAC, OGG, OPUS, M4A) using FFmpeg, resamples to 16kHz mono, and converts to log-mel spectrogram features (80 mel bins, 25ms window, 10ms stride) for model consumption. The pipeline is implemented in whisper.load_audio() and whisper.log_mel_spectrogram(), handling format normalization and feature extraction transparently.

Detects the spoken language in audio by analyzing the audio embeddings from the AudioEncoder and using the TextDecoder to predict language tokens, returning the identified language code and confidence score. This leverages the same Transformer architecture used for transcription but extracts language predictions from the first decoded token without generating full transcription.

Generates precise word-level timestamps by tracking the decoder's attention patterns and token positions during autoregressive decoding, enabling frame-accurate alignment of transcribed text to audio. The system maps each decoded token to its corresponding audio frame through the attention mechanism, producing start/end timestamps for each word without requiring separate alignment models.

Provides six model variants (tiny, base, small, medium, large, turbo) with explicit parameter counts, VRAM requirements, and relative speed metrics to enable developers to select the optimal model for their latency/accuracy constraints. Each model is pre-trained and available for download; the system includes English-only variants (tiny.en, base.en, small.en, medium.en) for faster inference on English-only workloads, and turbo (809M params) as a speed-optimized variant of large-v3 with minimal accuracy loss.

Processes audio longer than 30 seconds by automatically segmenting into overlapping 30-second windows, transcribing each segment independently, and merging results while handling segment boundaries to maintain context. The system uses the high-level transcribe() API which internally manages segmentation, padding, and result concatenation, avoiding manual segment management and enabling end-to-end processing of hour-long audio files.

Automatically detects CUDA-capable GPUs and offloads model computation to GPU, with built-in memory management that handles model loading, activation caching, and intermediate tensor allocation. The system uses PyTorch's device placement and automatic mixed precision (AMP) to optimize memory usage, enabling inference on GPUs with limited VRAM by trading compute precision for memory efficiency.

Provides a command-line interface that accepts audio file paths or directories, processes them with configurable model selection and task specification (transcribe/translate/language-id), and outputs results in multiple formats (JSON, VTT, SRT, plain text). The CLI wraps the Python API with argument parsing, file discovery, and result formatting, enabling non-programmatic users to run Whisper without writing code.

Exposes two decoding APIs: whisper.decode() for fine-grained control over decoding parameters (beam search width, temperature, language constraints) and model.transcribe() for high-level end-to-end transcription with automatic segmentation and result formatting. The low-level API accepts DecodingOptions objects specifying task, language, and decoding strategy; the high-level API abstracts these details and handles audio loading, segmentation, and output formatting automatically.

Implements task specification through special tokens prepended to the decoder input, enabling a single model to perform transcription, translation, and language identification without model switching. The decoder interprets task tokens (e.g., <|transcribe|>, <|translate|>, <|detect_language|>) to condition its output distribution, allowing the same AudioEncoder and TextDecoder weights to serve multiple tasks by changing only the task token.