whisperX

WhisperX achieves sub-second word-level timestamp precision by performing forced alignment using wav2vec2 acoustic models after ASR transcription. The system extracts phoneme sequences from the transcribed text, aligns them against the audio's acoustic features using dynamic time warping or similar alignment algorithms, and produces precise start/end timestamps for each word. This two-stage approach (ASR → alignment) decouples transcription quality from timestamp accuracy, enabling accurate timing even when Whisper's native utterance-level timestamps drift by seconds.

WhisperX implements batched transcription using faster-whisper (CTranslate2 backend) instead of OpenAI's sequential Whisper API, enabling parallel processing of multiple audio segments. The system performs VAD-based segmentation to identify speech regions, groups segments into batches, and processes them in a single forward pass through the model. This architecture reduces GPU memory footprint to <8GB for large-v2 model (vs 10-11GB for sequential Whisper) while achieving 70x realtime transcription speed by eliminating per-segment model loading overhead and leveraging CTranslate2's quantization and kernel optimizations.

WhisperX provides confidence scores for each transcribed segment, indicating the model's certainty in the transcription. These scores are derived from Whisper's logit outputs during decoding and reflect the probability of the predicted token sequence. Confidence scores are attached to each segment in the output, enabling downstream applications to filter low-confidence segments or flag them for manual review. Additionally, WhisperX can compute Word Error Rate (WER) if reference transcriptions are available, providing quantitative quality metrics for evaluation and benchmarking.

WhisperX supports multiple Whisper model sizes (tiny, base, small, medium, large) and enables users to specify custom model paths or Hugging Face model IDs. The system loads models on-demand and caches them locally to avoid repeated downloads. For alignment and diarization stages, users can specify alternative wav2vec2 or pyannote models, enabling experimentation with different model variants. Model selection is configurable via CLI flags or Python API parameters, and the system validates model compatibility before loading. This flexibility enables users to trade off accuracy vs speed/memory based on their constraints.

WhisperX integrates pyannote-audio's speaker diarization models to identify and label distinct speakers in multi-speaker audio. The system performs speaker embedding extraction on speech segments, clusters embeddings using agglomerative clustering, and assigns speaker IDs (speaker_0, speaker_1, etc.) to each transcribed segment. The diarization stage runs after ASR and alignment, enriching each word-level timestamp with speaker attribution. This enables downstream applications to track who said what and when, with speaker labels propagated through the entire transcript hierarchy.

WhisperX uses voice activity detection (VAD) to identify speech regions in audio before ASR, segmenting the audio into speech-only chunks. The VAD stage runs before transcription and filters out silence, background noise, and non-speech regions, reducing the input to the ASR model. This preprocessing step enables two benefits: (1) reduces hallucination artifacts where Whisper generates spurious text during silence, and (2) enables efficient batching by providing natural segment boundaries. The VAD model (typically Silero VAD or similar) produces confidence scores and segment timestamps that guide the ASR batching strategy.

WhisperX supports transcription in 99+ languages using Whisper's multilingual model, with automatic language detection via Whisper's encoder. The system detects the language from the first 30 seconds of audio by analyzing the acoustic features and comparing against language-specific phoneme distributions. Once detected, the appropriate language-specific tokenizer and decoder are loaded, and transcription proceeds with language-aware beam search. The language detection is automatic but can be overridden via configuration, enabling forced transcription in a specific language if detection fails.

WhisperX provides a comprehensive CLI that orchestrates the entire transcription pipeline (VAD → ASR → alignment → diarization) with a single command. The CLI accepts audio file paths or directories, applies configuration flags for model selection, language, speaker count, and output format, and produces structured output files (JSON, VTT, SRT, TSV). The CLI manages model lifecycle (loading, caching, unloading) and memory optimization automatically, enabling non-technical users to run complex multi-stage pipelines without writing code. Output can be written to multiple formats simultaneously, supporting downstream integrations with video editors, subtitle tools, and analytics platforms.

WhisperX exposes a Python API that enables fine-grained control over each pipeline stage (VAD, ASR, alignment, diarization) with conditional execution and custom model loading. The API provides a `load_model()` function to load ASR, alignment, and diarization models with configurable device placement and precision (FP32, FP16, INT8), and a `transcribe()` function that orchestrates the pipeline with optional stage skipping. Users can access intermediate outputs (VAD segments, raw ASR results, aligned timestamps, speaker labels) at each stage, enabling custom post-processing or integration with external systems. The API manages model caching and memory cleanup automatically, preventing GPU memory leaks in long-running applications.

WhisperX automatically handles audio preprocessing including format detection, resampling, and channel conversion. The system accepts audio in multiple formats (MP3, WAV, M4A, FLAC, OGG) and automatically resamples to 16kHz mono (Whisper's native sample rate) using librosa or ffmpeg. The preprocessing stage detects audio duration, validates sample rate, and handles edge cases like stereo-to-mono conversion with channel mixing. This preprocessing is transparent to users and runs before VAD, ensuring consistent input to downstream stages regardless of source audio characteristics.

WhisperX generates transcription output in multiple formats (JSON, VTT, SRT, TSV) from a single transcription run, enabling direct integration with video editors, subtitle tools, and analytics platforms. Each format preserves word-level timestamps and speaker labels where applicable. JSON output includes full metadata (confidence scores, segment boundaries, speaker IDs), while VTT and SRT formats are optimized for video players and subtitle editors. TSV format enables import into spreadsheet applications for manual review and editing. The output generation is decoupled from transcription, allowing users to regenerate outputs in different formats without re-running the pipeline.

WhisperX optimizes GPU memory usage through INT8 quantization (via CTranslate2) and selective model loading, reducing the large-v2 model footprint from 10-11GB to <8GB. The system loads only the required models for the enabled pipeline stages (ASR, alignment, diarization can be independently enabled/disabled), and unloads models after use to prevent memory accumulation in long-running applications. CTranslate2's quantization reduces model weights to 8-bit integers while maintaining accuracy, and the batching strategy ensures efficient GPU utilization by processing multiple segments per forward pass. Memory profiling is built-in, enabling users to monitor GPU usage and identify bottlenecks.