SpeechBrain

SpeechBrain uses a declarative YAML-based configuration system where all training hyperparameters, model architectures, and augmentation pipelines are defined in a single file per recipe. The Brain class accesses these via `self.hparams` namespace, and command-line arguments can override any YAML value at runtime (e.g., `--learning_rate=0.1`). This hybrid imperative-declarative approach separates configuration from training logic, enabling reproducibility and rapid experimentation without code changes.

SpeechBrain provides a `sb.Brain` base class that encapsulates the PyTorch training loop with explicit lifecycle methods: `compute_forward()` for forward pass definition, `compute_objectives()` for loss computation, and `compute_metrics()` for evaluation metrics. Developers subclass Brain and override these methods to define custom training logic, while the framework handles batching, device management, checkpointing, and validation loops. This abstraction eliminates boilerplate training code while maintaining full control over model behavior.

SpeechBrain includes recipes and pre-trained models for speech enhancement tasks like noise reduction, speech separation, and quality improvement. The framework provides models trained on noisy speech datasets that learn to suppress background noise while preserving speech quality. Enhancement can be applied as a preprocessing step before ASR or as a standalone task. Pre-trained models are available for common scenarios (office noise, street noise, etc.).

SpeechBrain provides recipes and pre-trained models for text-to-speech (TTS) synthesis, including acoustic modeling (text-to-mel-spectrogram) and vocoding (mel-spectrogram-to-waveform). The framework supports multiple TTS architectures and vocoder types, enabling end-to-end speech synthesis from text. Pre-trained models are available for multiple languages, and the framework supports fine-tuning on custom voice datasets.

SpeechBrain provides recipes for spoken language understanding (SLU) tasks that extract intents and entities directly from speech. The framework supports end-to-end SLU models that jointly perform ASR and semantic understanding, as well as pipeline approaches that apply NLU to ASR outputs. Pre-trained models and recipes are available for common SLU datasets and domains.

SpeechBrain provides recipes and models for sound event detection (identifying and localizing sounds in audio) and audio classification (categorizing audio into predefined classes). The framework supports both frame-level event detection and clip-level classification, with pre-trained models available for common sound events. Models can be fine-tuned on custom audio datasets for domain-specific classification.

SpeechBrain provides tools and recipes for multi-microphone signal processing, including beamforming for spatial filtering and microphone array processing. The framework supports various beamforming strategies (delay-and-sum, MVDR, etc.) and can be integrated into speech recognition pipelines to improve robustness in multi-microphone scenarios. Pre-trained models and recipes are available for common microphone array configurations.

SpeechBrain's Brain class provides hooks for custom loss function computation via `compute_objectives()` and custom metric computation via `compute_metrics()`. Developers can define task-specific loss functions (e.g., CTC loss for ASR, triplet loss for speaker verification) and evaluation metrics without modifying the training loop. This enables flexible optimization strategies and evaluation protocols for diverse speech tasks.

SpeechBrain's recipe system enables training by running a single command: `python train.py hparams/train.yaml`, with any YAML parameter overridable from the command line (e.g., `--learning_rate=0.1`). This pattern eliminates the need to edit YAML files for quick experiments and enables reproducible training across team members. The recipe structure (hparams/train.yaml + train.py) is standardized across all 200+ recipes, making it easy to switch between tasks.

SpeechBrain integrates feature extraction (MFCC, mel-filterbanks, etc.) and audio augmentation as composable pipeline stages accessed via `self.hparams.compute_features()` and `self.hparams.augment()`. These operations execute during the forward pass on raw waveforms, enabling dynamic augmentation strategies and avoiding pre-computation bottlenecks. The pipeline is defined declaratively in YAML, allowing researchers to swap feature extractors or augmentation strategies without modifying training code.

SpeechBrain provides 200+ pre-built recipes organized as `recipes/{dataset}/{task}/train/` directories, each containing a complete YAML hyperparameter file and training script ready to run on popular speech datasets (LibriSpeech, VoxCeleb, etc.). Each recipe encodes best-practice hyperparameters, model architectures, and augmentation strategies for that specific dataset-task combination, enabling researchers to reproduce published results or bootstrap new projects without manual hyperparameter tuning.

SpeechBrain integrates with HuggingFace Hub to distribute and load pre-trained models for tasks like transcription, speaker verification, speech enhancement, and source separation. Models are versioned, documented, and accessible via a unified interface, eliminating manual model hosting and enabling one-line model loading in inference scripts. The integration handles model downloading, caching, and version management automatically.

SpeechBrain provides a `self.modules` namespace where custom neural network components (encoders, decoders, attention layers, etc.) are registered and accessed during training. Components are instantiated from YAML configuration and composed together in the `compute_forward()` method, enabling flexible model architecture definition without hardcoding layer connections. This pattern separates model definition (YAML) from training logic (Python), making it easy to swap components or build ensemble models.

SpeechBrain's Brain class handles batching, device placement (CPU/GPU), and gradient updates automatically. Developers define batch structure in YAML (batch size, number of workers, etc.) and the framework orchestrates data loading, moving tensors to the correct device, and synchronizing gradients across batches. This eliminates manual device management code and enables seamless training on CPU or GPU without code changes.

SpeechBrain automatically saves model checkpoints during training and supports resuming from saved checkpoints. The Brain class manages checkpoint state (model weights, optimizer state, training step counter) and persists it to disk at configurable intervals. Developers can resume training from any checkpoint by specifying the checkpoint path, enabling long-running training jobs to survive interruptions and enabling hyperparameter search across multiple training runs.

SpeechBrain supports integration of language models (from basic n-gram LMs to modern large language models) into speech recognition pipelines for improved decoding. Language models can be used to rescore ASR hypotheses, guide beam search, or generate contextual predictions. The framework provides interfaces for loading pre-trained LMs and composing them with acoustic models, enabling end-to-end speech understanding systems.

SpeechBrain provides pre-built recipes and models for speaker verification (authentication) and speaker embedding extraction. The framework includes speaker embedding models trained on large speaker datasets (VoxCeleb, etc.) that map variable-length speech to fixed-size embeddings. These embeddings can be used for speaker identification, verification (1:1 matching), or speaker clustering. Pre-trained models are available via HuggingFace Hub for immediate use.