SpeechBrain

Users extend a base `Brain` class and override task-specific methods (`compute_forward()`, `compute_objectives()`, `compute_metrics()`) to implement custom speech processing pipelines. The framework orchestrates the training loop, gradient updates, and checkpoint management automatically. This pattern decouples model architecture from training orchestration, similar to PyTorch Lightning's LightningModule but specialized for speech tasks with built-in audio feature computation and augmentation hooks.

All training hyperparameters (learning rate, batch size, model architecture, augmentation strategies, feature extractors) are defined in a single YAML file per recipe. Parameters can be overridden at runtime via CLI flags (e.g., `python train.py hparams/train.yaml --learning_rate=0.001 --batch_size=32`) without modifying code. The framework loads YAML into a `hparams` object accessible throughout the Brain instance, enabling reproducible experiments and easy hyperparameter sweeps.

SpeechBrain provides speech separation models that isolate individual speakers from multi-speaker audio (cocktail party problem). Models are trained to estimate time-frequency masks or speaker-specific spectrograms from mixed audio. The framework includes pre-trained separation models and recipes for training on multi-speaker datasets. Users can separate speakers as a preprocessing step before ASR or speaker verification, or as a standalone application. The framework handles feature extraction and waveform reconstruction automatically.

SpeechBrain provides end-to-end SLU models that convert speech to structured semantic representations (intent + slots). Models combine ASR (speech-to-text) with NLU (intent/slot extraction) in a single neural network, avoiding cascading errors from separate ASR and NLU systems. The framework includes pre-trained SLU models and recipes for training on SLU datasets (ATIS, SNIPS, etc.). Users can fine-tune models on custom intents/slots or train from scratch on new datasets.

SpeechBrain provides sound event detection models that identify and classify acoustic events (e.g., dog barking, car horn, speech) in audio. Models are trained to predict event labels and timestamps from audio spectrograms. The framework includes pre-trained models for common sound events and recipes for training on sound event datasets (ESC-50, AudioSet, etc.). Users can detect events in continuous audio streams or classify individual audio clips. The framework handles feature extraction and event localization automatically.

SpeechBrain provides multi-microphone signal processing capabilities including beamforming (MVDR, superdirective) and source localization (direction of arrival estimation). The framework handles multi-channel audio input and applies beamforming to enhance speech from a target direction while suppressing noise and interference. Users can specify target direction or estimate it automatically. The framework integrates beamforming with downstream tasks (ASR, speaker verification) to improve performance on multi-microphone arrays.

SpeechBrain provides built-in metric computation for speech tasks including word error rate (WER) for ASR, equal error rate (EER) for speaker verification, mel-cepstral distortion (MCD) for TTS, and others. Metrics are computed automatically during training and evaluation via the `compute_metrics()` method in the Brain class. The framework handles metric aggregation across batches and epochs, and logs metrics to training logs. Users can define custom metrics by overriding the `compute_metrics()` method.

SpeechBrain automatically saves model checkpoints during training and enables resuming training from saved checkpoints. The framework saves model weights, optimizer state, and training metadata (epoch, step) to enable exact resumption. Users can specify checkpoint frequency and retention policy via YAML configuration. The framework handles checkpoint loading and state restoration automatically, allowing training to resume without code changes. Checkpoints include all information needed for inference and fine-tuning.

SpeechBrain's recipe system enables training by running a single command: `python train.py hparams/train.yaml`, with any YAML parameter overridable from the command line (e.g., `--learning_rate=0.1`). This pattern eliminates the need to edit YAML files for quick experiments and enables reproducible training across team members. The recipe structure (hparams/train.yaml + train.py) is standardized across all 200+ recipes, making it easy to switch between tasks.

Custom neural network components are registered in a `self.modules` dictionary within the Brain instance, allowing composition of complex models from reusable pieces. Each module is a standard PyTorch `nn.Module` that can be accessed and executed within the `compute_forward()` method (e.g., `output = self.modules.encoder(features)`). This pattern enables mixing pre-built components (provided by SpeechBrain) with custom layers while maintaining a clean, declarative model definition.

Audio features (MFCC, mel-filterbank energies, spectrograms) and augmentations (SpecAugment, time-stretching, pitch-shifting) are defined declaratively in YAML and applied on-the-fly during training via `self.hparams.compute_features(batch.wavs)` and `self.hparams.augment(features)`. The framework computes features in batches on GPU when available, avoiding pre-computation bottlenecks. Augmentations are applied stochastically during training and disabled during validation, with no additional code required.

SpeechBrain integrates with HuggingFace Model Hub to download pre-trained models (ASR, speaker verification, TTS, etc.) with a single function call. Models are cached locally and automatically loaded with their associated hyperparameters and tokenizers. Users can fine-tune pre-trained models by loading them into a custom Brain subclass and training on new data, with the framework handling gradient updates and checkpoint management. The integration includes automatic model versioning and reproducibility tracking.

SpeechBrain provides 200+ pre-built recipes organized by dataset and task (e.g., `recipes/LibriSpeech/ASR/train/`), each containing a `train.py` script and `hparams/train.yaml` configuration. Users can clone a recipe, modify hyperparameters in YAML, and run `python train.py hparams/train.yaml` to train on that dataset. Recipes include data loading, preprocessing, and evaluation scripts tailored to each dataset, eliminating the need to write custom data loaders or evaluation code.

SpeechBrain provides end-to-end ASR models (acoustic encoder + CTC/attention decoder) with optional integration of n-gram or neural language models for beam search decoding. Language models can be trained separately and loaded during inference to improve word error rate. The framework handles tokenization, decoding, and language model scoring automatically. Users can swap language models without retraining the acoustic model, enabling easy experimentation with different LM architectures.

SpeechBrain provides speaker verification models that extract speaker embeddings (d-vectors or x-vectors) from audio and compare them using cosine similarity or other distance metrics. The framework includes pre-trained speaker encoders trained on large speaker datasets (VoxCeleb, etc.). Users can extract embeddings from new speakers, build speaker databases, and perform 1-to-1 verification or 1-to-N identification. The framework handles feature extraction, embedding normalization, and similarity scoring automatically.

SpeechBrain provides end-to-end TTS models that convert text to mel-spectrograms (via Tacotron2, Glow-TTS, or similar) and neural vocoders (HiFi-GAN, WaveGlow) that convert spectrograms to waveforms. The framework handles text tokenization, phoneme conversion, and mel-spectrogram generation automatically. Users can train custom TTS models on new datasets or use pre-trained models for inference. The framework supports multiple speaker TTS by conditioning on speaker embeddings.

SpeechBrain provides speech enhancement models that suppress background noise, reverberation, and other artifacts from audio. Models are trained to estimate clean speech spectrograms or time-domain waveforms from noisy input. The framework includes pre-trained enhancement models and recipes for training on noisy datasets. Users can apply enhancement as a preprocessing step before ASR or other downstream tasks, or as a standalone application. The framework handles feature extraction and waveform reconstruction automatically.