SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language... (SpeechT5)

SpeechT5 implements a shared encoder-decoder architecture that processes both speech and text through a single semantic space using cross-modal vector quantization. The model uses six modal-specific pre/post-nets (speech and text variants) that interface with a unified latent representation, enabling the encoder-decoder to learn aligned representations across modalities through self-supervised pre-training on unlabeled speech and text corpora. Random mixing of speech/text states during training forces the model to develop modality-agnostic semantic understanding.

SpeechT5 performs ASR by encoding raw speech audio through the shared encoder and speech-specific pre-net, then decoding the resulting embeddings into text tokens using the shared decoder with text-specific post-net. The pre-trained cross-modal representations enable the model to recognize speech with minimal fine-tuning on labeled ASR data, leveraging the semantic alignment learned during self-supervised pre-training on unlabeled speech corpora.

SpeechT5 enables efficient fine-tuning on downstream speech tasks (ASR, TTS, translation, voice conversion, enhancement, speaker identification) by leveraging pre-trained cross-modal representations. The pre-trained encoder-decoder provides a strong initialization that captures general speech-text knowledge, allowing downstream tasks to achieve good performance with minimal labeled task-specific data. Fine-tuning typically involves adding task-specific heads or adapters while keeping most pre-trained weights frozen or using low-learning-rate updates.

SpeechT5 performs TTS by encoding text through the shared encoder and text-specific pre-net, then decoding the resulting embeddings into continuous speech waveforms using the shared decoder with speech-specific post-net. The cross-modal pre-training aligns text and speech representations, enabling the decoder to generate natural speech from text with minimal fine-tuning on labeled TTS data.

SpeechT5 performs speech translation by encoding source speech through the shared encoder and speech-specific pre-net, then decoding into target language text using the shared decoder with text-specific post-net. The cross-modal pre-training provides aligned speech-text representations that enable the model to translate speech across languages with minimal fine-tuning, effectively learning to map source speech to target text through the unified semantic space.

SpeechT5 performs voice conversion by encoding source speech through the shared encoder and speech-specific pre-net, then decoding with speaker embeddings or speaker-specific information to generate target speaker speech using the shared decoder and speech-specific post-net. The cross-modal pre-training provides robust speech representations that enable the model to separate speaker identity from linguistic content, allowing conversion of one speaker's voice to another while preserving speech content.

SpeechT5 performs speech enhancement by encoding noisy speech through the shared encoder and speech-specific pre-net to extract robust speech representations learned during cross-modal pre-training, then decoding into clean speech using the shared decoder with speech-specific post-net. The pre-trained representations provide noise-robust features that enable the model to separate speech from background noise with minimal fine-tuning on labeled noisy-clean speech pairs.

SpeechT5 performs speaker identification by encoding speech through the shared encoder and speech-specific pre-net to extract speaker-discriminative embeddings learned during cross-modal pre-training, then using these embeddings for speaker classification or verification. The pre-trained representations capture speaker characteristics while the unified architecture enables speaker identification to leverage representations learned across speech and text modalities.

SpeechT5 implements self-supervised pre-training using random mixing of speech and text latent states as the encoder-decoder interface, forcing the model to learn modality-agnostic semantic representations without labeled data. The pre-training objective uses cross-modal vector quantization to align speech and text embeddings in a shared latent space, enabling the model to learn from large unlabeled speech and text corpora and transfer these representations to downstream tasks with minimal fine-tuning.

SpeechT5 uses six modal-specific pre-nets and post-nets (three for speech, three for text) that convert between raw modality-specific representations and the unified latent space used by the shared encoder-decoder. Speech pre-nets convert raw waveforms to latent embeddings, text pre-nets convert token sequences to embeddings, and corresponding post-nets perform the reverse transformations. This architecture enables the shared encoder-decoder to operate on a unified representation while maintaining modality-specific input/output handling.

SpeechT5 uses cross-modal vector quantization as the mechanism for aligning speech and text representations in a shared latent space during pre-training. The vector quantization codebook discretizes continuous embeddings into discrete latent units, enabling the model to learn a shared vocabulary of semantic concepts that can be expressed in both speech and text modalities. Random mixing of speech/text states during training forces the model to learn representations that are invariant to modality.