Transformer Based Audio Synthesis

via “streaming transformer inference for long-form audio”

Meta's library for music and audio generation.

Unique: Implements rolling key-value cache for transformer attention, enabling efficient incremental generation of audio chunks without reprocessing previous context. Maintains generation coherence across chunk boundaries through overlapping context windows.

vs others: Enables generation of arbitrarily long audio without memory explosion; practical for streaming applications. More efficient than regenerating full sequences for each chunk.

via “multilingual text-to-speech synthesis with transformer architecture”

text-to-speech model by undefined. 2,95,715 downloads.

Unique: Uses a unified 3B transformer encoder-decoder trained on four typologically diverse languages (English, Mandarin, German, Korean) with shared phoneme embeddings, enabling cross-lingual transfer and language-agnostic prosody modeling rather than separate language-specific models

vs others: Smaller footprint than Tacotron2-based systems (3B vs 10B+ parameters) while maintaining multilingual support, and fully open-source unlike commercial APIs (Google Cloud TTS, Azure Speech), enabling on-device deployment without vendor lock-in

via “transformer-based text-to-speech synthesis with speaker embedding control”

text-to-speech model by undefined. 1,49,878 downloads.

Unique: Separates linguistic content processing from speaker identity via explicit speaker embedding conditioning, enabling flexible multi-speaker synthesis and voice cloning without model retraining — unlike single-speaker TTS models or those requiring speaker-specific fine-tuning

vs others: More flexible than Tacotron2 for speaker control and more efficient than autoregressive models due to non-autoregressive transformer decoder, while maintaining open-source accessibility with MIT license unlike commercial APIs

via “text-to-speech synthesis”

text-to-speech model by undefined. 1,70,084 downloads.

Unique: Utilizes a transformer architecture with a focus on prosody and phonetic nuances, unlike traditional TTS systems that rely on pre-recorded audio segments.

vs others: Produces more natural-sounding speech than older concatenative systems, making it preferable for professional audio applications.

via “lightweight transformer-based post-processing compression enhancement”

* ⭐ 12/2022: [Robust Speech Recognition via Large-Scale Weak Supervision (Whisper)](https://arxiv.org/abs/2212.04356)

Unique: Applies Transformer models specifically to the quantized latent space rather than raw audio, enabling learned redundancy removal in the compressed domain. Achieves 40% additional compression while maintaining faster-than-real-time operation — a rare combination in neural codecs where compression and speed typically trade off.

vs others: Achieves better compression-to-speed ratio than applying Transformers to raw audio or using traditional entropy coding, because it operates on already-quantized representations where Transformers can learn domain-specific redundancy patterns without the computational burden of processing high-dimensional audio.

via “efficient transformer architecture optimization for audio classification”

* ⭐ 04/2022: [MAESTRO: Matched Speech Text Representations through Modality Matching (Maestro)](https://arxiv.org/abs/2204.03409)

Unique: Combines patchout augmentation with architectural optimizations (attention pruning, parameter sharing) specifically tuned for audio spectrograms, creating a holistic training pipeline that improves both sample efficiency and computational efficiency simultaneously

vs others: Outperforms standard transformer baselines on audio tasks with 30-50% fewer parameters because it jointly optimizes data augmentation and model architecture, whereas most approaches apply augmentation and compression independently

via “audio quality and fidelity optimization”

A model by Google Research for generating high-fidelity music from text descriptions.

via “transformer-based audio synthesis”