Real Time Mel Spectrogram Generation With Attention Based Alignment

via “mel spectrogram generation from discrete audio tokens”

A generative speech model for daily dialogue.

Unique: Uses a DVAE (Discrete Variational Autoencoder) rather than a simple lookup table or continuous decoder, enabling learned, high-quality reconstruction of spectrograms from discrete tokens. The DVAE is trained end-to-end with the audio codec, ensuring that discrete tokens capture all information needed for high-fidelity spectrogram reconstruction.

vs others: More flexible than fixed codebooks because the DVAE decoder learns to reconstruct spectrograms from tokens, enabling better quality and smoother transitions between tokens. More efficient than storing spectrograms directly because discrete tokens are more compact and enable better generalization across speakers and content.