AudioLM: a Language Modeling Approach to Audio Generation (AudioLM)

Converts raw audio waveforms into discrete token sequences using a hybrid scheme combining masked language model activations (for long-term coherence and semantic structure) with neural audio codec codes (for acoustic fidelity). This dual-stream tokenization enables the language model to capture both structural continuity and high-quality synthesis, avoiding the quality degradation that occurs when using either codec tokens or LM tokens alone. The tokenization process discretizes continuous audio representations into a vocabulary suitable for autoregressive language modeling.

Generates coherent audio continuations by treating audio generation as a language modeling task: tokenizes a short audio prompt using the hybrid scheme, then autoregressively samples tokens from a transformer-based language model conditioned on the prompt tokens, finally decoding the generated token sequence back to raw waveform. The model learns to predict statistically plausible next tokens given preceding context, enabling it to extend audio with natural prosody, speaker consistency, and structural coherence without requiring transcripts or symbolic representations.

Maintains consistent speaker identity when generating audio continuations for speakers not seen during training, achieved through the language model's learned ability to capture speaker-specific acoustic patterns in the token sequence. The hybrid tokenization preserves speaker characteristics in both the masked LM tokens (which encode prosodic and structural patterns) and codec tokens (which encode acoustic timbre), allowing the model to implicitly learn speaker embeddings without explicit speaker conditioning or speaker ID inputs.

Generates speech continuations that preserve and extend the prosodic characteristics (intonation patterns, rhythm, stress, and timing) of the input prompt by learning prosodic patterns implicitly through the language model's token predictions. The masked LM tokens capture long-term prosodic structure (sentence-level intonation contours, stress patterns), while codec tokens preserve fine-grained acoustic prosody (pitch trajectories, duration variations). The autoregressive generation process naturally extends these prosodic patterns into the continuation.

Generates coherent piano music continuations from short audio prompts by applying the same language modeling approach used for speech, but trained on piano music audio without requiring MIDI, sheet music, or symbolic notation. The model learns musical structure (harmony, melody, rhythm, phrasing) directly from raw waveforms, discovering patterns in the acoustic signal that correspond to musical concepts. Generation proceeds autoregressively by predicting next tokens conditioned on the prompt, producing audio that maintains harmonic consistency and musical coherence.

Maintains long-term coherence and semantic plausibility in generated audio by leveraging a masked language model pre-trained on audio, which learns to predict missing tokens in the middle of sequences. This pre-training objective forces the model to understand long-range dependencies and global structure in audio, enabling it to generate continuations that are not just locally plausible but globally coherent. The masked LM tokens in the hybrid representation explicitly encode this long-range structure, which the autoregressive generation process extends naturally.

Generates audio without requiring transcripts, phonetic annotations, or any text-based metadata, operating entirely on raw waveforms. This eliminates the annotation bottleneck present in text-to-speech and phoneme-based systems, allowing the model to learn directly from unlabeled audio corpora. The language model operates on discrete audio tokens that implicitly encode linguistic and acoustic information, discovering phonetic and linguistic structure without explicit supervision.

Processes audio entirely in raw waveform form without converting to spectrograms, mel-frequency cepstral coefficients (MFCCs), or other intermediate acoustic features. The tokenization step converts raw waveforms directly to discrete tokens, and generation produces raw waveforms directly from tokens, avoiding information loss and artifacts introduced by intermediate representations. This end-to-end approach preserves fine-grained acoustic details and enables the model to learn directly from the raw signal.