High Fidelity Neural Audio Compression (EnCodec)

Encodes raw audio (24 kHz mono or 48 kHz stereo) into a compressed quantized latent space using a streaming encoder-decoder architecture trained end-to-end with adversarial loss. The encoder progressively downsamples audio while maintaining temporal coherence, outputting discrete codes that can be transmitted or stored at variable bitrates. Decoding reconstructs high-fidelity audio from these codes in real-time, with latency suitable for interactive applications.

Applies lightweight Transformer models as a post-processing stage after the base encoder-decoder to achieve up to 40% additional compression without sacrificing reconstruction quality. These Transformers operate on the quantized latent codes, learning to predict and remove redundancy in the compressed representation. The approach trades some computational cost for improved compression efficiency, enabling faster-than-real-time operation on standard hardware.

Evaluates codec performance across multiple audio domains (speech, noisy-reverberant speech, music) using MUSHRA (MUltiple Stimuli with Hidden Reference and Anchor) methodology, which produces Mean Opinion Scores (MOS) reflecting human perception of audio quality. The evaluation framework systematically tests codec performance at different bandwidth settings and audio domains, enabling comparative assessment against baseline methods and identification of domain-specific quality trade-offs.

Trains the encoder-decoder using adversarial loss with a single multiscale spectrogram discriminator that evaluates reconstructed audio quality at multiple frequency scales simultaneously. This replaces traditional multi-discriminator approaches with a more efficient single-discriminator architecture that examines spectral content across different time-frequency resolutions, enabling the encoder-decoder to learn perceptually-aligned compression without explicit perceptual loss functions.

Implements a novel loss balancer mechanism that decouples loss weight from loss scale during training, enabling stable multi-objective optimization of the encoder-decoder. Rather than directly weighting losses by their magnitude, the balancer defines weights as fractions of overall gradient representation, allowing different loss components (reconstruction, adversarial, perceptual) to contribute proportionally to gradient updates regardless of their absolute scale. This prevents large-magnitude losses from dominating training dynamics.

Supports encoding and decoding audio at multiple bandwidth settings, enabling variable bitrate compression where the same model can operate at different compression levels. The codec learns to gracefully degrade quality as bandwidth decreases, with performance evaluated across the full bandwidth range. This allows applications to dynamically adjust bitrate based on network conditions or storage constraints without requiring separate models.

Implements a streaming encoder-decoder architecture designed for real-time audio processing with minimal latency, enabling the codec to process audio samples incrementally without buffering entire segments. The encoder progressively downsamples audio while maintaining temporal coherence, and the decoder reconstructs audio from compressed codes with latency suitable for interactive applications. The base model operates in real-time, while the Transformer variant achieves faster-than-real-time performance.