TTS

Converts text input to natural-sounding speech across 1100+ languages using a unified TTS API that abstracts model selection, text processing, and vocoder execution. The system loads pre-trained model weights and configurations from a centralized catalog (.models.json), applies language-specific text normalization, generates mel-spectrograms via the selected TTS model (VITS, Tacotron2, GlowTTS, etc.), and converts spectrograms to audio waveforms using neural vocoders. The Synthesizer class orchestrates this pipeline, handling sentence segmentation, speaker/language routing, and audio post-processing in a single inference call.

Generates speech in specific speaker voices by routing speaker IDs or speaker embeddings through multi-speaker TTS models (VITS, Tacotron2) that were trained on datasets with multiple speakers. The system maintains speaker metadata in model configurations, validates speaker IDs at inference time, and passes speaker embeddings or speaker conditioning vectors to the model's speaker encoder layers. For models without pre-trained speaker support, the framework provides a Speaker Encoder training pipeline to learn speaker embeddings from custom voice data, enabling zero-shot speaker adaptation.

Uses YAML configuration files to define model architectures, training hyperparameters, and dataset specifications, decoupling configuration from code and enabling reproducible experiments without code changes. Each model architecture (Tacotron2, VITS, GlowTTS, etc.) has a corresponding config class (e.g., Tacotron2Config) that loads YAML files and validates parameters. Training scripts read configuration files to instantiate models, create data loaders, and configure optimizers and learning rate schedules. This approach allows users to experiment with different hyperparameters, model architectures, and datasets by modifying YAML files rather than editing Python code, improving reproducibility and reducing the barrier to entry for non-programmers.

Orchestrates the inference pipeline by automatically composing TTS models with compatible vocoders, handling text processing, spectrogram generation, and waveform synthesis in a single call. The Synthesizer class manages the pipeline: it loads the TTS model and its paired vocoder from configuration, applies text normalization and sentence segmentation, runs the TTS model to generate mel-spectrograms, applies vocoder-specific normalization, runs the vocoder to generate waveforms, and optionally applies post-processing (silence trimming, loudness normalization). The system validates model compatibility (e.g., spectrogram dimensions match between TTS and vocoder) and provides clear error messages if incompatible models are paired.

Maintains a centralized model catalog (.models.json) containing metadata for 100+ pre-trained TTS and vocoder models, enabling users to list available models, query by language/architecture/dataset, and automatically download model weights and configurations from remote repositories. The ModelManager class handles HTTP-based model fetching, local caching, configuration path updates, and version management. When a user requests a model by name, the system looks up the model in the catalog, downloads weights if not cached locally, and loads the configuration YAML file that specifies model architecture, hyperparameters, and vocoder pairing.

Preprocesses raw text input by applying language-specific text normalization (expanding abbreviations, converting numbers to words, handling punctuation) and splitting text into sentences to manage synthesis latency and memory usage. The system uses language-specific text processors (defined in TTS/tts/utils/text/) that handle character sets, phoneme conversion, and linguistic rules for each language. Sentence segmentation uses regex-based splitting with language-aware punctuation rules, preventing incorrect splits on abbreviations or decimal numbers. This preprocessing ensures consistent phoneme generation and prevents out-of-memory errors on very long texts.

Converts mel-spectrogram outputs from TTS models into high-quality audio waveforms using neural vocoder models (HiFi-GAN, Glow-TTS vocoder, WaveGlow). The vocoder inference pipeline takes spectrograms generated by the TTS model, applies optional normalization and denormalization based on vocoder-specific statistics, and passes them through the vocoder's neural network to produce raw audio samples. The system supports multiple vocoder architectures and automatically selects the appropriate vocoder based on the TTS model's configuration, ensuring spectral compatibility. Vocoders are loaded separately from TTS models, enabling vocoder swapping without retraining the TTS model.

Provides a complete training pipeline for building custom TTS models from scratch or fine-tuning pre-trained models on new datasets. The training system uses PyTorch-based model definitions (Tacotron2, VITS, GlowTTS, etc.), configuration files (YAML) that specify hyperparameters, and a DataLoader that handles audio preprocessing (mel-spectrogram computation), text normalization, and speaker/language conditioning. The training loop implements gradient accumulation, mixed precision training, learning rate scheduling, and checkpoint management. Users define custom datasets by creating metadata files (CSV with audio paths and transcriptions) and specifying dataset-specific configuration (sample rate, mel-spectrogram parameters, speaker count).

Provides a specialized training pipeline for building custom neural vocoders (HiFi-GAN, Glow-TTS vocoder) from raw audio data. The vocoder training system takes audio files and corresponding mel-spectrograms, trains the vocoder to minimize reconstruction error (L1 loss on waveforms), and optionally applies adversarial training (discriminator loss) for improved audio quality. The training loop handles audio preprocessing (normalization, mel-spectrogram computation), batch loading, and checkpoint management. Unlike TTS training, vocoder training does not require text transcriptions — only audio files and their spectrograms are needed.

Implements a specialized training pipeline for learning speaker embeddings from reference audio samples, enabling zero-shot speaker adaptation without retraining the TTS model. The Speaker Encoder is a neural network (typically a ResNet-based architecture) that maps audio samples to fixed-size speaker embedding vectors. During training, the encoder is optimized using triplet loss or similar metric learning objectives to ensure that embeddings from the same speaker are close together and embeddings from different speakers are far apart. Once trained, the encoder can generate embeddings for new speakers from just 5-10 minutes of reference audio, which are then passed to the TTS model's speaker conditioning layers.

Provides a command-line tool (tts command) that wraps the Python API for text-to-speech synthesis, model listing, and model downloading without requiring Python code. The CLI accepts text input via stdin or command-line arguments, model selection via --model_name flag, speaker/language selection via --speaker_idx or --language flags, and output file specification via --out_path. The CLI internally uses the TTS class and ModelManager to handle model loading and synthesis. Additional CLI commands support listing available models (tts --list_models), downloading models (tts --model_name <name> --download), and running a web server (tts-server) for browser-based synthesis.

Provides a tts-server command that launches a Flask/FastAPI web server exposing TTS functionality via HTTP endpoints. The server implements REST endpoints for text-to-speech synthesis (/tts), model listing (/models), and speaker listing (/speakers). Clients send POST requests with text, model name, speaker ID, and language parameters, and receive audio files or JSON responses. The server handles concurrent requests using a thread pool or async workers, manages model caching in memory, and provides a simple HTML interface for browser-based testing. The server internally uses the TTS class and Synthesizer for synthesis, ensuring consistency with the Python API.