Coqui TTS

Converts text input to natural-sounding speech across 1100+ languages using a modular pipeline that chains text normalization, phoneme conversion, spectrogram generation via TTS models (VITS, Tacotron, Glow-TTS), and vocoder-based waveform synthesis. The Synthesizer class orchestrates sentence segmentation, language-specific text processing, model inference, and audio post-processing in a unified workflow that abstracts away model architecture differences through a common BaseTTS interface.

Clones a target speaker's voice by extracting speaker embeddings from a reference audio sample using a pre-trained speaker encoder network, then conditioning the TTS model (particularly XTTS) on those embeddings during synthesis. The system uses speaker encoder training to learn speaker-discriminative representations that generalize to unseen speakers without fine-tuning, enabling voice cloning with just 5-10 seconds of reference audio.

Externalizes model architecture and training hyperparameters into Python dataclass-based configuration objects (e.g., VitsConfig, Tacotron2Config, TrainingConfig) that define model layers, dimensions, loss weights, and training parameters. Users modify config objects to change model architecture or training settings without editing model code. Configs are loaded from Python files or JSON, allowing reproducible experiments and easy hyperparameter sweeps.

Supports multi-speaker TTS models that condition on speaker ID embeddings or one-hot speaker vectors to generate speech in different voices. Speaker embeddings are learned during training via speaker embedding layers that map speaker IDs to continuous vectors. During inference, users specify speaker ID or speaker name, and the model conditions on the corresponding speaker embedding to generate speech in that speaker's voice.

Converts text to phoneme sequences using language-specific phoneme inventories and grapheme-to-phoneme (G2P) conversion rules. The system supports multiple phoneme sets (IPA, language-specific phoneme sets) and uses rule-based or neural G2P models to convert text to phonemes. Phoneme sequences are then used as input to TTS models instead of raw text, improving pronunciation accuracy.

Provides a unified interface to multiple TTS architectures (VITS, Tacotron, Tacotron2, Glow-TTS, FastPitch, FastSpeech, AlignTTS, SpeedySpeech) through a common BaseTTS base class that defines the inference contract. Each model architecture inherits from BaseTTS and implements forward() and inference() methods; the Synthesizer decouples TTS model selection from vocoder selection, allowing any TTS model to pair with any vocoder (HiFi-GAN, Glow-TTS vocoder, etc.) via a modular vocoder registry.

Enables training TTS models on custom datasets through a modular training system that handles data loading, preprocessing, loss computation, and checkpoint management. The training pipeline supports transfer learning by loading pre-trained model weights and fine-tuning on new data; it uses PyTorch Lightning for distributed training, supports mixed precision training, and includes data samplers for handling imbalanced datasets. Configuration-driven training allows users to specify hyperparameters, data paths, and model architecture via Python config classes without modifying training code.

Trains a speaker encoder network to extract speaker-discriminative embeddings using speaker verification losses (e.g., GE2E loss, Angular Prototypical loss). The trained encoder learns to map variable-length audio to fixed-size speaker embeddings that cluster speakers together and separate different speakers in embedding space. These embeddings are then used to condition TTS models for speaker-adaptive synthesis or voice cloning without per-speaker fine-tuning.

Converts mel-spectrograms generated by TTS models into high-quality audio waveforms using neural vocoder models (HiFi-GAN, Glow-TTS vocoder, WaveGlow, etc.). The vocoder system is decoupled from TTS models through a BaseVocoder interface and vocoder registry, allowing any TTS model to use any vocoder. Vocoder inference runs as the final stage of the synthesis pipeline, taking spectrograms and optional speaker embeddings as input and producing raw audio waveforms.

Provides a CLI tool (tts command) for text-to-speech synthesis, model listing, and model management without writing Python code. The CLI wraps the TTS API and supports batch processing (reading text from files or stdin), model selection, speaker selection, and output format configuration. The synthesize.py module implements the CLI with argument parsing, file I/O, and progress reporting.

Provides a tts-server command that runs an HTTP server exposing TTS functionality via REST API endpoints. The server handles text-to-speech requests, returns audio files or streams, and supports model selection and speaker selection via query parameters or request body. The server uses Flask or similar framework to handle HTTP routing, request validation, and response formatting.

Preprocesses input text by normalizing abbreviations, numbers, and special characters into phonetically appropriate forms, then segments text into sentences for synthesis. The text processing pipeline uses language-specific rules and regex patterns to handle contractions, currency symbols, dates, and other text variations. Sentence segmentation uses punctuation-based heuristics and optional neural segmentation for languages without clear punctuation boundaries.

Maintains a centralized catalog of pre-trained models in .models.json that lists available models with metadata (architecture, language, dataset, download URL, speaker count). The ModelManager class provides methods to list available models, download models from remote repositories, and load model configurations and weights. Model discovery is automatic — users can list models by language, architecture, or dataset without manual URL lookup.