TTS WebUI

Orchestrates 15+ TTS models (Bark, Tortoise, VALL-E X, StyleTTS2, MMS, SeamlessM4T, etc.) through a dynamic extension system that loads model implementations at runtime without core codebase modification. Each model is wrapped as an extension with standardized input/output contracts, allowing users to switch between models via a single web UI while the server coordinates model initialization, GPU memory management, and inference execution.

Implements a plugin system where extensions are discovered and loaded dynamically at server startup without hardcoding model implementations. Extensions register themselves with category tags (tts, audio_generation, audio_conversion, tools), and the server introspects extension metadata to auto-generate UI tabs and parameter controls. This allows third-party developers to add new models by dropping extension files into a directory without modifying core server logic.

Handles conversion between audio formats (WAV, MP3, FLAC, OGG, M4A) and sample rate normalization. The system accepts audio in various formats, detects format and sample rate, and converts to a standardized format (typically 16-bit WAV at 22050Hz or model-specific rate) for processing. Supports both lossless (FLAC, WAV) and lossy (MP3, OGG) formats with configurable quality settings.

Implements GPU memory management that tracks VRAM usage across loaded models and automatically offloads unused models to CPU or disk when memory is constrained. The system maintains a model cache with LRU (least-recently-used) eviction policy, preloads frequently-used models, and prevents out-of-memory errors by monitoring GPU utilization. Users can configure memory thresholds and offloading strategies.

Provides UI and backend support for systematically varying model parameters and comparing outputs. Users can define parameter ranges (e.g., temperature 0.1-0.9 in 0.1 increments), generate outputs for all combinations, and organize results by parameter values. The system tracks which parameters were used for each output, enabling retrospective analysis of parameter sensitivity.

Integrates Retrieval-based Voice Conversion (RVC) to transform audio from one speaker to another by extracting speaker embeddings and applying voice conversion models. The system accepts input audio (from TTS output or user uploads), extracts speaker characteristics using a pre-trained encoder, and applies a conversion model trained on target speaker data to produce output audio with the target speaker's voice characteristics while preserving linguistic content.

Integrates Demucs (Meta's music source separation model) to decompose audio into constituent tracks (vocals, drums, bass, other instruments). The system accepts mixed audio input, runs inference through the Demucs model to separate sources, and outputs individual audio tracks for each source. This enables downstream processing like isolated vocal extraction for voice conversion or instrumental-only background music.

Integrates generative audio models (MusicGen, MAGNeT, Stable Audio) that synthesize music and sound effects from text descriptions. The system accepts natural language prompts describing desired audio characteristics (genre, instruments, mood, duration), encodes the prompt into embeddings, and runs inference through the generative model to produce audio samples. Multiple samples can be generated per prompt for variation.

Implements a collections system that organizes generated audio files into categorized groups (Outputs, Favorites, custom collections). The system tracks metadata for each generated file (model used, parameters, timestamp, source text), enables users to mark outputs as favorites, and supports custom collection creation for project-based organization. Collections are persisted to disk and accessible through the UI for browsing and re-processing.

Provides two parallel web interfaces: a Gradio-based UI (localhost:7770) auto-generated from extension metadata for rapid prototyping, and a custom React UI (localhost:3000) with more sophisticated UX for production use. Both interfaces communicate with the same Python backend via HTTP/WebSocket APIs, allowing users to choose based on their needs (simplicity vs. polish). The server coordinates both interfaces and maintains state synchronization.

Implements a configuration system that loads settings from environment variables, config files, and command-line arguments with a precedence hierarchy. The system manages model paths, GPU allocation, API keys, UI ports, and extension directories without hardcoding values. Configuration is validated at startup and provides sensible defaults for common scenarios (local development, Docker deployment, Google Colab).

Supports processing multiple audio files or text inputs sequentially through a queue system. Users can submit multiple generation or conversion jobs, and the system processes them in order while managing GPU memory and preventing resource exhaustion. Progress tracking and cancellation are available for long-running batches. Results are collected and organized by batch ID.

Integrates OpenAI's Whisper model for automatic speech recognition (ASR) to transcribe audio files into text. The system accepts audio input (from user uploads or generated audio), runs inference through the Whisper model, and outputs transcribed text with optional timestamp alignment. Supports multiple languages and provides confidence scores for transcription accuracy assessment.