real-time websocket-based audio streaming and session management for esp32 devices
Implements a persistent WebSocket connection handler (ConnectionHandler class) that manages per-client session state, routes incoming audio frames at 60ms intervals via AudioRateController, and maintains bidirectional communication with ESP32 hardware. Uses frame-based timing synchronization to ensure consistent audio delivery rates and handles connection lifecycle events (hello handshake, authentication, disconnection). The architecture supports multiplexed concurrent device connections through async I/O patterns.
Unique: Uses frame-rate-controlled WebSocket streaming with per-device session handlers rather than request-response HTTP, enabling true real-time bidirectional audio without polling or connection re-establishment overhead. AudioRateController enforces 60ms frame timing to match ESP32 hardware capabilities.
vs alternatives: Achieves lower latency than REST-based polling approaches and simpler state management than raw socket implementations by leveraging WebSocket's persistent connection model with explicit frame timing synchronization.
multi-provider speech recognition (asr) with streaming audio processing
Integrates pluggable ASR providers (FunASR, Whisper, etc.) that process streaming audio frames in real-time, converting spoken input to text through provider-specific APIs. The system buffers incoming audio, detects speech boundaries via SileroVAD (Voice Activity Detection), and routes complete utterances to the configured ASR provider. Supports both cloud-based (OpenAI Whisper, Alibaba FunASR) and on-device (local Silero models) recognition with configurable fallback chains.
Unique: Implements provider-agnostic ASR abstraction with automatic VAD-based utterance segmentation, allowing seamless switching between cloud and local models without application-level code changes. Uses SileroVAD for hardware-efficient speech boundary detection rather than relying on provider-specific silence detection.
vs alternatives: More flexible than single-provider solutions (e.g., Whisper-only) by supporting provider chains and local fallbacks; more efficient than always-cloud approaches by enabling on-device ASR for privacy-sensitive deployments.
configuration management with yaml-based provider and model definitions
Implements centralized configuration loading from YAML files (config.yaml) that define AI providers (LLM, ASR, TTS), model parameters, device settings, and system behavior. The system supports environment variable substitution for sensitive data (API keys), configuration validation against schema, and hot-reload capabilities for non-critical settings. Configurations are hierarchically organized (global, per-user, per-device) with inheritance and override rules. Integrates with database for user-specific configuration overrides.
Unique: Implements hierarchical YAML-based configuration with environment variable substitution and database-backed per-user overrides, enabling flexible provider and model management without code changes. Supports configuration inheritance from global → user → device levels.
vs alternatives: More flexible than hardcoded configurations by supporting YAML definitions; more secure than storing API keys in code by using environment variables.
voice activity detection (vad) with silero vad for utterance boundary detection
Implements real-time voice activity detection using Silero VAD model, which processes streaming audio frames to identify speech boundaries (start/end of utterance). The system runs VAD on incoming audio, buffers frames until speech ends, and triggers ASR only on complete utterances. Silero VAD is lightweight (~40MB) and runs on CPU, making it suitable for edge deployment. Supports configurable sensitivity and frame-based processing at 16kHz sample rate.
Unique: Uses Silero VAD for lightweight, CPU-efficient voice activity detection with frame-based processing, enabling real-time utterance boundary detection without GPU acceleration. Integrates seamlessly with ASR pipeline to buffer frames until speech ends.
vs alternatives: More efficient than provider-specific VAD (e.g., Whisper's built-in VAD) by running locally on CPU; more accurate than simple energy-based detection by using neural network-based speech classification.
plugin system for custom function development with python function registry
Provides a plugin architecture that allows developers to create custom functions in Python and register them with the function registry for invocation via intent recognition. Plugins are stored in plugins_func directory, automatically discovered and loaded at startup, and can access system context (user_id, device_id, conversation history). Each plugin is a Python function with type hints and docstring documentation, which are automatically converted to JSON Schema for parameter validation. Supports both synchronous and asynchronous function execution with error handling and result serialization.
Unique: Implements automatic plugin discovery and schema generation from Python type hints, enabling developers to create custom functions without manual schema definition. Supports both sync and async execution with integrated error handling.
vs alternatives: More developer-friendly than manual schema definition by auto-generating JSON Schema from type hints; more flexible than hardcoded functions by supporting dynamic plugin loading.
multi-provider text-to-speech (tts) with voice cloning and streaming output
Provides pluggable TTS providers (Azure, Google Cloud, ElevenLabs, local TTS engines) that convert text responses into audio streams, with support for voice cloning and custom voice parameters. The system accepts text input from LLM responses, applies provider-specific voice selection and prosody controls, streams audio back to ESP32 clients in 60ms frames, and manages voice profile storage for user-specific voice preferences. Supports both streaming TTS (real-time audio generation) and batch synthesis with caching.
Unique: Implements provider-agnostic TTS abstraction with integrated voice profile management and streaming output synchronization to 60ms ESP32 frame boundaries. Supports voice cloning through provider-specific APIs (ElevenLabs, Azure) while maintaining fallback to standard voices.
vs alternatives: More flexible than single-provider TTS by supporting provider chains and voice customization; more efficient than batch-only approaches by streaming audio in real-time to reduce perceived latency.
intent recognition and function calling with plugin-based action execution
Processes LLM-generated intent outputs through a function registry that maps recognized intents to executable Python functions or MCP tool calls. The system parses LLM responses for intent names and parameters, validates them against a schema registry, and executes corresponding plugins (built-in or user-defined) with automatic error handling and result serialization. Supports both synchronous function calls and async task queuing for long-running operations. Integrates with MCP (Model Context Protocol) for standardized tool definitions.
Unique: Implements a schema-based function registry with MCP protocol support, allowing both built-in Python plugins and external MCP tools to be invoked through a unified intent interface. Uses JSON Schema validation for parameter type checking and automatic error serialization.
vs alternatives: More extensible than hardcoded intent handlers by supporting plugin discovery and dynamic registration; more standardized than custom function calling by using MCP protocol for tool definitions.
dialogue memory and context management with multi-turn conversation support
Maintains per-user conversation history with configurable context windows, storing previous user utterances, assistant responses, and execution results in a structured format. The system passes relevant context to the LLM for each turn, implements sliding-window context truncation to manage token budgets, and supports memory persistence across sessions via database storage. Integrates with knowledge base (RAG) to augment context with relevant documents and maintains dialogue state (current topic, user preferences, device state).
Unique: Implements sliding-window context management with integrated RAG augmentation, allowing dialogue history to be automatically truncated based on token budgets while relevant documents are injected from knowledge base. Stores conversation state in structured database format for multi-session persistence.
vs alternatives: More sophisticated than simple conversation history by implementing context truncation and RAG integration; more persistent than in-memory solutions by supporting database-backed storage across sessions.
+5 more capabilities