TrendRadar

Crawls 11+ Chinese social platforms (Zhihu, Weibo, Bilibili, Douyin, etc.) and RSS feeds simultaneously, normalizing heterogeneous data schemas into a unified NewsItem model with platform-agnostic metadata. Uses platform-specific adapters that extract title, URL, hotness rank, and engagement metrics, then merges results into a single deduplicated feed ordered by composite hotness score (rank × 0.6 + frequency × 0.3 + platform_hot_value × 0.1).

Filters aggregated news against user-defined keyword lists (frequency_words.txt) using regex pattern matching and boolean logic (required keywords AND, excluded keywords NOT). Implements a scoring engine that weights matches by keyword frequency tier and calculates relevance scores. Supports regex patterns, case-insensitive matching, and multi-language keyword sets. Articles matching filter criteria are retained; non-matching articles are discarded before analysis and notification stages.

Detects newly trending topics by comparing current aggregated feed against historical baseline (previous execution results). Marks new topics with 🆕 emoji and calculates trend velocity (rate of rank change) to identify rapidly rising topics. Implements configurable sensitivity thresholds to distinguish genuine new trends from noise. Stores historical snapshots to enable trend trajectory analysis and prediction.

Supports region-specific content filtering and display preferences (e.g., show only Mainland China trends, exclude Hong Kong/Taiwan content, or vice versa). Implements per-region keyword lists and notification channel routing (e.g., send Mainland China trends to WeChat, international trends to Telegram). Allows users to configure multiple region profiles and switch between them based on monitoring focus.

Abstracts deployment environment differences through unified execution mode interface. Detects runtime environment (GitHub Actions, Docker container, local Python) and applies mode-specific configuration (storage backend, notification channels, scheduling mechanism). Supports seamless migration between deployment modes without code changes. Implements environment-specific error handling and logging (e.g., GitHub Actions annotations for CI/CD visibility).

Sends filtered news articles to LiteLLM, which abstracts over multiple LLM providers (OpenAI, Anthropic, Ollama, local models, etc.) to generate structured analysis including sentiment classification, key entity extraction, trend prediction, and executive summaries. Uses configurable system prompts and temperature settings per provider. Results are cached to avoid redundant API calls and formatted as structured JSON for downstream processing and notification delivery.

Translates article titles and summaries from Chinese to English (or other target languages) using LiteLLM-abstracted LLM providers with automatic fallback to alternative providers if primary provider fails. Maintains translation cache to avoid redundant API calls for identical content. Supports batch translation of multiple articles in single API call to reduce latency and cost. Integrates with notification system to deliver translated content to non-Chinese-speaking users.

Distributes filtered and analyzed news to 9+ notification channels (WeChat, WeWork, Feishu, Telegram, Email, ntfy, Bark, Slack, etc.) using channel-specific adapters. Implements atomic message batching to group multiple articles into single notification payloads, respecting per-channel rate limits and message size constraints. Supports channel-specific formatting (Markdown for Slack, card format for WeWork, plain text for Email). Includes retry logic with exponential backoff for failed deliveries and delivery status tracking.

Exposes TrendRadar's aggregated news and analysis data as an MCP server with tool definitions for natural language queries. Allows AI agents and LLM-based applications to query trending topics, filter by keywords, retrieve sentiment analysis, and generate trend reports via conversational interface. Implements tools like 'analyze_trends', 'get_trending_topics', 'search_by_keyword' that accept natural language parameters and return structured JSON responses. Integrates with Claude and other MCP-compatible clients for seamless agent-based analysis.

Orchestrates periodic execution of the entire news aggregation and analysis pipeline using Python scheduler (APScheduler or similar) configured via timeline.yaml. Supports multiple execution schedules (e.g., hourly for stock monitoring, daily for general trends) with timezone awareness. Integrates with GitHub Actions for cloud-based scheduling without local infrastructure, storing results in cloud storage (S3, GitHub Releases, etc.). Implements graceful shutdown and error recovery to ensure reliable scheduled execution.

Provides unified storage interface supporting multiple backends: local filesystem (Docker volumes, mounted directories), cloud storage (S3, GitHub Releases, Alibaba OSS), and database storage (SQLite, PostgreSQL). Implements automatic data persistence for aggregated news, analysis results, and execution logs. Supports both streaming writes (for real-time updates) and batch writes (for periodic reports). Includes data retention policies and automatic cleanup of old data based on configurable TTL.

Packages TrendRadar as Docker image with docker-compose orchestration for multi-service deployment including news crawler, AI analysis service, notification service, and optional MCP server. Implements health checks, automatic restart policies, and volume mounts for persistent storage. Supports environment variable configuration for secrets management (API keys, credentials). Includes pre-built images for common deployment scenarios (GitHub Actions, local Docker, cloud platforms).

Provides web-based UI for editing configuration files (config.yaml, frequency_words.txt, timeline.yaml) with real-time validation, syntax highlighting, and preview of filter results. Implements live keyword filter testing against sample articles to show matching results before deployment. Supports configuration versioning and rollback to previous versions. Integrates with main application to apply configuration changes without restart (hot-reload for certain settings).