robin

Transforms raw user investigation queries into optimized search terms by routing them through a pluggable multi-provider LLM layer (OpenAI, Anthropic, Google, Ollama). The system uses prompt engineering to expand queries with domain-specific dark web terminology, synonyms, and alternative phrasings that improve hit rates across heterogeneous dark web search engines. Implementation delegates to llm.refine_query() which constructs a system prompt contextualizing the dark web domain, then streams the LLM response to generate semantically richer search queries.

Queries multiple dark web search engines (Torch, Ahmia, Candle, etc.) concurrently using a thread-pooled orchestration pattern implemented in search.py:get_search_results(). Each search engine query is wrapped in a timeout-protected thread to prevent hanging on slow .onion sites; results are aggregated into a unified list of URLs and titles. The system handles search engine-specific response formats through adapter patterns, normalizing heterogeneous HTML/JSON responses into a common data structure for downstream LLM filtering.

Manages Robin configuration through a two-tier system: environment variables for sensitive credentials (API keys, Tor proxy address) and YAML/JSON config files for operational settings (model selection, timeout values, search engine whitelist). The system reads environment variables first (highest priority), then falls back to config file values, then uses hardcoded defaults. Configuration is loaded at startup in main.py and passed through the investigation pipeline. This approach enables secure credential management (via environment variables in Docker/Kubernetes) while allowing flexible operational configuration (via config files for different investigation types).

Filters dark web search results using LLM-powered relevance scoring implemented in llm.py:filter_results(). The system constructs a prompt containing the original investigation query and candidate search results, then uses the LLM to score each result's relevance to the investigation objective. Results are ranked by LLM-assigned relevance scores and filtered to retain only high-confidence matches, reducing noise from off-topic .onion pages. This approach captures semantic relevance beyond keyword matching — e.g., identifying a marketplace listing as relevant to 'ransomware payment tracking' even if it doesn't contain the exact phrase.

Extracts HTML content from dark web .onion sites by routing HTTP requests through a Tor SOCKS5 proxy (127.0.0.1:9050) implemented in scrape.py:scrape_multiple(). The system uses a thread-pooled architecture to scrape multiple URLs concurrently with per-request timeout protection (default 30 seconds) to prevent hanging on slow/offline sites. Responses are parsed with BeautifulSoup to extract text content, and failures (connection timeouts, 404s, Tor circuit failures) are gracefully handled with fallback retry logic. The implementation maintains request anonymity by routing all HTTP traffic through Tor and rotating user agents to avoid fingerprinting.

Synthesizes raw scraped content, search results, and metadata into structured intelligence reports using LLM-powered summarization implemented in llm.py:generate_summary(). The system constructs a prompt containing the investigation query, filtered search results, and scraped page content, then uses the LLM to extract key findings, identify threat indicators (IOCs), and organize information into a structured report with sections like 'Threat Overview', 'Key Findings', 'Indicators of Compromise', and 'Recommendations'. The report is formatted as JSON or markdown for downstream consumption by SIEM systems, threat intelligence platforms, or human analysts.

Provides a pluggable abstraction layer for multiple LLM providers (OpenAI, Anthropic, Google, Ollama) implemented in llm_utils.py:get_llm(). The system uses a factory pattern to instantiate the appropriate LLM client based on environment variables or configuration, enabling seamless provider switching without modifying downstream code. Each provider is wrapped with a consistent interface supporting streaming responses, token counting, and error handling. Configuration is managed through environment variables (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.) and a config file, allowing users to specify model selection, temperature, and max tokens per provider.

Orchestrates a complete dark web OSINT investigation workflow through a six-stage pipeline implemented in main.py:cli(). The pipeline sequentially executes: (1) LLM initialization, (2) query refinement, (3) multi-engine search, (4) result filtering, (5) content scraping, and (6) report generation. Each stage is implemented as a modular function with clear input/output contracts, enabling easy insertion of custom stages or modification of existing ones. The orchestration layer handles error propagation, logging, and progress reporting across stages, with optional checkpointing to resume interrupted investigations.

Exposes the investigation pipeline through two distinct interfaces: command-line (main.py:cli()) for automation and scripting, and Streamlit web UI (ui.py) for interactive exploration. The CLI mode accepts query and model arguments, executes the investigation pipeline, and outputs results to stdout/files. The Streamlit UI provides a web dashboard with form inputs for query/model selection, real-time progress updates, and interactive result visualization. Both interfaces share the same underlying pipeline implementation, ensuring consistency while accommodating different user workflows (batch automation vs. interactive investigation).

Routes all HTTP traffic (search queries, content scraping) through a Tor SOCKS5 proxy running on 127.0.0.1:9050, implemented at the HTTP client level using the requests library with PySocks support. The system configures the Tor proxy globally for all outbound requests, ensuring that search engine queries and .onion site scraping are anonymized. Tor circuit failures are handled with retry logic, and user agents are rotated to avoid fingerprinting. The implementation assumes a local Tor service is running (typically via Docker or system package) and does not manage Tor lifecycle.

Packages Robin and its dependencies (Python, requests, BeautifulSoup, Streamlit) into a Docker image with an integrated Tor service, enabling single-command deployment without manual dependency installation. The Dockerfile installs Tor, configures SOCKS5 proxy on 127.0.0.1:9050, and starts both Tor and Robin services on container startup. Environment variables for LLM provider credentials are passed at runtime, allowing users to deploy the container without modifying the image. The Docker Compose configuration (if provided) orchestrates the Robin container with optional additional services (e.g., Redis for caching, PostgreSQL for result storage).