Topic Diverse Conversation Corpus For Domain Coverage

via “domain-specific parallel corpus selection and filtering”

Massive parallel corpus for machine translation.

Unique: Curates domain-specific corpora including medical (EMEA 282.5M pairs), patents (EuroPat 252.2M), legal/institutional (Europarl 217.4M, JRC-Acquis 215.9M, DGT 1.2B), and specialized sources (Bible translations 88.3M, Ubuntu documentation) alongside general-domain subtitle and web-crawled data, enabling users to select data by source type and implied domain rather than explicit domain labels.

vs others: Provides access to specialized domain corpora (medical, legal, patents) in a single interface, whereas generic parallel corpus repositories focus on general-domain data; however, lacks explicit domain tagging, quality metrics per domain, and domain-specific preprocessing that specialized MT data providers offer.

via “topic-diverse conversation corpus for domain coverage”

Real ChatGPT conversations used to train Vicuna.

Unique: Organically diverse domain coverage from real user interests rather than synthetic balancing, preserving authentic frequency distributions while spanning coding, creative writing, analysis, and problem-solving without artificial curation

vs others: More naturally balanced across domains than manually curated instruction datasets, but less systematically comprehensive than proprietary datasets with explicit domain sampling strategies

via “conversation metadata and filtering by task type and domain”

161K human-written messages in 35 languages with quality ratings.

Unique: Conversation diversity (creative writing, coding, Q&A, general knowledge) within a single dataset enables domain-specific analysis and filtering, though without explicit labels requiring custom classification.

vs others: Broader task coverage than single-domain datasets (e.g., code-specific or creative writing-specific), allowing multi-domain model training or domain-specific subset creation.

via “domain and use-case diversity sampling and stratification”

1M+ real user-AI conversations with demographic metadata.

Unique: Captures authentic domain diversity from real ChatGPT/GPT-4 users without synthetic prompt engineering, preserving natural distribution of use cases and user intents, though requiring post-hoc domain inference rather than explicit labels

vs others: More authentic domain diversity than synthetic instruction-tuning datasets, though less explicitly labeled and curated than purpose-built domain-specific corpora