Yi-34B

A 34-billion parameter decoder-only transformer model trained on 3 trillion tokens with native support for both English and Chinese language understanding and generation. The model uses standard transformer architecture with optimized attention mechanisms for efficient inference across both languages, leveraging balanced training data to maintain competitive performance in each language without degradation. Implements a unified vocabulary and embedding space that allows seamless code-switching and cross-lingual reasoning within single prompts.

Achieves 76.3% accuracy on the Massive Multitask Language Understanding (MMLU) benchmark, indicating strong performance across 57 diverse knowledge domains including STEM, humanities, social sciences, and professional fields. The model demonstrates broad factual knowledge and reasoning capability across these domains through transformer-based pattern matching and learned world knowledge from the 3 trillion token training corpus. Performance is competitive within the 34B parameter class, positioning it as a capable general-purpose reasoning engine for knowledge-intensive tasks.

Adapts to new tasks through in-context learning by observing examples in the prompt without parameter updates, enabling the model to generalize to unseen tasks by inferring patterns from provided examples. The transformer attention mechanisms learn to recognize task structure from examples and apply learned patterns to generate appropriate outputs for new instances of the same task.

Supports an extended context window variant with 200K token capacity (vs. 4K base variant), enabling processing of long-form documents, multi-turn conversations, and large code repositories within a single inference pass. The extended variant likely uses position interpolation, ALiBi, or similar techniques to extend the context window beyond the base training length without retraining. This allows models to maintain coherence and reference accuracy across significantly longer input sequences, critical for document analysis, code understanding, and multi-document reasoning tasks.

Demonstrates competitive performance on coding tasks (specific benchmarks undocumented) through transformer-based code understanding and generation. The model processes code as text tokens, leveraging the 3 trillion token training corpus which likely includes substantial code data from public repositories. Coding capability emerges from pretraining without documented specialized code fine-tuning, suggesting the base transformer architecture and training data composition are sufficient for code reasoning, completion, and generation tasks.

Demonstrates competitive performance on mathematical reasoning tasks (specific benchmarks undocumented) through transformer-based pattern matching and learned mathematical relationships. The model processes mathematical notation and reasoning as text tokens, leveraging training data that includes mathematical problems, proofs, and explanations. Mathematical capability emerges from pretraining without documented specialized math fine-tuning or chain-of-thought training, relying on the transformer's ability to learn mathematical patterns and reasoning from examples in the training corpus.

Distributed under Apache 2.0 license, enabling unrestricted commercial use, modification, and redistribution of model weights and architecture. The permissive license allows developers to integrate Yi-34B into proprietary products, fine-tune for specialized domains, and deploy in any environment (cloud, on-premise, edge) without licensing fees or usage restrictions. This open-source distribution model contrasts with closed-source commercial APIs and enables full model ownership and customization for organizations with specific requirements.

Serves as a foundation model for creating specialized variants through instruction tuning, domain-specific fine-tuning, and alignment training. The 34B base model provides a strong starting point for organizations to adapt to specific use cases (customer service, medical diagnosis, legal analysis, etc.) without training from scratch. This capability is evidenced by Yi-34B's role as the foundation for Yi-1.5 and subsequent models from 01.AI, demonstrating the model's suitability for downstream adaptation and specialization.

Supports seamless code-switching between English and Chinese within single prompts and responses, enabling cross-lingual reasoning and mixed-language outputs. The unified bilingual architecture processes both languages through a shared vocabulary and embedding space, allowing the model to understand relationships between English and Chinese concepts, translate between languages implicitly, and generate responses that mix both languages naturally. This capability is particularly valuable for applications serving bilingual users or requiring cross-lingual understanding.

Responds to natural language instructions and task specifications through learned instruction-following patterns in training data, enabling users to specify desired behavior through prompts without explicit fine-tuning. The model interprets instructions like 'summarize this text', 'translate to Chinese', or 'explain this code' and adapts its output format and content accordingly through attention mechanisms trained on instruction-response pairs.

Maintains conversation state across multiple turns through transformer attention mechanisms that reference previous messages in the conversation history, enabling coherent multi-turn dialogues where the model understands context, pronouns, and references to earlier statements. The model uses positional embeddings and attention patterns to weight recent messages more heavily while retaining access to earlier conversation context.