Baidu: ERNIE 4.5 21B A3BModel23/100 via “mixture-of-experts text generation with sparse activation”
A sophisticated text-based Mixture-of-Experts (MoE) model featuring 21B total parameters with 3B activated per token, delivering exceptional multimodal understanding and generation through heterogeneous MoE structures and modality-isolated routing. Supporting an...
Unique: Uses heterogeneous MoE structure with modality-isolated routing, meaning different expert subsets are specialized for different input modalities or semantic categories, rather than generic expert pools. This architectural choice enables the model to maintain multimodal understanding (text + image) while keeping sparse activation efficient.
vs others: Achieves lower per-token latency than dense 21B models (e.g., Llama 2 21B) while maintaining competitive quality through learned expert specialization, making it faster and cheaper than dense alternatives at similar parameter counts.