Visual Tokenization With Variable Resolution Vae Supporting 2 16 To 2 64 Vocabulary Sizes

via “vocabulary-constrained token prediction with 30k wordpiece vocabulary”

fill-mask model by undefined. 40,14,871 downloads.

Unique: Uses a shared 30,522-token WordPiece vocabulary across 104 languages, enabling consistent subword tokenization and vocabulary-constrained predictions without language-specific token sets. The vocabulary includes multilingual character coverage and subword units learned from joint pretraining, providing deterministic and reproducible token predictions.

vs others: Shared vocabulary enables cross-lingual consistency and transfer learning; however, language-specific BERT models (e.g., RoBERTa for English) achieve higher vocabulary coverage and prediction accuracy for single-language tasks due to language-optimized tokenization.