Unigram Language Model Tokenization With Probability Based Selection

via “token-level probability and uncertainty estimation”

text-generation model by undefined. 72,54,558 downloads.

Unique: Exposes full vocabulary probability distributions at inference time without requiring model modification, enabling post-hoc confidence filtering and uncertainty quantification that works with any decoding strategy (greedy, beam, sampling)

vs others: More transparent than black-box confidence scoring but less calibrated than ensemble methods or Bayesian approaches; faster than external uncertainty quantification but requires manual threshold tuning

via “unigram language model tokenization with probability-based selection”

Python AI package: tokenizers

Unique: Uses probabilistic loss-based token selection instead of greedy matching, enabling graceful handling of unknown characters through byte-level fallback without [UNK] tokens; EM-based training iteratively optimizes vocabulary for corpus-specific loss minimization

vs others: Better multilingual support than WordPiece (no language-specific preprocessing needed) and more principled than BPE (probability-based vs heuristic merge frequency), though slower than BPE at inference time