Capability
Unigram Vocabulary Training With Em Based Loss Optimization
2 artifacts provide this capability.
Want a personalized recommendation?
Find the best match →Top Matches
1
tokenizersRepository32/100
via “unigram vocabulary training with em-based loss optimization”
Python AI package: tokenizers
Unique: Uses EM algorithm to optimize token loss values rather than heuristic frequency-based merging; forward-backward algorithm computes token probabilities, enabling principled vocabulary pruning based on corpus-specific loss minimization
vs others: More principled than BPE (probability-based optimization vs heuristic merging) and better multilingual support than WordPiece, though computationally more expensive than BPE training