Verified record · from the answer index

sentencepiece

Language-independent tokenizer treating text as raw Unicode. Supports BPE and Unigram algorithms. Fast (50k sentences/sec), lightweight (6MB memory), deterministic vocabulary. Used by T5, ALBERT, XLNet, mBART. Train on raw text without pre-

skill:orchestra-research/ai-research-skills/sentencepiece · published by orchestra-research · source repository