Capability
Tokenization And Detokenization With Chatglm Vocabulary
3 artifacts provide this capability.
Want a personalized recommendation?
Find the best match →Top Matches
1
ChatGLM-4Model45/100
Tsinghua's bilingual dialogue model.
Unique: Provides ChatGLMTokenizer with bilingual vocabulary optimized for Chinese-English text, using special dialogue tokens ([gMASK], [eos_token]) that are integrated into the tokenization process rather than added post-hoc
vs others: More efficient Chinese tokenization than generic BPE tokenizers (fewer tokens per character); built-in dialogue special tokens eliminate manual token management compared to generic tokenizers
2