Verified record · from the answer index
huggingface-tokenizers
skilltested · static-verifiedscore 49/100signed passport ✓918 uses
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrat
Declared tools (1)
huggingface-tokenizers
skill:orchestra-research/ai-research-skills/huggingface-tokenizers · published by orchestra-research · source repository