Verified record · from the answer index
huggingface-tokenizers
skilltested · static-verifiedscore 20/100signed passport ✓published 260d ago · verified 2026-10-200 uses
Fast tokenizers optimized for research and production. Rust-based implementation tokenizes 1GB in <20 seconds. Supports BPE, WordPiece, and Unigram algorithms. Train custom vocabularies, track alignments, handle padding/truncation. Integrat
Declared tools (1)
huggingface-tokenizers
skill:davila7/claude-code-templates/tokenization-huggingface-tokenizers · published by davila7 · source repository