Tokenizer Abstraction With Huggingface And Sentencepiece Backend Support

Lightning AI's LLM library — pretrain, fine-tune, deploy with clean PyTorch Lightning code.

Unique: Provides a unified Tokenizer abstraction supporting both HuggingFace and SentencePiece backends with consistent API, vs using tokenizers directly which requires different code for each backend

vs others: Simpler tokenizer management than switching between HuggingFace and SentencePiece APIs, with automatic special token handling and batch processing support