Speculative Decoding With Draft Model Acceleration

via “speculative decoding for latency reduction in batch inference”

1.1B model pre-trained on 3T tokens for edge use.

Unique: Leverages TinyLlama's 10x smaller size and 10x faster inference speed as draft model for speculative decoding, enabling 30-50% latency reduction for batch inference while maintaining output quality of larger models — unique positioning as draft model rather than standalone inference

vs others: More practical than self-speculative decoding (using same model for draft/verify) due to TinyLlama's speed advantage, and lower memory overhead than ensemble methods (two models vs three+)