Trident: Accelerating Speculative Decoding with Similarity-Driven Context Reuse

Abstract: Speculative decoding (SD) reduces the autoregressive latency of large language model (LLM) inference, but tree-based verification remains costly in this paradigm. We present Trident,a similarity-driven context-reuse framework that accelerates SD by exploiting semantic locality across independent requests. Trident retrieves the closest historical prefix for each incoming prefix and dynamically selects among Sequence Reuse, Probability Reuse, and SD-based Recomputation. The reuse paths leverage historical sequences, target probabilities, and KV cache through fuzzy matching, guided by an approximation that constrains reuse to cases with bounded generation loss. To scale historical-context storage, Trident stores contexts in an RDMA-enabled remote layer indexed with OMH-verified LSH and hides access latency through asynchronous retrieval and layer-wise context restoration. Evaluations demonstrate that Trident reduces TBT by 69.2% on average over autoregressive engines across models and datasets. It also outperforms existing training-free SD frameworks by an average of 30.3% across the same scenarios.