Specter: Fast MoE Inference via Quantization-Enabled Multi-Step Speculative Offloading
Abstract
Specter accelerates single-GPU mixture-of-experts inference by combining quantized speculative decoding with expert offloading to hide PCIe latency and improve throughput.
Mixture-of-experts (MoE) models often exceed a single accelerator's memory capacity. Offloading experts to host memory creates a severe PCIe I/O bottleneck because data-dependent routing keeps transfers on the critical path. We present Specter, an inference system that uses quantization to combine speculative decoding with expert offloading. Its lightweight quantized draft model runs efficiently on device while preserving high-fidelity expert activation patterns. This provides accurate multi-step lookahead over future expert usage, allowing the runtime to organize expert movement and execution ahead of verification and improve both communication and execution efficiency. Under single-request inference, Specter achieves an average 1.50x (up to 1.88x) speedup in time per output token (TPOT) over the state-of-the-art system. Specter makes memory-constrained, single-GPU MoE inference more practical on commodity hardware. Our code is available at https://github.com/wwwwwf/specter_asplos.
Community
ASPLOS 2027
Using the quantized draft model to predict which experts will be needed a few steps ahead is a nice use of speculative decoding beyond just saving verification passes. How often does the draft predict the wrong experts, and what does a miss cost in TPOT compared with plain on-demand offloading? I would also like to know if the approach still helps when host memory is ordinary DDR4 on a desktop rather than a server.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Mira: Memory-Efficient MoE Inference Using Adaptive Caching and Predictive Expert Staging (2026)
- Efficient CPU-GPU collaborative MoE inference via Adaptive Expert Assignment (2026)
- RapidMoE: Exploiting Cross-Asymmetry via Adaptive Residual Offloading for Large-Scale MoE Inference (2026)
- MoE-CORE: Coordinated Expert Offloading and Residency for Memory-Constrained MoE Inference (2026)
- Resource-Efficient Speculative Decoding for Long-Context LLM Serving (2026)
- SSD-LLaMA: SSD-Native Inference for Trillion-Parameter MoE at 1+ Token/s on a Consumer PC (2026)
- EdgeAgent: Orchestrating On-Device LLM inference for End-User Multi-Agent Systems on CPU-GPU Unified Memory Architectures (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper