Speculative Decoding
This covers three main approaches to making LLMs generate text faster without quality loss: classic speculative decoding with a small draft model that guesses tokens for a large target model to verify in parallel (1.5-2× speedup), Medusa which adds extra prediction heads to guess multiple future tokens at once using tree-based attention (2.3-3.6× speedup), and Lookahead Decoding which reformulates generation as parallel Jacobi iteration over n-grams. The implementations are practical, with working code for transformers' assisted generation, the Medusa library, and lookahead parameters. Best when you need lower latency for chatbots or code completion and have the memory to load draft models or train lightweight heads. The math checks out and these actually work in production.
- skills.sh installs
- 1K
- GitHub stars
- 14.4K
- last updated
- Aug 21, 2026
Install
$ npx -y skills add orchestra-research/ai-research-skills --skill speculative-decoding --agent claude-codeInstalls into .claude/skills of the current project.
Files
Speculative Decoding: Accelerating LLM Inference
When to Use This Skill
Use Speculative Decoding when you need to:
- Speed up inference by 1.5-3.6× without quality loss
- Reduce latency for real-time applications (chatbots, code generation)
- Optimize throughput for high-volume serving
- Deploy efficiently on limited hardware
- Generate faster without changing model architecture
Key Techniques: Draft model speculative decoding, Medusa (multiple heads), Lookahead Decoding (Jacobi iteration)
Papers: Medusa (arXiv 2401.10774), Lookahead Decoding (ICML 2024), Speculative Decoding Survey (ACL 2024)
Installation
# Standard speculative decoding (transformers)
pip install transformers accelerate
# Medusa (multiple decoding heads)
git clone https://github.com/FasterDecoding/Medusa
cd Medusa
pip install -e .
# Lookahead Decoding
git clone https://github.com/hao-ai-lab/LookaheadDecoding
cd LookaheadDecoding
pip install -e .
# Optional: vLLM with speculative decoding
pip install vllm
Quick Start
Basic Speculative Decoding (Draft Model)
from transformers import AutoModelForCausalLM, AutoTokenizer
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/orchestra-research/ai-research-skills
- Categories
- AI & Agent Building, Data Science & ML, Automation & Workflows
- First seen
- Jun 3, 2026
More from orchestra-research/ai-research-skills
All 95- Llama Factory1K
- Model Merging1K
- Nemo Curator1K
- Transformer Lens Interpretability1K
- Distributed Llm Pretraining Torchtitan1K
- Openrlhf Training1K
- Evaluating Code Models1K
- Gptq1K
- Sentencepiece1K
- Outlines1K
- Llamaindex1K
- Pyvene Interventions1K
- Simpo Training1K
- Slime Rl Training1K
- Training Llms Megatron1K
- Guidance1K
- Mamba Architecture1K
- Model Pruning1K
- Sparse Autoencoder Training1K
- Implementing Llms Litgpt1K
- Llamaguard1K
- Miles Rl Training1K
- Torchforge Rl Training1K
- Nemo Guardrails1K
Recommended
More AI & Agent Building
simpo-trainingorchestra-research/ai-research-skillsSimple Preference Optimization for LLM alignment. Reference-free alternative to DPO with better performance (+6.4 points on AlpacaEval 2.0). No reference model 1Kskills.sh installsguidanceorchestra-research/ai-research-skillsControl LLM output with regex and grammars, guarantee valid JSON/XML/code generation, enforce structured formats, and build multi-step workflows with Guidance -1Kskills.sh installsmodel-pruningorchestra-research/ai-research-skillsReduce LLM size and accelerate inference using pruning techniques like Wanda and SparseGPT. Use when compressing models without retraining, achieving 50% sparsi1Kskills.sh installstorchforge-rl-trainingorchestra-research/ai-research-skillsProvides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy1Kskills.sh installsnemo-guardrailsorchestra-research/ai-research-skillsNVIDIA's runtime safety framework for LLM applications. Features jailbreak detection, input/output validation, fact-checking, hallucination detection, PII filte1Kskills.sh installsconstitutional-aiorchestra-research/ai-research-skillsAnthropic's method for training harmless AI through self-improvement. Two-phase approach - supervised learning with self-critique/revision, then RLAIF (RL from 1Kskills.sh installs