Turboquant Pytorch

This is a from-scratch PyTorch implementation of Google's TurboQuant algorithm for compressing LLM key-value caches down to 2-4 bits per coordinate. It uses two-stage vector quantization: random rotation plus Lloyd-Max scalar quantization for the first pass, then QJL residual correction to make inner product estimates unbiased. The trick is it preserves attention scores, not individual vector fidelity. At 3-bit you get 5x compression with 99.5% attention accuracy on Qwen2.5-3B, which is the practical sweet spot. Use it when you're running long-context inference and need to fit more tokens in VRAM. The repo includes both synthetic tests and real model validation, plus production compressors that let you compute attention scores directly from compressed keys without decompressing.

skills.sh installs: 772
GitHub stars: 61
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/trending-skills --skill turboquant-pytorch --agent claude-code

Installs into .claude/skills of the current project.

Files

TurboQuant PyTorch

Skill by ara.so — Daily 2026 Skills collection.

From-scratch PyTorch implementation of Google's TurboQuant (ICLR 2026) for compressing LLM KV caches. Achieves 5x compression at 3-bit with 99.5% attention fidelity via two-stage vector quantization.

What It Does

TurboQuant compresses LLM key-value caches to 2–4 bits per coordinate:

Stage 1: Random orthogonal rotation + Lloyd-Max scalar quantization (MSE-optimal)
Stage 2: QJL residual correction — 1-bit sign projection that makes inner product estimates unbiased

Result: attention scores remain accurate even when individual vectors look quite different from originals. The algorithm preserves inner products, not vector fidelity.

Compression ratios at 8K context on Qwen2.5-3B (289 MB FP16 baseline):

4-bit → 76 MB (3.8x)
3-bit → 58 MB (5.0x) ← practical sweet spot
2-bit → 40 MB (7.3x)

Installation

git clone https://github.com/tonbistudio/turboquant-pytorch
cd turboquant-pytorch
pip install -r requirements.txt

# For CUDA PyTorch:
pip install torch --index-url https://download.pytorch.org/whl/cu128

SKILL.md on GitHub