Awq Quantization

This is the MLSys 2024 Best Paper winner for 4-bit quantization, and it holds up in practice. AWQ protects about 1% of salient weights identified by activation patterns, which is why you get 3x speedup with under 5% accuracy loss on instruction-tuned models. It's faster than GPTQ and generalizes better to chat models, though GPTQ still has wider tool support. The vLLM integration is native and smooth. Worth noting that AutoAWQ itself is officially deprecated, so you'll want to migrate to vLLM's llm-compressor for new projects. If you're deploying 7B to 70B models on consumer GPUs or need production inference with tight memory budgets, this is still the technique to beat.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill awq-quantization --agent claude-code

Installs into .claude/skills of the current project.

Files

AWQ (Activation-aware Weight Quantization)

4-bit quantization that preserves salient weights based on activation patterns, achieving 3x speedup with minimal accuracy loss.

When to use AWQ

Use AWQ when:

Need 4-bit quantization with <5% accuracy loss
Deploying instruction-tuned or chat models (AWQ generalizes better)
Want ~2.5-3x inference speedup over FP16
Using vLLM for production serving
Have Ampere+ GPUs (A100, H100, RTX 40xx) for Marlin kernel support

Use GPTQ instead when:

Need maximum ecosystem compatibility (more tools support GPTQ)
Working with ExLlamaV2 backend specifically
Have older GPUs without Marlin support

Use bitsandbytes instead when:

Need zero calibration overhead (quantize on-the-fly)
Want to fine-tune with QLoRA
Prefer simpler integration

Quick start

Installation

# Default (Triton kernels)
pip install autoawq

# With optimized CUDA kernels + Flash Attention
pip install autoawq[kernels]

# Intel CPU/XPU optimization
pip install autoawq[cpu]

Requirements: Python 3.8+, CUDA 11.8+, Compute Capability 7.5+