Quantizing Models Bitsandbytes

This wraps bitsandbytes for quantizing large language models directly in your Claude Code workflow. You get 8-bit quantization for 50% memory savings or 4-bit for 75% reduction, both with less than 1% accuracy loss according to the docs. Useful when you're trying to run models locally or fit bigger ones into limited VRAM. The skill includes the standard transformers integration setup, so you're basically getting the same quantization approach that's become common in the open source LLM space. Originally from ovachiever/droid-tings, now maintained under orchestra-research. It passed Gen Agent Trust Hub and Socket audits but has a Snyk warning, so check dependencies if that matters for your project.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill quantizing-models-bitsandbytes --agent claude-code

Installs into .claude/skills of the current project.

Files

bitsandbytes - LLM Quantization

Quick start

bitsandbytes reduces LLM memory by 50% (8-bit) or 75% (4-bit) with <1% accuracy loss.

Installation:

pip install bitsandbytes transformers accelerate

8-bit quantization (50% memory reduction):

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

config = BitsAndBytesConfig(load_in_8bit=True)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=config,
    device_map="auto"
)

# Memory: 14GB → 7GB

4-bit quantization (75% memory reduction):

config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    quantization_config=config,
    device_map="auto"
)

# Memory: 14GB → 3.5GB

Common workflows

Workflow 1: Load large model in limited GPU memory

Copy this checklist:

Quantization Loading:
- [ ] Step 1: Calculate memory requirements
- [ ] Step 2: Choose quantization level (4-bit or 8-bit)
- [ ] Step 3: Configure quantization
- [ ] Step 4: Load and verify model

SKILL.md on GitHub