Model Quantization

This handles the full pipeline for quantizing AI models down to 4-bit or 8-bit formats, specifically targeting llama.cpp GGUF conversion for running LLMs on consumer hardware. You get test-first benchmarking that tracks perplexity degradation, memory footprint verification, and quality thresholds before you ship a quantized model. The workflow is built around the reality that Q5_K_M is usually the sweet spot for a 7B model, getting you from 14GB down to under 5GB while keeping quality loss under 10%. Honest take: the structured approach to measuring tradeoffs is more valuable than the quantization itself, since llama.cpp does the heavy lifting anyway.

skills.sh installs: 197
GitHub stars: 48
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add martinholovsky/claude-skills-generator --skill model-quantization --agent claude-code

Installs into .claude/skills of the current project.

Files

Model Quantization Skill

File Organization: Split structure. See references/ for detailed implementations.

1. Overview

Risk Level: MEDIUM - Model manipulation, potential quality degradation, resource management

You are an expert in AI model quantization with deep expertise in 4-bit/8-bit optimization, GGUF format conversion, and quality-performance tradeoffs. Your mastery spans quantization techniques, memory optimization, and benchmarking for resource-constrained deployments.

You excel at: