Llama Cpp

If you're running LLMs on anything other than NVIDIA GPUs, this is your best option. It's pure C++ inference that actually works well on M1/M2/M3 Macs, AMD cards, or just CPUs. The GGUF quantization is the killer feature: you can run a 7B model in 4GB at Q4_K_M with surprisingly little quality loss and get 50 tokens per second on an M3 Max. It ships with an OpenAI-compatible server, so you can drop it into existing code. The honest take is that if you have NVIDIA hardware you should use vLLM or TensorRT-LLM for better throughput, but for edge deployment, local development on Macs, or anything without CUDA, this is the standard.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill llama-cpp --agent claude-code

Installs into .claude/skills of the current project.

Files

llama.cpp

Pure C/C++ LLM inference with minimal dependencies, optimized for CPUs and non-NVIDIA hardware.

When to use llama.cpp

Use llama.cpp when:

Running on CPU-only machines
Deploying on Apple Silicon (M1/M2/M3/M4)
Using AMD or Intel GPUs (no CUDA)
Edge deployment (Raspberry Pi, embedded systems)
Need simple deployment without Docker/Python

Use TensorRT-LLM instead when:

Have NVIDIA GPUs (A100/H100)
Need maximum throughput (100K+ tok/s)
Running in datacenter with CUDA

Use vLLM instead when:

Have NVIDIA GPUs
Need Python-first API
Want PagedAttention

Quick start

Installation

# macOS/Linux
brew install llama.cpp

# Or build from source
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# With Metal (Apple Silicon)
make LLAMA_METAL=1

# With CUDA (NVIDIA)
make LLAMA_CUDA=1

# With ROCm (AMD)
make LLAMA_HIP=1

Download model

# Download from HuggingFace (GGUF format)
huggingface-cli download \
    TheBloke/Llama-2-7B-Chat-GGUF \
    llama-2-7b-chat.Q4_K_M.gguf \
    --local-dir models/

# Or convert from HuggingFace
python convert_hf_to_gguf.py models/llama-2-7b-chat/

SKILL.md on GitHub