Serving Llms Vllm

If you need to serve LLMs at scale without melting your infrastructure, this is the go-to solution. vLLM delivers 24x higher throughput than standard transformers by using PagedAttention for memory-efficient KV caching and continuous batching that mixes prefill and decode requests. The skill wraps the Python library so you can spin up high-performance inference servers or run offline batches. It's built for production workloads where you're actually paying attention to tokens per second and GPU utilization. The installation is straightforward, and you get access to the same engine that powers a lot of commercial LLM APIs. Worth noting the skill comes from orchestra-research's AI research collection, so expect research-grade tooling rather than hand-holding docs.

skills.sh installs: 1.1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill serving-llms-vllm --agent claude-code

Installs into .claude/skills of the current project.

Files

vLLM - High-Performance LLM Serving

Quick start

vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests).

Installation:

pip install vllm

Basic offline inference:

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3-8B-Instruct")
sampling = SamplingParams(temperature=0.7, max_tokens=256)

outputs = llm.generate(["Explain quantum computing"], sampling)
print(outputs[0].outputs[0].text)

OpenAI-compatible server:

vllm serve meta-llama/Llama-3-8B-Instruct

# Query with OpenAI SDK
python -c "
from openai import OpenAI
client = OpenAI(base_url='http://localhost:8000/v1', api_key='EMPTY')
print(client.chat.completions.create(
    model='meta-llama/Llama-3-8B-Instruct',
    messages=[{'role': 'user', 'content': 'Hello!'}]
).choices[0].message.content)
"