Rwkv Architecture

RWKV is a hybrid architecture that trains like a Transformer (parallel) but runs like an RNN (sequential), giving you O(n) complexity instead of O(n²). The big win is constant memory per token during inference, no KV cache bloat, which means you can actually handle 100K+ token contexts without exploding your VRAM. It's a Linux Foundation project already running in production at Microsoft (Windows, Office) and NVIDIA NeMo. If you're hitting memory walls with long context or building streaming apps, this is worth trying. The 14B models are available now, and RWKV-7 just dropped in March 2025. Performance is comparable to Transformers but inference gets dramatically cheaper as sequences grow.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill rwkv-architecture --agent claude-code

Installs into .claude/skills of the current project.

Files

RWKV - Receptance Weighted Key Value

Quick start

RWKV (RwaKuv) combines Transformer parallelization (training) with RNN efficiency (inference).

Installation:

# Install PyTorch
pip install torch --upgrade --extra-index-url https://download.pytorch.org/whl/cu121

# Install dependencies
pip install pytorch-lightning==1.9.5 deepspeed wandb ninja --upgrade

# Install RWKV
pip install rwkv

Basic usage (GPT mode + RNN mode):

import os
from rwkv.model import RWKV

os.environ["RWKV_JIT_ON"] = '1'
os.environ["RWKV_CUDA_ON"] = '1'  # Use CUDA kernel for speed

# Load model
model = RWKV(
    model='/path/to/RWKV-4-Pile-1B5-20220903-8040',
    strategy='cuda fp16'
)

# GPT mode (parallel processing)
out, state = model.forward([187, 510, 1563, 310, 247], None)
print(out.detach().cpu().numpy())  # Logits

# RNN mode (sequential processing, same result)
out, state = model.forward([187, 510], None)  # First 2 tokens
out, state = model.forward([1563], state)      # Next token
out, state = model.forward([310, 247], state)  # Last tokens
print(out.detach().cpu().numpy())  # Same logits as above!