Spark Training Gotchas

If you're about to kick off a multi-hour training run on a DGX Spark, this skill is worth running first. It walks through ten known failure modes for the GB10 chip, numbered G1 through G10, covering things like the CUDA 12/13 wheel mismatch, unified-memory OOMs that show up well under the 128GB cap, the roughly 100W sustained power ceiling that causes mid-run throttling, and the fact that two linked Sparks should only use DDP or FSDP, never tensor parallelism. There's a preflight script that prints one PASS/FAIL/WARN line per check, which is handy for catching the G1 and G7 problems in seconds. My honest take: the bandwidth and power numbers alone justify reading it, since budgeting from the 273 GB/s spec figure instead of the measured 180 to 192 GB/s is an easy way to waste a day.

skills.sh installs: 2.9K
GitHub stars: 42.4K
last updated: Jul 22, 2026

View on GitHub

Install

$ npx -y skills add wshobson/agents --skill spark-training-gotchas --agent claude-code

Installs into .claude/skills of the current project.

Files

Spark Training Gotchas

DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified memory, aarch64) has ten recurring failure modes across launch, memory, thermals, bandwidth, and precision. Each is named G1–G10 so it can be checked by number — the numbering is load-bearing for tooling that runs these checks. Read this before a long run, not after hour six.

When to Use This Skill

A training run fails to start, with an import error or a segfault that doesn't point at the real cause.
A run OOMs while nvidia-smi still shows headroom.
Throughput degrades partway through a run that started fine.
Before any multi-hour or multi-epoch job on GB10.
Wiring two Sparks together, before picking a parallelism strategy.
Choosing between FP8 and NVFP4 for a Spark-hosted run.