Hyperpod Nccl

When your distributed training job hangs at AllReduce or you see EFA TCP fallback warnings, this is the skill to reach for. It runs a read-only diagnostic script that probes your HyperPod cluster (EKS or Slurm) via AWS APIs, kubectl, and SSM, then prints every issue with a direct pointer to the fix: security group missing self-references, NCCL version mismatches across pods, memlock limits, NetworkPolicy blocks, GPU OOM versus container OOM. The workflow is strict about not running destructive commands without your approval, which matters when training state is on the line. It won't help with single-node hardware faults or cluster creation failures, but for multi-node NCCL and rendezvous problems it cuts straight to root cause.

GitHub stars: 783
last updated: Jun 11, 2026

View on GitHub

Install

$ npx -y skills add awslabs/agent-plugins --skill hyperpod-nccl --agent claude-code

Installs into .claude/skills of the current project.

Files

HyperPod NCCL Debugger

Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation.

Diagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). scripts/nccl-diagnose.sh reads state via AWS APIs, kubectl, and SSM, then prints each issue as [FAIL] ... → references/<file>.md § <section>. Read-only.

Signal sourcing: list-cluster-events carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does not carry NCCL timeouts, GPU XID/ECC, or per-pod training signals — those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. "No events" on a training-time NCCL issue is expected, not a clean bill of health.