Hyperpod Slurm Debugger

A read-only diagnostic that walks you through Slurm scheduler and daemon problems on HyperPod clusters. It covers the usual suspects: nodes stuck in down or drain, jobs pending forever despite idle capacity, GPU counts wrong, slurmctld unresponsive, and the "Node unexpectedly rebooted" message that shows up after auto-repair. The procedure runs a bash script that checks slurmd status, disk usage, RPC layer health, and controller state, then maps findings to the official AWS troubleshooting guide without printing any remediation commands. You still have to run fixes yourself. Requires the SSM plugin and unbuffer, which is non-negotiable because start-session will silently fail otherwise. Good for teams managing Slurm on HyperPod who need structured diagnosis before escalating or running state-changing commands.

GitHub stars: 783
last updated: Jun 11, 2026

View on GitHub

Install

$ npx -y skills add awslabs/agent-plugins --skill hyperpod-slurm-debugger --agent claude-code

Installs into .claude/skills of the current project.

Files

HyperPod Slurm Debugger

Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on HyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command. For remediation, link to the official AWS or Slurm documentation.

When to invoke

Invoke when the user reports any of the symptoms in the decision table.

When NOT to invoke

Cluster has Orchestrator.Eks — invoke hyperpod-node-debugger or hyperpod-nccl.
Single-node hardware fault with healthy Slurm scheduler — invoke hyperpod-node-debugger.
NCCL training-hang investigation — invoke hyperpod-nccl.
Node unreachable via SSM — invoke hyperpod-ssm.

Constraints

Read-only. Do not run, recommend, or print state-mutating commands.
For any remediation, link to AWS or Slurm docs. The user authorizes and executes.
IaC-managed cluster (Terraform / CloudFormation / CDK): warn that direct mutation drifts the live state from the IaC plan.

Canonical recovery URLs: references/slurm-details.md → Authoritative recovery documentation.