Prometheus Cardinality Troubleshooter
This is a triage guide for when Prometheus is actively on fire: OOMing, queries timing out, or your Grafana Cloud bill just doubled. It walks you through the tsdb status endpoints, shows you how to drill into per-metric and per-label breakdowns, and points you at the usual suspects (unbounded path labels, trace_ids leaking into metrics, histogram explosions). The best part is the opening rule: it explicitly warns you not to labeldrop distinguishing labels at scrape time, which looks like it fixes cardinality but actually merges distinct series and silently breaks rate() calculations. If you've ever stared at a cardinality fire and wondered where to start clicking, this gives you the runbook.
- skills.sh installs
- 4.1K
- GitHub stars
- 307
- last updated
- Aug 21, 2026
Install
$ npx -y skills add grafana/skills --skill prometheus-cardinality-troubleshooter --agent claude-codeInstalls into .claude/skills of the current project.
Files
Prometheus Cardinality Troubleshooter
You are an expert in diagnosing live Prometheus cardinality problems. When a user reports a Prometheus performance, memory, or cost issue that smells like cardinality, use this guide to triage systematically.
This skill is diagnostic and operational. For schema design and prevention, route to prometheus-label-strategy.
Before You Remediate: The One Rule
Under pressure, the tempting move is to labeldrop the high-cardinality label at scrape time. Do not. You cannot remove, at scrape time, any label that makes a series unique — not pod, not instance, not anything that distinguishes one real series from another. It looks like it stops the bleeding; it actually breaks the data:
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/grafana/skills
- Categories
- Documentation, Cloud & Infrastructure
- First seen
- Jul 14, 2026
More from grafana/skills
All 42- K6 Perf Test Website3.7K
- Dashboarding9.5K
- Promql9.1K
- Loki8.5K
- Loki Label Analyzer7.2K
- Opentelemetry6.6K
- Grafana Oss6K
- Prometheus5.6K
- Alerting Irm5.5K
- Assistant Mcp5.4K
- Alloy5.4K
- K65.3K
- Tempo5.2K
- Beyla5.1K
- Mimir5K
- Pyroscope5K
- App Observability4.7K
- Infrastructure4.6K
- Database Observability4.6K
- Adaptive Metrics4.5K
- App Sdk Concepts4.5K
- K6 Docs4.5K
- Reconciler Logic4.5K
- Cue Kind Definition4.5K
Recommended
More Documentation
mtp-hot-reloaddotnet/skillsSuggests using Microsoft Testing Platform (MTP) hot reload to iterate fixes on failing tests without rebuilding. Use when user says "hot reload tests", "iterate4.1Kskills.sh installs
literature-reviewlingzhi227/agent-research-skillsConduct comprehensive literature reviews using multi-perspective dialogue simulation. Generate diverse expert personas, conduct grounded Q&A conversations, and 4.1Kskills.sh installs
spark-engineerjeffallan/claude-skillsUse when writing Spark jobs, debugging performance issues, or configuring cluster settings for Apache Spark applications, distributed data processing pipelines,4.1Kskills.sh installsresolve-project-referencesdotnet/skillsGuide for interpreting ResolveProjectReferences time in MSBuild performance summaries. Activate when ResolveProjectReferences appears as the most expensive targ4.1Kskills.sh installs
ui-typographybencium/bencium-marketplaceProfessional typography rules for UI design, web applications, software interfaces, and all screen-based text. Enforces timeless typographic correctness that LL4Kskills.sh installscreate-skill-testdotnet/skillsScaffolds eval.yaml evaluation specs for agent skills in the dotnet/skills repository. Use when creating skill tests, writing evaluation stimuli, defining grade4Kskills.sh installs