Site Crawlability

Guides you through fixing the technical issues that prevent search engines from crawling your site properly. Covers redirect chains, broken links, site architecture, orphan pages, and crawl budget optimization. Also includes AI crawler optimization with server-side rendering considerations. The skill checks for project context files first to understand your site structure, then walks you through an assessment and action items. If you're seeing indexing problems or want to audit how efficiently bots are crawling your site, this gives you a structured checklist. It's focused on the plumbing, not content strategy.

skills.sh installs: 1.2K
GitHub stars: 1.1K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add kostja94/marketing-skills --skill site-crawlability --agent claude-code

Installs into .claude/skills of the current project.

Files

SEO Technical: Crawlability

Guides crawlability improvements: robots, X-Robots-Tag, site structure, and internal linking.

When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.

Scope (Technical SEO)

Redirect chains & loops: Fix multi-hop redirects; point directly to final URL
Broken links (4xx): Fix broken internal/external links; 301 or remove
Site architecture: Logical hierarchy; pages within 3–4 clicks from homepage
Orphan pages: Add internal links to pages with no incoming links
Pagination: Prefer pagination over infinite scroll for crawlability
Crawl budget: Reduce waste on duplicates, redirects, low-value URLs (see below)
AI crawler optimization: SSR for critical content; URL management; reduce 404/redirect waste (see below)