Robots Txt

Configures and audits robots.txt for search engine and AI crawler control. Handles the usual suspects like blocking /admin/ and /api/, but the real value is in the AI crawler strategy table that breaks down which bots respect robots.txt and which don't (ChatGPT-User stopped respecting it in December 2025, for instance). Includes clean separation between crawl control (robots.txt) and index control (noindex meta tags), with clear guidance on when to use each. The critical blocker list is helpful: don't block CSS/JS, don't block pages that use noindex, don't block Next.js asset paths. Pairs with the indexing skill for the full picture.

skills.sh installs: 1.3K
GitHub stars: 1.1K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add kostja94/marketing-skills --skill robots-txt --agent claude-code

Installs into .claude/skills of the current project.

Files

SEO Technical: robots.txt

Guides configuration and auditing of robots.txt for search engine and AI crawler control.

When invoking: On first use, if helpful, open with 1–2 sentences on what this skill covers and why it matters, then provide the main output. On subsequent use or when the user asks to skip, go directly to the main output.

Scope (Technical SEO)

Robots.txt: Configure Disallow/Allow, Sitemap, Clean-param; audit for accidental blocks
Crawler access: Path-level crawl control; AI crawler allow/block strategy
Differentiation: robots.txt = crawl control (who accesses what paths); noindex = index control (what gets indexed). See indexing for page-level exclusions.

Initial Assessment

Project context: Read root {project}.md when present and load only the modules relevant to this task. Without Contextus, use available project material or user-provided facts and ask for missing information; do not create a parallel context system.

SKILL.md on GitHub