Geo Crawlers

This analyzes whether AI crawlers like GPTBot, ClaudeBot, and PerplexityBot can actually access your site by checking robots.txt, meta tags, and headers. Useful when you're wondering why your content isn't showing up in ChatGPT Search or Perplexity results. The skill knows the difference between crawlers that power live AI search (high priority to allow) versus training-only bots (your call), and apparently over a third of top sites accidentally block major AI crawlers through legacy SEO configs. It gives you a complete access map and specific recommendations. Honestly pretty essential if you care about GEO, since being blocked is the fastest way to become invisible in AI-generated answers regardless of content quality.

skills.sh installs: 979
GitHub stars: 11.6K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add zubair-trabzada/geo-seo-claude --skill geo-crawlers --agent claude-code

Installs into .claude/skills of the current project.

Files

AI Crawler Access Analysis Skill

Purpose

This skill analyzes a website's accessibility to AI crawlers -- the bots that AI companies use to discover, index, and train on web content. If AI crawlers are blocked, the site's content cannot appear in AI-generated responses regardless of its quality. Crawler access is the foundational technical requirement for GEO.

Key Insight

As of early 2026, many websites inadvertently block AI crawlers through overly aggressive robots.txt rules, inherited from legacy SEO configurations. An Originality.ai 2025 study found that over 35% of the top 1,000 websites block at least one major AI crawler, and 5-10% block all AI crawlers. Blocking AI crawlers is the single fastest way to become invisible in AI-generated search results.

Complete AI Crawler Reference

Tier 1: Critical for AI Search Visibility (RECOMMEND: ALLOW)

These crawlers power the AI search products where users actively look for answers. Blocking them directly reduces your visibility in AI-generated responses.

SKILL.md on GitHub