Robotstxt Ai
Gives Claude the ability to analyze, generate, and check robots.txt files with a focus on AI crawler detection. Exposes five tools: fetch_robots pulls and parses robots.txt from any URL, analyze_robots checks pasted content for bot blocking rules, generate_robots creates new files with custom bot selections, list_ai_bots returns a database of 25+ crawlers including GPTBot, ClaudeBot, Google-Extended, and PerplexityBot, and check_bot_status verifies if a specific bot is blocked on a given site. Useful when you need to audit AI crawler access across domains, quickly generate blocking rules without memorizing user agent strings, or help clients understand which bots are hitting their sites. The bot database is the real value here since these user agents change frequently and most devs don't track them.
- tools
- 5
- transport
- STDIO
- last updated
- Mar 21, 2026
Tools
5 toolsPublic tool metadata for what this MCP can expose to an agent.
fetch_robotsFetch and analyze a robots.txt from any URL
Fetch and analyze a robots.txt from any URL
No parameter schema in public metadata yet.
analyze_robotsAnalyze pasted robots.txt content for AI bot blocking status
Analyze pasted robots.txt content for AI bot blocking status
No parameter schema in public metadata yet.
generate_robotsGenerate a robots.txt with specified blocked bots and custom rules
Generate a robots.txt with specified blocked bots and custom rules
No parameter schema in public metadata yet.
list_ai_botsList all known AI bots with user-agents, companies, and descriptions
List all known AI bots with user-agents, companies, and descriptions
No parameter schema in public metadata yet.
check_bot_statusCheck if a specific bot is blocked on a given website
Check if a specific bot is blocked on a given website
No parameter schema in public metadata yet.
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageReadme
Manage which AI bots can crawl your website — visually.
Toggle GPTBot, ClaudeBot, PerplexityBot, and 20+ AI crawlers on/off with a simple UI. Analyze any site's robots.txt instantly.
Features
Details
- Repository
- github.com/sharozdawa/robotstxt-ai
- Category
- Search & Web Crawling
- Package
robotstxt-ai-mcp- Transport
- STDIO
- Registry
- MCP registry, active
More from sharozdawa
Related Search & Web Crawling MCP servers
View all
ClawPageskypher/clawpage-mcpExtract and structure any web page into clean JSON. Free tier: 10/day.0stars
Linkupso.linkup.mcp/linkup-mcp-serverSearch the web in real time to get trustworthy, source-backed answers.0stars
Suparse Document Processingsuparse/suparse-mcpExtract data, create schemas, and export results from PDFs and documents with Suparse AI OCR.0stars
Slipstreamtathagat22/slipstreamShared distillation cache for AI agents — every fetch ~73-89% fewer tokens via a shared cache.0stars
Apollo Intelligencebnmbnmai/mcp-proxy36 tools: intel feeds, DeFi, crypto, OSINT, NLP, scraping, proxy. x402 micropayments.0stars
Superjob Mcptheyahia/superjob-mcpMCP server for SuperJob.ru API — vacancy search, employers. Requires API key.0stars