Site Architecture

Covers the technical foundation that makes your content discoverable by both traditional search engines and AI crawlers like GPTBot, ClaudeBot, and PerplexityBot. You get templates for robots.txt with specific AI bot configurations, dynamic sitemap generation for Next.js, and comprehensive meta tag setups including Open Graph and Twitter Cards. Use this when you're setting up a new site's technical SEO or need to configure how different bots crawl your content. The philosophy here is solid: great content buried in poor architecture won't be found, so this handles the infrastructure that surfaces your work to search engines and AI assistants alike.

skills.sh installs: 233
GitHub stars: 685
last updated: Jun 3, 2026

View on GitHub

Install

$ npx -y skills add alinaqi/claude-bootstrap --skill site-architecture --agent claude-code

Installs into .claude/skills of the current project.

Files

Site Architecture Skill

For technical website structure that enables discovery by search engines AND AI crawlers (GPTBot, ClaudeBot, PerplexityBot).

Philosophy

Content is king. Architecture is the kingdom.

Great content buried in poor architecture won't be discovered. This skill covers the technical foundation that makes your content findable by:

Google, Bing (traditional search)
GPTBot (ChatGPT), ClaudeBot, PerplexityBot (AI assistants)
Social platforms (Open Graph, Twitter Cards)

robots.txt

Basic Template

# robots.txt

# Allow all crawlers by default
User-agent: *
Allow: /
Disallow: /api/
Disallow: /admin/
Disallow: /private/
Disallow: /_next/
Disallow: /cdn-cgi/

# Sitemap location
Sitemap: https://yoursite.com/sitemap.xml

# Crawl delay (optional - be careful, not all bots respect this)
# Crawl-delay: 1

AI Bot Configuration

# robots.txt with AI bot rules

# === SEARCH ENGINES ===
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# === AI ASSISTANTS (Allow for discovery) ===
User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: ClaudeBot
Allow: /

SKILL.md on GitHub