Site Crawler

Site Crawler Skill

Respectfully crawl documentation sites and web content for RAG ingestion.

Overview

Documentation sites, blogs, and knowledge bases contain valuable structured content. This skill covers:

• Respectful crawling (robots.txt, rate limiting)
• Structure-preserving extraction
• Incremental updates (only fetch changed pages)
• Sitemap-based discovery

Prerequisites

# HTTP client
pip install httpx

# HTML parsing
pip install beautifulsoup4 lxml

# Clean article extraction
pip install trafilatura

# Markdown conversion
pip install markdownify

Crawling Principles

1. Be Respectful

• Always check robots.txt
• Rate limit requests (1-2 seconds between)
• Identify yourself with a User-Agent
• Don't overload servers

2. Be Efficient

• Use sitemaps when available
• Track what's been crawled
• Only re-fetch changed content
• Skip non-content pages (login, search results)

3. Be Smart

• Preserve document structure
• Extract meaningful content only
• Handle pagination
• Detect and follow documentation structure

Core Implementation

Robots.txt Handling

#!/usr/bin/env python3
"""Robots.txt compliance."""