Web Scraper

Straightforward Python script that fetches web pages and converts them to markdown. Good for when you need to read articles or extract content from URLs, especially if Claude's built-in fetching hits network restrictions. It strips out navigation and ads, preserves formatting like headings and code blocks, and can handle multiple URLs in parallel with basic shell scripting. Supports custom timeouts and has a raw HTML mode if you need it. Won't help with JavaScript-heavy sites since it's just HTTP requests and HTML parsing, but for static content it gets the job done without dependencies on external services.

skills.sh installs: 264
GitHub stars: 310
last updated: Jun 3, 2026

View on GitHub

Install

$ npx -y skills add zephyrwang6/myskill --skill web-scraper --agent claude-code

Installs into .claude/skills of the current project.

Files

Web Scraper

Fetch web page content and convert to clean markdown format.

Usage

Run the fetch script to get web content:

python3 scripts/fetch_url.py <url> [options]

Options

--timeout <seconds>: Request timeout (default: 30)
--max-length <chars>: Maximum output length (default: 100000)
--raw: Output raw HTML instead of markdown

Examples

Fetch single URL:

python3 scripts/fetch_url.py "https://example.com/article"

Fetch with custom timeout:

python3 scripts/fetch_url.py "https://example.com/article" --timeout 60

Fetch multiple URLs in parallel:

for url in "https://url1.com" "https://url2.com"; do
  python3 scripts/fetch_url.py "$url" &
done
wait

Workflow

Single URL: Run fetch_url.py with the URL
Multiple URLs: Run multiple fetch commands in parallel using background processes
Handle errors: If a URL fails, check:
- Network connectivity
- URL validity
- Website may block automated requests (try different User-Agent or use browser automation)