Spider

This is a solid web scraping toolkit that combines curl-based crawling with Gemini AI analysis. You get practical scripts for extracting links, scanning security headers, monitoring page changes, and pulling structured data from HTML. The AI integration is the interesting bit here: instead of fighting with regex and BeautifulSoup for every edge case, you can throw page content at Gemini to extract products, find API endpoints in JavaScript, or spot security issues. It's bash-heavy and assumes you're comfortable piping commands together. Good for quick reconnaissance, content monitoring, or building lightweight scrapers without heavyweight frameworks. Just respect robots.txt and rate limits.

skills.sh installs: 208
GitHub stars: 27
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add johnlindquist/claude --skill spider --agent claude-code

Installs into .claude/skills of the current project.

Files

Web Spider

Crawl and analyze websites.

Prerequisites

# curl for fetching
curl --version

# Gemini for analysis
pip install google-generativeai
export GEMINI_API_KEY=your_api_key

# Optional: better HTML parsing
npm install -g puppeteer
pip install beautifulsoup4

Basic Crawling

Fetch Page

# Simple fetch
curl -s "https://example.com"

# With headers
curl -s -H "User-Agent: Mozilla/5.0" "https://example.com"

# Follow redirects
curl -sL "https://example.com"

# Save to file
curl -s "https://example.com" -o page.html

# Get headers only
curl -sI "https://example.com"

# Get headers and body
curl -sD - "https://example.com"

Extract Links

# Extract all links from a page
curl -s "https://example.com" | grep -oE 'href="[^"]*"' | sed 's/href="//;s/"$//'

# Filter to specific domain
curl -s "https://example.com" | grep -oE 'href="https?://example\.com[^"]*"'

# Unique links
curl -s "https://example.com" | grep -oE 'href="[^"]*"' | sort -u

Quick Site Scan

#!/bin/bash
URL=$1
echo "=== Scanning: $URL ==="

echo ""
echo "### Response Info ###"
curl -sI "$URL" | head -20

SKILL.md on GitHub

Details

Repository: github.com/johnlindquist/claude
Category: Security
First seen: Jun 3, 2026

More from johnlindquist/claude

All 7