Web Scraping

This is a systematic web scraping workflow that starts with a single curl request and only escalates to browser automation when needed. It runs adaptive reconnaissance phases with quality gates, so you're not burning browser resources if the data is already in the raw HTML. The framework detection tables help it recognize Next.js, WordPress, Shopify and others to skip irrelevant patterns. What's smart is the validation phase that actually tests every selector and API endpoint before claiming it works. It includes traffic interception to sniff APIs during page loads and has a full decision tree for when to test anti-bot protections versus when to skip it. Built for Apify Actor development with TypeScript templates included.

skills.sh installs: 415
GitHub stars: 110
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add yfe404/web-scraper --skill web-scraping --agent claude-code

Installs into .claude/skills of the current project.

Files

Web Scraping with Intelligent Strategy Selection

When This Skill Activates

Activate automatically when user requests:

"Scrape [website]"
"Extract data from [site]"
"Get product information from [URL]"
"Find all links/pages on [site]"
"I'm getting blocked" or "Getting 403 errors" (loads strategies/anti-blocking.md)
"Make this an Apify Actor" (loads apify/ subdirectory)
"Productionize this scraper"

Input Parsing

Determine reconnaissance depth from user request:

User Says	Mode	Phases Run
"quick recon", "just check", "what framework"	Quick	Phase 0 only
"scrape X", "extract data from X" (default)	Standard	Phases 0-3 + 5, Phase 4 only if protection signals detected
"full recon", "deep scan", "production scraping"	Full	All phases (0-5) including protection testing

Default is Standard mode. Escalate to Full if protection signals appear during any phase.