Web Crawler

This is the fallback scraper you reach for when web_fetch hits a wall or you need social media data. It routes to ScrapeCreators for 27+ platforms (TikTok, Instagram, YouTube, LinkedIn, Reddit, Threads, the whole roster), SerpApi for YouTube transcripts, and Firecrawl when a page blocks basic fetching. The OpenAPI routing is smart: you specify intent (profile, posts, single video), it fetches the spec for that endpoint, then makes the call. Costs money per request, so use native fetch first and invoke this when you actually need profiles, comments, transcripts, or anti-bot evasion. The scope is huge but the use case is clear: structured extraction from platforms that don't want to be scraped.

skills.sh installs: 3.9K
GitHub stars: 32
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add starchild-ai-agent/official-skills --skill web-crawler --agent claude-code

Installs into .claude/skills of the current project.

Files

Preferred entry: call exports.py (don't hand-roll requests)

Ready-made helpers live in skills/web-crawler/exports.py. Prefer them over writing your own proxied_get/proxied_post calls — they already inject the proxy credentials, so there is no API key to find (don't read $SCRAPECREATORS_API_KEY / $FIRECRAWL_API_KEY, don't check .env, don't ask the user).

import sys; sys.path.insert(0, "/data/workspace/skills/web-crawler")
from exports import scrape_markdown, youtube_transcript, sc_get
scrape_markdown("https://example.com/article")          # Firecrawl fallback
youtube_transcript("https://youtube.com/watch?v=ID")     # ScrapeCreators
youtube_video("https://youtube.com/watch?v=ID")          # metadata: title/description/uploader (+ transcript)
sc_get("/v1/tiktok/profile", handle="charlidamelio")     # any SC endpoint

from exports import archive_fallback                      # paywall / Firecrawl-403
archive_fallback("https://www.nytimes.com/.../article.html")  # archive snapshot