Web Scraper
A straightforward Python scraper that saves web pages as HTML or Markdown with downloaded images, no browser automation required. Just requests and BeautifulSoup4. Single page mode is fast for archiving articles or documentation. Recursive mode follows links with sensible guardrails: depth limits, rate limiting, robots.txt compliance by default. Images get saved to a local folder automatically, which is nice for offline viewing. The recursive crawling is sequential rather than parallel, so it's polite but not blazing fast. Good for documentation snapshots or building local archives when you need actual files on disk, not just viewing in a browser.
- skills.sh installs
- 230
- GitHub stars
- 50
- last updated
- Aug 21, 2026
Install
$ npx -y skills add agentbay-ai/agentbay-skills --skill web-scraper --agent claude-codeInstalls into .claude/skills of the current project.
Files
Web Scraper
Fetch web page content (text + images) and save as HTML or Markdown locally.
Minimal dependencies: Only requires requests and beautifulsoup4 - no browser automation.
Default behavior: Downloads images to local images/ directory automatically.
Quick start
Single page
{baseDir}/scripts/scrape.py --url "https://example.com" --format html --output /tmp/page.html
{baseDir}/scripts/scrape.py --url "https://example.com" --format md --output /tmp/page.md
Recursive (follow links)
{baseDir}/scripts/scrape.py --url "https://docs.example.com" --format md --recursive --max-depth 2 --output ~/Downloads/docs-archive
Setup
Requires Python 3.8+ and minimal dependencies:
cd {baseDir}
pip install -r requirements.txt
Or install manually:
pip install requests beautifulsoup4
Note: No browser or driver needed - uses pure HTTP requests.
Inputs to collect
Single page mode
- URL: The web page to scrape (required)
- Format:
htmlormd(default:html)
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/agentbay-ai/agentbay-skills
- Category
- Frontend Development
- First seen
- Jun 3, 2026
More from agentbay-ai/agentbay-skills
All 6Recommended
More Frontend Development
optimizing-performancecloudai-x/claude-workflow-v2Analyzes and optimizes application performance across frontend, backend, and database layers. Use when diagnosing slowness, improving load times, optimizing que225skills.sh installs
ln-832-bundle-optimizerlevnikolaevich/claude-code-skillsReduces JS/TS bundle size via tree-shaking, code splitting, and unused dependency removal. Use when optimizing frontend bundle size.225skills.sh installs
hyperframesnexu-io/open-designCreate video compositions, animations, title cards, overlays, captions, voiceovers, audio-reactive visuals, and scene transitions in HyperFrames HTML. Use when 220skills.sh installs
cucumber-gherkinel-feo/ai-contextBDD testing with Cucumber and Gherkin for Ruby and Rails applications. Use when writing feature files (.feature), step definitions, hooks, or implementing Behav215skills.sh installs
axiom-ios-uicharleswiltgen/axiomUse when building, fixing, or improving ANY iOS UI including SwiftUI, UIKit, layout, navigation, animations, design guidelines. Covers view updates, layout bugs213skills.sh installsreplit-decknexu-io/open-designSingle-file horizontal-swipe HTML deck in the style of Replit Slides's landing-page template gallery. Eight distinct themes (helix, holm, vance, bevel, world-da209skills.sh installs