Claude MarketplacesSubmit
liranudi avatar

Web Scraper

liranudi/openclaw-web-scraper

This is a practical web scraping toolkit that runs entirely locally with Playwright and Chromium, no API keys needed. You get four Python scripts: multi-page search across DuckDuckGo, Brave, or Google; one-shot page reading with automatic cookie banner dismissal; a persistent browser session that handles tabs, clicks, screenshots, and text search; and a file downloader that extracts text from PDFs. Everything outputs JSON by default, with markdown and plain text options. It's straightforward if you need to scrape data or automate browser tasks without managing headless browser boilerplate yourself. The persistent session mode is especially handy for multi-step interactions where you need to click around and extract data progressively.

skills.sh installs
250
GitHub stars
2
last updated
Jun 3, 2026
View on GitHub

Install

$ npx -y skills add liranudi/openclaw-web-scraper --skill web-scraper --agent claude-code

Installs into .claude/skills of the current project.

Files

Web Scraper

Four scripts, zero API keys. All output is JSON by default.

Dependencies: requests, beautifulsoup4, playwright (with Chromium). Optional: pdfplumber or PyPDF2 for PDF text extraction.

Install: pip install requests beautifulsoup4 playwright && playwright install chromium

1. Search the Web

python3 scripts/google_search.py "query" --pages N --engine ENGINE
  • --engine — duckduckgo (default), brave, or google
  • Returns [{title, url, snippet}, ...]

2. Read a Page (one-shot)

python3 scripts/read_page.py "https://url" [--max-chars N] [--visible] [--format json|markdown|text] [--no-dismiss]
  • --format — json (default), markdown, or text
  • Auto-dismisses cookie consent banners (skip with --no-dismiss)

3. Persistent Browser Session

SKILL.md on GitHub

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language

Details

Repository
github.com/liranudi/openclaw-web-scraper
First seen
Jun 3, 2026

Recommended

1juliusbrussee avatarcavemanjuliusbrussee/cavemanUltra-compressed communication mode. Cuts output tokens 65% (measured) by speaking like caveman while keeping full technical accuracy. Supports intensity levels618.3Kskills.sh installs
2mattpocock avatargrill-memattpocock/skillsA relentless interview to sharpen a plan or design.1.5Mskills.sh installs
3shadcn avatarimproveshadcn/improveSurvey any codebase as a senior advisor and produce prioritized, self-contained implementation plans for OTHER models/agents to execute. Strictly read-only on s40Kskills.sh installs
4obra avatarsystematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes326.7Kskills.sh installs
5forrestchang avatarkarpathy-guidelinesforrestchang/andrej-karpathy-skillsBehavioral guidelines to reduce common LLM coding mistakes through explicit assumptions, simplicity, and verifiable success criteria.17.9Kskills.sh installs
6vercel-labs avatarfind-skillsvercel-labs/skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express int4.3Mskills.sh installs