Claude MarketplacesSubmit
brettdavies avatar

Crawl4ai

brettdavies/crawl4ai-skill

This is a full web scraping toolkit that converts websites to markdown, extracts structured data, and handles JavaScript-heavy pages. The standout feature is schema-based CSS extraction: you generate a JSON schema once using an LLM, then scrape repeatedly without any API costs. It's got both a CLI for quick jobs and a Python SDK for complex workflows. The content filtering is solid, using BM25 to strip irrelevant sections before you even process the text. If you're building documentation scrapers, price monitors, or any pipeline that needs clean structured data from messy web pages, this covers the spectrum from simple markdown conversion to authenticated multi-page crawls with proxy rotation.

skills.sh installs
1.2K
GitHub stars
50
last updated
Jul 14, 2026
View on GitHub

Install

$ npx -y skills add brettdavies/crawl4ai-skill --skill crawl4ai --agent claude-code

Installs into .claude/skills of the current project.

Files

Crawl4AI

Verified against crawl4ai VERSION. PEP 723 pins in scripts/*.py and tests/*.py floor at that version.

Overview

Crawl4AI wraps a headless browser (Playwright) plus a markdown-aware content pipeline. Use it when defuddle/curl can't reach the content — JavaScript-rendered pages, login-gated content, infinite scroll, multi-URL concurrency, repeatable schema-based extraction.

This skill exposes both interfaces of the underlying library:

  • CLI (crwl) — quick, scriptable commands: CLI Guide
  • Python SDK — full programmatic control: SDK Guide

Invoked with a URL argument

When the user runs /crawl4ai <url> with a single URL and no further qualifier, treat it as the JS-heavy fetch case and default to:

crwl <url> -c "wait_until=networkidle,page_timeout=60000" -o markdown
SKILL.md on GitHub

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language

Details

Repository
github.com/brettdavies/crawl4ai-skill
First seen
May 16, 2026

Recommended

1juliusbrussee avatarcavemanjuliusbrussee/cavemanUltra-compressed communication mode. Cuts output tokens 65% (measured) by speaking like caveman while keeping full technical accuracy. Supports intensity levels618.3Kskills.sh installs
2mattpocock avatargrill-memattpocock/skillsA relentless interview to sharpen a plan or design.1.5Mskills.sh installs
3shadcn avatarimproveshadcn/improveSurvey any codebase as a senior advisor and produce prioritized, self-contained implementation plans for OTHER models/agents to execute. Strictly read-only on s40Kskills.sh installs
4obra avatarsystematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes326.7Kskills.sh installs
5forrestchang avatarkarpathy-guidelinesforrestchang/andrej-karpathy-skillsBehavioral guidelines to reduce common LLM coding mistakes through explicit assumptions, simplicity, and verifiable success criteria.17.9Kskills.sh installs
6vercel-labs avatarfind-skillsvercel-labs/skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express int4.3Mskills.sh installs