Firecrawl Crawl

The firecrawl-crawl skill enables developers and content extraction teams to bulk extract structured content from entire websites or specific site sections by following links up to configurable depth and path limits. It solves the problem of efficiently scraping multiple interdependent pages at scale, handling concurrent extraction, and managing credit consumption for large-scale data collection tasks. The skill is typically used as the final step in a workflow after searching, scraping individual pages, and mapping site structure to extract complete documentation sections, archives, or site-wide content.

skills.sh installs: 81.2K
GitHub stars: 682
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add firecrawl/cli --skill firecrawl-crawl --agent claude-code

Installs into .claude/skills of the current project.

Files

firecrawl crawl

Bulk extract content from a website. Crawls pages following links up to a depth/limit.

Prerequisite: crawl requires authentication (no keyless free tier); without credentials the CLI prompts an interactive login.

Quick start

# Crawl a docs section
firecrawl crawl "<url>" --include-paths /docs --limit 50 --wait -o .firecrawl/crawl.json

# Full crawl with depth limit
firecrawl crawl "<url>" --max-depth 3 --wait --progress -o .firecrawl/crawl.json

# Check status of a running crawl
firecrawl crawl <job-id>

Run firecrawl crawl --help for the full option list.

Done when: the crawl reaches a terminal status and the saved output under .firecrawl/ contains the expected pages.

Tips

Use --wait when you need the results immediately. It has no default timeout; use --timeout <seconds> to bound polling. Without --wait, crawl returns a job ID for async polling.
Scope crawls with --include-paths whenever the request names a section — crawl only the pages you need.
Crawl consumes credits per page. Check firecrawl credit-usage before large crawls (credit-usage requires authentication).