Firecrawl Knowledge Ingest

This is purpose-built for scraping documentation portals that normal scrapers choke on: JavaScript-heavy sites, login-gated content, paginated help centers, or anything with complex navigation. It uses Firecrawl's browser automation to actually navigate the site like a human would, following sidebars and pagination controls while extracting clean markdown or structured JSON. The workflow interviews you briefly for the portal URL and auth requirements, then crawls methodically while preserving code blocks and tables. Honest take: if you're trying to ingest Notion docs, enterprise knowledge bases, or modern SaaS documentation into a RAG system, this beats fighting with Puppeteer yourself. Outputs include metadata tracking, failed page reports, and a clean JSON shape ready for vector storage.

skills.sh installs: 31.6K
GitHub stars: 221
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add firecrawl/firecrawl-workflows --skill firecrawl-knowledge-ingest --agent claude-code

Installs into .claude/skills of the current project.

Files

Firecrawl Knowledge Ingest

Use this when a docs portal needs browser navigation, auth, pagination, or JS rendering.

Onboarding Interview

Infer the portal URL, output format, auth needs, and page limit from context. If the portal is clear, proceed immediately.

Ask at most 1-3 concise questions only if blocked, such as the portal URL, whether authentication is required, or the desired output format.

Firecrawl Collection Plan

Use Firecrawl browser to:

open the portal and inspect navigation
identify sections, categories, sidebar links, and article URLs
follow sidebar navigation, next links, pagination, load-more controls, or search
scrape article content as markdown
extract metadata such as title, section, last updated date, author, and tags

Try Firecrawl map as a supplement for public URLs, but use browser navigation for auth-gated or JS-heavy content.

Final Deliverable

# Knowledge Ingest: [Portal]

## Summary
[Pages extracted, sections covered, limitations]

## Output
[JSON/markdown/merged file path or content]

## Sections
[Section names and article counts]

## Failed Or Restricted Pages
[Any access/loading issues]

SKILL.md on GitHub