Xcrawl Map

This is a pre-crawl reconnaissance tool that hits XCrawl's Map API to discover URLs across a site before you commit to a full scrape. You give it an entry point, optional regex filters, and limits, and it returns a raw list of discovered links with credit costs. It's built around curl and node with local API key config, no Python required. Useful when you need to estimate scope, validate URL patterns, or cherry-pick sections of a large site instead of crawling blindly. The skill explicitly returns raw API responses by default, which is refreshing if you'd rather handle your own analysis than get an AI summary of what URLs look like.

skills.sh installs: 217
GitHub stars: 449
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add xcrawl-api/xcrawl-skills --skill xcrawl-map --agent claude-code

Installs into .claude/skills of the current project.

Files

XCrawl Map

Overview

This skill uses XCrawl Map API to discover URLs for a site. Default behavior is raw passthrough: return upstream API response bodies as-is.

Required Local Config

Before using this skill, the user must create a local config file and write XCRAWL_API_KEY into it.

Path: ~/.xcrawl/config.json

{
  "XCRAWL_API_KEY": "<your_api_key>"
}

Read API key from local config file only. Do not require global environment variables.

Credits and Account Setup

Using XCrawl APIs consumes credits. If the user does not have an account or available credits, guide them to register at https://dash.xcrawl.com/. After registration, they can activate the free 1000 credits plan before running requests.