Crawlforge Mcp Server
A comprehensive scraping toolkit that brings 26 web extraction tools into Claude and other MCP clients. The server includes both free local operations (text extraction, link parsing, metadata scraping) and metered premium features (deep crawling, autonomous research agent, stealth browsing). Notable is the extract_with_llm tool that runs against local Ollama models by default, keeping your data on your machine with no per-token cost. The autonomous agent tool is interesting: describe what you need in natural language and it plans and executes the research without requiring URLs upfront. Includes pre-built templates for common sites like Amazon, GitHub, and YouTube. Free tier starts at 1,000 credits with rollover, and the 15 local tools require no API key at all.
- GitHub stars
- 6
- tools
- 31
- auth
- Secret needed
- transport
- STDIO
- env vars
- 3
- last updated
- Nov 12, 2026
Tools
31 toolsPublic tool metadata for what this MCP can expose to an agent.
fetch_urlFetch content from any URL
Fetch content from any URL
No parameter schema in public metadata yet.
extract_textExtract clean text from web pages
Extract clean text from web pages
No parameter schema in public metadata yet.
extract_linksGet all links from a page
Get all links from a page
No parameter schema in public metadata yet.
extract_metadataExtract page metadata (title, OG tags, schema.org)
Extract page metadata (title, OG tags, schema.org)
No parameter schema in public metadata yet.
scrape_templateStructured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors
Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors
No parameter schema in public metadata yet.
list_ollama_modelsList the Ollama models installed locally (helps you pick a model for extract_with_llm)
List the Ollama models installed locally (helps you pick a model for extract_with_llm)
No parameter schema in public metadata yet.
get_batch_resultsRetrieve paginated results for a batch_scrape job by batchId
Retrieve paginated results for a batch_scrape job by batchId
No parameter schema in public metadata yet.
read_resultSearch, slice, read lines or a JSON path from a result a tool returned with truncated: true and a result_handle
Search, slice, read lines or a JSON path from a result a tool returned with truncated: true and a result_handle
No parameter schema in public metadata yet.
scrapeUnified single-fetch, multi-format extraction with a formats array and onlyMainContent
Unified single-fetch, multi-format extraction with a formats array and onlyMainContent
No parameter schema in public metadata yet.
scrape_structuredExtract structured data with CSS selectors
Extract structured data with CSS selectors
No parameter schema in public metadata yet.
extract_embedded_stateRead a page's embedded JavaScript state such as __NEXT_DATA__, RSC payloads, Nuxt, SvelteKit, Apollo, Redux and JSON data attributes
Read a page's embedded JavaScript state such as __NEXT_DATA__, RSC payloads, Nuxt, SvelteKit, Apollo, Redux and JSON data attributes
No parameter schema in public metadata yet.
extract_contentEnhanced content extraction
Enhanced content extraction
No parameter schema in public metadata yet.
map_siteDiscover and map website structure (optional search= ranks the discovered URLs)
Discover and map website structure (optional search= ranks the discovered URLs)
No parameter schema in public metadata yet.
process_documentMulti-format document processing
Multi-format document processing
No parameter schema in public metadata yet.
localizationLook up a country's locale settings (Accept-Language, timezone, currency) to pass to other tools
Look up a country's locale settings (Accept-Language, timezone, currency) to pass to other tools
No parameter schema in public metadata yet.
track_changesMonitor content changes over time
Monitor content changes over time
No parameter schema in public metadata yet.
analyze_contentComprehensive content analysis
Comprehensive content analysis
No parameter schema in public metadata yet.
extract_structuredLLM-powered schema-driven extraction
LLM-powered schema-driven extraction
No parameter schema in public metadata yet.
extract_with_llmNatural-language extraction using a local Ollama model or cloud LLM providers
Natural-language extraction using a local Ollama model or cloud LLM providers
No parameter schema in public metadata yet.
browser_sessionA browser page that stays open across calls, keeping cookies and login between calls (open, snapshot, act, read, close)
A browser page that stays open across calls, keeping cookies and login between calls (open, snapshot, act, read, close)
No parameter schema in public metadata yet.
summarize_contentGenerate intelligent summaries
Generate intelligent summaries
No parameter schema in public metadata yet.
crawl_deepDeep crawl entire websites
Deep crawl entire websites
No parameter schema in public metadata yet.
search_webSearch the live web through the CrawlForge API
Search the live web through the CrawlForge API
No parameter schema in public metadata yet.
reddit_searchSearch Reddit posts/comments or read a full thread via community archives
Search Reddit posts/comments or read a full thread via community archives
No parameter schema in public metadata yet.
serp_rankCheck where a domain ranks in Google's organic SERP for a keyword
Check where a domain ranks in Google's organic SERP for a keyword
No parameter schema in public metadata yet.
batch_scrapeProcess multiple URLs simultaneously
Process multiple URLs simultaneously
No parameter schema in public metadata yet.
scrape_with_actionsBrowser automation chains
Browser automation chains
No parameter schema in public metadata yet.
generate_llms_txtGenerate AI interaction guidelines
Generate AI interaction guidelines
No parameter schema in public metadata yet.
stealth_modeAnti-detection browser management
Anti-detection browser management
No parameter schema in public metadata yet.
agentAutonomous research/extraction from a natural-language prompt without URLs required
Autonomous research/extraction from a natural-language prompt without URLs required
No parameter schema in public metadata yet.
deep_researchMulti-stage research with source verification
Multi-stage research with source verification
No parameter schema in public metadata yet.
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageConfiguration
CRAWLFORGE_API_KEY*secretCrawlForge API key for authentication and usage/credit tracking. Get a free key at https://www.crawlforge.dev
DATAFORSEO_LOGINsecretDataForSEO account login. Optional — only needed to enable the serp_rank tool (real Google organic rank lookups); billed separately to the caller's own DataForSEO account.
DATAFORSEO_PASSWORDsecretDataForSEO account password. Optional — only needed to enable the serp_rank tool.
Readme
31 web scraping, crawling, deep-research & autonomous-extraction tools for Claude, Cursor & any MCP client.
Clean Markdown & structured JSON from any site. Get started with 1,000 free credits — no credit card required.
Details
- Repository
- github.com/mysleekdesigns/crawlforge-mcp
- Website
- www.crawlforge.dev
- Categories
- Documents & Knowledge, Search & Web Crawling, Data & Analytics
- Package
crawlforge-mcp-server- Transport
- STDIO
- Auth
- Required
- Registry
- MCP registry, active
Related Documents & Knowledge MCP servers
View all
RAGMapkhalidsaidi/ragmapFind a retrieval MCP server that works today: a daily-checked list of servers that search knowledge.6stars
SideButton MCP Serversidebutton/sidebuttonOpen-source MCP server with browser automation and knowledge packs for AI agents6stars
grokipedia-mcpskymoore/grokipedia-mcpSearch and retrieve Grokipedia articles with filters, full content, and citations. Explore related pages and extract sections to quickly gather relevant knowled6stars
Wickwickproject/wickBrowser-grade web access for AI agents. Fetches any webpage, returns clean markdown.6stars
Langchain Mcpbaixianger/langchain-mcpLangChain/LangGraph/DeepAgents knowledge + LangGraph agent debugging (Polly-like)5stars
Scrapibamchi/scrapi-mcp-serverWeb scraping for AI agents. Converts URLs to clean, LLM-ready Markdown with anti-bot bypass.5stars