ShadowCrawl
Built for agents that hit rate limits and bot challenges during web research. Exposes unified tools over MCP stdio: web_search with optional inline content extraction, web_fetch with single/batch/crawl modes, extract_fields for structured data, and a stateful browser_automate omni-tool that replaces Playwright with one-shot step arrays covering navigation, locators, assertions, network mocking, and storage state. The CDP engine runs headless in an isolated agent profile that persists cookies and sessions across calls. When automation hits a CAPTCHA or OAuth wall, scout_agent_profile_auth opens a visible window for manual solve, then returns to silent operation. Includes hitl_web_fetch for challenge and auth workflows, plus memory_search backed by LanceDB for retrieval history. Handles Cloudflare, DataDome, reCAPTCHA, and Turnstile through progressive fallbacks and proxy rotation. Written in Rust, ships as a single binary.
- GitHub stars
- 66
- tools
- 10
- last updated
- Feb 18, 2026
Tools
10 toolsPublic tool metadata for what this MCP can expose to an agent.
web_searchURL discovery, or search plus content in one call with include_content=true.
URL discovery, or search plus content in one call with include_content=true.
No parameter schema in public metadata yet.
web_fetchUnified fetch family with mode="single", "batch", or "crawl".
Unified fetch family with mode="single", "batch", or "crawl".
No parameter schema in public metadata yet.
extract_fieldsPrimary structured extraction tool.
Primary structured extraction tool.
No parameter schema in public metadata yet.
scout_browser_automateStateful omni-tool for browser automation (also callable as browser_automate).
Stateful omni-tool for browser automation (also callable as browser_automate).
No parameter schema in public metadata yet.
scout_agent_profile_authAgent profile authentication for browser automation.
Agent profile authentication for browser automation.
No parameter schema in public metadata yet.
scout_browser_closeCloses the browser automation session.
Closes the browser automation session.
No parameter schema in public metadata yet.
visual_scoutHuman-in-the-loop visual scouting tool.
Human-in-the-loop visual scouting tool.
No parameter schema in public metadata yet.
hitl_web_fetchHuman-in-the-loop fetch with auth_mode="challenge" or "auth".
Human-in-the-loop fetch with auth_mode="challenge" or "auth".
No parameter schema in public metadata yet.
memory_searchSearches LanceDB-backed research history.
Searches LanceDB-backed research history.
No parameter schema in public metadata yet.
deep_researchMulti-hop search, scrape, and synthesis.
Multi-hop search, scrape, and synthesis.
No parameter schema in public metadata yet.
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageReadme
CortexScout (cortex-scout) — Search and Web Extraction Engine for AI Agents
CortexScout is the Deep Research & Web Extraction module within the Cortex-Works ecosystem.
Designed for agent workloads that require token-efficient web retrieval, reliable anti-bot handling, and optional Human-in-the-Loop (HITL) fallback.
Overview
CortexScout provides a single, self-hostable Rust binary that exposes search, extraction, and stateful browser automation capabilities over MCP (stdio) and an optional HTTP server. Output formats are structured and optimized for downstream LLM use.
Details
- Repository
- github.com/devshero/shadowcrawl
- Category
- Search & Web Crawling
- Registry
- MCP registry, active
More from devshero
Related Search & Web Crawling MCP servers
View all
Olostep MCP Serverolostep/olostep-mcp-serverLive web search, scraping and crawling API for AI. Get structured, LLM-ready data from any site.25stars
Zenrows Mcpzenrows/zenrows-mcpZenrows MCP server: Fetch, Extract, Batch, and Browser Sessions for AI agents21stars
Mcpscrapfly/scrapfly-mcpScrape any website, extract structured data, and collect web content at scale with AI agents15stars
CodeWiki MCP — AI-generated docs for GitHub reposizzzzzi/codewiki-mcpMCP server for codewiki.google: search repos, fetch wiki docs, ask questions9stars
Webshiftannibale-x/webshiftRust MCP server for denoised web search — fetch, clean, and rerank web content for AI agents.7stars
Arxiv Mcp Servercyanheads/arxiv-mcp-serverSearch arXiv, fetch paper metadata, and read full-text content.7stars