ShadowCrawl
Built for AI agents that need to scrape protected sites without getting blocked. Exposes a unified MCP toolkit covering search, extraction, stateful browser automation, and human-in-the-loop fallback for captchas and auth walls. The browser automation layer replaces Playwright with a single omni-tool that handles navigation, clicks, waits, assertions, network mocking, and screenshot capture in one call, all running in a persistent headless profile. When automated flow hits Cloudflare Turnstile or DataDome, you can invoke the HITL auth portal to solve it once, then the agent resumes silently. Ships as a self-contained Rust binary with validation artifacts for LinkedIn, Ticketmaster, Airbnb, and other heavily protected targets. Reach for this when web scraping needs to survive real-world bot detection at agent speed.
- GitHub stars
- 66
- tools
- 11
- transport
- HTTP
- last updated
- Feb 14, 2026
Tools
11 toolsPublic tool metadata for what this MCP can expose to an agent.
web_searchURL discovery, or search plus content in one call with include_content=true.
URL discovery, or search plus content in one call with include_content=true.
No parameter schema in public metadata yet.
web_fetchUnified fetch family with mode single, batch or crawl.
Unified fetch family with mode single, batch or crawl.
No parameter schema in public metadata yet.
extract_fieldsPrimary structured extraction.
Primary structured extraction.
No parameter schema in public metadata yet.
scout_browser_automateStateful omni-tool for browser automation.
Stateful omni-tool for browser automation.
No parameter schema in public metadata yet.
browser_automateStateful omni-tool for browser automation (alias of scout_browser_automate).
Stateful omni-tool for browser automation (alias of scout_browser_automate).
No parameter schema in public metadata yet.
scout_agent_profile_auth
No parameter schema in public metadata yet.
scout_browser_close
No parameter schema in public metadata yet.
visual_scout
No parameter schema in public metadata yet.
hitl_web_fetchHuman-in-the-loop fetch with auth_mode challenge or auth.
Human-in-the-loop fetch with auth_mode challenge or auth.
No parameter schema in public metadata yet.
memory_searchSearches LanceDB-backed research history.
Searches LanceDB-backed research history.
No parameter schema in public metadata yet.
deep_researchMulti-hop search, scrape and synthesis.
Multi-hop search, scrape and synthesis.
No parameter schema in public metadata yet.
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageReadme
CortexScout (cortex-scout) — Search and Web Extraction Engine for AI Agents
CortexScout is the Deep Research & Web Extraction module within the Cortex-Works ecosystem.
Designed for agent workloads that require token-efficient web retrieval, reliable anti-bot handling, and optional Human-in-the-Loop (HITL) fallback.
Overview
CortexScout provides a single, self-hostable Rust binary that exposes search, extraction, and stateful browser automation capabilities over MCP (stdio) and an optional HTTP server. Output formats are structured and optimized for downstream LLM use.
Details
- Repository
- github.com/devshero/shadowcrawl
- Categories
- Cloud & Infrastructure, Search & Web Crawling
- Transport
- HTTP
- Registry
- MCP registry, active
More from devshero
Related Cloud & Infrastructure MCP servers
View all
Porkbun (official)oborseth/porkbun-mcpOfficial Porkbun MCP server: domains, DNS, SSL, hosting and Cloudflare via the Porkbun API.36stars
Awesome Agent Skillsshadowrootdev/awesome-agent-skills-mcpMCP server with 100+ AI agent skills from Anthropic, Vercel, Trail of Bits, Hugging Face & more.23stars
OneTool MCPbeycom/onetool-mcpOne MCP for devs incl Brave,Google,Tavily,Context7,AWS,Excalidraw,DB,GitHub,DevTools,Playwright20stars
Interactive Terminalamol21p/mcp-interactive-terminalMCP server for real interactive terminal sessions — REPLs, SSH, databases, Docker15stars
Synapse Layer — Trust Infrastructure for AI Agentssynapselayer/synapse-layerMCP-native Trust Infrastructure for AI Agents. Persistent encrypted memory with Trust Quotient.13stars
Lumino MCP Serverspre-sre/lumino-mcp-serverAI-powered SRE observability for Kubernetes/OpenShift with 40+ Tekton debugging tools8stars