Crawl4AI RAG
The Crawl4AI RAG MCP server enables AI agents and coding assistants to crawl websites using Crawl4AI and store the content in a Supabase vector database for retrieval-augmented generation (RAG). It provides tools for web scraping and knowledge retrieval, supporting advanced RAG strategies including contextual embeddings, hybrid search, agentic RAG, reranking, and knowledge graphs to enhance retrieval quality and reduce hallucinations. The server solves the problem of integrating web crawling with semantic search capabilities, allowing AI agents to leverage real-time web content as external knowledge sources.
- GitHub stars
- 2.2K
- tools
- 8
- last updated
- Dec 15, 2025
Tools
8 toolsPublic tool metadata for what this MCP can expose to an agent.
crawl_single_pageQuickly crawl a single web page and store its content in the vector database
Quickly crawl a single web page and store its content in the vector database
No parameter schema in public metadata yet.
smart_crawl_urlIntelligently crawl a full website based on the type of URL provided (sitemap, llms-full.txt, or a regular webpage that needs to be crawled recursively)
Intelligently crawl a full website based on the type of URL provided (sitemap, llms-full.txt, or a regular webpage that needs to be crawled recursively)
No parameter schema in public metadata yet.
get_available_sourcesGet a list of all available sources (domains) in the database
Get a list of all available sources (domains) in the database
No parameter schema in public metadata yet.
perform_rag_querySearch for relevant content using semantic search with optional source filtering
Search for relevant content using semantic search with optional source filtering
No parameter schema in public metadata yet.
search_code_examplesSearch specifically for code examples and their summaries from crawled documentation. This tool provides targeted code snippet retrieval for AI coding assistants.
Search specifically for code examples and their summaries from crawled documentation. This tool provides targeted code snippet retrieval for AI coding assistants.
No parameter schema in public metadata yet.
parse_github_repositoryParse a GitHub repository into a Neo4j knowledge graph, extracting classes, methods, functions, and their relationships for hallucination detection
Parse a GitHub repository into a Neo4j knowledge graph, extracting classes, methods, functions, and their relationships for hallucination detection
No parameter schema in public metadata yet.
check_ai_script_hallucinationsAnalyze Python scripts for AI hallucinations by validating imports, method calls, and class usage against the knowledge graph
Analyze Python scripts for AI hallucinations by validating imports, method calls, and class usage against the knowledge graph
No parameter schema in public metadata yet.
query_knowledge_graphExplore and query the Neo4j knowledge graph with commands like `repos`, `classes`, `methods`, and custom Cypher queries
Explore and query the Neo4j knowledge graph with commands like `repos`, `classes`, `methods`, and custom Cypher queries
No parameter schema in public metadata yet.
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageReadme
Crawl4AI RAG MCP Server
Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants
A powerful implementation of the Model Context Protocol (MCP) integrated with Crawl4AI and Supabase for providing AI agents and AI coding assistants with advanced web crawling and RAG capabilities.
With this MCP server, you can scrape anything and then use that knowledge anywhere for RAG.
The primary goal is to bring this MCP server into Archon as I evolve it to be more of a knowledge engine for AI coding assistants to build AI agents. This first version of the Crawl4AI/RAG MCP server will be improved upon greatly soon, especially making it more configurable so you can use different embedding models and run everything locally with Ollama.
Consider this GitHub repository a testbed, hence why I haven't been super actively address issues and pull requests yet. I certainly will though as I bring this into Archon V2!
Overview
Details
- Repository
- github.com/coleam00/mcp-crawl4ai-rag
- Categories
- Databases, AI & LLM Tools, Documents & Knowledge, Search & Web Crawling
More from coleam00
Related Databases MCP servers
View all
Monitorbetterdb-inc/monitorBetterDB MCP server - Valkey and Redis observability for Claude Code and other MCP clients1.3Kstars
Queryweaverfalkordb/queryweaverAn MCP server for Text2SQL: transforms natural language into SQL using graph schema understanding.1Kstars
Mcp Server Motherduckmotherduckdb/mcp-server-motherduckSQL analytics and data engineering for AI Assistants and IDEs540stars
Memory Vaultmihaibuilds/memory-vaultLocal-first AI memory layer with hybrid search. Postgres + pgvector. Self-hosted, MIT.67stars
Orionbelt Analyticsralfbecher/orionbelt-analyticsOntology-based MCP server for database schema analysis and RDF/OWL ontology generation32stars
Greptimedb Mcp Servergreptimeteam/greptimedb-mcp-serverQuery GreptimeDB metrics, logs and traces with SQL, TQL and RANGE. Manage pipelines and dashboards.29stars