Data Scraper Agent

This is a complete blueprint for building scrapers that run themselves. It walks you through scraping any public source (job boards, prices, news, repos), enriching results with Gemini Flash, storing to Notion or Sheets, and running it all free on GitHub Actions. The architecture is smart: batch API calls to stay under rate limits, model fallback chains when quotas hit, and a feedback loop so the agent learns from your decisions over time. You get actual Python templates for HTML scraping, API parsing, and RSS feeds. Use this when you want to monitor something continuously without paying for servers or when you need an agent that gets better at filtering signal from noise.

skills.sh installs: 4.7K
GitHub stars: 229.1K
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add affaan-m/everything-claude-code --skill data-scraper-agent --agent claude-code

Installs into .claude/skills of the current project.

Files

Data Scraper Agent

Build a production-ready, AI-powered data collection agent for any public data source. Runs on a schedule, enriches results with a free LLM, stores to a database, and improves over time.

Stack: Python · Gemini Flash (free) · GitHub Actions (free) · Notion / Sheets / Supabase

When to Activate

User wants to gather or monitor any public website or API
User says "build a bot that checks...", "monitor X for me", "collect data from..."
User wants to track jobs, prices, news, repos, sports scores, events, listings
User asks how to automate data collection without paying for hosting
User wants an agent that gets smarter over time based on their decisions

Core Concepts

The Three Layers

Every data collection agent has three layers:

COLLECT → ENRICH → STORE
  │           │        │
Scraper    AI (LLM)  Database
runs on    scores/   Notion /
schedule   summarises Sheets /
           & classifies Supabase