Web Scraping Automation

This handles web scraping and API automation with Python or JavaScript stacks like requests, BeautifulSoup, Selenium, and Playwright. It'll parse HTML, call REST endpoints, handle dynamic pages, and even deal with anti-scraping measures like request throttling and header spoofing. The workflow is straightforward: analyze the target, pick your tools, write the scraper, test and optimize. One thing worth noting is the explicit focus on resource cleanup, especially killing Chrome processes after browser automation runs, which shows this was built with real production headaches in mind. Includes practical examples for static scraping, API calls, and dynamic content extraction.

skills.sh installs: 580
GitHub stars: 74
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aaaaqwq/claude-code-skills --skill web-scraping-automation --agent claude-code

Installs into .claude/skills of the current project.

Files

网站爬取与 API 自动化

功能说明

此技能专门用于自动化网站数据爬取和 API 接口调用，包括：

分析和爬取网站结构
调用和测试 REST/GraphQL API
创建自动化爬虫脚本
数据解析和清洗
处理反爬虫机制
定时任务和数据存储

使用场景

"爬取这个网站的产品信息"
"帮我调用这个 API 并解析返回数据"
"创建一个脚本定时抓取新闻"
"分析这个网站的 API 接口文档"
"绕过这个网站的反爬虫限制"

技术栈

⚠️ 资源清理原则（强制）

所有涉及浏览器的爬取任务完成后，必须自动关闭 Chrome/Selenium 进程！

# Playwright 示例
from playwright.sync_api import sync_playwright

def scrape_website():
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page()
        # ... 爬取逻辑 ...
        browser.close()

    # ⚠️ 强制清理残留进程
    import subprocess
    subprocess.run(['pkill', '-f', 'chrome'], capture_output=True)

# Selenium 示例
from selenium import webdriver

driver = webdriver.Chrome()
try:
    # ... 爬取逻辑 ...
    pass
finally:
    driver.quit()
    # ⚠️ 确保清理
    import subprocess
    subprocess.run(['pkill', '-f', 'chrome'], capture_output=True)

原因: 避免内存泄漏和资源占用，防止 Gateway CPU 100% 过载