News Extractor

This pulls article content from 12 news platforms (WeChat, Toutiao, BBC, CNN, Twitter, Quora, and six others) and spits out both JSON and Markdown files. It auto-detects the platform from the URL, handles the scraping with curl_cffi for browser simulation, and includes retry logic via tenacity. Twitter support works for public posts without auth, but protected tweets need cookies. The whole thing is self-contained with no external dependencies beyond the Python packages, so you can drop it into other projects. Main use case is when you need structured article data or clean Markdown instead of dealing with different site layouts yourself. Just run it with uv, point it at a URL, and it handles the extraction.

skills.sh installs: 408
GitHub stars: 639
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add nanmicoder/newscrawler --skill news-extractor --agent claude-code

Installs into .claude/skills of the current project.

Files

News Extractor Skill

从主流新闻平台提取文章内容，输出 JSON 和 Markdown 格式。

标准兼容、独立可迁移：本目录遵循 Agent Skills 规范，包含完整的 SKILL.md、脚本与依赖，不依赖 NewsCrawler 的其他模块。通过兼容的 Skills 安装器安装后，运行一次 uv sync 即可使用。

运行环境需要 Python 3.9+、uv 和目标站点的网络访问。

支持平台 (12)

中文平台

平台	ID	URL 示例
微信公众号	wechat	`https://mp.weixin.qq.com/s/xxxxx`
今日头条	toutiao	`https://www.toutiao.com/article/123456/`
网易新闻	netease	`https://www.163.com/news/article/ABC123.html`
搜狐新闻	sohu	`https://www.sohu.com/a/123456_789`
腾讯新闻	tencent	`https://news.qq.com/rain/a/20251016A07W8J00`

国际平台

平台	ID	URL 示例
BBC News	bbc	`https://www.bbc.com/news/articles/c797qlx93j0o`
CNN News	cnn	`https://edition.cnn.com/2025/10/27/uk/article-slug`
Twitter/X	twitter	`https://x.com/user/status/123456789`
Lenny's Newsletter	lenny	`https://www.lennysnewsletter.com/p/article-slug`
Naver Blog	naver	`https://blog.naver.com/username/123456`
Detik News	detik	`https://news.detik.com/internasional/d-123456/slug`
Quora	quora	`https://www.quora.com/question/answers/123456`