Engineering Data Pipelines

This is a reference guide for building production data pipelines, covering both batch orchestration (Airflow, Dagster, Prefect) and stream processing (Kafka Streams, Flink, Spark Streaming). It gives you checklists for the things that actually matter: idempotency patterns, incremental loads, watermark tuning for out-of-order events, state management in streaming jobs, and layered data quality checks with tools like Great Expectations and dbt tests. The bilingual formatting is a bit quirky, but the technical checklists are solid. Use this when you're setting up ETL workflows or real-time streaming and need a quick reference for best practices around exactly-once semantics, backpressure monitoring, or cross-DAG dependencies without digging through framework docs.

GitHub stars: 224
last updated: Jun 10, 2026

View on GitHub

Install

$ npx -y skills add telagod/code-abyss --skill engineering-data-pipelines --agent claude-code

Installs into .claude/skills of the current project.

Files

数据工程域 · Data Engineering

判断先于执行：决定「是否做 / 选什么 / 如何取舍」（栈、方案、架构、权衡）前，先读领域判断内核 skills/_kernel/backend/SKILL.md——它管 judgment，本秘典管 execution；冲突时以内核判断为准。

编排：Airflow(调度) | Dagster(资产) | Prefect(现代流)
流处理：Kafka Streams(嵌入式) | Flink(集群) | Spark Streaming
质量：Great Expectations | dbt tests | Soda Core

编排检查项

幂等(UPSERT/分区覆盖) | 增量(WHERE updated_at > last_run) | 事件驱动触发 | 跨 DAG 依赖 | 数据血缘(ref()/Asset deps)

流处理检查项

质量检查项

分层验证(源→转换→目标) | 完整性+准确性+一致性 | 及时性阈值 | 加权评分 | 告警(Slack/PagerDuty)

工具对比、API 用法、质量维度详见 references/details.md

SKILL.md on GitHub

Details

Repository: github.com/telagod/code-abyss
Category: Code Review & Quality
License: MIT
First seen: Jun 11, 2026