Data Pipeline Engineer

This one's for building and maintaining data pipelines with the modern stack: dbt, Airflow, Spark, Kafka. It pushes the medallion architecture pattern (Bronze/Silver/Gold layers) and emphasizes data quality gates at every step. The anti-patterns section is genuinely useful, calling out stuff like full table refreshes and hardcoded dates that I've seen break production pipelines. Comes with reference examples for complete dbt projects, Airflow DAGs with sensors, and Spark streaming jobs. The quality checklist is comprehensive without being preachy. If you're doing ETL work beyond basic scripts and need structure around incremental processing, orchestration, and data testing, this gives you that framework.

skills.sh installs: 119
GitHub stars: 113
last updated: Jun 3, 2026

View on GitHub

Install

$ npx -y skills add erichowens/some_claude_skills --skill data-pipeline-engineer --agent claude-code

Installs into .claude/skills of the current project.

Files

Data Pipeline Engineer

Expert data engineer specializing in ETL/ELT pipelines, streaming architectures, data warehousing, and modern data stack implementation.

Quick Start

Identify sources - data formats, volumes, freshness requirements
Choose architecture - Medallion (Bronze/Silver/Gold), Lambda, or Kappa
Design layers - staging → intermediate → marts (dbt pattern)
Add quality gates - Great Expectations or dbt tests at each layer
Orchestrate - Airflow DAGs with sensors and retries
Monitor - lineage, freshness, anomaly detection

Core Capabilities

Capability	Technologies	Key Patterns
Batch Processing	Spark, dbt, Databricks	Incremental, partitioning, Delta/Iceberg
Stream Processing	Kafka, Flink, Spark Streaming	Watermarks, exactly-once, windowing
Orchestration	Airflow, Dagster, Prefect	DAG design, sensors, task groups
Data Modeling	dbt, SQL	Kimball, Data Vault, SCD
Data Quality	Great Expectations, dbt tests	Validation suites, freshness