Senior Data Engineer

This one generates Airflow DAGs, validates data quality with Great Expectations, and optimizes SQL/Spark queries with specific recommendations. It's built around three Python tools that handle pipeline orchestration, quality checks, and performance tuning. The workflows are solid: batch ETL with dbt and Snowflake, real-time streaming with Kafka and Spark, plus a data contract validation framework. The decision trees for batch vs streaming and warehouse vs lakehouse are actually helpful. If you're wiring together modern data stacks and need generated boilerplate plus validation hooks, this covers the repetitive parts. The SQL optimization tool estimates costs too, which is nice for keeping BigQuery bills in check.

skills.sh installs: 195
GitHub stars: 1.1K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add borghei/claude-skills --skill senior-data-engineer --agent claude-code

Installs into .claude/skills of the current project.

Files

Senior Data Engineer

Generate pipeline configurations (Airflow, Prefect, Dagster), validate data quality with profiling and anomaly detection, and optimize SQL/Spark performance with actionable recommendations.

Core Capabilities

Pipeline generation — Airflow/Prefect/Dagster DAG code for batch and incremental loads, with DAG validation.
Data quality — schema validation, profiling, anomaly detection, data contracts, and Great Expectations suite generation.
ETL/ELT optimization — SQL and Spark analysis, partition strategy, and query cost estimation per warehouse.
Architecture decisions — batch vs streaming and warehouse vs lakehouse trade-off frameworks.
Reliability patterns — incremental watermarks, dead letter queues, freshness checks, and schema-drift detection.

When to Use

Designing a data architecture or choosing batch vs streaming / warehouse vs lakehouse.
Building or generating Airflow/Spark/dbt pipelines.
Adding data-quality checks or data contracts.
Optimizing slow ETL/ELT queries or troubleshooting pipeline failures.