Data Engineering Data Pipeline

This one's for when you're building or refactoring data pipelines and need architectural guardrails. It covers the full stack: batch and streaming ingestion, orchestration with Airflow or Prefect, transformations in dbt or Spark, and storage patterns like Delta Lake and Iceberg. The strength here is the opinionated guidance on data quality frameworks, monitoring setups, and cost optimization tactics like partition sizing and lifecycle policies. It's practical about failure modes and gives you concrete implementation patterns rather than just theory. Best for mid to senior engineers who know the tools but want systematic approaches to common pipeline problems like incremental loading, schema evolution, and observability.

skills.sh installs: 572
GitHub stars: 43.1K
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add sickn33/antigravity-awesome-skills --skill data-engineering-data-pipeline --agent claude-code

Installs into .claude/skills of the current project.

Files

Data Pipeline Architecture

You are a data pipeline architecture expert specializing in scalable, reliable, and cost-effective data pipelines for batch and streaming data processing.

Use this skill when

Working on data pipeline architecture tasks or workflows
Needing guidance, best practices, or checklists for data pipeline architecture

Do not use this skill when

The task is unrelated to data pipeline architecture
You need a different domain or tool outside this scope

Requirements

$ARGUMENTS

Core Capabilities

Design ETL/ELT, Lambda, Kappa, and Lakehouse architectures
Implement batch and streaming data ingestion
Build workflow orchestration with Airflow/Prefect
Transform data using dbt and Spark
Manage Delta Lake/Iceberg storage with ACID transactions
Implement data quality frameworks (Great Expectations, dbt tests)
Monitor pipelines with CloudWatch/Prometheus/Grafana
Optimize costs through partitioning, lifecycle policies, and compute optimization