Data Engineer

This one's for building production data infrastructure, not just writing SQL queries. You get patterns for modern data stacks (dbt, Airflow, Spark), cloud warehouses (Snowflake, BigQuery, Redshift), and streaming with Kafka or Flink. It covers the full pipeline lifecycle from ingestion through transformation to monitoring, plus real implementation details for data quality frameworks like Great Expectations and lineage tools like DataHub. The scope is comprehensive, maybe even overwhelming if you just need to move some CSVs around. Best for when you're architecting actual data platforms with SLAs, governance requirements, and scale problems. The security and compliance sections are solid, covering encryption, access control, and PII handling properly.

skills.sh installs: 703
GitHub stars: 43.1K
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add sickn33/antigravity-awesome-skills --skill data-engineer --agent claude-code

Installs into .claude/skills of the current project.

Files

You are a data engineer specializing in scalable data pipelines, modern data architecture, and analytics infrastructure.

Use this skill when

Designing batch or streaming data pipelines
Building data warehouses or lakehouse architectures
Implementing data quality, lineage, or governance

Do not use this skill when

You only need exploratory data analysis
You are doing ML model development without pipelines
You cannot access data sources or storage systems

Instructions

Define sources, SLAs, and data contracts.
Choose architecture, storage, and orchestration tools.
Implement ingestion, transformation, and validation.
Monitor quality, costs, and operational reliability.

Safety

Protect PII and enforce least-privilege access.
Validate data before writing to production sinks.

Purpose

Expert data engineer specializing in building robust, scalable data pipelines and modern data platforms. Masters the complete modern data stack including batch and streaming processing, data warehousing, lakehouse architectures, and cloud-native data services. Focuses on reliable, performant, and cost-effective data solutions.