Data Engineering Medallion Pipeline
This is a complete local data engineering stack that implements medallion architecture using Docker Compose. You get MinIO for object storage, Airbyte for ingestion, PostgreSQL as your warehouse, DBT for transformations across bronze/silver/gold layers, and Airflow for orchestration. The repo includes a Makefile with commands for everything from `make dbt-run` to `make trigger-dag`, plus Grafana dashboards for monitoring. It's genuinely end-to-end, meaning you can clone it, run `make setup && make start`, and have a working ELT pipeline with data quality tests and SCD Type 2 snapshots. Best for learning the full data engineering workflow locally or prototyping pipelines before moving to cloud infrastructure. The DBT models show realistic patterns like deduplication and profit margin calculations.
- skills.sh installs
- 1.5K
- GitHub stars
- 2
- last updated
- Jul 14, 2026
Install
$ npx -y skills add aradotso/data-skills --skill data-engineering-medallion-pipeline --agent claude-codeInstalls into .claude/skills of the current project.
Files
Data Engineering Medallion Pipeline Skill
Skill by ara.so — Data Skills collection.
This skill enables AI agents to work with a complete data engineering pipeline implementing the Medallion Architecture (Bronze → Silver → Gold) using modern open-source tools: MinIO (S3-compatible storage), Airbyte (data ingestion), PostgreSQL (data warehouse), DBT (transformations), Apache Airflow (orchestration), and Grafana (monitoring).
What This Project Does
The data-engineering-medallion project provides a complete end-to-end data pipeline that:
- Ingests raw data from MinIO object storage into PostgreSQL using Airbyte
- Transforms data through three layers (Bronze/Silver/Gold) using DBT
- Orchestrates the entire pipeline with Apache Airflow DAGs
- Validates data quality with automated DBT tests
- Monitors infrastructure health with Prometheus and Grafana
- Visualizes business metrics in Power BI dashboards
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/aradotso/data-skills
- Categories
- DevOps & CI/CD, Automation & Workflows, Sales & Marketing
- First seen
- Jul 14, 2026
More from aradotso/data-skills
All 60- Data Engineering Study Material1.5K
- Employee Performance Analytics Hr1.5K
- Enterprise Data Engineering Pipeline Ssis Pyspark1.5K
- Harvard Art Museum Data Pipeline1.5K
- Harvard Art Museum Etl Analytics1.5K
- Harvard Art Museums Data Pipeline1.5K
- Harvard Art Museums Etl Analytics1.5K
- Harvard Artifacts Data Engineering Analytics1.5K
- Harvard Artifacts Data Engineering App1.5K
- Harvard Artifacts Data Engineering Pipeline1.5K
- Harvard Artifacts Etl Analytics1.5K
- Llm Intelligent Public Opinion Analytics1.5K
- Options Analytics Agent Langgraph1.5K
- Realtime Cinema Data Engineering Pipeline1.5K
- Harvard Artifacts Collection Analytics Pipeline1.4K
- Harvard Artifacts Collection Data Engineering1.4K
- Harvard Artifacts Collection Data Engineering Analytics1.4K
- Harvard Artifacts Collection Etl Analytics1.4K
- Harvard Artifacts Etl Streamlit Analytics1.4K
- Harvard Art Museums Data Engineering App1.3K
- Harvard Art Museums Etl Pipeline1.3K
- Harvard Artifacts Data Pipeline1.3K
- Iac Data Engineering Terraform1.3K
- Iac Terraform Data Engineering1.3K
Recommended
More DevOps & CI/CD
agent-performance-monitorruvnet/rufloAgent skill for performance-monitor - invoke with $agent-performance-monitor1.2Kskills.sh installs
netlify-cli-and-deploynetlify/context-and-toolsGuide for using the Netlify CLI and deploying sites. Use when installing the CLI, linking sites, deploying (Git-based or manual), managing environment variables1Kskills.sh installs
docker-expertdavila7/claude-code-templatesDocker containerization expert with deep knowledge of multi-stage builds, image optimization, container security, Docker Compose orchestration, and production d899skills.sh installs
data-engineering-data-pipelinesickn33/antigravity-awesome-skillsdata engineering data pipeline572skills.sh installsmachine-learning-ops-ml-pipelinesickn33/antigravity-awesome-skillsmachine learning ops ml pipeline508skills.sh installs
uptime-monitoringaj-geddes/useful-ai-promptsImplement uptime monitoring and status page systems for tracking service availability. Use when monitoring application uptime, creating status pages, or impleme490skills.sh installs