Realtime Cinema Data Engineering Pipeline

This is a full-stack streaming pipeline tutorial that wires together Kafka, Postgres, Airflow, and Streamlit to process cinema transaction events through a bronze/silver/gold medallion architecture. You get a Faker-based event generator, a Kafka consumer that dumps JSON into Postgres, Airflow DAGs that normalize the data into 3NF tables, and a Streamlit dashboard with Plotly charts. It's designed to handle 1M+ events and teaches you how the pieces fit together in a production-style setup. The Docker Compose file spins up the infrastructure, but you'll need to wait a few minutes for Airflow to initialize before running the pipeline. Good for learning streaming ETL patterns without the AWS bill.

skills.sh installs: 1.5K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill realtime-cinema-data-engineering-pipeline --agent claude-code

Installs into .claude/skills of the current project.

Files

CinéWorld Real-Time Data Engineering Pipeline Skill

Skill by ara.so — Data Skills collection.

Overview

This project implements an end-to-end real-time data engineering pipeline using Apache Kafka for event streaming, PostgreSQL for data warehousing with Medallion Architecture (Bronze/Silver/Gold layers), Apache Airflow for ELT orchestration, and Streamlit for live visualization. Perfect for learning how to build production-grade streaming data pipelines that process 1M+ events.

Installation

Prerequisites

Docker and Docker Compose
Python 3.8+
Virtual environment (recommended)

Setup Steps

# Clone the repository
git clone https://github.com/BaidaneAyoub/realtime-cinema-data-engineering.git
cd realtime-cinema-data-engineering

# Create and activate virtual environment
python -m venv myenv
source myenv/bin/activate  # On Windows: myenv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

# Start infrastructure (Kafka, PostgreSQL, Airflow)
docker-compose up -d

Important: Wait 2-3 minutes for Airflow to fully initialize before proceeding.