Harvard Artifacts Etl Pipeline

Here's a solid reference implementation for building an ETL pipeline on top of the Harvard Art Museums API. It walks through the full stack: paginated API calls, transforming nested JSON into normalized SQL tables (metadata, media, colors), loading into MySQL/TiDB, and surfacing analytics through a Streamlit dashboard with Plotly charts. The code is practical and well-structured, showing real patterns like batch inserts with upsert logic and proper environment variable handling. If you're learning data engineering fundamentals or need a template for API-to-dashboard workflows, this covers the essentials without overcomplicating things. The museum domain makes it more interesting to work with than yet another e-commerce dataset.

skills.sh installs: 1.2K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-artifacts-etl-pipeline --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Artifacts ETL Pipeline

Skill by ara.so — Data Skills collection.

This skill enables AI coding agents to help developers build end-to-end data engineering pipelines using the Harvard Art Museums API. The project demonstrates real-world ETL patterns, SQL database design, analytical queries, and interactive visualization using Streamlit.

What This Project Does

The Harvard Artifacts Collection application:

Fetches artifact data from Harvard Art Museums API with pagination
Transforms nested JSON into relational database tables
Loads data into MySQL/TiDB Cloud databases
Executes analytical SQL queries on artifact metadata, media, and colors
Visualizes results through interactive Plotly charts in Streamlit

Architecture: API → ETL → SQL → Analytics → Visualization

Installation

# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt

# Required packages
pip install streamlit pandas requests mysql-connector-python plotly python-dotenv

SKILL.md on GitHub