Harvard Art Museums Etl Pipeline

This is a practical reference for building a complete ETL pipeline against the Harvard Art Museums API. You get working code for paginated extraction, JSON normalization into three relational tables (metadata, media, colors), and a Streamlit dashboard with Plotly visualizations. The schema design is sensible with proper foreign keys, and the transformation logic handles nested API responses cleanly. Good if you're learning data engineering patterns with real cultural data or need a template for similar API-to-database projects. The code shows batch inserts and upsert logic, though you'll need your own Harvard API key and MySQL instance to run it.

skills.sh installs: 1.3K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-art-museums-etl-pipeline --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Art Museums ETL Pipeline

Skill by ara.so — Data Skills collection.

This project provides an end-to-end data engineering solution for extracting, transforming, and analyzing artifacts from the Harvard Art Museums API. It demonstrates real-world ETL patterns, SQL analytics, and interactive visualization using Streamlit.

What It Does

Extracts artifact data from Harvard Art Museums API with pagination and rate limiting
Transforms nested JSON into normalized relational tables (metadata, media, colors)
Loads data into MySQL/TiDB Cloud with proper foreign key relationships
Analyzes data using predefined SQL queries for cultural insights
Visualizes results through interactive Plotly charts in a Streamlit dashboard

Installation

# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt

# Required packages
pip install streamlit pandas requests mysql-connector-python plotly python-dotenv