Harvard Art Museum Data Pipeline

This is a complete ETL reference implementation for the Harvard Art Museums API that goes beyond toy examples. It shows you how to handle pagination and rate limiting when extracting thousands of artifacts, normalize nested JSON into a three-table relational schema (metadata, media, colors), and batch load into MySQL or TiDB Cloud. The included Streamlit dashboard with 20+ analytical queries gives you working examples of how to actually use the data once it's loaded. If you're learning data engineering or need a starting point for museum or cultural heritage data projects, this covers the full pipeline with production considerations like connection pooling and error handling. The schema design is sensible and the transform functions handle missing fields gracefully.

skills.sh installs: 1.5K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-art-museum-data-pipeline --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Art Museum Data Pipeline

Skill by ara.so — Data Skills collection.

This project provides an end-to-end data engineering solution for collecting, transforming, storing, and analyzing artifact data from the Harvard Art Museums API. It demonstrates production-ready ETL pipelines, SQL analytics, and interactive visualization using Streamlit.

What It Does

The Harvard Art Museum Data Pipeline:

Extracts artifact data from the Harvard Art Museums API with pagination and rate limiting
Transforms nested JSON into normalized relational tables (metadata, media, colors)
Loads data into MySQL/TiDB Cloud with batch inserts for performance
Analyzes data using predefined SQL queries for business insights
Visualizes results through interactive Streamlit dashboards with Plotly charts

Architecture

Harvard Art Museums API → Python ETL → MySQL/TiDB → SQL Analytics → Streamlit Dashboard

Key Components:

API integration with secure key management
Three-table relational schema: artifactmetadata, artifactmedia, artifactcolors
20+ analytical SQL queries
Real-time interactive visualizations