Harvard Art Museums Data Engineering App

This is a complete reference implementation for building an ETL pipeline around the Harvard Art Museums API. It walks you through fetching paginated artifact data, normalizing nested JSON into three related MySQL tables (metadata, media, colors), and surfacing the results through a Streamlit dashboard with Plotly visualizations. The code includes proper rate limiting, batch inserts, and 20+ analytical SQL queries. Use this if you're learning data engineering fundamentals or need a template for similar museum/cultural API integrations. The schema design is clean and the transformation logic handles real world messiness in the JSON responses, making it a solid starting point for adaptation.

skills.sh installs: 1.3K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-art-museums-data-engineering-app --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Art Museums Data Engineering App

Skill by ara.so — Data Skills collection.

This project provides an end-to-end data engineering and analytics application built on the Harvard Art Museums API. It demonstrates real-world ETL pipelines, SQL database design, analytical queries, and interactive visualization using Streamlit.

What This Project Does

The application follows a complete data pipeline: API → ETL → SQL → Analytics → Visualization

Collects artifact data from Harvard Art Museums API with pagination and rate limiting
Transforms nested JSON into normalized relational tables
Loads data into MySQL/TiDB Cloud databases
Analyzes with 20+ predefined SQL queries
Visualizes results through interactive Plotly dashboards in Streamlit

Installation

# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt

SKILL.md on GitHub