Harvard Art Museums Data Pipeline

This walks you through building a complete data pipeline from API to dashboard using the Harvard Art Museums collection as your dataset. You'll hit their API with pagination and rate limiting, normalize the JSON into MySQL tables (metadata, media, colors), and build a Streamlit dashboard with Plotly visualizations. It's a solid reference implementation if you're learning data engineering patterns or need to spin up something similar for a different API. The code is straightforward Python with pandas, mysql-connector, and requests. Worth noting it uses TiDB Cloud compatible schemas, so you can run it on their free tier without setting up local MySQL.

skills.sh installs: 1.5K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-art-museums-data-pipeline --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Art Museums Data Pipeline

Skill by ara.so — Data Skills collection.

This skill enables you to build production-ready data engineering pipelines using the Harvard Art Museums API. It covers ETL workflows, relational database design, SQL analytics, and interactive Streamlit dashboards for artifact data visualization.

What This Project Does

The Harvard-Artifacts-Collection-Data-Engineering-Analytics-App demonstrates a complete data pipeline:

Extract: Fetches artifact data from Harvard Art Museums API with pagination and rate limiting
Transform: Converts nested JSON into normalized relational tables
Load: Batch inserts data into MySQL/TiDB Cloud databases
Analyze: Executes analytical SQL queries for insights
Visualize: Renders interactive dashboards with Plotly and Streamlit

The architecture follows: API → ETL → SQL → Analytics → Visualization

Installation

git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt