Harvard Artifacts Data Pipeline

This is a complete reference implementation for building ETL pipelines against the Harvard Art Museums API. You get working code for pagination and rate limiting, transformation logic that normalizes nested JSON into a three-table relational schema (metadata, media, colors), and MySQL insert patterns with proper foreign keys. The included Streamlit dashboard has 20+ prebuilt analytical queries and Plotly visualizations. Use this when you need to see how a full data engineering workflow fits together, from API extraction through SQL analytics. The schema design is sensible and the transformation functions are clean enough to adapt to similar museum or collection APIs. It's educational first, but the patterns are production-adjacent.

skills.sh installs: 1.3K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-artifacts-data-pipeline --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Artifacts Data Pipeline

Skill by ara.so — Data Skills collection.

What This Project Does

The Harvard Artifacts Collection Data Engineering & Analytics App is an end-to-end data pipeline that demonstrates professional ETL workflows. It fetches artifact data from the Harvard Art Museums API, transforms it into structured relational tables, stores it in MySQL/TiDB Cloud, and provides interactive analytics through a Streamlit dashboard with Plotly visualizations.

Key capabilities:

API data extraction with pagination and rate limiting
ETL transformations for nested JSON to relational schema
SQL database design with proper foreign keys
20+ predefined analytical queries
Interactive visualization dashboards

Installation

# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt

SKILL.md on GitHub