Harvard Artifacts Collection Data Engineering

This is a complete reference implementation for building an ETL pipeline on top of the Harvard Art Museums API. You get the full stack: API client with pagination handling, transformation logic that flattens nested JSON into three normalized tables (metadata, media, colors), MySQL batch loading, and a Streamlit dashboard with Plotly visualizations. The code samples are detailed enough to actually use, showing exactly how to structure the database schema, handle the API's nested color and media arrays, and implement proper foreign key relationships. Good starting point if you need to build similar museum data pipelines or want a realistic example of API-to-warehouse data engineering without the usual tutorial hand-waving.

skills.sh installs: 1.4K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-artifacts-collection-data-engineering --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Artifacts Collection Data Engineering

Skill by ara.so — Data Skills collection.

This project provides an end-to-end data engineering and analytics application built on the Harvard Art Museums API. It demonstrates real-world ETL pipelines, SQL database design, analytical queries, and interactive visualization using Streamlit. The architecture follows: API → ETL → SQL → Analytics → Visualization.

What This Project Does

API Integration: Fetches artifact data from Harvard Art Museums API with pagination and rate limiting
ETL Pipeline: Extracts, transforms, and loads nested JSON into relational database tables
SQL Database: Stores structured data across artifactmetadata, artifactmedia, and artifactcolors tables
Analytics: Executes 20+ predefined SQL queries for insights
Visualization: Interactive dashboards using Plotly and Streamlit

Installation

# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt

SKILL.md on GitHub