Harvard Artifacts Data Engineering Analytics

If you're learning data engineering or need a reference implementation for API-to-database pipelines, this walks through the full stack with a real dataset. It pulls from Harvard Art Museums API, transforms nested JSON into a three-table relational schema (metadata, media, colors), batch loads into MySQL, and builds a Streamlit dashboard with 20+ analytical queries and Plotly visualizations. The code handles pagination and rate limiting properly, shows actual SQL DDL and insert patterns, and demonstrates end-to-end ETL flow. Best used as a template for similar API ingestion projects or as teaching material for Python/SQL workflows. The schema design is straightforward and the transformation logic is clear enough to adapt to other museum or catalog APIs.

skills.sh installs: 1.5K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-artifacts-data-engineering-analytics --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Artifacts Data Engineering Analytics

Skill by ara.so — Data Skills collection

This project provides an end-to-end data engineering and analytics application for the Harvard Art Museums API. It demonstrates real-world ETL pipelines, SQL database design, analytical queries, and interactive visualization using Streamlit.

What This Project Does

The application implements a complete data pipeline:

Extract: Fetches artifact data from Harvard Art Museums API with pagination and rate limiting
Transform: Processes nested JSON into relational database tables (metadata, media, colors)
Load: Batch inserts transformed data into MySQL/TiDB Cloud
Analyze: Executes 20+ predefined SQL queries for insights
Visualize: Renders interactive dashboards with Plotly charts in Streamlit

Installation

# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt

SKILL.md on GitHub