Harvard Artifacts Data Engineering Pipeline

This is a full ETL tutorial that walks you through pulling artifact data from Harvard Art Museums API, normalizing it into MySQL tables, and building a Streamlit dashboard with Plotly charts. You get the complete pipeline: API pagination with rate limiting, transforming nested JSON into three relational tables (metadata, media, colors), and 20+ predefined SQL analytics queries. The code is production-oriented with proper error handling and environment variable management. If you're learning data engineering or need a reference implementation for API-to-database workflows, this covers the fundamentals end to end. The Streamlit frontend is basic but functional for exploring the collection data you've loaded.

skills.sh installs: 1.5K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-artifacts-data-engineering-pipeline --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Artifacts Data Engineering Pipeline

Skill by ara.so — Data Skills collection.

This project provides an end-to-end data engineering solution for collecting, transforming, storing, and analyzing artifact data from the Harvard Art Museums API. It demonstrates production-ready ETL pipelines, relational database design, SQL analytics, and interactive Streamlit dashboards.

What This Project Does

API Integration: Fetches artifact data from Harvard Art Museums API with pagination and rate limiting
ETL Pipeline: Extracts nested JSON, transforms into relational schema, loads into SQL database
Database Design: Implements normalized tables (artifactmetadata, artifactmedia, artifactcolors)
SQL Analytics: Executes 20+ predefined analytical queries
Visualization: Interactive Plotly charts rendered through Streamlit

Installation

# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt