Harvard Art Museum Data Engineering

This is a complete ETL pipeline that pulls artifact data from Harvard's Art Museums API and loads it into a MySQL or TiDB database with a proper relational schema. You get three normalized tables (metadata, media, colors) and a Streamlit dashboard for analytics. The setup is straightforward if you've done any API work before: grab a free Harvard API key, configure your database credentials, and run the extraction scripts. It handles pagination, transforms nested JSON into flat tables, and uses batch inserts for performance. Good reference implementation if you're teaching data engineering concepts or need to build something similar for another museum or cultural API. The code is clean enough to fork and adapt.

skills.sh installs: 831
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-art-museum-data-engineering --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Art Museum Data Engineering Skill

Skill by ara.so — Data Skills collection.

Overview

This project provides an end-to-end data engineering solution for the Harvard Art Museums API. It implements a complete ETL pipeline that extracts artifact data, transforms it into relational tables, loads it into SQL databases (MySQL/TiDB), and provides interactive analytics dashboards using Streamlit.

Architecture Flow: API → ETL → SQL → Analytics → Visualization

Installation

# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt

# Required packages
pip install streamlit pandas requests mysql-connector-python plotly python-dotenv

Configuration

Environment Variables

Create a .env file in the project root:

# Harvard API Configuration
HARVARD_API_KEY=your_api_key_here

# Database Configuration
DB_HOST=your_database_host
DB_PORT=3306
DB_USER=your_username
DB_PASSWORD=your_password
DB_NAME=harvard_artifacts

SKILL.md on GitHub