Harvard Artifacts Data Engineering App

This is a complete ETL pipeline that pulls artifact data from the Harvard Art Museums API and builds a proper relational database with MySQL or TiDB. It handles the full transformation from nested JSON to normalized tables (metadata, media, colors), then layers on a Streamlit dashboard with 20+ analytical queries and Plotly visualizations. Good for learning data engineering fundamentals with real cultural data, or if you're prototyping museum analytics. The code is straightforward, handles pagination and rate limits, and includes all the SQL setup. It's more educational than production-ready, but that's the point. You get hands-on practice with API integration, database normalization, and building dashboards without wrestling with authentication complexities or messy data cleaning.

skills.sh installs: 1.5K
GitHub stars: 2
last updated: Jul 14, 2026

View on GitHub

Install

$ npx -y skills add aradotso/data-skills --skill harvard-artifacts-data-engineering-app --agent claude-code

Installs into .claude/skills of the current project.

Files

Harvard Artifacts Data Engineering App

Skill by ara.so — Data Skills collection.

This project demonstrates end-to-end data engineering using the Harvard Art Museums API. It extracts artifact data, transforms it into relational tables, loads it into SQL databases (MySQL/TiDB), and provides interactive analytics through Streamlit dashboards with Plotly visualizations.

What It Does

API Integration: Fetches paginated artifact data from Harvard Art Museums API
ETL Pipeline: Transforms nested JSON into normalized relational tables
SQL Storage: Creates and populates artifactmetadata, artifactmedia, and artifactcolors tables
Analytics Queries: 20+ predefined SQL queries for artifact insights
Interactive Visualization: Streamlit dashboard with Plotly charts

Installation

# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App

# Install dependencies
pip install -r requirements.txt

SKILL.md on GitHub