Harvard Art Museums Data Engineering Pipeline
This is a reference implementation for building a complete data pipeline around the Harvard Art Museums API. You get the full stack: paginated API extraction with rate limiting, ETL transforms that normalize nested JSON into a proper relational schema (separate tables for metadata, media, and color data), and a Streamlit dashboard with 20+ analytical queries and Plotly visualizations. It's overkill if you just want to pull some museum data, but if you're learning data engineering patterns or need to pitch pipeline architecture to stakeholders, having a working example with real API integration, foreign keys, and batch loading is genuinely useful. The code samples are thorough enough to fork and adapt to other APIs.
- skills.sh installs
- 1.2K
- GitHub stars
- 2
- last updated
- Jul 14, 2026
Install
$ npx -y skills add aradotso/data-skills --skill harvard-art-museums-data-engineering-pipeline --agent claude-codeInstalls into .claude/skills of the current project.
Files
Harvard Art Museums Data Engineering Pipeline
Skill by ara.so — Data Skills collection.
What This Project Does
The Harvard-Artifacts-Collection-Data-Engineering-Analytics-App is a complete data engineering solution that demonstrates:
- API Integration: Fetching artifact data from Harvard Art Museums API with pagination and rate limiting
- ETL Pipeline: Extracting, transforming, and loading nested JSON into normalized SQL tables
- Database Design: Multi-table relational schema with proper foreign keys
- SQL Analytics: 20+ predefined analytical queries for insights
- Interactive Visualization: Streamlit dashboards with Plotly charts
This project serves as a reference architecture for building production-grade data pipelines.
Installation
# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App
# Install dependencies
pip install -r requirements.txt
Required packages:
streamlit
pandas
requests
mysql-connector-python
plotly
python-dotenv
Configuration
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/aradotso/data-skills
- Categories
- DevOps & CI/CD, Automation & Workflows, Sales & Marketing
- First seen
- Jul 14, 2026
More from aradotso/data-skills
All 60- Harvard Artifacts Etl Pipeline1.2K
- Snowflake Dbt Airbnb Analytics1.2K
- Data Engineering Patterns Fabric Databricks1.1K
- Game Analytics Platform Computer Vision1.1K
- Infrastructure Cicd Data Engineering1.1K
- Harvard Artifacts Collection Analytics App907
- Harvard Artifacts Etl Streamlit833
- Harvard Art Museum Data Engineering831
- Data Analytics Skills Claude735
- Car Sales Data Engineering Analytics713
- Google Cloud Data Engineering Hub662
- Harvard Art Museums Data Engineering Analytics628
- Harvard Artifacts Data Engineering Streamlit623
- Coinmarketcap Diamonds Premium Analytics608
- Harvard Art Museums Data Engineering536
- Apache Airflow Orchestration1.9K
- Datatalks Data Engineering Zoomcamp1.9K
- Llm Public Opinion Analytics Assistant1.9K
- Roblox Mm2 Analytics Toolkit1.9K
- Mm2 Analytics Dashboard Roblox1.8K
- Mm2 Analytics Roblox Toolkit1.8K
- Mm2 Analytics Roblox Tracker1.8K
- Mm2 Roblox Analytics Toolkit1.8K
- Mm2 Roblox Analytics Tracker1.8K
Recommended
More DevOps & CI/CD
harvard-artifacts-etl-pipelinearadotso/data-skillsharvard artifacts etl pipeline1.2Kskills.sh installs
build-mcpneolabhq/context-engineering-kitGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when b1.2Kskills.sh installs
terraform-infrastructureaj-geddes/useful-ai-promptsInfrastructure as Code using Terraform with modular components, state management, and multi-cloud deployments. Use for provisioning and managing cloud resources1.2Kskills.sh installs