Harvard Artifacts Collection Data Engineering Analytics
This is a complete ETL pipeline tutorial for pulling artifact data from Harvard Art Museums' API into MySQL, then visualizing it with Streamlit. It handles the full workflow: pagination and rate limiting on API calls, transforming nested JSON into three relational tables (metadata, media, colors), and running 20+ analytical queries with Plotly charts. The code examples are production-ready with proper error handling and batch inserts. You'd use this as a reference implementation if you're building similar museum or cultural data pipelines, or if you need a working example of API-to-database workflows with visualization. It's batteries-included with schema definitions, environment config, and transform functions that handle real-world messiness like field length limits.
- skills.sh installs
- 1.4K
- GitHub stars
- 2
- last updated
- Jul 14, 2026
Install
$ npx -y skills add aradotso/data-skills --skill harvard-artifacts-collection-data-engineering-analytics --agent claude-codeInstalls into .claude/skills of the current project.
Files
Harvard Artifacts Collection Data Engineering Analytics
Skill by ara.so — Data Skills collection.
Overview
This project demonstrates a complete data engineering workflow: extracting artifact data from the Harvard Art Museums API, transforming it into structured relational tables, loading it into SQL databases (MySQL/TiDB Cloud), and building interactive analytics dashboards with Streamlit and Plotly.
The application handles:
- API pagination and rate limiting
- ETL pipeline for nested JSON to relational data
- SQL database design with proper relationships
- 20+ analytical SQL queries
- Interactive visualizations
Installation
# Clone the repository
git clone https://github.com/Manali0711/Harvard-Artifacts-Collection-Data-Engineering-Analytics-App.git
cd Harvard-Artifacts-Collection-Data-Engineering-Analytics-App
# Install dependencies
pip install -r requirements.txt
Required dependencies:
streamlit
pandas
requests
mysql-connector-python
plotly
python-dotenv
Configuration
Environment Variables
Create a .env file in the project root:
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/aradotso/data-skills
- Categories
- Data Science & ML, Marketing & SEO
- First seen
- Jul 14, 2026
More from aradotso/data-skills
All 60- Harvard Artifacts Collection Etl Analytics1.4K
- Harvard Artifacts Etl Streamlit Analytics1.4K
- Harvard Art Museums Data Engineering App1.3K
- Harvard Art Museums Etl Pipeline1.3K
- Harvard Artifacts Data Pipeline1.3K
- Iac Data Engineering Terraform1.3K
- Iac Terraform Data Engineering1.3K
- Retail Etl Medallion Pipeline1.3K
- Retail Etl Pipeline Medallion1.3K
- Terraform Data Engineering Iac1.3K
- Terraform Data Engineering Infrastructure1.3K
- Terraform Iac Data Engineering1.3K
- Harvard Art Museums Data Engineering Pipeline1.2K
- Harvard Artifacts Etl Pipeline1.2K
- Snowflake Dbt Airbnb Analytics1.2K
- Data Engineering Patterns Fabric Databricks1.1K
- Game Analytics Platform Computer Vision1.1K
- Infrastructure Cicd Data Engineering1.1K
- Harvard Artifacts Collection Analytics App907
- Harvard Artifacts Etl Streamlit833
- Harvard Art Museum Data Engineering831
- Data Analytics Skills Claude735
- Car Sales Data Engineering Analytics713
- Google Cloud Data Engineering Hub662
Recommended
More Data Science & ML
harvard-artifacts-collection-etl-analyticsaradotso/data-skillsharvard artifacts collection etl analytics1.4Kskills.sh installs
huggingface-zerogpuhuggingface/skillsAI demos and GPU compute with Gradio Spaces and Hugging Face Spaces ZeroGPU. Use when writing or reviewing code that uses `@spaces.GPU`, configuring `python_ver1.4Kskills.sh installs
academic-plottingorchestra-research/ai-research-skillsGenerates publication-quality figures for ML papers from research context. Given a paper section or description, extracts system components and relationships to1.3Kskills.sh installs
setup-arxiv-mcpneolabhq/context-engineering-kitGuide for setup arXiv paper search MCP server using Docker MCP1.2Kskills.sh installs
agent-neural-networkruvnet/rufloAgent skill for neural-network - invoke with $agent-neural-network1.1Kskills.sh installs
youtube-researchpostplusai/postplus-skillsResearch YouTube channel summaries, audience comment samples, downloadable video records, and public videos using PostPlus Cloud collection service. Use this wh1.1Kskills.sh installs