Spark Engineer

This is your go-to when you're writing Spark jobs or debugging why your pipeline is spilling to disk. It walks you through the full workflow from analyzing requirements to validating in Spark UI, with solid reference guides for DataFrames, RDDs, partitioning, and performance tuning. The examples are practical stuff you'll actually use: broadcast joins for dimension tables, salting patterns for skewed data, proper caching with unpersist. What I appreciate is the constraints section that tells you the real gotchas, like not using collect() on large datasets and why UDFs are 10-100x slower than built-ins. It assumes you know what Spark is but need help doing it right at scale.

skills.sh installs: 4.1K
GitHub stars: 12.3K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add jeffallan/claude-skills --skill spark-engineer --agent claude-code

Installs into .claude/skills of the current project.

Files

Spark Engineer

Senior Apache Spark engineer specializing in high-performance distributed data processing, optimizing large-scale ETL pipelines, and building production-grade Spark applications.

Core Workflow

Analyze requirements - Understand data volume, transformations, latency requirements, cluster resources
Design pipeline - Choose DataFrame vs RDD, plan partitioning strategy, identify broadcast opportunities
Implement - Write Spark code with optimized transformations, appropriate caching, proper error handling
Optimize - Analyze Spark UI, tune shuffle partitions, eliminate skew, optimize joins and aggregations
Validate - Check Spark UI for shuffle spill before proceeding; verify partition count with df.rdd.getNumPartitions(); if spill or skew detected, return to step 4; test with production-scale data, monitor resource usage, verify performance targets