Spark Optimization

If your Spark jobs are crawling or crashing, this gives you the exact patterns to fix them. It covers the critical stuff: calculating optimal partition sizes, implementing broadcast joins versus sort-merge joins, setting up proper caching with storage levels, and tuning executor memory configurations. The salting technique for handling data skew is especially useful when you have hot keys killing performance. It includes actual configuration values and memory breakdowns, not just theory. Most helpful when you're dealing with production workloads that need to scale beyond toy datasets.

skills.sh installs: 11.1K
GitHub stars: 42.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add wshobson/agents --skill spark-optimization --agent claude-code

Installs into .claude/skills of the current project.

Files

Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

When to Use This Skill

Optimizing slow Spark jobs
Tuning memory and executor configuration
Implementing efficient partitioning strategies
Debugging Spark performance issues
Scaling Spark pipelines for large datasets
Reducing shuffle and data skew

Core Concepts

1. Spark Execution Model

Driver Program
    ↓
Job (triggered by action)
    ↓
Stages (separated by shuffles)
    ↓
Tasks (one per partition)

2. Key Performance Factors

Factor	Impact	Solution
Shuffle	Network I/O, disk I/O	Minimize wide transformations
Data Skew	Uneven task duration	Salting, broadcast joins
Serialization	CPU overhead	Use Kryo, columnar formats
Memory	GC pressure, spills	Tune executor memory
Partitions	Parallelism	Right-size partitions