Spark Optimization

This is a solid reference for tuning Spark jobs when they're running slow or eating memory. You get concrete patterns for partitioning strategies, join optimization (broadcast, bucketed, skew handling), caching decisions, and memory configuration with actual code examples in PySpark. It covers the execution model, shows you how to calculate optimal partition sizes, and includes adaptive query execution settings. The implementation playbook apparently has more detailed examples if you need them. It's community sourced so treat it as a starting point rather than gospel, but the patterns here address the common bottlenecks like shuffle overhead, data skew, and GC pressure that actually slow down production Spark pipelines.

skills.sh installs: 182
GitHub stars: 39.4K
last updated: Jun 3, 2026

View on GitHub

Install

$ npx -y skills add sickn33/antigravity-awesome-skills --skill spark-optimization --agent claude-code

Installs into .claude/skills of the current project.

Files

Apache Spark Optimization

Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.

Do not use this skill when

The task is unrelated to apache spark optimization
You need a different domain or tool outside this scope

Instructions

Clarify goals, constraints, and required inputs.
Apply relevant best practices and validate outcomes.
Provide actionable steps and verification.

Use this skill when

Optimizing slow Spark jobs
Tuning memory and executor configuration
Implementing efficient partitioning strategies
Debugging Spark performance issues
Scaling Spark pipelines for large datasets
Reducing shuffle and data skew

Core Concepts

1. Spark Execution Model

Driver Program
    ↓
Job (triggered by action)
    ↓
Stages (separated by shuffles)
    ↓
Tasks (one per partition)

2. Key Performance Factors

SKILL.md on GitHub

Details

Repository: github.com/sickn33/antigravity-awesome-skills
First seen: Jun 3, 2026