Apache Spark Data Processing
This is a tier-1 reference for Apache Spark that covers the full stack: RDDs, DataFrames, Spark SQL, streaming, and MLlib. It walks through core concepts like lazy evaluation, partitioning strategies, and the Catalyst optimizer with practical guidance on when to cache data and how to choose between RDDs and DataFrames. The skill includes code examples for broadcast variables, accumulators, and connecting to various data sources. Useful if you're processing terabyte-scale datasets across a cluster or building ETL pipelines that need distributed computing. The source is honest about what Spark isn't good for, like small datasets under 100GB or ultra-low latency requirements under 10ms where you'd want a specialized stream processor instead.
- skills.sh installs
- 481
- GitHub stars
- 63
- last updated
- Aug 21, 2026
Install
$ npx -y skills add manutej/luxor-claude-marketplace --skill apache-spark-data-processing --agent claude-codeInstalls into .claude/skills of the current project.
Files
Apache Spark Data Processing
A comprehensive skill for mastering Apache Spark data processing, from basic RDD operations to advanced streaming, SQL, and machine learning workflows. Learn to build scalable, distributed data pipelines and analytics systems.
When to Use This Skill
Use Apache Spark when you need to:
- Process Large-Scale Data: Handle datasets too large for single-machine processing (TB to PB scale)
- Perform Distributed Computing: Execute parallel computations across cluster nodes
- Real-Time Stream Processing: Process continuous data streams with low latency
- Complex Data Analytics: Run sophisticated analytics, aggregations, and transformations
- Machine Learning at Scale: Train ML models on massive datasets
- ETL/ELT Pipelines: Build robust data transformation and loading workflows
- Interactive Data Analysis: Perform exploratory analysis on large datasets
- Unified Data Processing: Combine batch and streaming workloads in one framework
Sponsors
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any languageDetails
- Repository
- github.com/manutej/luxor-claude-marketplace
- Categories
- Documentation, Databases
- First seen
- Jun 3, 2026
More from manutej/luxor-claude-marketplace
All 66- Pydantic476
- Kubernetes Orchestration474
- Nodejs Development474
- Nextjs Development473
- React Patterns473
- Pandas470
- React Development468
- Database Management Patterns467
- Angular Development463
- Tailwind Css463
- Javascript Fundamentals461
- Vector Database Management460
- Performance Benchmark Specialist457
- Pytest457
- Observability Monitoring456
- Terraform Infrastructure As Code454
- Fastapi Development452
- Langchain Orchestration452
- Aws Cloud Services451
- Express Microservices Architecture449
- Graphql Api Development449
- Microservices Patterns447
- Linear Dev Accelerator446
- Mlops Workflows445
Recommended
More Documentation
agent-package-managerwebmaxru/ai-native-devInstalls, configures, audits, and operates Agent Package Manager (APM) in repositories. Use when initializing apm.yml, installing or updating packages, validati481skills.sh installs
flash-fictionjwynia/agent-skillsDiagnose and fix problems in flash fiction and micro fiction. Use when flash fiction feels weak, when writing stories under 1500 words, when working with micro 479skills.sh installsrevisionjwynia/agent-skillsGuide the edit pass after drafting. Use when revision feels overwhelming, when changes cascade unpredictably, when you can't see problems anymore, or when editi479skills.sh installsagent-package-managerwebmaxru/enonic-agent-skillsInstalls, configures, audits, and operates Agent Package Manager (APM) in repositories. Use when initializing apm.yml, installing or updating packages, validati479skills.sh installs
tiptapjezweb/claude-skillsBuild rich text editors with Tiptap - headless editor framework with React and Tailwind v4. Covers SSR-safe setup, image uploads, prose styling, and collaborati478skills.sh installs
content-idea-generatorbrianrwagner/ai-marketing-claude-code-skillsGenerate content ideas rooted in positioning. Use when someone needs "content ideas," "what should I post," "blog topics," "LinkedIn ideas," or is stuck on what477skills.sh installs