Ray Data

A distributed data processing library built on Ray for ML workloads that actually need to scale beyond a single machine. If you're loading 100GB+ datasets for training, running batch inference across a cluster, or preprocessing multi-modal data (images, video, audio), this handles the streaming execution and GPU acceleration. It integrates cleanly with PyTorch, TensorFlow, and Ray Train, so you can read from S3, transform in parallel, and feed directly into training loops. The API feels like pandas but executes lazily across hundreds of nodes. Companies like Pinterest and Spotify use it for production ML pipelines. Below 100GB you're probably better off with pandas or Dask, but once you need real distributed processing for ML, this is the practical choice in the Ray ecosystem.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill ray-data --agent claude-code

Installs into .claude/skills of the current project.

Files

Ray Data - Scalable ML Data Processing

Distributed data processing library for ML and AI workloads.

When to use Ray Data

Use Ray Data when:

Processing large datasets (>100GB) for ML training
Need distributed data preprocessing across cluster
Building batch inference pipelines
Loading multi-modal data (images, audio, video)
Scaling data processing from laptop to cluster

Key features:

Streaming execution: Process data larger than memory
GPU support: Accelerate transforms with GPUs
Framework integration: PyTorch, TensorFlow, HuggingFace
Multi-modal: Images, Parquet, CSV, JSON, audio, video

Use alternatives instead:

Pandas: Small data (<1GB) on single machine
Dask: Tabular data, SQL-like operations
Spark: Enterprise ETL, SQL queries

Quick start

Installation

pip install -U 'ray[data]'

Load and transform data

import ray

# Read Parquet files
ds = ray.data.read_parquet("s3://bucket/data/*.parquet")

# Transform data (lazy execution)
ds = ds.map_batches(lambda batch: {"processed": batch["text"].str.lower()})

SKILL.md on GitHub