Vaex

If you're hitting memory limits with pandas on multi-gigabyte datasets, this is the out-of-core alternative that actually delivers. It uses lazy evaluation and memory mapping to let you run aggregations and filters on billions of rows at laptop scale, with a familiar DataFrame API. The workflow is straightforward: load your massive CSV or Parquet file, create virtual columns for feature engineering without copying data, run your groupbys and stats, then export to HDF5 or Arrow for instant future access. Real win is interactive exploration of datasets that would normally require a cluster. Just know you're trading pandas' ecosystem breadth for raw performance on big tabular data.

skills.sh installs: 211
GitHub stars: 33.2K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add davila7/claude-code-templates --skill vaex --agent claude-code

Installs into .claude/skills of the current project.

Files

Vaex

Overview

Vaex is a high-performance Python library designed for lazy, out-of-core DataFrames to process and visualize tabular datasets that are too large to fit into RAM. Vaex can process over a billion rows per second, enabling interactive data exploration and analysis on datasets with billions of rows.

When to Use This Skill

Use Vaex when:

Processing tabular datasets larger than available RAM (gigabytes to terabytes)
Performing fast statistical aggregations on massive datasets
Creating visualizations and heatmaps of large datasets
Building machine learning pipelines on big data
Converting between data formats (CSV, HDF5, Arrow, Parquet)
Needing lazy evaluation and virtual columns to avoid memory overhead
Working with astronomical data, financial time series, or other large-scale scientific datasets