Vaex

When you're staring down a CSV file with billions of rows that won't fit in RAM, this is your way out. Vaex handles out-of-core DataFrames through memory-mapped HDF5 and Arrow files, letting you filter, aggregate, and visualize datasets at a billion rows per second on a single machine. The virtual columns are clever since they compute on the fly without eating memory, and the lazy evaluation means you can chain operations without waiting. It's not as fast as Polars if your data fits in RAM, and it's not distributed like Dask, but for single-machine analytics on tabular data that exceeds your memory, it hits a sweet spot that the other tools don't really cover.

skills.sh installs: 2K
GitHub stars: 56.6K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add k-dense-ai/scientific-agent-skills --skill vaex --agent claude-code

Installs into .claude/skills of the current project.

Files

Vaex

When to use

Use Vaex for columnar analysis on a single machine when data exceeds RAM, especially repeated reductions and histograms over local Vaex HDF5 or Arrow files. Expressions and virtual columns defer computation; reductions normally execute immediately. Out-of-core storage does not make every operation memory bounded: sorting, joins, large group dictionaries, materialization, and many estimator fits need substantial RAM.

Installation and verified scope

Use a separate environment; the repository's default Python is newer than this release supports:

uv venv --python 3.12 .venv-vaex
uv pip install --python .venv-vaex/bin/python "vaex-core==4.19.0" "vaex-hdf5==0.15.0" "vaex-viz==0.6.0"
# Optional ML (also installs its declared estimator dependencies):
uv pip install --python .venv-vaex/bin/python "vaex-ml==0.19.0"

SKILL.md on GitHub