Sn Da Large File Analysis

This is a performance rescue kit for Excel files that choke pandas. It routes you through three tiers: under 10k rows, use normal read_excel; 10k to 100k, cache to Parquet; over 100k, stream with openpyxl's read_only mode and iter_rows to avoid loading everything into memory. The streaming converter writes chunks as Parquet with all columns cast to string to dodge PyArrow schema conflicts, then you convert types after. Also includes dtype downcasting that claims 50 to 80 percent memory savings and a write_only mode for exports over a million rows. The trigger logic is baked in, so it activates on row counts or keywords like OOM, streaming read, or big data. Good for when your data analysis suddenly becomes a memory management problem.

skills.sh installs: 135
GitHub stars: 6.1K
last updated: Jun 11, 2026

View on GitHub

Install

$ npx -y skills add opensensenova/sensenova-skills --skill sn-da-large-file-analysis --agent claude-code

Installs into .claude/skills of the current project.

Files

Large Scale Excel Analysis Skill

Mandatory Rules

When total rows >= 10,000, you MUST use the methods in this skill.

Data Scale	Read Strategy	Reason
< 10k rows	`pd.read_excel()` directly	No memory pressure
10k–100k rows	`pd.read_excel()` → convert to Parquet → `pd.read_parquet()` for analysis	Avoid repeated slow reads
100k–1M rows	openpyxl `read_only` + `iter_rows` streaming → Parquet	`pd.read_excel()` will OOM or timeout
> 1M rows	Streaming read + multi-sheet split (Excel max 1,048,576 rows per sheet)	Must chunk

Prohibited:

Do NOT use pd.read_excel() to fully load 100k+ row files
Do NOT search for fonts with fc-list, find ... fonts, or install packages with pip install
Do NOT use df.iterrows() on large DataFrames (use itertuples() or vectorized ops)
Do NOT use df.apply(lambda...) for operations that can be vectorized

Environment Setup

import pandas as pd
import numpy as np
import os
import gc

pd.options.mode.copy_on_write = True

SKILL.md on GitHub

Details

Repository: github.com/opensensenova/sensenova-skills
Categories: Debugging, Automation & Workflows, Office & Documents
License: MIT
First seen: Jun 11, 2026

issue-reportercherryhq/cherry-studio帮助用户提交 Bug Report 或 Feature Request。支持 GitHub Issue（有账户）和本地存档（无账户）两种模式。当诊断发现是代码 Bug 时主动提议，或当用户说"帮我提 issue"、"这是个 bug"、"我想要这个功能"、"submit a bug"、"feature request"时0skills.sh installs