Sentencepiece

Google's language-independent tokenizer that treats text as raw Unicode and works without pre-tokenization. It's the engine behind T5, ALBERT, and XLNet, supporting both BPE and Unigram algorithms. The big win is multilingual support, especially for CJK languages where you can't just split on spaces. It's deterministic, lightweight (6MB memory), and processes 50k sentences per second. Training takes a few minutes on 100MB of text. HuggingFace Tokenizers are 4x faster if you need speed, but SentencePiece is the standard when you need reproducible tokenization across languages or want to train on raw text without language-specific rules.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill sentencepiece --agent claude-code

Installs into .claude/skills of the current project.

Files

SentencePiece - Language-Independent Tokenization

Unsupervised tokenizer that works on raw text without language-specific preprocessing.

When to use SentencePiece

Use SentencePiece when:

Building multilingual models (no language-specific rules)
Working with CJK languages (Chinese, Japanese, Korean)
Need reproducible tokenization (deterministic vocabulary)
Want to train on raw text (no pre-tokenization needed)
Require lightweight deployment (6MB memory, 50k sentences/sec)

Performance:

Speed: 50,000 sentences/sec
Memory: ~6MB for loaded model
Languages: All (language-independent)

Use alternatives instead:

HuggingFace Tokenizers: Faster training, more flexibility
tiktoken: OpenAI models (GPT-3.5/4)
BERT WordPiece: English-centric tasks

Quick start

Installation

# Python
pip install sentencepiece

# C++ (requires CMake)
git clone https://github.com/google/sentencepiece.git
cd sentencepiece
mkdir build && cd build
cmake .. && make -j $(nproc)
sudo make install