tokenization
Text tokenization for LLMs including HuggingFace Tokenizers and SentencePiece. Use when training custom tokenizers or handling multilingual text.
- GitHub stars
- 14.4K
Install
/plugin install tokenization@ai-research-skillsRun it inside Claude Code after adding the marketplace with /plugin marketplace add Orchestra-Research/AI-Research-SKILLs.
Sponsors
Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthly
Agent CaffeineSponsoredClose your lid, come back to finished work
SnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthly
WaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthly
knobsSponsoredSee your site on any phone, any day, in any language
Agent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any languageDetails
- Marketplace
- Orchestra-Research/AI-Research-SKILLs
- Category
- Community
More from Orchestra-Research/AI-Research-SKILLs
All 23- agent-native-research-artifact
- agents
- autoresearch
- data-processing
- distributed-training
- emerging-techniques
- evaluation
- fine-tuning
- ideation
- inference-serving
- infrastructure
- mechanistic-interpretability
- ml-paper-writing
- mlops
- model-architecture
- multimodal
- observability
- optimization
- post-training
- prompt-engineering
- rag
- safety-alignment