Huggingface Tokenizers

davila7/claude-code-templates

This wraps the HuggingFace tokenizers library with its Rust-powered performance into a Claude Code skill. You're looking at sub-20 second tokenization times for a full gigabyte of text, which makes it worth considering if you're training custom tokenizers or processing large corpora where pure Python implementations bog down. The alignment tracking feature is handy when you need to map tokens back to their original positions in the source text. It's production-ready tooling that's been battle-tested across the NLP community, now packaged for quick integration. The skill originates from ovachiever/droid-tings and has been picked up by davila7's template collection.

skills.sh installs

340

GitHub stars

33.2K

last updated

Aug 21, 2026

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language

Recommended

More Data Science & ML

1arxiv-searchvarde80/arxiv-zotero-obsidianSearch arXiv for academic papers. Use when users want to find research papers, preprints, or academic articles on any topic. Supports filtering by date, categor321skills.sh installs

2deep-researchfeiskyer/claude-code-settingsMulti-agent research orchestration: split a research goal into parallel sub-goals, run each via headless `claude -p` subprocesses, aggregate results into a poli304skills.sh installs

3chat-with-arxivqodex-ai/ai-agent-skillsBuild interactive chat agents for exploring and discussing academic research papers from ArXiv. Covers paper retrieval, content processing, question-answering, 303skills.sh installs

4statistics-verifiertravisjneuman/.claudeVerify statistics from raw data with methodology checking, significance testing, claim validation, and bias detection. Use when fact-checking statistical claims218skills.sh installs

5wiki-research-looprohitg00/pro-workflowAuto-grow a pro-workflow wiki by running a budget-capped BFS research loop over pluggable source fetchers (web, arXiv, GitHub). Each iteration pops a seed from 154skills.sh installs

6deep-researchactionbook/actionbookDeep research and analysis tool. Generates comprehensive HTML reports on any topic, domain, paper, or technology. Use when user asks to research, analyze, inves148skills.sh installs