Claude MarketplacesSubmit
aradotso avatar

Flash Moe Inference

aradotso/trending-skills

This is a pure C/Metal inference engine that runs Qwen3.5-397B (397B parameters, 209GB on disk) on a MacBook Pro with 48GB RAM at 4.4 tokens per second. It streams expert weights from SSD on demand using parallel pread calls, no Python or ML frameworks at runtime. The repo includes hand-tuned Metal shaders with fused dequantization FMA kernels and uses Accelerate BLAS for the GatedDeltaNet attention layers. You'll need an M3 Max or similar, 1TB SSD, and patience for the weight extraction and repacking process. There's an optional 2-bit quantization path that's faster but breaks tool calling. This is what happens when someone decides PyTorch is too slow and writes 7000 lines of Objective-C instead.

skills.sh installs
1.1K
GitHub stars
61
last updated
Jul 14, 2026
View on GitHub

Install

$ npx -y skills add aradotso/trending-skills --skill flash-moe-inference --agent claude-code

Installs into .claude/skills of the current project.

Files

Flash-MoE Inference Engine

Skill by ara.so — Daily 2026 Skills collection.

Flash-MoE is a pure C/Objective-C/Metal inference engine that runs Qwen3.5-397B-A17B (397B parameter Mixture-of-Experts) on a MacBook Pro with 48GB RAM at 4.4+ tokens/second. It streams 209GB of expert weights from NVMe SSD on demand — no Python, no ML frameworks, just C, Objective-C, and hand-tuned Metal shaders.

Requirements

  • Hardware: Apple Silicon Mac (M3 Max or similar), 48GB+ unified memory, 1TB+ SSD with ~210GB free
  • OS: macOS 26+ (Darwin 25+)
  • Tools: Xcode Command Line Tools, Python 3.x (for weight extraction only)
  • Model: Qwen3.5-397B-A17B safetensors weights (download separately from HuggingFace)

Installation & Build

# Clone the repo
git clone https://github.com/danveloper/flash-moe
cd flash-moe/metal_infer

# Build everything
make

# Verify build artifacts
ls infer chat main

The Makefile compiles infer.m, chat.m, main.m with Metal shader compilation for shaders.metal.

Weight Preparation

Step 1: Extract non-expert weights

SKILL.md on GitHub

Sponsors

Promote your product hereReach 125k+ visitors monthlyPromote your product hereReach 125k+ visitors monthlyAgent CaffeineSponsoredClose your lid, come back to finished workSnugSponsoredThe clipboard manager for the AI eraPromote your product hereReach 125k+ visitors monthlyWaxSponsoredBlock anything with supervision, without erasing your iPhonePromote your product hereReach 125k+ visitors monthlyknobsSponsoredSee your site on any phone, any day, in any language

Details

Repository
github.com/aradotso/trending-skills
First seen
Apr 16, 2026

More from aradotso/trending-skills

All 251

Recommended

1juliusbrussee avatarcavemanjuliusbrussee/cavemanUltra-compressed communication mode. Cuts output tokens 65% (measured) by speaking like caveman while keeping full technical accuracy. Supports intensity levels618.3Kskills.sh installs
2mattpocock avatargrill-memattpocock/skillsA relentless interview to sharpen a plan or design.1.5Mskills.sh installs
3shadcn avatarimproveshadcn/improveSurvey any codebase as a senior advisor and produce prioritized, self-contained implementation plans for OTHER models/agents to execute. Strictly read-only on s40Kskills.sh installs
4obra avatarsystematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes326.7Kskills.sh installs
5forrestchang avatarkarpathy-guidelinesforrestchang/andrej-karpathy-skillsBehavioral guidelines to reduce common LLM coding mistakes through explicit assumptions, simplicity, and verifiable success criteria.17.9Kskills.sh installs
6vercel-labs avatarfind-skillsvercel-labs/skillsHelps users discover and install agent skills when they ask questions like "how do I do X", "find a skill for X", "is there a skill that can...", or express int4.3Mskills.sh installs