Moe Training

This walks you through training Mixture of Experts models with DeepSpeed or HuggingFace, covering everything from basic MoE layers to production configurations like Mixtral 8x7B. You get working code for routing mechanisms (top-1, top-2, expert choice), load balancing losses to prevent expert collapse, and expert parallelism configs for distributing 128 experts across multiple GPUs. The real value is in the details: capacity factors, auxiliary loss coefficients, router z-loss for stability. Use this when you need to train larger models on limited compute, the claimed 5x cost reduction versus dense models is legit because you only activate a fraction of parameters per token. The DeepSpeed configurations are production ready, not toy examples.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill moe-training --agent claude-code

Installs into .claude/skills of the current project.

Files

MoE Training: Mixture of Experts

When to Use This Skill

Use MoE Training when you need to:

Train larger models with limited compute (5× cost reduction vs dense models)
Scale model capacity without proportional compute increase
Achieve better performance per compute budget than dense models
Specialize experts for different domains/tasks/languages
Reduce inference latency with sparse activation (only 13B/47B params active in Mixtral)
Implement SOTA models like Mixtral 8x7B, DeepSeek-V3, Switch Transformers

Notable MoE Models: Mixtral 8x7B (Mistral AI), DeepSeek-V3, Switch Transformers (Google), GLaM (Google), NLLB-MoE (Meta)

Installation

# DeepSpeed with MoE support
pip install deepspeed>=0.6.0

# Megatron-DeepSpeed for large-scale training
git clone https://github.com/microsoft/Megatron-DeepSpeed
cd Megatron-DeepSpeed
pip install -r requirements.txt

# Alternative: HuggingFace Transformers
pip install transformers accelerate

Quick Start

Basic MoE Architecture

import torch
import torch.nn as nn

class MoELayer(nn.Module):
    """Sparse Mixture of Experts layer."""

SKILL.md on GitHub

Details

Repository: github.com/orchestra-research/ai-research-skills
Categories: AI & Agent Building, Data Science & ML
First seen: Jun 3, 2026