Training Llms Megatron

This is NVIDIA's Megatron-Core wrapped as a Claude skill for training massive language models at scale. It handles everything from 2B to 462B parameters and claims 47% Model FLOP Utilization on H100s through tensor, pipeline, and data parallelism. You'd reach for this if you're actually training foundation models with multi-GPU setups, not fine-tuning or running inference. The Docker container route is cleaner than pip since it bundles the full NVIDIA PyTorch stack. Fair warning: this is infrastructure for serious compute budgets, not something you spin up for experiments. The skill passes security audits and comes from orchestra-research's AI toolkit, though the original repo is from ovachiever/droid-tings.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill training-llms-megatron --agent claude-code

Installs into .claude/skills of the current project.

Files

Megatron-Core - Large-Scale LLM Training

Quick start

Megatron-Core trains LLMs from 2B to 462B parameters with up to 47% Model FLOP Utilization on H100 GPUs through advanced parallelism strategies.

Installation:

# Docker (recommended)
docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:25.04-py3

# Or pip
pip install megatron-core

Simple distributed training:

# Train with 2 GPUs using data parallelism
torchrun --nproc_per_node=2 examples/run_simple_mcore_train_loop.py

# Or LLaMA-3 8B training
./examples/llama/train_llama3_8b_fp8.sh

Common workflows

Workflow 1: Train LLaMA-style model with 3D parallelism

Copy this checklist:

LLaMA Training Setup:
- [ ] Step 1: Choose parallelism configuration
- [ ] Step 2: Configure training hyperparameters
- [ ] Step 3: Launch distributed training
- [ ] Step 4: Monitor performance metrics

Step 1: Choose parallelism configuration

Model size determines parallelism strategy:

SKILL.md on GitHub