Nemo Mbridge Mlm Bridge Training

This handles correlation testing between Megatron-LM and Megatron Bridge training runs, which matters when you need to verify that loss curves match after code changes or when translating MLM command-line arguments to Bridge recipe configs. It walks you through running both frameworks with identical hyperparameters (2 layer, 256 hidden, mock data) and checking that BF16 losses agree within rounding error. The multi-GPU examples show tensor parallelism setup for both sides. One thing that will save you time: it front-loads the gotchas like always nuking nemo_experiments before fresh runs because Bridge silently resumes from stale checkpoints, and remembering to set PYTHONPATH for MLM or nothing works.

skills.sh installs: 1.9K
GitHub stars: 3.9K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add nvidia/skills --skill nemo-mbridge-mlm-bridge-training --agent claude-code

Installs into .claude/skills of the current project.

Files

MLM vs Bridge Training

For how they differ, the arg mapping tables, gotchas, and translation script, see:

@docs/megatron-lm-to-megatron-bridge.md

First Answer Checklist

For MLM-vs-Bridge correlation questions, always name these items up front:

Bridge recipe: vanilla_gpt_pretrain_config.
Bridge entry point: scripts/training/run_recipe.py.
MLM entry point: 3rdparty/Megatron-LM/pretrain_gpt.py.
Launch wrapper for both: uv run python -m torch.distributed.run.
Fresh-run cleanup: rm -rf nemo_experiments before the Bridge run.

Also state that MLM needs PYTHONPATH=3rdparty/Megatron-LM:$PYTHONPATH, matched Bridge and MLM losses should agree within BF16 rounding, and files under 3rdparty/Megatron-LM/ should not be modified from this repo.

Correlation Testing

Use vanilla_gpt_pretrain_config for loss-correlation testing. This recipe uses bare GPTModelProvider defaults (LayerNorm, GeLU, learned_absolute position embeddings, vocab_size inherited from tokenizer) — matching MLM pretrain_gpt.py defaults with no args.