Tao Generate Referring Expressions

This is NVIDIA's four-stage pipeline for turning images with KITTI bounding boxes into rich referring expressions. A VLM (Gemini, NIM, or any OpenAI-compatible endpoint) generates per-object descriptions, scene captions, grouped grounding phrases tied to bboxes, and an optional verification pass that catches mismatches. It's built for auto-labeling traffic and scene datasets where you already have basic bbox annotations but need natural language references. The workflow is resumable, skips already-processed images, and writes everything to a unified JSONL. Start with 5 to 10 images, check if the groupings and spatial relationships make sense, swap to a stronger model if needed, then scale up. Runs in the TAO Toolkit container with decent parallelism controls.

skills.sh installs: 1.6K
GitHub stars: 3.9K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add nvidia/skills --skill tao-generate-referring-expressions --agent claude-code

Installs into .claude/skills of the current project.

Files

Image Referring Expression Pipeline

Standalone install? If this session was not initialized by the TAO skill bank plugin, run the tao-setup skill first (host preflight, credentials, cross-skill discovery).

Generate referring-expression and grounding annotations from images with KITTI-format bounding box labels. A single VLM (Gemini or any OpenAI-compatible endpoint) runs four steps: per-object region descriptions, holistic image captions, grouped grounding expressions tied to bboxes, and an optional double-check verification pass.

Purpose

Transform (image, KITTI labels) pairs into a unified annotations.jsonl containing rich, grounded referring expressions. The VLM acts as a "teacher" annotator: Steps 0-1 see the image; Step 2 groups Step 0 outputs into grouping phrases with bbox lists; Step 3 (optional) re-examines those bboxes against the image and corrects mismatches.

Pipeline Architecture

Step 0: Region expression  ──┐
                              ├──▶  Step 2: Grounding expression  ──▶  [Step 3: Double check]
Step 1: Image caption  ──────┘                                                   (optional)

SKILL.md on GitHub