Tensorrt Llm

If you're serving LLMs on NVIDIA GPUs in production and need serious performance, this is the tool to reach for. It delivers 100x speedups over PyTorch on models like Llama 3 through aggressive optimizations like in-flight batching, FP8 quantization, and multi-GPU scaling. The tradeoff is complexity: you're dealing with model compilation, CUDA dependencies, and TensorRT's learning curve. But if you're running A100s or H100s and need to squeeze out 24,000 tokens per second with low latency, the performance gains justify the setup cost. For simpler deployments or non-NVIDIA hardware, vLLM or llama.cpp are easier choices.

skills.sh installs: 1.1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill tensorrt-llm --agent claude-code

Installs into .claude/skills of the current project.

Files

TensorRT-LLM

NVIDIA's open-source library for optimizing LLM inference with state-of-the-art performance on NVIDIA GPUs.

When to use TensorRT-LLM

Use TensorRT-LLM when:

Deploying on NVIDIA GPUs (A100, H100, GB200)
Need maximum throughput (24,000+ tokens/sec on Llama 3)
Require low latency for real-time applications
Working with quantized models (FP8, INT4, FP4)
Scaling across multiple GPUs or nodes

Use vLLM instead when:

Need simpler setup and Python-first API
Want PagedAttention without TensorRT compilation
Working with AMD GPUs or non-NVIDIA hardware

Use llama.cpp instead when:

Deploying on CPU or Apple Silicon
Need edge deployment without NVIDIA GPUs
Want simpler GGUF quantization format

Quick start

Installation

# Docker (recommended)
docker pull nvidia/tensorrt_llm:latest

# pip install
pip install tensorrt_llm==1.2.0rc3

# Requires CUDA 13.0.0, TensorRT 10.13.2, Python 3.10-3.12

Basic inference

from tensorrt_llm import LLM, SamplingParams

# Initialize model
llm = LLM(model="meta-llama/Meta-Llama-3-8B")

SKILL.md on GitHub

Details

Repository: github.com/orchestra-research/ai-research-skills
Categories: AI & Agent Building, Data Science & ML
First seen: Jun 3, 2026