Clip

CLIP lets you match images and text without training a classifier first. You give it an image and a list of labels ("a dog", "a cat", "a sunset"), and it tells you which ones match using cosine similarity between embeddings. Works surprisingly well for semantic image search, content moderation, and zero-shot classification. Trained on 400M image-text pairs from the web. The ViT-B/32 model is the sweet spot for speed and quality. Main limitation is that it only understands whole images, not regions or fine-grained details, and vague text prompts give you vague results. If you need actual captions or chat, use BLIP-2 or LLaVA instead.

skills.sh installs: 1K
GitHub stars: 14.4K
last updated: Aug 21, 2026

View on GitHub

Install

$ npx -y skills add orchestra-research/ai-research-skills --skill clip --agent claude-code

Installs into .claude/skills of the current project.

Files

CLIP - Contrastive Language-Image Pre-Training

OpenAI's model that understands images from natural language.

When to use CLIP

Use when:

Zero-shot image classification (no training data needed)
Image-text similarity/matching
Semantic image search
Content moderation (detect NSFW, violence)
Visual question answering
Cross-modal retrieval (image→text, text→image)

Metrics:

25,300+ GitHub stars
Trained on 400M image-text pairs
Matches ResNet-50 on ImageNet (zero-shot)
MIT License

Use alternatives instead:

BLIP-2: Better captioning
LLaVA: Vision-language chat
Segment Anything: Image segmentation

Quick start

Installation

pip install git+https://github.com/openai/CLIP.git
pip install torch torchvision ftfy regex tqdm

Zero-shot classification

import torch
import clip
from PIL import Image

# Load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)

# Load image
image = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)

# Define possible labels
text = clip.tokenize(["a dog", "a cat", "a bird", "a car"]).to(device)

SKILL.md on GitHub