name: clip

description: OpenAI's model connecting vision and language. Enables zero-shot image classification, image-text matching, and cross-modal retrieval. Trained on 400M image-text pairs. Use for image search, content moderation, or vision-language tasks without fine-tuning. Best for general-purpose image understanding.

version: 1.0.0

author: Orchestra Research

license: MIT

dependencies: [transformers, torch, pillow]

metadata:

hermes:

tags: [Multimodal, CLIP, Vision-Language, Zero-Shot, Image Classification, OpenAI, Image Search, Cross-Modal Retrieval, Content Moderation]

CLIP - Contrastive Language-Image Pre-Training

OpenAI's model that understands images from natural language.

When to use CLIP

Use when:

• - Zero-shot image classification (no training data needed)
• - Image-text similarity/matching
• - Semantic image search
• - Content moderation (detect NSFW, violence)
• - Visual question answering
• - Cross-modal retrieval (image→text, text→image)

Metrics:

• - 25,300+ GitHub stars
• - Trained on 400M image-text pairs
• - Matches ResNet-50 on ImageNet (zero-shot)
• - MIT License

Use alternatives instead:

• - BLIP-2: Better captioning
• - LLaVA: Vision-language chat
• - Segment Anything: Image segmentation

Quick start

Installation

`bash

pip install git+https://github.com/openai/CLIP.git

pip install torch torchvision ftfy regex tqdm

`

Zero-shot classification

`python

import torch

import clip

from PIL import Image

Load model

device = "cuda" if torch.cuda.is_available() else "cpu"

model, preprocess = clip.load("ViT-B/32", device=device)

Load image

image = preprocess(Image.open("photo.jpg")).unsqueeze(0).to(device)

Define possible labels

text = clip.tokenize(["a dog", "a cat", "a bird", "a car"]).to(device)

Compute similarity

with torch.no_grad():

image_features = model.encode_image(image)

text_features = model.encode_text(text)

# Cosine similarity

logits_per_image, logits_per_text = model(image, text)

probs = logits_per_image.softmax(dim=-1).cpu().numpy()

Print results

labels = ["a dog", "a cat", "a bird", "a car"]

for label, prob in zip(labels, probs[0]):

print(f"{label}: {prob:.2%}")

`

Available models

`python

Models (sorted by size)

models = [

"RN50", # ResNet-50

"RN101", # ResNet-101

"ViT-B/32", # Vision Transformer (recommended)

"ViT-B/16", # Better quality, slower

"ViT-L/14", # Best quality, slowest

]

model, preprocess = clip.load("ViT-B/32")

`

Image-text similarity

`python

Compute embeddings

image_features = model.encode_image(image)

text_features = model.encode_text(text)

Normalize

image_features /= image_features.norm(dim=-1, keepdim=True)

text_features /= text_features.norm(dim=-1, keepdim=True)

Cosine similarity

similarity = (image_features @ text_features.T).item()

print(f"Similarity: {similarity:.4f}")

`

Semantic image search

`python

Index images

image_paths = ["img1.jpg", "img2.jpg", "img3.jpg"]

image_embeddings = []

for img_path in image_paths:

image = preprocess(Image.open(img_path)).unsqueeze(0).to(device)

with torch.no_grad():

embedding = model.encode_image(image)

embedding /= embedding.norm(dim=-1, keepdim=True)

image_embeddings.append(embedding)

image_embeddings = torch.cat(image_embeddings)

Search with text query

query = "a sunset over the ocean"

text_input = clip.tokenize([query]).to(device)

with torch.no_grad():

text_embedding = model.encode_text(text_input)

text_embedding /= text_embedding.norm(dim=-1, keepdim=True)

Find most similar images

similarities = (text_embedding @ image_embeddings.T).squeeze(0)

top_k = similarities.topk(3)

for idx, score in zip(top_k.indices, top_k.values):

print(f"{image_paths[idx]}: {score:.3f}")

`

Content moderation

`python

Define categories

categories = [

"safe for work",

"not safe for work",

"violent content",

"graphic content"

]

text = clip.tokenize(categories).to(device)

Check image

with torch.no_grad():

logits_per_image, _ = model(image, text)

probs = logits_per_image.softmax(dim=-1)

Get classification

max_idx = probs.argmax().item()

max_prob = probs[0, max_idx].item()

print(f"Category: {categories[max_idx]} ({max_prob:.2%})")

`

Batch processing

`python

Process multiple images

images = [preprocess(Image.open(f"img{i}.jpg")) for i in range(10)]

images = torch.stack(images).to(device)

with torch.no_grad():

image_features = model.encode_image(images)

image_features /= image_features.norm(dim=-1, keepdim=True)

Batch text

texts = ["a dog", "a cat", "a bird"]

text_tokens = clip.tokenize(texts).to(device)

with torch.no_grad():

text_features = model.encode_text(text_tokens)

text_features /= text_features.norm(dim=-1, keepdim=True)

Similarity matrix (10 images × 3 texts)

similarities = image_features @ text_features.T

print(similarities.shape) # (10, 3)

`

Integration with vector databases

`python

Store CLIP embeddings in Chroma/FAISS

import chromadb

client = chromadb.Client()

collection = client.create_collection("image_embeddings")

Add image embeddings

for img_path, embedding in zip(image_paths, image_embeddings):

collection.add(

embeddings=[embedding.cpu().numpy().tolist()],

metadatas=[{"path": img_path}],

ids=[img_path]

)

Query with text

query = "a sunset"

text_embedding = model.encode_text(clip.tokenize([query]))

results = collection.query(

query_embeddings=[text_embedding.cpu().numpy().tolist()],

n_results=5

)

`

Best practices

1. Use ViT-B/32 for most cases - Good balance

2. Normalize embeddings - Required for cosine similarity

3. Batch processing - More efficient

4. Cache embeddings - Expensive to recompute

5. Use descriptive labels - Better zero-shot performance

6. GPU recommended - 10-50× faster

7. Preprocess images - Use provided preprocess function

Performance

Model	Parameters	Speed	Quality
-------	------------	-------	---------
RN50	102M	Fast	Good
ViT-B/32	151M	Medium	Better
ViT-L/14	428M	Slow	Best
Operation	CPU	GPU (V100)
-----------	-----	------------
Image encoding	~200ms	~20ms
Text encoding	~50ms	~5ms
Similarity compute	<1ms	<1ms

Limitations

1. Not for fine-grained tasks - Best for broad categories

2. Requires descriptive text - Vague labels perform poorly

3. Biased on web data - May have dataset biases

4. No bounding boxes - Whole image only

5. Limited spatial understanding - Position/counting weak

Resources

• - GitHub: https://github.com/openai/CLIP ⭐ 25,300+
• - Paper: https://arxiv.org/abs/2103.00020
• - Colab: https://colab.research.google.com/github/openai/clip/
• - License: MIT