jina-ai/clip-as-service

★ 12,836PythonAudience · developerComplexity · 3/5Setup · hard

Mindmap

mindmap
  root((clip-as-service))
    What it does
      Image embeddings
      Text embeddings
      Image-text ranking
    Architecture
      Server component
      Client component
      Load balancing
    Protocols
      gRPC
      HTTP
      WebSocket
    Runtimes
      PyTorch
      ONNX Runtime
      TensorRT
    Use cases
      Visual search
      Caption ranking
      Batch processing

mindmap root((clip-as-service)) What it does Image embeddings Text embeddings Image-text ranking Architecture Server component Client component Load balancing Protocols gRPC HTTP WebSocket Runtimes PyTorch ONNX Runtime TensorRT Use cases Visual search Caption ranking Batch processing

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Build an image search engine that ranks photos by how closely they match a text query using CLIP embeddings.

USE CASE 2

Run visual reasoning tasks that score competing text captions against an image to find the best match.

USE CASE 3

Generate embeddings for large image datasets using the server's async batch request mode for non-blocking throughput.

USE CASE 4

Host a GPU-accelerated CLIP model on Google Colab and connect a Python client to it over gRPC or HTTP.

Tech stack

PythonPyTorchONNX RuntimeTensorRTgRPC

Getting it running

Difficulty · hard Time to first run · 1h+

Server requires a GPU, TensorRT runtime needs additional NVIDIA tooling on top of standard CUDA.

In plain English

CLIP-as-service is a Python tool that turns images and text into numerical vectors (called embeddings) and compares them to each other. It is built around a model called CLIP, which was originally developed at OpenAI and understands the relationship between images and natural language descriptions. The practical result is that you can give it an image and several text captions, and it will rank the captions by how well they describe the image. The system is split into a server component and a client component, each installed as a separate Python package. You start the server on a machine with access to a GPU, and the client connects to it to send images or text and receive embeddings or rankings back. The server supports three different runtimes: standard PyTorch, ONNX Runtime for better efficiency, and TensorRT for the fastest throughput. Requests can be sent over gRPC, HTTP, or WebSocket, with optional TLS encryption. The client supports async (non-blocking) requests, which the README describes as designed for large amounts of data or long-running tasks. The server can also scale horizontally and run multiple CLIP model replicas on a single GPU with automatic load balancing. The README demonstrates a few use cases: generating embeddings for images and text sentences, and visual reasoning tasks where the model is asked questions about an image by providing competing text descriptions. For example, you can send an image of berries with captions like "this is a photo of three berries" versus "this is a photo of four berries," and the model returns a confidence score for each. Installation is handled through pip. The server can also be hosted on Google Colab using its free GPU resources.

Copy-paste prompts

Prompt 1

Using clip-as-service, write Python code to start the CLIP server and send 100 product photos to get embeddings, then find the 5 most visually similar images to a query image.

Prompt 2

I have a clip-as-service server running. Help me write async Python client code that sends 10,000 images in parallel and collects all embeddings into a NumPy array.

Prompt 3

Using clip-as-service visual reasoning, write code that takes a photo and 5 competing captions and returns which caption best describes the image with a confidence score for each.

Prompt 4

How do I configure the clip-as-service server to use TensorRT runtime instead of standard PyTorch for faster GPU throughput on my NVIDIA card?

Open on GitHub → Explain another repo

← jina-ai on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.