faizan711/multimodal-search

Multimodal Search is a small learning project that lets you search a collection of images using either a text description or another image as the query. Instead of relying on filenames or manual tags, the app converts both text and images into numeric vectors that sit in the same shared space, so a phrase like "sunset over mountains" can be compared directly against the visual content of a photo. Under the hood it uses OpenAI's CLIP model to turn text and images into these vectors, and it stores and searches them with Qdrant, a vector database built for fast similarity lookups. The backend is written in Python with FastAPI, and the frontend is plain HTML, CSS, and JavaScript, no framework required. One notable feature is a real time animation in the browser that visually walks through each step of a search as it happens, showing the query being tokenized, passed through the CLIP encoder, turned into a vector embedding, searched against Qdrant, and finally ranked into results. To run it yourself you need Python 3.10 or newer and Docker, since Qdrant runs as a local Docker container. The project ships with a pre-built set of vectors for 541 sample images, so on first run the app automatically loads that data into your local Qdrant instance and downloads the CLIP model weights before you can start searching from your browser. The project structure separates concerns cleanly: the FastAPI routes live in one file, the CLIP model setup and inference code in another, the Qdrant connection and search logic in a third, and configuration settings in a fourth, with a scripts folder for utilities that handle importing and exporting the vector data. The README does not mention a license, so it is unclear under what terms this code can be reused.