mercuriustech/odyseus-spatial-vlm

Odyseus Spatial VLM is an experiment that combines a vision language model with a depth estimation model to give AI a better understanding of 3D space from a single photograph. A vision language model, or VLM, is an AI that can analyze images and answer questions about them, but it typically only understands things in terms of flat 2D screen coordinates: it can say the chair is in the upper left of the image, but not how far away the chair actually is. This project adds monocular depth estimation, meaning figuring out distance using just one camera, to translate the VLM's 2D outputs into real 3D positions. The workflow is simple. You upload an image and type a prompt describing objects you want to locate, for example select the chair near the desk and the closest door. The VLM identifies the 2D pixel locations of those objects. A separate depth model called Depth-Anything-3 then estimates how far each pixel is from the camera. The system combines these two outputs to calculate real 3D coordinates and displays them as an interactive point cloud in the browser, complete with labeled targets and camera orientation guides, rendered using Three.js, a JavaScript library for 3D graphics. This project is aimed at people building physical robots or autonomous agents that need spatial awareness, since it lets an AI reason about where objects actually sit in 3D space rather than just identifying them on a flat image. It currently runs on Linux, requires running two separate server processes locally, and also has a hosted live demo you can try without any setup.