robbyant/lingbot-vision

LingBot-Vision is a family of image understanding models built around Vision Transformers, ranging from a small version up to a large 1.1 billion parameter one. All of them are trained without labeled data, using a technique called self supervised learning, where the model teaches itself by comparing different views of the same image rather than being told what each image contains. The main model is trained with an approach the project calls masked boundary modeling, which pushes the model to notice where one object or shape ends and another begins, while still keeping a good general sense of what is in the image. This produces features that line up well with actual object edges, useful for anything working with the physical layout of a scene rather than just recognizing what an image shows. Because of this design, LingBot-Vision can act as a ready made building block for other systems that need to understand images at a detailed, per region level. The README lists uses such as producing readable visual maps directly from the model's raw features, estimating depth in a scene, splitting an image into semantic regions, tracking and separating objects across video frames, and completing missing depth data. That last use case connects directly to a companion project, LingBot-Depth 2.0, which swaps in LingBot-Vision as its image encoder and shows large accuracy gains as a result, including filling in depth information on tricky surfaces like glass and mirrors where normal depth sensors fail. Four model sizes are offered, from small to giant, each distilled down from the largest trained version, so a user can trade off accuracy against speed depending on their hardware. Setting it up follows a standard Python workflow: clone the repository, create a conda environment with Python 3.10, and install the dependencies. Model weights download automatically from Hugging Face the first time they are used. A short example script shows how to load an image, run it through the small model, and get back a grid of feature tokens, plus a demo script that visualizes those features as a colored map of an image. The project is released under the Apache 2.0 license.