somnusochi/vlm-autoyolo

This is an end-to-end platform for creating object detection models without manually drawing bounding boxes on every image. You feed it images or video clips, describe what you want to find in plain language, and an AI model called LocateAnything-3B automatically detects and marks those objects. From there you can refine the results by hand, then export the labeled data and train a YOLO object detection model, all within the same web interface. The pipeline runs in stages. First you upload images or video and type a description such as "red car" or "fire, smoke." The AI draws bounding boxes around matching objects. Optionally, a second model called SAM2 or SAM3 refines those boxes into pixel-precise outlines rather than just rectangles. You review the results on a canvas, adjust thresholds, hide or show individual detections, and correct anything the AI got wrong. When the labels look right, you export the dataset in one of several formats: YOLO, COCO JSON, Pascal VOC XML, or CreateML JSON. Training a model from the labeled data is a one-click step inside the same interface. The system uses the Ultralytics YOLO library and streams training progress in real time. After training you can test the resulting model on new images or video directly in the app. For video input, the tool extracts keyframes automatically using scene detection, motion analysis, or fixed time intervals, with duplicate-frame filtering to avoid redundant labels. The system runs as a local web application with a Python backend and a React frontend. On Linux or Windows with an NVIDIA GPU, you can start everything with a single Docker Compose command. On macOS, a manual setup path is provided. A GPU with at least 12 GB of video memory is required for the AI models, or an Apple Silicon Mac with sufficient unified memory. Python 3.12 or newer and Node.js 22 or newer are needed for the manual setup path.