shure-dev/small-vlm-sop-check

This project is a research toolkit for catching mistakes in factory work procedures before they turn into accidents or quality problems. It uses small vision-language models, the kind of AI that can look at video and answer questions about it, to watch first-person video from a worker's camera and figure out what step of a procedure is happening. The core task the project focuses on is called temporal grounding. You give the model a video and a description of an action, such as a worker tipping a bag to drop parts into a container, and the model returns the exact start and end time of that action in the video, or says it does not happen at all. Knowing precisely when each step starts and ends makes it possible to detect skipped steps, wrong order, or actions that take unusually long, which points to a procedure error. To test this, the project uses twenty short first-person factory videos from a public dataset called Egocentric-10K, each hand-labeled in Japanese with events and exact time ranges. It compares several small AI models, all four billion parameters or smaller, on how closely their predicted time ranges match the human-labeled ones. In the current results, a model called Marlin-2B performs noticeably better than several Qwen and Gemma variants that were also tested. The repository includes a web app for annotating videos and reviewing model predictions side by side on a timeline, plus command line tools for fetching the dataset, running evaluations, and exporting labeled data for fine-tuning models with a tool called ms-swift. You can also bring your own video clips instead of using the built-in dataset. This is aimed at researchers and engineers working on factory safety or industrial AI, not general users. The authors are clear that the project currently only covers offline annotation, evaluation, and training data preparation, not real-time deployment on wearable devices, and that it should not be used on its own to automate safety-critical decisions. The code is released under the MIT license, while the video dataset and any pretrained models follow their own separate licenses.