hiteshhedwig/scenesense-rgb-reliability

SceneSense is a research project that tries to answer a practical question for self-driving cars: can a camera image be trusted right now, or is the scene too visually difficult for the car's perception system to rely on? Conditions like night driving, fog, glare, tunnels, or heavy snow can all make a normal camera image unreliable, and this project builds a model that scores how challenging a given frame is, using only the plain RGB camera image at the moment it makes that prediction. The model is trained using a technique called weak supervision. Instead of having humans carefully label every video frame by hand, the project uses text captions that were automatically generated for driving video clips, and pulls training labels out of those captions, such as whether a caption mentions night, fog, or a tunnel. Because these captions are not perfect, the authors are careful about which mentions they trust: things like snow, glare, and rain are treated as unreliable label sources and left out of the main difficulty score, while more consistently described conditions like night, low light, fog, and tunnels are used to build the target label. Under the hood, the system uses a ResNet18 image model, a common building block in computer vision, with two output heads: one that predicts general scene conditions like day, night, or snow, and another that predicts the simple RGB reliability judgment. To check that the model is actually learning something more useful than basic image brightness and contrast, the authors compare it against a simple handcrafted statistics baseline, test it on video the model has never seen before, and use a visualization technique called Grad-CAM to see which parts of an image the model is focusing on when it makes its judgment. On its test data, the reliability model reaches strong accuracy and balanced accuracy scores above ninety percent. This project is aimed at researchers and engineers working on autonomous driving perception or computer vision reliability, and is best understood as a research prototype and case study rather than a ready-made production tool.