joliang17/vcsd

This repository contains code for a research paper called "Visual Contrast Self-Distillation for Vision-Language Models." The paper and code are both works in progress, meaning the authors are still actively developing and refining them. The project addresses a specific problem in AI: how to train models that understand both images and text to pay closer attention to the visual information they receive. The core idea is a training technique called self-distillation through visual contrast. Here is how it works in plain terms. The model processes an image and produces predictions about what text should follow. Then, the model processes the same input but with the image removed, blacked out, or degraded. By comparing these two passes, the system identifies which predictions actually depend on seeing the image. The training process then sharpens exactly those image-dependent predictions. A safety mechanism called a plausibility guard ensures the model only redistributes probability among tokens it already considered reasonable, so it never learns to produce something the teacher model would not have generated anyway. The implementation is built on top of an existing reinforcement learning framework called verl. The authors describe their additions as minimal: a few files that add the contrast target and a loss function to a standard training loop. The repository includes scripts for launching training runs, preparing data, and handling degraded images for the control passes. Training requires an NVIDIA GPU and specific dependencies pinned in a requirements file. Data must be provided as Parquet files containing prompts and images. No model checkpoints, datasets, or pre-generated outputs are included, users must supply their own data. Configuration options let users adjust the tilt strength (how aggressively to sharpen image-dependent tokens), the plausibility threshold, and the type of control input used (black image, degraded image, or no image). The project is derived from two prior efforts: Vision-OPD and the verl framework. It is released under the Apache 2.0 license.