liangnjupt/vistouch

VisTouch is a dataset project that captures what happens when a robot hand slides across everyday materials like silk, wood, brass, and stone. While the robot presses and slides, three sensors record the same event at the same time: a camera films it, a microphone picks up the sound, and a force sensor measures the physical pressure. The result is a collection of matched video, audio, and touch recordings that researchers can use to study how different senses relate to each other. The full research corpus covers 47 material categories and contains millions of raw sensor readings. This public release is a smaller, curated slice: 2000 timestamp-aligned triplets covering 8 materials, with a predefined split between training and test data. The split is designed so that models trained on lower pressures get tested on higher pressures, which measures whether they can generalize to conditions they have not seen. Every sample is a real sensor capture, with no synthetic data mixed in. The repository includes scripts for four benchmark tasks. Material recognition tries to identify what the robot is touching based on the fused audio, tactile, and video signals, reaching 76 percent accuracy on the sliding test compared with a 12.5 percent chance baseline. Tactile super-resolution takes a noisy, low-rate force measurement and reconstructs a clean signal. Cross-modal retrieval asks whether a touch recording can be used to find the matching sound clip. Cross-modal generation attempts to produce a force curve from sound alone. The data files are not stored in the repository itself. Instead, the repo provides an empty dataset folder and links to external downloads on Google Drive and Baidu Netdisk. Once downloaded, the files follow a simple folder structure and require no additional configuration. Baseline models are intentionally lightweight, trainable on a CPU in minutes, and serve as starting points rather than state-of-the-art results. The project roadmap mentions plans to release more material categories, additional sliding paths and camera views, and deeper model architectures in future updates. The full README is longer than what was shown.