danieldoradotalaveron-rb/yolosegment-2d-to-3d-rebotarm_pick_and_place

This project is a robotic vision system that allows a robot arm to see objects on a table and pick them up. It combines computer vision and robot control: a camera identifies what objects are present and where they are, then the robot arm moves to grasp and place them. The pipeline has four stages. First, an Intel RealSense camera captures a color image along with depth information, meaning it can measure how far away each pixel is. Second, a YOLO object detection model identifies and outlines individual objects in the color image. Third, those outlines are combined with the depth data to determine where each object sits in three-dimensional space relative to the robot. Fourth, a ROS 2 motion planning system called MoveIt takes those 3D positions and plans a sequence of movements to pick up the object and put it down somewhere else. The example in this repository uses three object types: corks, lighters, and pompoms, with the pick-and-place demonstration set up for corks and pompoms. Pre-trained model weights and a labeled dataset are available on Hugging Face, so you can run the pipeline with the included objects without training from scratch. If you want to use different objects, the repository includes tools for capturing your own images, labeling them, and training a new detection model. One notable aspect is how the system handles different object shapes. Symmetrical objects like corks and pompoms are always grasped straight down from above, since their rotation does not matter. Non-symmetrical objects can be grasped at a specific angle. The 3D position tracking also smooths detections over time to avoid jitter from single noisy frames. The project requires a reBot 601-DM arm, an Intel RealSense camera, and a ROS 2 environment with MoveIt. The repository's own code is under the MIT License. Parts of the underlying 3D detection tooling are adapted from another open-source project under separate licensing terms.