cyrildiagne/ar-cutpaste

AR Cut and Paste is a research prototype that lets you point your phone camera at a real object in the room, cut it out of the live view, and have it appear as a new layer in Photoshop on your computer. The README is clear that this is a research demo, not a polished product. It combines augmented reality on the phone with machine learning that figures out which pixels belong to the object you want, then a small server that hands the result over to Photoshop. The README points readers who want a finished consumer tool toward clipdrop.co, which the same author started as a follow-up. For now, this repo only talks to Photoshop, though the project notes other outputs may be supported later. A Twitter thread linked in the README has a video demo of the workflow. The code is split into three modules. The mobile app, under the /app folder, runs on a phone and provides the camera view that the user aims at the object. The local server, under /server, is the glue between the phone and Photoshop. It uses a helper library called screenpoint to figure out where on the Photoshop document the phone camera is currently pointing. The third piece is an external object detection and background removal service, which receives an image and returns the same image with everything except the chosen object cut away. Setting it up takes several steps. In Photoshop you open Preferences, then Plug-ins, enable Remote Connection, and set a password that the local server will use to talk to it. The document size in Photoshop has to match settings in server/src/ps.py, or only an empty layer arrives. The README also warns that the document needs some background texture, because the SIFT feature matcher inside screenpoint will fail to align the camera view against a blank canvas. For the background removal model you have two options. You can host the BASNet model yourself using a BASNet-HTTP wrapper, which requires a CUDA GPU, and pass the resulting URL into the local server. Or you can use a community-provided endpoint hosted on CoreWeave, passed in with the --basnet_service_ip flag. After that, you follow the per-folder READMEs to start the local server and deploy the app. The README ends with thanks to the BASNet authors, to RunwayML for the Photoshop paste code, and to CoreWeave for hosting the public endpoint.