gabriel-kahen/hyprland-codex-background-computer-use

This project lets an AI automation agent called Codex control applications that are already running on someone's real desktop, without stealing keyboard focus or moving windows around. It works specifically with Hyprland, a Wayland-based desktop environment on Linux, and its goal is to interact with a user's actual signed-in session, files, and open windows instead of spinning up a separate virtual machine or a fresh, disconnected environment. The tool can list all the windows currently open on the desktop and which workspace they belong to, then take a screenshot of a specific window even if that window is sitting on a workspace the user is not currently looking at, all without bringing it to the front. It can send keyboard shortcuts to a specific window by address, and it can click, scroll, and drag inside background windows, whether those windows are native to Wayland or running through XWayland, a compatibility layer for older X11 style applications. For actions that need deeper understanding of what is on screen, it hands off to AT-SPI, an accessibility system built into Linux that lets software read and control user interface elements semantically rather than by guessing pixel coordinates. Some applications will not cooperate with background control no matter what, so the project includes a fallback: it can temporarily create a hidden virtual screen, briefly move the target application there to interact with it, and then restore everything, including its original screen, workspace, and focus, once finished. The project is careful to document that this fallback is the one case where there could be brief interference with whatever the user is actually doing on their real screen. Setting it up requires building a native Hyprland extension from source on the machine that will run it, since the extension has to exactly match that machine's Hyprland version. Beyond that, it needs several system-level dependencies like a screenshot tool, X11 automation tooling, and Python. It also includes clear safety limits: it will not act while the screen is locked, a physical mouse button is held down, or during drag-and-drop, and it is explicitly not meant to bypass logins or anti-cheat systems.