stevibe/local-screen-agent

Local Screen Agent is a macOS app that lets an AI model control your computer's screen by looking at it and clicking on things. It is built in Swift and made by a developer called stevibe, and it currently has 65 stars on GitHub. Here is how it works. The app takes a screenshot of your main display, then sends information about that screen to an AI model that can make tool calls, meaning the model can decide to run specific actions instead of just chatting. The model figures out what needs to happen next. A separate tool called LocateAnything looks at the screenshot and works out exactly where a specific item on the screen sits, giving back coordinates. The app then uses those coordinates to move the mouse and click in the right spot, using macOS's own system for sending click events. To run this project you need a Mac running macOS 13 or newer, the Swift 6.2 toolchain installed, access to an AI chat model that supports tool calling, and a separately hosted LocateAnything server. You also need to grant the app Screen Recording and Accessibility permissions in macOS settings, since it needs to see your screen and control mouse clicks. The project's own creator describes it as a proof of concept, meaning it is experimental and meant for local testing and research rather than real world production use. Right now it can only click the mouse, no other automation actions are supported, and it only captures your main display if you have more than one screen connected. It also has not been through Apple's notarization process, which is normally required before distributing macOS apps outside of local development. The code is organized into an app and interface layer, a core runtime layer that handles screenshots and click logic, and a script for building a local app bundle. It is released under the MIT license, a permissive license that allows free use, including commercial use.