gigglewang/mobileagent-android

MobileAgent Android is an app that lets your Android phone operate itself using an AI vision model. You type a task in plain language, such as "open Settings and turn on Dark Mode," and the app takes a screenshot, sends it to an AI model (such as GPT-4o or Claude), and executes whatever action the model decides on. This cycle repeats until the task is done or a configurable step limit is reached. No PC, USB connection, or ADB tool is required. The app runs a four-phase loop on each step. The Manager phase analyzes the screen and creates a numbered plan of subgoals. The Executor phase selects one concrete action, such as tapping a button, swiping, or typing text. The Reflector phase compares before-and-after screenshots to check whether the action had the intended effect. An optional Notetaker phase extracts useful on-screen information into running notes. If an action fails, the loop tries to replan rather than stopping immediately. Interactive elements on screen are found using Android's Accessibility API, which produces a tree of visible UI components. The app annotates the screenshot with numbered bounding boxes around those elements before sending the image to the AI model, giving the model precise targets to reference when choosing the next action. Setup requires granting several Android permissions: the Accessibility Service for performing taps and typing, an overlay permission for a floating status window that shows live progress, and a battery optimization exemption to prevent the system from stopping the accessibility service mid-task. The app supports OpenAI-compatible API endpoints as well as Anthropic Claude directly, and the interface is available in English and Simplified Chinese. It is licensed under MIT.