android-notes/shadowauto

ShadowAuto is an open source prototype for Android phones that runs apps quietly in the background while you keep using your screen normally. It works by starting a special process through Android's developer tools known as adb, then launching a real Android app inside a hidden virtual display that the phone owner cannot see on the normal screen. An AI model then controls that hidden app the same way a person would: tapping buttons, typing text, scrolling, and reading what is on the virtual screen, while your actual screen stays free for you to use the phone as normal. To understand what is happening on the hidden screen, ShadowAuto reads the same information Android's accessibility tools use, listing windows, buttons, and text fields. When that information is missing or incomplete, it falls back to optical character recognition to read text directly from the screen image. The AI repeats a loop of choosing an action such as tap, type, scroll, go back, search, or wait, until the task is finished. The project has three main pieces: a shell process that lives on the phone and does the actual automation, a controller app used to type in a goal and watch progress, and a browser based launcher that lets you connect a phone over USB and start everything without touching a terminal. It works on any Android 10 or newer device, needs USB debugging turned on, and does not require the phone to be rooted. You also need your own key for an AI service that supports tool calls, since ShadowAuto sends the model instructions and receives back the actions to take. An example task shown in the project is asking the app to order food through a delivery app on your behalf. Multiple tasks can run at once, though the project notes that running too many at the same time can slow the phone down or make it warm. The interface and much of the documentation are available in both English and Chinese.