521xueweihan/hgdoll

HGDoll is a fully open-source Android app that acts as an AI companion while you play mobile games. Instead of playing alongside you in the game itself, it watches your phone screen in real time, listens to you through the microphone, and responds with voice chat, commentary, and encouragement. Think of it as a friend sitting next to you who can see what you are doing and react to it. The app is split into two parts. The Android client, written in Kotlin, handles screen recording and speech recognition on the phone. It takes periodic screenshots and converts your voice to text, then sends both to a separate backend server. The backend, written in Python using FastAPI, receives that data, builds a running picture of the conversation context, and passes it to a set of AI models. One model looks at the screenshots to understand what is on screen, another generates a text response, and a third converts that response back into spoken audio that plays through the app. All of the AI processing relies on Doubao, a family of large language and vision models from ByteDance, accessed through the Volcano Ark platform. You need API keys for these services to run the project. Both the client and server can be run locally, and setup instructions live in their respective subdirectories. The project is described by its author as a small toy still full of bugs and rough edges, actively being improved. It is released under the MIT license, so anyone can read the code, run it, or contribute changes. If you want an AI that reacts to your gameplay in real time through voice, this is a working proof of concept you can self-host.