anikesh0415/forge

Forge is a tool for controlling a Windows computer using voice commands and hand gestures, instead of a mouse and keyboard. It runs entirely on your own machine, so no cloud services see your screen or your data. Under the hood, Forge combines a few different local AI models. A language model called Hermes 2 Pro (run through a program named LM Studio) listens to what you ask for and turns it into a step by step plan written as structured data. A vision model called Moondream looks at the screen to check whether an action actually worked, for example confirming that an app finished loading before Forge tries to click something in it. A separate tool, Tesseract, reads text on screen so Forge can find and click things by their exact position, even in apps that do not have predictable button layouts. The project also gives Forge a kind of memory. It can store useful steps it has learned so it does not have to figure them out again, and it can remember facts and preferences about the user across sessions. There is also a plugin system: dropping a Python script into a specific folder lets Forge pick up new abilities without changing the core program. A text to speech feature lets it talk back while it works, without freezing up the rest of the process. Setting it up takes several steps. You need to install LM Studio and load a specific compatible language model, install Ollama and download the Moondream vision model, and install Tesseract for reading on screen text. After that you clone the repository, set up a Python virtual environment, and install the listed dependencies before running the included server script and opening its dashboard. This is an early stage, single developer project (zero stars, created this week) rather than a polished consumer app, and it currently only supports Windows. It is released under the MIT license, which allows free use, including commercial use, as long as the copyright notice is kept. The project description notes it is a work in progress, and the README invites contributions to strengthen the system and add new plugins.