aleksandern/assistant-ai

AssistantAI is a desktop application that listens in on a selected app, such as Zoom, Teams, Telegram, or a browser, and picks up on spoken conversation happening there in real time. It can then generate context aware responses based on a document the user has attached, and it can also analyze screenshots through a built in web interface. The author describes it as a personal engineering project meant to explore low latency desktop AI, speech processing, and screenshot analysis, rather than a finished commercial product. A short demo video linked in the README shows the assistant reacting to a live conversation as it happens. Under the hood it captures audio from a specific application using Apple's ScreenCaptureKit, so it appears to be built primarily for macOS. It uses Silero VAD to detect when someone is actually speaking, and then runs the audio through whisper.cpp locally to turn speech into text. From there it can bring in an attached document to shape its responses to the conversation, keep a local history of past conversations, and expose a remote web interface so the assistant can be reached from elsewhere, rather than only being usable on the same machine it runs on. The project is currently written in Python. The README points to a setup script for the Python environment and a separate Python README for fuller installation instructions and OpenAI configuration details, meaning it currently relies on OpenAI's models for at least part of its response generation rather than running a local language model. The roadmap lists plans to support additional local model backends like Ollama and llama.cpp, handle more document formats, and improve the screenshot analysis feature, suggesting the current version is an early, actively evolving build rather than a finished release.