microsoft/omniparser

OmniParser is a Microsoft research tool that can look at a screenshot of any computer interface and break it down into a structured list of elements, buttons, icons, text fields, menus, telling an AI agent exactly what is on screen and where each element is located. Think of it as giving an AI "eyes" that can read a graphical user interface the same way a human would. The core problem it solves: AI models like GPT-4V can see images, but they struggle to accurately identify and click on specific small elements within a complex app interface. OmniParser first detects all the interactive regions in a screenshot, then generates text descriptions of what each icon or element does. This structured output makes it much easier for a vision-based AI agent to understand the screen and take correct actions. A companion tool called OmniTool lets you actually control a Windows 11 virtual machine using OmniParser combined with an AI model of your choice, including OpenAI, DeepSeek, Qwen, or Anthropic's Computer Use. The result is an agent that can operate a real computer based on plain-language instructions. Researchers and developers working on AI computer-use agents, systems that can autonomously navigate apps and perform tasks, use OmniParser as a foundational component. It is a Microsoft Research project built in Python and distributed as Jupyter Notebooks, with model weights available on Hugging Face.