qwenlm/qwen3-omni

Qwen3-Omni is an AI model released by Alibaba Cloud that can understand and respond to text, images, audio, and video all in one system. Unlike tools that handle only one type of input, this model takes in a spoken question, a photo, a video clip, or plain text and responds either by typing or by speaking back in real time. The model streams its replies as it generates them, so the experience feels closer to a live conversation than waiting for a finished response. The model supports a wide range of languages: 119 languages for reading and writing, 19 languages for understanding speech input, and 10 languages for generating spoken output. The speech input list includes English, Chinese, Japanese, French, German, Spanish, Arabic, and several others, while the spoken output covers a similar set of major languages. This breadth makes it practical for multilingual applications without needing separate models for each language. Technically, the model uses a design the team calls Thinker and Talker. The Thinker handles reasoning and text or image understanding, while the Talker is responsible for generating speech. They run together rather than being piped sequentially, which is what keeps latency low enough for real-time back-and-forth interaction. The repository includes code for running the model via the Transformers Python library, via vLLM (a high-throughput inference server), and via Alibaba Cloud's DashScope API. A set of Jupyter notebook cookbooks walks through specific use cases: speech recognition, speech translation, music analysis, image description, video question answering, and more. Each notebook includes actual execution logs so you can see what the output looks like before running anything yourself. A Docker image is available for those who want a pre-packaged environment, and a web UI demo can be run locally for interactive testing. The full README is longer than what was shown.