alibaba/mnn

MNN is a deep learning framework built by Alibaba whose job is to run machine learning models on individual devices, such as phones, PCs, and small embedded computers, rather than on a server in the cloud. The technical term for this is on-device inference. Once a model has been trained somewhere else, MNN takes that trained model and executes it efficiently on whatever hardware the user has in their hand or on their desk. The README says MNN also supports training, but inference is the headline use case. The README backs up its claims with deployment numbers from inside Alibaba. MNN is integrated into more than 30 Alibaba apps, including Taobao, Tmall, Youku, DingTalk, and Xianyu, across over 70 distinct usage scenarios such as live broadcasts, short-video capture, search and recommendation, visual product search, interactive marketing, and risk control. MNN is also the basic compute module of a system Alibaba calls Walle, described in a paper at the OSDI 2022 systems conference as the first large-scale production system for device-cloud collaborative machine learning. The README provides the BibTeX entry for citing that paper. Two sub-projects sit on top of MNN. MNN-LLM is a runtime for large language models that aims to run LLMs locally on phones, PCs, and Internet-of-Things devices. It claims support for several open-weight model families including Qianwen, Baichuan, Zhipu, and LLAMA. MNN-Diffusion is the same idea for stable-diffusion image-generation models. The repository also ships several reference applications: an Android chat app that bundles text, image, audio, and image-generation models, an iOS multimodal chat app, a 3D-avatar app called TaoAvatar that talks back using on-device speech recognition, language modelling, and text-to-speech, and a cartoon-style photo editor named Sana. The key features section emphasises small size and broad hardware support. On iOS the static library, with all options enabled for armv7 and arm64, comes to about 12 megabytes, and on Android the core shared library is around 800 kilobytes. A reduced build option called MNN_BUILD_MINI shrinks this another 25 percent at the cost of requiring fixed model input sizes. MNN also supports half-precision (FP16) and 8-bit integer quantization, which the README says can cut model size by 50 to 70 percent. The news section at the top tracks recent additions: support for the Qwen 3.5 series, the Qwen3-VL vision-language series, DeepSeek R1 1.5B, and Qwen 2.5 Omni 3B and 7B. The full README is longer than what was shown.