oaid/tengine

Tengine is a lightweight inference engine developed by OPEN AI LAB, designed to run trained AI models on embedded and edge devices in IoT scenarios. The typical use case is taking a neural network model that was trained on a powerful server and deploying it on low-power hardware such as ARM-based processors, where resources are constrained. The core problem it addresses is fragmentation: AI models are trained using various frameworks like Caffe, ONNX, TensorFlow, and MXNet, and they need to run on a wide variety of hardware. Tengine can load models from all of those formats and execute them across different chip architectures, handling the translation and optimization work so developers do not have to do it separately for each target device. Internally the engine is split into five modules: a core module providing basic system components, an operator module that defines the mathematical building blocks of neural networks (convolution, pooling, activation functions, and more), a serializer module for loading saved models, an executor module that actually runs the computation with optimizations for multi-core ARM processors, and a driver module that interfaces with specific hardware. The benchmark table in the README shows inference times for two popular lightweight models on an RK3399 processor with a single A72 core. Running Mobilenet v1 in 32-bit floating point takes about 109ms, dropping to around 64ms in 8-bit integer mode. These numbers give a sense of the kind of hardware and latency the project targets. The project is licensed under Apache 2.0 and offers support through GitHub issues, a QQ group, and email. A companion repository shares Android and Linux application examples built on top of Tengine.