google-ai-edge/mediapipe

MediaPipe is Google's open-source framework for building machine learning pipelines that work directly on a device, your phone, laptop, or browser, rather than needing to send data to a remote server. It solves the problem of integrating real-time AI perception features (things like detecting hands, faces, poses, or objects in live video) into apps without requiring deep ML expertise or cloud infrastructure. The way it works is through a graph-based pipeline concept: you connect small processing units called "calculators" together in a directed graph (a flowchart where data moves from one step to the next). A camera frame enters the graph, travels through calculators that resize, normalize, run inference (the AI prediction step), and extract landmarks (key points like finger joints), and the result comes out the other end ready to display or use. This design makes it easy to build custom workflows by connecting existing building blocks. MediaPipe Ships two main layers. The higher-level "MediaPipe Tasks" provides pre-built, ready-to-use solutions, hand landmark detection, face detection, object detection, audio classification, and more, that you can drop into an app with just a few lines of code. The lower-level "MediaPipe Framework" exposes the full graph pipeline so developers can build custom perception systems from scratch. You would use MediaPipe when building apps that need to understand what the camera sees in real time: AR effects that track your face, fitness apps that analyze body pose, sign-language tools, gesture controls, or video filters. It is particularly valuable because it runs on-device with no internet required, keeping user data private and keeping latency low. The tech stack is primarily C++ at the core, with official SDKs and bindings for Android (Java/Kotlin), iOS (Swift/Objective-C), Python, and JavaScript (for web browsers). Models are pre-trained and bundled with each solution.