mattmireles/magenta-realtime-2-iphone

This project takes Google DeepMind's Magenta RealTime 2 music generation model and makes it run on an iPhone without using the GPU. That matters because music generation is continuous: the model has to produce a new frame of audio every 40 milliseconds and keep doing so indefinitely. A phone's GPU can handle that pace for short bursts, but over ten minutes of sustained use it draws too much power and heats up. Apple's Neural Engine, a separate chip inside iPhones designed for machine learning tasks, can run the same work at a fraction of the power draw. The challenge is that Apple's Neural Engine has strict requirements: it only accepts fixed input shapes, it does not tolerate branching logic, and its compiler can silently appear to accept a model while actually routing it to the CPU instead, where it runs 16 times too slow without any error message. The author hit that limit, measured it carefully, and published the result. The solution was to split the model into three separate graphs and route each one to the chip that suits it best. The outcome on an iPhone 12 Pro from 2020: ten unbroken minutes of 48 kHz stereo audio at 25 frames per second, with zero recorded GPU usage from the music process. The project includes validation measurements comparing the output numerically against Google's own reference implementation, showing an output correlation of 0.999985 and a signal quality of 118.85 dB. This repository contains the Python exporter scripts that convert the original model into Apple's Core ML format, along with documentation of the decisions made, including several approaches that were tried and rejected with measurements attached. Pre-converted model files ready to use in an iOS app are hosted separately on Hugging Face. The README is unusually detailed about what was proven versus what remains unvalidated.