tensor-master/edgesync-llm

EdgeSync-LLM is a Go library that makes AI language models respond faster on mobile devices, particularly Android phones running ARM-based chips. It works by saving and reusing pieces of the heavy computation that AI models perform when processing a prompt, so the model does not have to repeat that work from scratch every time. When an AI model processes text, it generates large internal tables of numbers called the attention cache. Normally, every new prompt triggers the model to rebuild this cache from zero, which is the slowest part of generating a response. EdgeSync-LLM stores slices of those computations, and when a new prompt arrives, it searches for a close match among what it has already computed. If the match is close enough, it injects the saved computation directly and skips most of the heavy lifting. The system classifies each incoming prompt into three categories based on how similar it is to something already stored. An exact match (above 92% similarity) skips nearly all computation and gets a response in roughly 8 milliseconds. A partial match (75 to 92% similarity) reuses the overlapping portion and fills in only the difference, taking around 280 milliseconds. A total miss runs the full computation as normal at roughly 1800 milliseconds, then saves the result for future reuse. The library is designed to slot into three popular engines used to run AI on phones and small devices: llama.cpp, MLC-LLM, and ONNX Runtime. It is written in Go and includes a bridge for integrating into Android apps via Kotlin. A built-in benchmark tests all three modes across 1000 requests drawn from 8 prompt clusters to give developers a realistic picture of the speedup. This is a developer-facing library aimed at engineers building on-device AI apps. The README is detailed and technical, covering the internal data structures, adapter interface, and build instructions for Android cross-compilation.