thestageai/edge-lm

edge-lm provides heavily compressed versions of Google's Gemma 4 language models designed to run locally on Apple Silicon Macs and iPhones rather than on remote servers. Language models are large by default: the original Gemma 4 E2B model weighs around 9 gigabytes. The compressed checkpoints in this repository bring that down to 1.44 gigabytes, which fits within the memory budgets of mobile devices and downloads in a reasonable amount of time. The project ships two model sizes: E2B (around 2 billion parameters before compression) and E4B (around 4 billion). Each comes in two operating points called M and L, where M is the smaller and faster version and L trades some size back for higher quality. The models are designed to preserve performance on three specific tasks: following instructions, answering general knowledge questions, and making tool calls, which the README treats as the most important capabilities for an on-device assistant. The underlying framework is MLX, Apple's machine learning library built for Apple Silicon chips. The compression technique uses a combination of per-group quantization for the main decoder weights and a more specialized codec for a component called PLE tables, which the project claims are the bottleneck that standard quantization tools handle poorly. The result at the 6.4x compression level is reported to outperform other compressed Gemma checkpoints from tools like Unsloth on instruction following and tool use benchmarks, while being smaller. Installation involves cloning the repository, creating a Python virtual environment, and running pip install. Model weights download automatically from Hugging Face on first use. The package provides a simple Python API for text generation plus example scripts for chat, image description, and audio transcription. Benchmark scripts for both quality and speed are included so you can reproduce the reported numbers on your own machine. The code is MIT licensed. The compressed model weights are derivatives of Gemma 4 and are also subject to Google's Gemma Terms of Use.