vaibhavpandeyvpz/mlex

mlex is a Rust runtime for running large language models on Apple Silicon computers using Apple's MLX framework, built without needing Python, a system MLX install, or PyTorch. Pointing it at a model folder downloaded from the Hugging Face Hub loads that model directly onto the GPU through Apple's Metal graphics technology, ready to generate text, call tools, and in some cases process images, audio, or video. The project is organized as a single Rust crate published as mlex, with a companion Node.js package called mlex.js built from the same underlying code, so the same core logic is usable from either language. It supports a range of model families including several generations of Qwen, Gemma4, NemotronH, and any model shaped like a standard Llama checkpoint, along with a wide variety of quantization formats used to shrink model file size, which it detects automatically by reading each model's own configuration file rather than relying on a hardcoded list. Beyond basic text generation, mlex supports system prompts in the same style as the OpenAI and Anthropic chat APIs, an optional reasoning or thinking mode on models that support it, and tool calling where a program supplies function definitions and receives structured calls back. It also includes an automatic caching system: instead of managing a session handle, a caller sends the whole conversation on every request, and mlex quietly reuses whatever previously computed state matches the shared prefix, even across separate calls that share the same system prompt. Every generation call supports streaming output token by token. Getting started in Rust is a single cargo add command followed by loading a model folder and calling a generate function with a list of chat messages. The Node.js version works the same way with an install through npm and an async generate call, with the same features, including tool calling and multi turn conversations, available in both languages.