weklund/mlx-inference-workbench

mlx-inference-workbench is a personal research project for measuring and improving how large language models run for text generation on Apple Silicon Macs, specifically the author's Mac with an M5 Max chip and a large amount of shared memory. The goal is to build a single, consistent way of benchmarking different inference setups so that experiments can be fairly compared against each other, and eventually to use what is learned to write custom, faster low level graphics chip code for running AI models locally, with the longer term aim of running capable coding assistants entirely on a local Mac rather than through a cloud service. The project follows a deliberately staged approach rather than jumping straight into optimization. Early phases focus on understanding an existing fast inference library's custom low level code and validating that benchmark measurements are actually reproducible from one test run to another, since heat buildup in a laptop can otherwise quietly skew results. Only once measurements pass a strict reproducibility check does the project move on to building out the actual benchmarking tools, establishing baseline performance numbers across several different inference backends and language models, and then digging into individual pieces of GPU code to find where time is actually being spent. Later planned phases involve building parts of the system in Rust and writing custom graphics chip shaders by hand, followed by a series of specific, hypothesis driven optimization experiments. The repository is organized around a shared benchmarking toolset, a folder of versioned test prompts, individual numbered experiment folders, and a growing set of custom low level graphics code. It uses a Python based build tool for dependency management and ships with a bundled set of commands for linting, running tests, checking code coverage, and running a quick smoke test of the whole benchmarking pipeline end to end. As a personal, actively evolving workbench rather than a finished tool, it is aimed at someone already comfortable with performance engineering and Apple's machine learning framework rather than a general audience.