poolsideai/llama.cpp

llama.cpp is a tool for running large language models, the same kind of AI behind chatbots like ChatGPT, directly on your own computer or server. It is written in C and C++, which are programming languages known for running fast and efficiently. The main goal is to let you use these AI models with minimal setup while getting strong performance across a wide variety of hardware, whether you are on a Mac, a Windows PC, or a cloud server. The project focuses on being self-contained. It does not depend on external software libraries, which keeps things simple. It is especially well-optimized for Apple computers with M-series chips, taking advantage of their built-in capabilities. It also supports a range of other processors, including those from Intel and AMD, and can run on NVIDIA graphics cards. One notable feature is its support for quantization, a technique that compresses models from their standard precision down to as few as 1.5 bits per parameter. This reduces memory usage and speeds up performance, making it feasible to run large models on machines with limited resources. llama.cpp supports a long list of AI models. The README lists dozens of compatible architectures, including the LLaMA family, Mistral, Falcon, Phi, Gemma, GPT-2, Grok, and many others. It also supports multimodal models, meaning it can handle not just text but also images and other input types. Recent updates added support for the gpt-oss model with a specific compressed format. Getting started involves installing the software through a package manager like Homebrew, downloading pre-built binaries, or building it from source. Once installed, you point it at a model file and it runs. It can also act as a server that mimics the OpenAI API, so other software can connect to it as if it were talking to a cloud-based AI service. There are also editor plugins for VS Code and Vim that use it to provide code completion suggestions. The full README is longer than what was shown.