sapid-labs/llama.cpp

This repository is a copy of llama.cpp, a project written in plain C and C++ that lets you run large language models on your own computer instead of relying on a cloud API. Its main goal is to make running these models fast and practical on ordinary hardware, from laptops to servers, without pulling in a big pile of external dependencies. The project supports a very wide range of hardware. It has optimized code paths for Apple silicon Macs, for x86 processors using their AVX instruction sets, for RISC-V chips, and for GPUs from NVIDIA, AMD, and Moore Threads, plus general Vulkan and SYCL backends. It can also split a model between CPU and GPU when the model is too large to fit entirely in the graphics card's memory. To keep models small enough to run locally, it supports many levels of quantization, a technique that shrinks a model's numbers down to fewer bits at some cost to precision, ranging from 1.5-bit up to 8-bit. Installation can go a few different ways. Someone can install a pre-built version through common package managers like brew, winget, or conda-forge, run it inside Docker, download a ready made binary from the releases page, or build it themselves from source. Once installed, the tool needs an actual model file in a format called GGUF, which can either be a file already on your computer or one downloaded directly from Hugging Face by name. A simple command line tool called llama-cli runs a model for chatting, and a separate llama-server program exposes a local web server with an API compatible with OpenAI's, plus a browser based web interface. The project supports an enormous list of model families, including several generations of LLaMA, Mistral, Mixtral, Gemma, Qwen, DeepSeek, Phi, GPT-2, and dozens more, with support for adding new ones documented for contributors. It also has features like tools for editors, a VS Code extension and Vim and Neovim plugins for code completion, and multimodal support for models that understand images as well as text. The project is released under the MIT license and serves as the main proving ground for a lower level numerical library called ggml that other projects also build on.