alange/llama.cpp

llama.cpp is a project that lets you run large language models, the kind of AI systems behind chatbots, directly on your own computer or server, without needing to depend on a cloud AI provider. It is written in plain C and C++ with no outside dependencies, which is part of why it runs efficiently on so many different kinds of hardware. The project supports a wide range of chips and setups: Apple computers with Apple Silicon, regular x86 processors from Intel and AMD, and even RISC-V chips. It can also use graphics cards for extra speed, with support for NVIDIA GPUs through CUDA, AMD GPUs through HIP, and other options through Vulkan and SYCL. If your model is too large to fit entirely in your graphics card's memory, llama.cpp can split the work between your GPU and regular computer memory so it still runs. One of its key tricks is quantization, a way of shrinking a model's size by storing its numbers with less precision, from very small down to 8-bit integers. This means models that would normally need a lot of memory can run on more modest machines, at some cost to a small amount of accuracy. Getting started involves either downloading a pre-built version for your operating system, using a package manager like brew or winget, running it through Docker, or building the code yourself from source. Once installed, you point it at a model file, either one saved on your computer or one pulled directly from Hugging Face, a popular site for sharing AI models. You can then run models directly from the command line or start a server that behaves like the well known OpenAI API, letting other applications talk to your locally running model. The README lists an extensive set of supported model families, including LLaMA, Mistral, Gemma, and dozens of others, showing this is a mature and actively maintained tool for anyone who wants to experiment with running AI models on their own hardware instead of renting cloud compute. The full README is longer than what was shown.