ggml-org/ggml

ggml is a small low level library written in C++ for doing the math that machine learning models, including large language models, depend on. Its short description calls it a tensor library, where a tensor is just a multidimensional array of numbers, the basic building block of any neural network. The README links out to a manifesto post in the llama.cpp project, and notes that ggml is still under active development, with some of the work happening inside the related projects llama.cpp and whisper.cpp. The README lists a short set of features. It is a low level, cross platform implementation, meaning it should run on many operating systems and chips. It supports integer quantization, which is a technique for shrinking models so they run faster and use less memory by storing weights as small integers instead of full precision numbers. It claims broad hardware support, automatic differentiation (the math that lets a model learn), and includes two classic optimizers called ADAM and L-BFGS. It is built with no third party dependencies and avoids allocating memory at runtime, which keeps it predictable for embedded or performance sensitive use. Building ggml means cloning the repo, optionally setting up a Python virtual environment to install some helper requirements, then making a build directory and running cmake followed by a release build. The README is upfront that this is a developer level library: you build the examples yourself, you do not get a polished installer. The one example walked through is GPT inference. After running a small shell script to download GPT-2 small (the 117 million parameter version), you call a binary called gpt-2-backend with the model file and a text prompt. It then continues the text using the loaded model. For other examples the README points readers to the examples folder and to two outside resources: a Hugging Face blog post titled Introduction to ggml, and documentation describing the GGUF file format, which is the standard format used to ship models for ggml-based tools.