harrrshall/numpygrad

This project is a complete, ground-up reimplementation of GPT-2, a well-known AI language model, using only NumPy, a foundational mathematics library for Python. GPT-2 is the kind of model that predicts and generates text, what makes this project unusual is that it rebuilds every piece of that system from scratch using basic numerical operations rather than relying on a modern AI framework. The core components are all hand-built: an automatic differentiation engine (which computes the gradients that teaching a neural network requires), the neural network layers, the full GPT-2 model, an optimizer (the algorithm that updates the model's parameters during training), and the training loop itself. PyTorch, a popular AI library, is used only as a reference to check correctness, not to do any of the actual computation. The implementation was built in nine steps, and at each step it was verified to match PyTorch's output exactly. When loaded with the real GPT-2 124M weights from OpenAI, this NumPy version reproduces PyTorch's benchmark scores on two standard tests, WikiText-103 and LAMBADA, to the same decimal place. The codebase runs entirely on the CPU in 64-bit floating-point arithmetic, and the core implementation requires only NumPy and SciPy, a small scientific computing library. All the math behind each training step is derived and documented in the repository. You would use this project if you want to understand how large language models actually work at the mathematical level, without the abstraction that modern frameworks add. It is a learning and research resource. The full README is longer than what was provided.