raiyanyahya/how-to-train-your-gpt

How to Train Your GPT is an interactive textbook that teaches you how to build a modern AI language model from scratch. Instead of just explaining the theory, it walks you through writing every single line of code yourself, from breaking down text into tokens to running the final training loop. The project is structured as a 12-chapter guide with companion coding notebooks you can run in your browser. Each chapter starts with a simple, everyday analogy, moves to a step-by-step example using real numbers, and then shows the actual code with a comment on every single line explaining what it does and why it is there. By the end, you have built a working 151-million-parameter language model using the same modern architecture that powers open-source models like LLaMA and Mistral. This is built for Python developers, students, or anyone curious about how tools like ChatGPT actually work under the hood. If you know basic Python, how to write functions and use lists, but have zero experience with machine learning, this guide is designed for you. It is also great for engineers who want to understand the specific tradeoffs in modern AI design, like why newer models rotate word positions instead of just numbering them, or why they use a specific math trick to stabilize training in very deep networks. What makes this project notable is its commitment to filling the gap between shallow tutorials that just call pre-built APIs and dense academic papers that assume you already have a PhD. It does not use any shortcuts or pre-packaged training tools, you write the entire training pipeline yourself. It also focuses on the latest publicly known techniques rather than older approaches, so what you learn reflects how today's state-of-the-art models are actually built. Beyond the core chapters, it includes 27 standalone deep-dive explainers on individual concepts like attention mechanisms and sampling methods, plus narrative walkthroughs that trace a single sentence through the entire model. It is a learning resource, not a production tool, but it leaves you with a thorough mental model of how every piece fits together.