rasbt/llms-from-scratch

LLMs-from-scratch is the official code repository accompanying Sebastian Raschka's book "Build a Large Language Model (From Scratch)." Its purpose is to teach how a ChatGPT-style large language model actually works by walking the reader through building a small but fully functional version of one, line by line, using PyTorch, a popular Python framework for deep learning. Rather than calling someone else's pretrained model, the reader codes the whole pipeline themselves and runs it on their own machine. The repository is organised by book chapters. After an introductory chapter explaining what LLMs are, later chapters guide the reader through working with text data, coding the attention mechanism that lets the model look at different parts of an input at once, building a GPT-style model architecture, pretraining the model on unlabelled text, fine-tuning it for text classification, and fine-tuning it again so it can follow instructions like a chat assistant. Appendices add an introduction to PyTorch, references, exercise solutions, extras for the training loop, and a parameter-efficient fine-tuning method called LoRA. Each chapter ships as Jupyter notebooks plus standalone Python scripts and exercise solutions. The README also notes that the code can load weights from larger pretrained models so readers can experiment with fine-tuning a real model after building their own. People typically use this repository as study material, alongside the book or on its own, to gain intuition about how modern language models are built, trained, and adapted.