karpathy/llm101n

LLM101n is a planned course by Andrej Karpathy (a well-known AI researcher and educator) that intends to teach students how to build a large language model, the type of AI system that powers ChatGPT, from the ground up. The end product of the course would be a "Storyteller" AI that can create, refine, and illustrate short stories. The stated goal is to build everything end-to-end, from basic mathematics to a working web application, using Python, C, and CUDA (the programming language used to run code on NVIDIA GPUs). The README describes a detailed 17-chapter syllabus starting from the simplest possible language model (a bigram model, which just looks at pairs of words) and progressively building up through neural network backpropagation, attention mechanisms, the transformer architecture (the foundation of modern LLMs), tokenization, optimization techniques, GPU acceleration, distributed training across multiple GPUs, inference optimization including quantization and KV-caching, fine-tuning with human feedback, and finally deployment as a web app. The appendix lists supplementary topics like tensor mechanics, different neural network architectures, and multimodal AI. Important note from the README: as of the time this README was written, the course does not yet exist. It is being developed by Eureka Labs and the repository is archived until the course is ready. You would follow this project if you are interested in learning how LLMs work at a deep technical level through a hands-on build-it-yourself approach, rather than just using existing models. No primary programming language is assigned to the repository since course materials have not yet been released.