w3cj/how-llms-work

This repository is an interactive web application that teaches you how large language models work by walking through every stage of the pipeline, from simple pattern matching all the way up to training a transformer from scratch. The app is built with TypeScript using a framework called Hono for the server and Vite for bundling. You need pnpm and Node.js 20 or higher to run it locally. The app is organized into five sections that build on each other. The first section is a simple chatbot that responds using basic if-statements, similar to the classic ELIZA program from the 1960s. It streams responses word by word, using the same streaming mechanism as ChatGPT but with no actual intelligence behind it. The second section trains a small neural network live in your browser to solve a problem called XOR, showing why a single layer of neurons fails but a multi-layer network succeeds using backpropagation, the core learning algorithm behind modern neural networks. The third section implements a tokenization algorithm called BPE from scratch. Tokenization is how text gets broken into pieces before a model processes it. You can watch the algorithm build a vocabulary step by step, starting from individual characters and merging them into larger units. The fourth section trains word embeddings, which are numerical representations of words, using a technique called Word2Vec skip-gram. You can see how words used in similar contexts naturally cluster together as the training runs. The fifth and final section is the most involved. It trains a decoder-only transformer, the architecture behind models like GPT, entirely from scratch without using any machine learning libraries. Every piece is implemented by hand, including the attention mechanism, layer normalization, feed-forward layers, backpropagation, and an optimizer called Adam. It uses shared memory buffers for multi-threaded training speed. Each section follows the same structure: a server route processes input and streams results, a client hook manages state and connects the stream to the interface, and a component renders the results as a visualization. The README also includes references to eleven research papers that correspond to the concepts demonstrated in each section, from the original ELIZA paper to the transformer architecture and sampling techniques.