jiahongsigma/efficient-llm-inference-serving-systems

This repository is a hands-on course explaining why large language models are slow to generate text and how modern serving systems make them faster. The course builds everything from a single core idea: producing one token means reading nearly every one of a model's weights out of GPU memory while doing very little math with each one, so a GPU's expensive number crunching hardware spends most of its time waiting on memory instead of computing. That single mismatch, the course argues, is the reason almost every optimization technique used in practice exists. From there the material works through a sequence of techniques, each framed as a direct answer to a problem the previous one creates. Batching multiple requests together spreads the cost of reading each weight across more work. That batching then strains memory because every sequence needs its own cache of previous computations, which leads into cache management and paging techniques. Shrinking the size of each weight through quantization directly reduces how much has to be read per token, improving speed rather than only saving memory. Later modules cover splitting a model across multiple GPUs, serving very long contexts efficiently, breaking the strictly sequential nature of generation with techniques like speculative decoding, and forcing output to follow a strict format. The final modules cover the practical layers a real deployment needs on top of raw speed: serving frameworks such as vLLM and SGLang, exposing everything through an API, keeping it observable and reliable, and honestly measuring whether a change actually helped. The course is aimed at people who already have working knowledge of Python, Linux, basic deep learning concepts, and fundamentals of how computer memory and processors interact, and it separates any heavier mathematics into optional appendices so the main path stays approachable. Every module has GPU labs on real open models so the ideas can be tested directly rather than only read about. It is released under the MIT license.