forge-the-kingdom/inference-serving-recipes

inference-serving-recipes is a collection of documentation, not a program you install and run. It is a set of Markdown files that each walk through one specific way to run large language models on your own multi-GPU computer, using tools like llama.cpp and vLLM. Each recipe includes the exact command used, an explanation of why certain less obvious settings matter, what tradeoff each setting buys you, and measured performance numbers, with each number clearly labeled as measured, observed, or estimated so nothing is presented as more certain than it is. The recipes were developed and tested on a specific machine with four RTX 3090 graphics cards, a multi-core AMD processor, and a large amount of RAM, and that exact hardware setup is documented so readers can judge how relevant the numbers are to their own systems. Individual recipes cover topics such as offloading only part of a model to the CPU to save GPU memory while keeping speed high, pinning software processes to specific processor cores to improve performance when running multiple models at once, and serving very large models that only fit in memory when spread carefully across all four graphics cards. The project explicitly invites people with different hardware to reproduce the same tests and submit their own results as a pull request, as long as they include the conditions under which they measured them, since a performance number without its context cannot be fairly compared to another. The repository itself is released under the MIT license, meaning the written recipes, commands, and explanations can be freely used, copied, and modified. It notes that the underlying AI model weights and the software tools mentioned, like llama.cpp and vLLM, each carry their own separate licenses.