jamesob/local-llm

This repository is one person's detailed personal guide to running powerful AI language models on your own hardware instead of through a cloud service. It walks through the actual computer parts the author bought, why they chose them, and how to configure everything to run large models locally, plus a setup for running speech to text locally as well. The guide breaks things down by budget. Spending around 2,000 dollars gets you two older gaming graphics cards with a combined 48GB of video memory, enough to run a solid open source model called Qwen along with local speech to text. Spending around 40,000 dollars buys four much more powerful workstation graphics cards with 384GB of video memory total, enough to run a huge open model that the author says comes close to matching Claude Opus in capability. Much of the guide covers the fine details of building a machine with four graphics cards: choosing an older but cheaper server motherboard and processor, adding a specialized PCIe switch so the graphics cards can talk to each other directly and quickly, building a custom wooden enclosure to physically fit everything, and tuning several low level system settings so the cards cooperate properly instead of the machine hanging. Once the hardware works, the author stores downloaded model files on a shared drive and runs each model inside its own Docker container using a ready made configuration file, so trying a different model does not mean reconfiguring the whole machine. Those containers serve the model over the network, and the author then connects to it from another computer using a coding assistant tool. The project is aimed at people who already have some technical background and are seriously considering building their own AI hardware rather than paying for cloud access, and it includes ready-to-run configuration files and a benchmarking script for testing GPU to GPU network speed.